enhance: reduce Tantivy index build write amplification [3.0]

Project: milvus-io/milvus | Language: Go, C++, Rust | Concepts: indexing, algorithmic-optimization, memory-management | Difficulty: mid
Link: https://github.com/milvus-io/milvus/pull/53057

The problem

When building text indexes with Tantivy in Milvus, the system would flush segments to disk based on a memory budget, then merge all these segments into one at the end of the build. This final "merge-all" operation caused significant write amplification: the entire index was rewritten, resulting in unnecessary I/O and slower builds, especially for large datasets.

Why it's non-trivial

A naive fix might disable the merge-all step, but this could leave too many small segments, hurting query performance. The solution needs to balance segment count (for performance) and write amplification (for efficiency), and must work differently for "growing" (mutable) and "sealed" (immutable) index builds. It also needs to coordinate memory budgets and background merging policies across C++, Go, and Rust components.

The approach

1. Differentiate memory budgets for growing vs sealed index builds

The constructor for TextMatchIndex now chooses the memory budget based on whether the index is "growing" (mutable) or "sealed" (immutable):

+    auto memory_budget_in_bytes =
+        commit_interval_in_ms == std::numeric_limits<int64_t>::max()
+            ? milvus::tantivy::DEFAULT_OVERALL_MEMORY_BUDGET_IN_BYTES
+            : milvus::tantivy::GROWING_TEXT_MEMORY_BUDGET_IN_BYTES;

Annotation: For sealed builds (commit interval is max), use a large memory budget to minimize the number of segments, reducing the need for later merges. For growing builds, use a smaller budget to avoid using too much memory.

2. Disable background merges for sealed builds and avoid final merge-all

In the Rust index writer implementations, the code disables background merging for sealed builds and removes the explicit "merge-all" at the end:

-        if !enable_background_merge {
-            // Sealed index builds end with an explicit merge-all in finish();
-            // background policy-driven merges would only waste IO and race
-            // with it, so disable them entirely for build-mode writers.
-            index_writer.set_merge_policy(Box::new(tantivy_5::merge_policy::NoMergePolicy));
-        }
+        if !enable_background_merge {
+            // Sealed index builds keep the segments produced by the writer's
+            // memory-budget flushes. Disable policy-driven background merges to
+            // avoid additional write amplification during the build.
+            index_writer.set_merge_policy(Box::new(tantivy_5::merge_policy::NoMergePolicy));
+        }

Annotation: The merge policy is set to NoMergePolicy for sealed builds, so no background or final merges are performed. The segments created during the build are preserved.

In the finish method, the explicit merge-all is removed:

-        if !self.enable_background_merge {
-            // Build-mode writers use NoMergePolicy (set in new()), so no
-            // background merge can race this explicit merge-all. Collapse the
-            // auto-flushed segments into a single one. Background-merge writers
-            // (e.g. growing segments) are left to their own policy and are not
-            // forced to a single segment here.
-            let segment_ids = self.index.searchable_segment_ids()?;
-            if segment_ids.len() > 1 {
-                self.index_writer.merge(&segment_ids).wait()?;
-            }
-        }

Annotation: This block, which used to merge all segments into one at the end of a sealed build, is now deleted. The system now keeps the (few) segments produced by memory-budget flushes.

3. Update documentation and tests for new segment behavior

Comments and tests are updated to reflect that sealed builds now preserve segments, rather than merging all into one:

-    /// A build-mode (enable_background_merge == false) V7 writer must collapse the
-    /// auto-flushed segments into exactly one searchable segment in finish().
+    /// A build-mode (enable_background_merge == false) V7 writer must preserve the
+    /// segments produced by memory-budget flushes instead of merging them all in
+    /// finish(). The production budget is large enough to keep the segment count
+    /// low, while avoiding a full-index rewrite at the end of the build.

Annotation: The test now checks that the number of segments is reasonable, not necessarily one.

From the review thread

No notable review discussion on this PR.

Takeaways

• Tune memory budgets and merge policies according to workload: Use larger memory budgets for bulk/sealed builds to minimize segment count, and disable expensive merges that offer little benefit for immutable data.
• Avoid unnecessary full rewrites in index builds: Merging all segments at the end of a build can cause significant write amplification; it's often better to keep a small number of segments if performance is not impacted.
• Align code, documentation, and tests when changing core behaviors: When changing how segments are managed, update comments and tests to reflect the new behavior and ensure future maintainers understand the rationale.