enhance: reduce Tantivy index build write amplification

Project: milvus-io/milvus | Language: Go, C++, Rust | Concepts: memory-management, algorithmic-optimization, indexing | Difficulty: mid
Link: https://github.com/milvus-io/milvus/pull/53035

The problem

Building text indexes with Tantivy in Milvus was causing excessive write amplification. During index construction, especially for sealed (immutable) segments, background merge operations would rewrite data unnecessarily, increasing IO and slowing down builds. The system needed to avoid unnecessary merges and segment rewrites, especially since the build is already bounded and the segment count is manageable.

Why it's non-trivial

A naive fix might simply disable background merges, but this could break assumptions elsewhere or lead to unbounded segment counts for growing (mutable) segments. The correct solution must distinguish between growing and sealed segments, only disabling background merges for the latter, and ensure that memory budget constraints are respected without introducing races or data loss.

The approach

1. Use a smaller, specialized memory budget for growing segments:
The constructor for TextMatchIndex is updated to use a different memory budget constant for growing segments, reducing memory pressure and segment flush frequency for those cases.

-        milvus::tantivy::DEFAULT_OVERALL_MEMORY_BUDGET_IN_BYTES,
+        milvus::tantivy::GROWING_TEXT_MEMORY_BUDGET_IN_BYTES,

This change ensures that growing segments use a tighter memory budget, which controls how often segments are auto-flushed.

2. Clarify and update documentation/comments:
Comments are rewritten to clarify the distinction between growing and sealed segments, and to document why background merges are disabled for sealed builds:

-    // for growing segment.
-    // In-memory writer. enable_background_merge must be true only for a
-    // long-lived growing segment (periodic commits would otherwise grow the
-    // segment count unbounded); a sealed interim index passes false so that
-    // finish()'s explicit merge-all is the only merge and cannot race a
-    // background policy merge.
+    // In-memory writer used by both growing segments and sealed interim index
+    // builds. Growing segments enable background merge because periodic
+    // commits would otherwise grow the segment count unbounded; sealed builds
+    // disable it to avoid merge write amplification.

This makes the intent and safety of the approach clearer for future maintainers.

3. Remove forced "merge-all" at finish for sealed builds:
Previously, the finish() method would always merge all segments into one for sealed builds. The new approach preserves the segments produced by memory-budget flushes and avoids a full rewrite at the end:

-        if !enable_background_merge {
-            // Build-mode writers use NoMergePolicy (set in new()), so no
-            // background merge can race this explicit merge-all. Collapse
-            // the auto-flushed segments into a single one. Background-merge
-            // writers (e.g. growing segments) keep their own policy and are
-            // not forced to a single segment here.
-            let segment_ids = index_writer.index().searchable_segment_ids()?;
-            if segment_ids.len() > 1 {
-                index_writer.merge(&segment_ids).wait()?;
-            }
-        }

By removing this block, the system avoids rewriting all data at the end of a sealed index build, reducing IO.

4. Consistently disable background merges for sealed builds:
Wherever a writer is created for a sealed build, the merge policy is set to NoMergePolicy to prevent background merges:

         if !enable_background_merge {
-            // Sealed index builds end with an explicit merge-all in finish();
-            // background policy-driven merges would only waste IO and race
-            // with it, so disable them entirely for build-mode writers.
+            // Sealed index builds keep the segments produced by the writer's
+            // memory-budget flushes. Disable policy-driven background merges to
+            // avoid additional write amplification during the build.
             index_writer.set_merge_policy(Box::new(tantivy::merge_policy::NoMergePolicy));
         }

This ensures that only the segments naturally produced by memory flushes are kept, with no extra background merging.

5. Update and add tests:
The test for sealed builds is updated to check that multiple segments are preserved after the build, not merged into one:

-    /// A build-mode (enable_background_merge == false) V7 writer must collapse the
-    /// auto-flushed segments into exactly one searchable segment in finish().
+    /// A build-mode (enable_background_merge == false) V7 writer must preserve the
+    /// segments produced by memory-budget flushes instead of merging them all in
+    /// finish().

This verifies that the new behavior is correct and prevents regressions.

From the review thread

No notable review discussion on this PR.

Takeaways

• Avoid unnecessary background work for bounded, one-shot operations: Disabling background merges for sealed (immutable) index builds prevents redundant IO and reduces write amplification.
• Tailor memory budgets and merge policies to workload type: Growing (mutable) and sealed (immutable) segments have different needs; using different memory budgets and merge policies for each improves both performance and resource usage.
• Tests should reflect real-world production constraints: Updating tests to check for multi-segment outputs ensures that code changes match the intended optimization and prevent silent regressions.