enhance: reduce Tantivy index build write amplification [2.6]

Project: milvus-io/milvus | Language: Go, C++, Rust | Concepts: algorithmic-optimization, memory-management | Difficulty: mid
Link: https://github.com/milvus-io/milvus/pull/53063

The problem

Building text indexes with Tantivy in Milvus could cause excessive write amplification. This happened because background segment merges were enabled even during bounded, one-off index builds ("sealed" builds), causing unnecessary disk writes and extra I/O. The system needed a way to avoid background merging in these cases while still allowing it for "growing" (mutable, in-memory) segments.

Why it's non-trivial

A naive fix—just disabling merges everywhere—would break the "growing" segment case, where background merges are necessary to keep the number of segments manageable over time. The solution needed to distinguish between sealed and growing index builds and apply the right merge policy for each, across several layers (C++, Rust, and C FFI).

The approach

1. Add a parameter to control background merging

The C++ index constructor now takes a bool enable_background_merge argument, which is passed through to the Tantivy writer layer:

-    explicit TextMatchIndex(int64_t commit_interval_in_ms,
-                            const char* unique_id,
-                            const char* tokenizer_name,
-                            const char* analyzer_params);
+    explicit TextMatchIndex(int64_t commit_interval_in_ms,
+                            const char* unique_id,
+                            const char* tokenizer_name,
+                            const char* analyzer_params,
+                            bool enable_background_merge);

2. Propagate the parameter through the stack

This new argument is threaded through all the relevant code paths: C++ index construction, test code, segment creation, and down to the Rust FFI boundary:

-RustResult tantivy_create_text_writer(const char *field_name,
-                                      const char *analyzer_params,
-                                      uintptr_t num_threads,
-                                      uintptr_t overall_memory_budget_in_bytes,
-                                      bool in_ram);
+RustResult tantivy_create_text_writer(const char *field_name,
+                                      const char *analyzer_params,
+                                      uintptr_t num_threads,
+                                      uintptr_t overall_memory_budget_in_bytes,
+                                      bool in_ram,
+                                      bool enable_background_merge);

3. Set merge policy in Rust based on the parameter

The Rust index writer implementation now respects the enable_background_merge flag, using Tantivy's NoMergePolicy when background merging should be disabled (sealed builds):

let index_writer =
    index.writer_with_num_threads(num_threads, overall_memory_budget_in_bytes)?;
// Ngram writers are only used for sealed index builds. Keep
// memory-budget-flushed segments and avoid background merge write
// amplification.
index_writer.set_merge_policy(Box::new(tantivy::merge_policy::NoMergePolicy));

Similar logic is applied for scalar writers:

let index_writer =
    index.writer_with_num_threads(num_threads, overall_memory_budget_in_bytes)?;
// Scalar writers are only used for sealed index builds. Keep the
// segments produced by memory-budget flushes and avoid background
// merge write amplification.
index_writer.set_merge_policy(Box::new(tantivy_5::merge_policy::NoMergePolicy));

4. Remove forced merges in sealed builds

The old code forced a merge of all segments after committing, which is now removed to avoid unnecessary write amplification:

-                // merge all segments
-                let segment_ids = index_writer.index().searchable_segment_ids()?;
-                if segment_ids.len() > 1 {
-                    let _ = index_writer.merge(&segment_ids).wait();
-                }

5. Update all call sites and tests

All index creation sites (both in production and test code) are updated to pass the correct value of enable_background_merge:

-    auto index = std::make_unique<Index>(
-        200, "test_multi_batch", "milvus_tokenizer", "{}");
+    auto index = std::make_unique<Index>(200,
+                                         "test_multi_batch",
+                                         "milvus_tokenizer",
+                                         "{}",
+                                         /*enable_background_merge=*/true);

From the review thread

No notable review discussion on this PR.

Takeaways

• Make resource-intensive behaviors configurable: Background merging is useful in some scenarios but harmful in others. Expose configuration options to control such behaviors based on use case.
• Thread configuration through all layers: When adding a new option, ensure it is consistently propagated through all relevant code paths and language boundaries.
• Avoid unnecessary work in batch/one-off operations: For one-time or bounded tasks, disable background processes (like merges) that are only beneficial for long-running or streaming workloads.