unlock unlimited code runs for $79, limited time →
NOSSLETTER.techLIVE
back to OSSpedia
milvus-io/milvus
mid8/10
backwards-compatibilityindexingdata-structures

fix: [3.0] gate nested HYBRID high-cardinality STL_SORT on scalar index version

Project: milvus-io/milvus | Language: Go, C++ | Concepts: backwards-compatibility, indexing, data-structures | Difficulty: mid
Link: https://github.com/milvus-io/milvus/pull/52959

The problem

When building scalar indexes for nested (struct sub-field) HYBRID fields, Milvus would always select the STL_SORT index type for high-cardinality data if the field was nested, regardless of the scalar index engine version. However, older Milvus versions (before support for nested STL_SORT) cannot read such indexes. This means that after a rollback or downgrade, segments could become unreadable if they contained these newer index types, breaking backward compatibility.

Why it's non-trivial

A naive fix might just change the index selection logic for nested fields, but would miss that the correct behavior depends on the cluster's scalar index engine version. The fix must ensure STL_SORT is only selected for nested, high-cardinality HYBRID indexes if the cluster is on a version that can read them, or else fall back to the legacy INVERTED type. This requires careful threading of version information and test coverage for both new and legacy scenarios.

The approach

1. Define a minimum version constant for nested STL_SORT

+// Version 4 additionally allows nested (struct sub-field) HYBRID indexes to
+// select STL_SORT for high-cardinality data instead of INVERTED. Below this
+// version, nested HYBRID indexes always keep INVERTED at high cardinality:
+// an older reader's ScalarIndexSort predates nested-index support and cannot
+// load a nested STL_SORT physical index.
+constexpr int32_t kNestedHybridStlSortMinVersion = 4;

What this does:
Introduces a constant to represent the minimum engine version that supports nested STL_SORT indexes. This is used as a gate to prevent incompatible index types from being created on older versions.

2. Thread the scalar index version through the index selection logic

+    scalar_index_version_ = scalar_index_version;

What this does:
Stores the resolved scalar index engine version in the HybridScalarIndex object during the Build() call, so it can be used later in index type selection.

3. Gate STL_SORT selection for nested high-cardinality data on version

-        internal_index_type_ = is_nested_index_ ? ScalarIndexType::STLSORT
-                                                : ScalarIndexType::INVERTED;
+        internal_index_type_ =
+            (is_nested_index_ &&
+             scalar_index_version_ >= kNestedHybridStlSortMinVersion)
+                ? ScalarIndexType::STLSORT
+                : ScalarIndexType::INVERTED;

What this does:
Only selects STL_SORT for nested, high-cardinality HYBRID indexes if the engine version is at or above the minimum supported version. Otherwise, falls back to the legacy INVERTED type.

4. Add and update tests to verify correct behavior

+// Below kNestedHybridStlSortMinVersion (e.g. the default v3 used by 3.0.x),
+// nested high-cardinality HYBRID indexes must keep selecting INVERTED so
+// that an older reader (e.g. 2.6, whose ScalarIndexSort predates nested-index
+// support) can still load the index after a rollback. Regression test for
+// https://github.com/milvus-io/milvus/issues/52893.
+TEST(BitmapIndexArrayNestedTest,
+     HybridNestedHighCardinalityBelowVersionKeepsInverted) {
+    ...
+    for (int32_t version :
+         {0, kHybridIndexConfigVersion, kNestedHybridStlSortMinVersion - 1}) {
+        TestHybridScalarIndexString nested(7, ctx, true);
+        nested.scalar_index_version_ = version;
+        EXPECT_EQ(nested.SelectIndexBuildTypePublic(
+                      std::vector<FieldDataPtr>{high_card_field_data}),
+                  ScalarIndexType::INVERTED)
+            << "scalar_index_version_=" << version;
+    }
+    ...
+}

What this does:
Adds a regression test to ensure that for all versions below the minimum, nested high-cardinality HYBRID indexes select INVERTED, not STL_SORT.

5. Update Go constants and documentation

+	// MinScalarIndexVersionForNestedHybridStlSort is the minimum scalar index
+	// engine version at which nested (struct sub-field) HYBRID indexes may
+	// select STL_SORT for high-cardinality data instead of INVERTED. Below
+	// this version, nested HYBRID indexes always keep INVERTED at high
+	// cardinality so that an older reader (e.g. 2.6, whose ScalarIndexSort
+	// predates nested-index support) can still load the index.
+	MinScalarIndexVersionForNestedHybridStlSort = int32(4)

What this does:
Documents and exposes the new minimum version constant in Go as well, for consistency and clarity across the codebase.

From the review thread

No notable review discussion on this PR.

Takeaways

  • Always gate new on-disk formats or index types on explicit version checks to ensure backward compatibility in distributed systems.
  • Propagate configuration and version information through all relevant code paths, not just the top-level logic, to avoid subtle compatibility bugs.
  • Regression tests for downgrade/rollback scenarios are essential when evolving storage formats or index selection logic.