enhance: [2.6] isolate online-write cgo pool from segment loading (#50812)
Project: milvus-io/milvus | Language: Go | Concepts: concurrency, configuration-management, observability | Difficulty: mid
Link: https://github.com/milvus-io/milvus/pull/53139
The problem
On Milvus QueryNode, the same goroutine pool was used for both online write operations (Insert/Delete) and load-time delta-log replay (LoadDeltaData/LoadDeletedRecord). After a compaction, a burst of delta-log replay could saturate the pool, starving online Insert/Delete operations. This caused the "channel tsafe stalled" error, where strong-consistency queries would fail because tSafe (timestamp safe) advancement was blocked for several seconds.
Why it's non-trivial
A naive fix might just increase the pool size or try to prioritize certain tasks, but that doesn't prevent starvation when load bursts occur. The real issue is resource contention: both critical online writes and background loading compete for the same limited pool. Without isolation, background tasks can still block latency-sensitive operations, regardless of pool size.
The approach
1. Add separate, configurable pools for different task types
A new pool, MutatePool, is introduced specifically for online-write CGO operations (Insert/Delete), separating them from load/management work:
+ // mutatePool serves the online write CGO path (segment Insert/Delete).
+ // It is isolated from the load/management work on DynamicPool so that a
+ // burst of segment loading (e.g. delta-log replay after compaction) cannot
+ // starve online insert/delete, which would otherwise stall tSafe advancement.
+ mutatePool atomic.Pointer[conc.Pool[any]]
+ mutatePoolOnce sync.Once
This declares the new pool and ensures it's only initialized once.
2. Make pool sizes configurable and hot-reloadable
Pool sizes are now determined by CPU count times a configurable factor, with new config keys and hot-reload support:
+func dynamicPoolSize() int {
+ size := hardware.GetCPUNum() * paramtable.Get().QueryNodeCfg.DynamicPoolSizeFactor.GetAsInt()
+ if size < 1 {
+ size = hardware.GetCPUNum()
+ }
+ return size
+}
This function computes the dynamic pool size based on configuration, clamping to at least one CPU.
The same pattern is used for MutatePool and DeletePool:
+func mutatePoolSize() int {
+ size := hardware.GetCPUNum() * paramtable.Get().QueryNodeCfg.MutatePoolSizeFactor.GetAsInt()
+ if size < 1 {
+ size = hardware.GetCPUNum()
+ }
+ return size
+}
3. Route operations to the correct pools
Online Insert/Delete now use the new MutatePool:
-func (s *LocalSegment) Insert(ctx context.Context, rowIDs []int64, timestamps []uint64, data *segcore.InsertData) error {
- GetDynamicPool().Submit(func() (any, error) {
+ GetMutatePool().Submit(func() (any, error) {
-func (s *LocalSegment) Delete(ctx context.Context, primaryKeys storage.PrimaryKeyData, timestamps []uint64) error {
- GetDynamicPool().Submit(func() (any, error) {
+ GetMutatePool().Submit(func() (any, error) {
Load-time delta replay is explicitly routed to the load pool:
- GetDynamicPool().Submit(func() (any, error) {
+ GetLoadPool().Submit(func() (any, error) {
4. Expose pool configuration and metrics
New configuration keys are added and documented:
+ mutatePoolSizeFactor: 2 # size factor (CPUNum * factor) of the online-write cgo pool for segment insert/delete; isolated from the load/management pool so segment loading cannot starve online writes
+ dynamicPoolSizeFactor: 1 # size factor (CPUNum * factor) of the dynamic cgo pool for segment create/release/statistics/index-info
+ deletePoolSizeFactor: 1 # size factor (CPUNum * factor) of the DeleteBatch dispatch pool
And the MutatePool is now included in pool metrics:
- refs := []poolRef{
- {"SQPool", GetSQPool()},
- {"DynamicPool", GetDynamicPool()},
+ refs := []poolRef{
+ {"SQPool", GetSQPool()},
+ {"DynamicPool", GetDynamicPool()},
+ {"MutatePool", GetMutatePool()},
5. Comprehensive tests for pool resizing and config edge cases
Unit tests ensure that pools resize correctly and never drop below CPU count:
+ // factor <= 0 clamps to CPUNum so the mutate pool is never zero-sized.
+ pt.Save(pt.QueryNodeCfg.MutatePoolSizeFactor.Key, "0")
+ ResizeMutatePool(&config.Event{
+ HasUpdated: true,
+ })
+ assert.Equal(t, hardware.GetCPUNum(), GetMutatePool().Cap())
From the review thread
No notable review discussion on this PR.
Takeaways
- Resource isolation is critical for latency-sensitive operations: Separate pools for background and online tasks prevent resource starvation and ensure predictable performance for critical paths.
- Configurable concurrency allows tuning for different workloads: Making pool sizes adjustable (and hot-reloadable) enables operators to adapt to hardware and workload changes without code changes or restarts.
- Observability and test coverage are essential for reliability: Exposing metrics and thoroughly testing configuration edge cases helps prevent regressions and aids in diagnosing production issues.