[FIXED] Dynamic max_file_store shrinks on restart as the store grows

Project: nats-io/nats-server | Language: Go | Concepts: memory-management, error-handling, testing-strategy | Difficulty: mid
Link: https://github.com/nats-io/nats-server/pull/8503

The problem

When NATS JetStream is configured with a dynamic max_file_store (i.e., the server automatically sets its storage cap based on available disk space), a restart would cause the calculated limit to shrink as more data was written. This happened because the calculation only considered free disk space, not the space already occupied by JetStream data. As a result, account or stream limits that were valid before a restart could become invalid after, preventing the server from starting or recovering data.

Why it's non-trivial

A naive fix might just recalculate the available disk space on restart, but this would ignore data already present on disk (from JetStream streams), causing the dynamic limit to drop unexpectedly. The solution needs to ensure that the dynamic limit remains stable across restarts, accounting for both free space and existing JetStream data, and must do so in a way that is safe under concurrent recovery operations.

The approach

1. Track when the dynamic max store is provisional

A new field is added to the JetStream config to indicate when the dynamic limit is still provisional (i.e., before recovery has accounted for existing data):

+	// maxStorePending is set when MaxStore was derived from the available disk
+	// space rather than configured, and has not been adjusted yet for the space
+	// that recovered streams already occupy. See finalizeDynamicMaxStore.
+	maxStorePending bool

2. Set the provisional flag when using a dynamic limit

When the server starts and calculates a dynamic MaxStore, it sets the maxStorePending flag:

-	jsc.MaxStore = diskAvailable(jsc.StoreDir)
+	jsc.MaxStore = diskAvailable(jsc.StoreDir)
+	jsc.maxStorePending = true

3. After recovery, finalize the dynamic limit by adding back recovered data

Once all file-based streams have been recovered, the server adjusts the dynamic limit to add back the space already used by JetStream, scaled by the same factor as the original calculation (75%):

+// finalizeDynamicMaxStore settles a dynamic max store limit once all file based
+// streams have been recovered. diskAvailable only reports free space, so add back
+// what we occupy ourselves to keep the limit stable across restarts.
+func (js *jetStream) finalizeDynamicMaxStore() {
+	js.mu.Lock()
+	if !js.config.maxStorePending {
+		js.mu.Unlock()
+		return
+	}
+	// From here on the limit is real and can be enforced, see sufficientResources.
+	js.config.maxStorePending = false
+	recovered := atomic.LoadInt64(&js.storeUsed)
+	if recovered <= 0 {
+		js.mu.Unlock()
+		return
+	}
+	// diskAvailable is already scaled down to 75% of what is free, so scale the
+	// recovered bytes the same way before adding them back.
+	maxStore := addSaturate(js.config.MaxStore, recovered/4*3)
+	js.config.MaxStore = maxStore
+	atomic.StoreInt64(&js.storeMax, maxStore)
+	js.mu.Unlock()
+}

This ensures the dynamic limit reflects both free and already-used space, keeping it stable across restarts.

4. Only enforce the finalized limit after recovery is complete

While the dynamic limit is still provisional (maxStorePending is true), resource checks skip enforcing the store limit:

-	if js.storeReserved+totalMaxStore > js.config.MaxStore {
+	// A dynamic limit is still provisional until recovery has run.
+	recovering := js.config.maxStorePending
+	if !recovering && js.storeReserved+totalMaxStore > js.config.MaxStore {
 		return NewJSStorageResourcesExceededError()
 	}

This prevents false failures during recovery, before the real limit is known.

5. Test: Ensure the dynamic limit is stable across restarts

A new test verifies that after writing a significant amount of data, the dynamic limit does not shrink unexpectedly on restart:

func TestJetStreamDynamicMaxStoreStableAcrossRestart(t *testing.T) {
	// ... setup omitted ...
	before := s.JetStreamConfig().MaxStore
	// ... write data ...
	s.Shutdown()
	// ... restart with a limit that would be invalid if the dynamic limit shrank ...
	after := s.JetStreamConfig().MaxStore
	if slack := int64(written / 4); before-after > slack {
		t.Fatalf("Dynamic max store shrank after writing %v: before %v, after %v",
			friendlyBytes(int64(written)), friendlyBytes(before), friendlyBytes(after))
	}
}

This ensures the fix works as intended and prevents regression.

From the review thread

synadia-claude-reviewer[bot]:
The fix is correct and the approach is well-designed.
...
The fix adds back recovered * 0.75 (the same 75 % factor used by diskAvailable) to restore the limit to approximately what it was before data was written.
...
The bypass in sufficientResources: Skipping the store check while maxStorePending is true is necessary and the blast radius is small — the server is not accepting client connections during this window, so only the internal recovery path runs.
...
Test: TestJetStreamDynamicMaxStoreStableAcrossRestart covers both failure modes — server starts successfully with the previously-valid account limit, and MaxStore does not shrink by more than written/4. The math is consistent with the fix: without it before − after ≈ written × 0.75 = 48 MB > slack 16 MB; with it after ≥ before so the slack condition is trivially met.

MauriceVanVeen:
This is aligned with us now always recovering what's already on disk in other places as well.

Takeaways

• When enforcing resource limits based on available system resources, always account for resources already consumed by your own application, especially after a restart.
• If a provisional configuration is used during recovery or initialization, defer strict enforcement until the final, accurate limit is established.
• Automated tests should simulate both the "before" and "after" states (e.g., pre- and post-restart) to verify that resource calculations remain stable and correct across application lifecycle events.