scrape: support zstd-compressed responses

Project: prometheus/prometheus | Language: Go | Concepts: memory-management, error-handling, configuration-management | Difficulty: mid
Link: https://github.com/prometheus/prometheus/pull/19502

The problem

Prometheus could only decompress scrape responses using gzip or no compression. Some exporters now support Zstandard (zstd) compression, which is faster and more efficient. Prometheus needed to support zstd-compressed responses, but only when explicitly enabled, and without risking excessive memory use or breaking existing behavior.

Why it's non-trivial

• Zstd decoders can use a large amount of memory by default (up to 512MB), so a safe limit must be enforced.
• The feature must be opt-in via a feature flag, not enabled for all users.
• The Accept-Encoding negotiation must be correct and not break existing gzip/identity handling.
• Decoder pooling and reuse must be handled safely and efficiently.
• Error handling for malformed or oversized compressed responses must be robust.

The approach

1. Add feature flag and configuration plumbing

The feature flag is introduced and wired through the configuration:

+			case "zstd-scrape":
+				c.scrape.EnableZstdScrape = true
+				logger.Info("Experimental zstd scrape compression enabled")

This snippet adds the zstd-scrape feature flag, enabling zstd support when set.

The flag is documented and added to the CLI and docs:

-| <code class="text-nowrap">--enable-feature</code> <code class="text-nowrap">...</code> | Comma separated feature names to enable. Valid options: ... |  |
+| <code class="text-nowrap">--enable-feature</code> <code class="text-nowrap">...</code> | Comma separated feature names to enable. Valid options: ..., zstd-scrape. ... |  |

This keeps documentation and CLI help accurate and up to date.

2. Accept-Encoding negotiation

The Accept-Encoding header is set based on the feature flag:

-func acceptEncodingHeader(enableCompression bool) string {
-	if enableCompression {
-		return "gzip"
-	}
-	return "identity"
+func acceptEncodingHeader(enableCompression, enableZstd bool) string {
+	if enableCompression {
+		if enableZstd {
+			return "zstd,gzip"
+		}
+		return "gzip"
+	}
+	return "identity"
 }

This ensures that zstd is only advertised if the feature is enabled, preserving existing behavior otherwise.

3. Zstd decoder pooling and memory window limit

A sync.Pool is used to efficiently reuse zstd decoders, and the decoder window is capped to prevent excessive memory usage:

+const zstdMaxWindowSize = 8 << 20
+
+var zstdDecoderPool = sync.Pool{
+	New: func() any {
+		decoder, err := zstd.NewReader(
+			nil,
+			zstd.WithDecoderConcurrency(1),
+			zstd.WithDecoderMaxWindow(zstdMaxWindowSize),
+		)
+		if err != nil {
+			panic(err)
+		}
+		return decoder
+	},
+}

This creates a pool of zstd decoders with a maximum window size of 8 MiB, mitigating the risk of high memory usage.

4. Response decompression logic

The main decompression logic is updated to handle zstd, with robust error handling and debug logging:

+	switch resp.Header.Get("Content-Encoding") {
+	case "gzip":
+		// existing gzip logic...
+	case "zstd":
+		if !s.enableZstd {
+			break
+		}
+		if s.Target == nil {
+			s.logger.Debug("Using zstd-compressed scrape response")
+		} else {
+			s.logger.Debug("Using zstd-compressed scrape response", "target", s.URL().Redacted())
+		}
+		zstdr := zstdDecoderPool.Get().(*zstd.Decoder)
+		if err := zstdr.Reset(resp.Body); err != nil {
+			_ = zstdr.Reset(nil)
+			zstdDecoderPool.Put(zstdr)
+			return "", err
+		}
+		defer func() {
+			_ = zstdr.Reset(nil)
+			zstdDecoderPool.Put(zstdr)
+		}()
+		reader = zstdr
+	}

This handles zstd responses only if enabled, emits a debug log, and ensures decoders are properly reset and returned to the pool even on error.

5. Comprehensive tests

Tests are added for:

• Negotiation and decoding of identity/gzip/zstd responses
• Enforcing decompressed body size limits
• Handling malformed compressed responses
• Enforcing zstd window limit
• Decoder pool reuse/reset

Example test for zstd window limit:

+func TestTargetScraperZstdWindowLimit(t *testing.T) {
+	encoder, err := zstd.NewWriter(nil, zstd.WithWindowSize(2*zstdMaxWindowSize))
+	require.NoError(t, err)
+	t.Cleanup(func() { encoder.Close() })
+
+	compressed := encoder.EncodeAll(bytes.Repeat([]byte("a"), zstdMaxWindowSize+1), nil)
+	resp := &http.Response{
+		StatusCode: http.StatusOK,
+		Header: http.Header{
+			"Content-Encoding": []string{"zstd"},
+		},
+		Body: io.NopCloser(bytes.NewReader(compressed)),
+	}
+	ts := &targetScraper{
+		bodySizeLimit: math.MaxInt64,
+		metrics:       newTestScrapeMetrics(t),
+		enableZstd:    true,
+		logger:        promslog.NewNopLogger(),
+	}
+	var buf bytes.Buffer
+	_, err = ts.readResponse(context.Background(), resp, &buf)
+	require.Error(t, err)
+}

This verifies that the decoder rejects zstd frames with a window size above the configured limit.

From the review thread

krajorama: gzip has a sane preset buffer limit (32KB), but zstd decoder can use 512MB by default. So maybe we want to set a lower WithDecoderMaxWindow(???) here to protect against too big decode window? I'm not familiar with zstd, but maybe a couple of megabytes would be ok? Also, not sure if Reset later would change the window size of concurrency?

fancive: Good point. I added WithDecoderMaxWindow(8 << 20) to cap the decoder window at 8 MiB, together with a test verifying that frames exceeding the limit are rejected.

fancive: Reset preserves the decoder options configured by NewReader. ResetWithOptions is only needed when changing resettable options between streams, so the existing concurrency and maximum window settings remain in effect.

Takeaways

• Always enforce resource limits when adding support for new compression formats, to guard against excessive memory usage.
• Gate experimental or potentially risky features behind explicit configuration flags, preserving default stability.
• Use pooling for expensive objects (like decoders) but ensure proper reset and error handling to avoid resource leaks or stale state.
• Comprehensive tests should cover not just the happy path, but also error cases, resource limits, and negotiation logic.