unlock unlimited code runs for $79, limited time →
NOSSLETTER.techLIVE
back to OSSpedia
envoyproxy/envoy
mid7/10
configuration-managementapi-designerror-handlingtesting-strategy

ai_protocol_manager: make inline string threshold configurable

Project: envoyproxy/envoy | Language: C++ | Concepts: configuration-management, api-design, error-handling, testing-strategy | Difficulty: mid
Link: https://github.com/envoyproxy/envoy/pull/47075

The problem

The AI Protocol Manager filter uses a JSON parser (JsonWithExtBufParser) that keeps small string values inline and offloads larger ones to an external buffer, based on a threshold. This threshold was hardcoded to 1KiB because the parser was always constructed with a default config, making it impossible to adjust via configuration.

Why it's non-trivial

A naive fix would simply add a configuration field for the threshold, but:

  • The threshold needs to be validated (not too small or too large).
  • The value should be overridable per endpoint schema, not just globally.
  • The code must ensure the threshold is single-sourced and defaults are consistent.
  • Tests must verify both defaulting and override behaviors, as well as error handling for out-of-range values.

The approach

Expose the threshold in the API and config:

+  // Request-path parsing limits. Defaults apply when unset.
+  RequestParsingLimits limits = 2;
+}
+
+// Per-stream limits for request payload parsing.
+message RequestParsingLimits {
+  ...
+  google.protobuf.UInt32Value inline_string_threshold_bytes = 1
+      [(validate.rules).uint32 = {lte: 1048576 gte: 64}];
+}

Adds a new RequestParsingLimits message to the filter's config proto, with inline_string_threshold_bytes (min 64, max 1MiB) as a field. This allows users to configure the threshold.

Read and apply the config in the filter:

+      inline_string_threshold_bytes_(PROTOBUF_GET_WRAPPED_OR_DEFAULT(
+          proto.request_handling().limits(), inline_string_threshold_bytes,
+          JsonWithExtBufParser::kDefaultInlineStringThresholdBytes)),

Reads the configured value, or falls back to the parser's default if unset.

Pass the threshold to the parser:

-  request_parser_ = std::make_unique<JsonWithExtBufParser>(JsonWithExtBufParser::Config{});
+  JsonWithExtBufParser::Config parser_config;
+  parser_config.inline_string_threshold_bytes = inlineStringThresholdBytes();
+  request_parser_ = std::make_unique<JsonWithExtBufParser>(parser_config);

Ensures the parser is constructed with the correct threshold for each request.

Allow per-schema override of the threshold:

+uint32_t AiProtocolManagerFilter::inlineStringThresholdBytes() const {
+  if (isAiEndpoint()) {
+    if (const PayloadSchema* payload_schema =
+            AdapterRegistry::get(route_request_protocol_).schema();
+        payload_schema != nullptr) {
+      if (const std::optional<uint32_t> pinned =
+              payload_schema->requestInlineStringThresholdBytes();
+          pinned.has_value()) {
+        return *pinned;
+      }
+    }
+  }
+  return config_->inlineStringThresholdBytes();
+}

Checks if the request's schema pins its own threshold; if so, uses it, otherwise uses the filter-level default.

Support per-schema threshold in schema classes:

+  RequestSchema& inlineStringThresholdBytes(uint32_t bytes) {
+    inline_string_threshold_bytes_ = bytes;
+    return *this;
+  }
+  std::optional<uint32_t> inlineStringThresholdBytes() const {
+    return inline_string_threshold_bytes_;
+  }

Allows schemas to declare their own inline string thresholds.

Test configuration, defaults, overrides, and error handling:

+TEST(AiProtocolManagerConfigTest, InlineStringThresholdDefaultsAndOverrides) {
+  ...
+  {
+    envoy::extensions::filters::http::ai_protocol_manager::v3::AiProtocolManager proto_config;
+    proto_config.mutable_request_handling();
+    const FilterConfig config(proto_config, *stats_store.rootScope());
+    EXPECT_EQ(config.inlineStringThresholdBytes(),
+              JsonWithExtBufParser::kDefaultInlineStringThresholdBytes);
+  }
+  {
+    envoy::extensions::filters::http::ai_protocol_manager::v3::AiProtocolManager proto_config;
+    proto_config.mutable_request_handling()
+        ->mutable_limits()
+        ->mutable_inline_string_threshold_bytes()
+        ->set_value(4096);
+    const FilterConfig config(proto_config, *stats_store.rootScope());
+    EXPECT_EQ(config.inlineStringThresholdBytes(), 4096);
+  }
+}
+
+TEST(AiProtocolManagerConfigTest, RejectsOutOfRangeInlineStringThreshold) {
+  ...
+  for (const uint32_t value : {0u, 32u, 2u * 1024u * 1024u}) {
+    ...
+    EXPECT_THROW(factory.createFilterFactoryFromProto(proto_config, "stats", context).IgnoreError(),
+                 EnvoyException);
+  }
+}

Tests that the filter uses the correct default, applies overrides, and rejects out-of-range values.

From the review thread

tyxia: This inline_string_threshold_bytes here is identical to max_per_scalar_bytes that was introduced in https://github.com/envoyproxy/envoy/pull/46300.

From design point of view, We should unify the limits and connect 3 layers: ai_protocol_manager, JsonWithExtBufParser and Json::Wuffs, to have single Source of Truth and to make it extensible for different handler types

tyxia: this gte: 1 is probably confusing,

Setting a small threshold that is larger than 1 but lower than the standard field like model names 30–80 bytes, roles like "assistant, will cause the legit requests to fail with HTTP 400.

We probably should have a more realistic lower bound like 64

penguingao: do we think this belongs to a filter level config, or individual schema? I feel like this can be a per-schema thing. I guess this can be an overall upper-bound, but individual schema should be able to override this.

tyxia: I think it makes sense to have a upper-bound in the filter level as coarse-grained threshold , and individual schema override as fine-grained control if needed

Takeaways

  • When exposing resource or size limits via configuration, always validate the range to prevent both accidental breakage and resource exhaustion.
  • Allowing both global (filter-level) defaults and per-endpoint (schema-level) overrides provides flexibility for deployment-wide and fine-grained tuning.
  • Tests for configuration should cover defaulting, overrides, and error handling for invalid values.
  • When introducing new configuration fields, ensure documentation and code comments are clear but concise to avoid confusion for future maintainers.