kube-scheduler: redact extender TLS key data from the component config log

Project: kubernetes/kubernetes | Language: Go | Concepts: security, logging, error-handling, testing-strategy | Difficulty: mid
Link: https://github.com/kubernetes/kubernetes/pull/140519

The problem

When kube-scheduler is run with a verbosity level of 2 or higher, it logs its full resolved configuration as YAML. If an extender is configured with inline TLS credentials, the private key (tlsConfig.keyData) is included in the log as base64. This exposes sensitive key material in logs, which is a security risk.

Why it's non-trivial

A naive fix might try to use struct field tags (like datapolicy:"security-key") to prevent logging sensitive fields. However, this approach does not work here because the entire configuration is serialized to a YAML string before logging, so field-level policies are bypassed. The fix needs to ensure that secrets are redacted before serialization and logging, without mutating the original configuration or affecting config file output.

The approach

1. Define a redacted placeholder for secrets:

+// redactedBytes serializes to the literal string "REDACTED" when encoded as
+// base64 in YAML or JSON output.
+var redactedBytes, _ = base64.StdEncoding.DecodeString("REDACTED")

Defines a byte slice that, when base64-encoded, produces the string "REDACTED". This will be used as a placeholder for redacted key data in logs.

2. Redact secrets before logging:

+	if loggerV.Enabled() {
+		buf, err := encodeConfig(redactSecrets(cfg))
+		if err != nil {
+			return err
+		}
+		loggerV.Info("Using component config", "config", buf.String())
+	}

When logging the config, a copy with secrets redacted is encoded and logged. The original config is not modified.

3. Only redact for logging, not for writing config files:

 	if len(fileName) > 0 {
+		buf, err := encodeConfig(cfg)
+		if err != nil {
+			return err
+		}
 		configFile, err := os.Create(fileName)
 		if err != nil {
 			return err
 		}
 		defer configFile.Close()
 		_, err = io.Copy(configFile, buf)
 		return err
 	}

When writing the config to a file (for reuse), the full, unredacted config is used.

4. Implement the redaction logic:

+// redactSecrets returns a copy of cfg with inline extender TLS key material
+// replaced by a placeholder, so that the config can be logged without
+// disclosing secrets. The datapolicy tag on KeyData cannot take effect here
+// because the whole config is flattened into a single string before it
+// reaches the logger.
+func redactSecrets(cfg *config.KubeSchedulerConfiguration) *config.KubeSchedulerConfiguration {
+	redacted := cfg.DeepCopy()
+	for i := range redacted.Extenders {
+		if tlsConfig := redacted.Extenders[i].TLSConfig; tlsConfig != nil && len(tlsConfig.KeyData) > 0 {
+			tlsConfig.KeyData = redactedBytes
+		}
+	}
+	return redacted
+}

Makes a deep copy of the config and replaces any extender TLS key data with the redacted placeholder.

5. Add a test to verify redaction:

+func TestLogOrWriteConfigRedactsExtenderKeyData(t *testing.T) {
+	logger := ktesting.NewLogger(t, ktesting.NewConfig(ktesting.Verbosity(2), ktesting.BufferLogs(true)))
+	keyData := []byte("-----BEGIN PRIVATE KEY-----\nsecret-bytes\n-----END PRIVATE KEY-----")
+	cfg := &config.KubeSchedulerConfiguration{
+		TypeMeta: metav1.TypeMeta{APIVersion: configv1.SchemeGroupVersion.String()},
+		Extenders: []config.Extender{{
+			URLPrefix:   "https://extender.example.com",
+			EnableHTTPS: true,
+			TLSConfig:   &config.ExtenderTLSConfig{KeyData: keyData},
+		}},
+	}
+
+	if err := LogOrWriteConfig(logger, "", cfg, nil); err != nil {
+		t.Fatal(err)
+	}
+
+	output := logger.GetSink().(ktesting.Underlier).GetBuffer().String()
+	if strings.Contains(output, base64.StdEncoding.EncodeToString(keyData)) {
+		t.Error("extender TLS key data leaked into the logged config")
+	}
+	if !strings.Contains(output, "keyData: REDACTED") {
+		t.Errorf("expected redacted keyData in the logged config, got:\n%s", output)
+	}
+	if !bytes.Equal(cfg.Extenders[0].TLSConfig.KeyData, keyData) {
+		t.Error("LogOrWriteConfig mutated the caller's config")
+	}
+}

Tests that the log output does not contain the actual key data, does contain the "REDACTED" placeholder, and that the original config is not mutated.

From the review thread

pacoxu:
/ok-to-test
This appears to be a security-sensitive bug because extender TLS private key material may be exposed in kube-scheduler logs.

Could SIG Scheduling leads @kubernetes/sig-scheduling-leads please help assess the severity, affected versions(probably introduced long time ago), and whether this should be treated as a v1.37 release blocker?

If confirmed, I think the fix should also be considered for cherry-picking to the supported release branches.

sanposhiho:
/lgtm
/approve

@pacoxu This has existed for a long time not a newly introduced issue. Also, I expect only a few people use extenders. So, it is not a hard blocker for the release. But, I support cherry-picking to old versions.
@kubernetes/release-managers in this case, should we merge this PR to 1.37.0, or should we just wait for 1.37.0 is out and cherry-pick to 1.37.1 (w/ other old versions) later?

liggitt:
this overlaps with https://github.com/kubernetes/kubernetes/pull/140162 which is addressing the same issue, but systemically

I agree this is not new and not a regression, I don't think it's release-blocking for 1.37

Takeaways

Always sanitize or redact sensitive data before logging, especially when logs are generated from serialized configuration objects.
Field-level data policies (like struct tags) may not be effective if the entire structure is serialized before logging; explicit redaction logic may be necessary.
When redacting sensitive information, ensure that the original data structure is not mutated unless intended—work on a copy.
Automated tests should verify both that secrets are not leaked and that the redaction does not alter the original data.