unlock unlimited code runs for $79, limited time →
NOSSLETTER.techLIVE
back to OSSpedia
argoproj/argo-cd
mid7/10
observabilitytesting-strategyapi-design

feat(hydrator): add metrics to app info

Project: argoproj/argo-cd | Language: Go | Concepts: observability, testing-strategy, api-design | Difficulty: mid
Link: https://github.com/argoproj/argo-cd/pull/29514

The problem

Argo CD's Prometheus metrics for applications (argocd_app_info) did not expose any information about the status of the application's "hydrator" feature. This made it difficult for operators to monitor or alert on the hydration state of applications, especially when troubleshooting or building dashboards.

Why it's non-trivial

A naive fix might simply add a hydrator field to the metric without considering:

  • The difference between apps that do not use a hydrator at all, apps that use it but have no status yet, and apps with a known hydrator phase.
  • The need to update all metric consumers and documentation to reflect the new label.
  • Ensuring tests cover all possible hydrator states and that the metric remains backward-compatible for apps without hydrators.

The approach

1. Add a new hydrator_status label to the metric descriptor:

-	append(descAppDefaultLabels, "autosync_enabled", "repo", "dest_server", "dest_namespace", "sync_status", "health_status", "operation", "phase"),
+	append(descAppDefaultLabels, "autosync_enabled", "repo", "dest_server", "dest_namespace", "sync_status", "health_status", "hydrator_status", "operation", "phase"),

This expands the metric's label set to include the hydrator status, so every emitted argocd_app_info metric will have this label (even if empty).

2. Populate the hydrator_status value during metric collection:

+	var hydratorStatus string
+	if app.Spec.SourceHydrator != nil {
+		hydratorStatus = string(argoappv1.HydrateOperationPhaseUnknown)
+		if op := app.Status.SourceHydrator.CurrentOperation; op != nil && op.Phase != "" {
+			hydratorStatus = string(op.Phase)
+		}
+	}

If the application configures a hydrator, the code checks for a current operation phase:

  • If set, it uses that phase (e.g., "Hydrating", "Hydrated", "Failed").
  • If not set, it defaults to "Unknown".
  • If the hydrator is not configured, the label is left empty.

3. Pass the new label to the metric emission:

-	addGauge(descAppInfo, 1, strconv.FormatBool(autoSyncEnabled), git.NormalizeGitURL(app.Spec.GetSource().RepoURL), destServer, app.Spec.Destination.Namespace, string(syncStatus), string(healthStatus), operation, operationPhase)
+	addGauge(descAppInfo, 1, strconv.FormatBool(autoSyncEnabled), git.NormalizeGitURL(app.Spec.GetSource().RepoURL), destServer, app.Spec.Destination.Namespace, string(syncStatus), string(healthStatus), hydratorStatus, operation, operationPhase)

This ensures every metric line includes the new label, preserving label order and compatibility.

4. Update tests to cover all hydrator states:

+const fakeAppHydratorHydrated = `...phase: Hydrated...`
+const fakeAppHydratorHydrating = `...phase: Hydrating...`
+const fakeAppHydratorFailed = `...phase: Failed...`
+const fakeAppHydratorUnknown = `...sourceHydrator: {}...`

Four new test fixtures are added for each possible hydrator state.

+func TestMetricsHydratorStatus(t *testing.T) {
+	testApp(t, []string{
+		fakeAppHydratorHydrated,
+		fakeAppHydratorHydrating,
+		fakeAppHydratorFailed,
+		fakeAppHydratorUnknown,
+	}, `
+# HELP argocd_app_info Information about application.
+# TYPE argocd_app_info gauge
+argocd_app_info{...,hydrator_status="Failed",...} 1
+argocd_app_info{...,hydrator_status="Hydrated",...} 1
+argocd_app_info{...,hydrator_status="Hydrating",...} 1
+argocd_app_info{...,hydrator_status="Unknown",...} 1
+`)
+}

This test ensures that the metric output contains the correct hydrator_status label value for each scenario.

5. Update documentation to describe the new label:

-| `argocd_app_info`                                 |   gauge   | Information about Applications. It contains labels such as `sync_status` and `health_status` that reflect the application state in Argo CD.                                                             |
+| `argocd_app_info`                                 |   gauge   | Information about Applications. It contains labels such as `sync_status`, `health_status`, and `hydrator_status` that reflect the application state in Argo CD.                                                             |

+| hydrator_status    | Failed                          | Source Hydrator phase for apps with `spec.sourceHydrator` set. Empty when hydrator is not configured on the Application. `Unknown` when hydrator is configured but `status.sourceHydrator.currentOperation.phase` is absent or empty. Possible values: `Hydrating`, `Failed`, `Hydrated`, `Unknown`.         |

Documentation now explains the semantics and possible values for the new label.

From the review thread

No notable review discussion on this PR.

Takeaways

  • Metrics should distinguish between "unset" and "unknown" states: When adding observability features, it's important to differentiate between an option not being configured at all (empty label) and an option being configured but in an unknown state ("Unknown").
  • Tests must cover all new code paths and edge cases: When introducing a new metric label, add test cases for every possible value, including empty and unknown.
  • Documentation must be kept in sync with code changes: Any externally observable change (like a new metric label) should be reflected in user-facing documentation to prevent confusion and ensure adoption.