bpf, datapath: move IPv4 SNAT exclusion to runtime config

Project: cilium/cilium | Language: Go, C (eBPF) | Concepts: configuration-management, refactoring | Difficulty: mid
Link: https://github.com/cilium/cilium/pull/48065

The problem

The IPv4 SNAT exclusion CIDR (range of destination addresses for which SNAT is skipped) was previously hardcoded via macros (IPV4_SNAT_EXCLUSION_DST_CIDR, etc.) in the BPF code. This made it inflexible and difficult to change at runtime or configure per node, limiting operational flexibility and maintainability.

Why it's non-trivial

A naive fix might simply replace the macro with a variable, but this would not integrate with the node-level runtime configuration system. Properly addressing the problem requires:

• Defining a new runtime config structure that can be updated per node.
• Ensuring type definitions are available in all relevant BPF and Go code.
• Avoiding initialization order issues with C struct definitions.
• Updating all code paths (Go and C) to use the new config field instead of the macro.

The approach

1. Remove the hardcoded macros and their usage.

The old macros are deleted:

-# ifdef ENABLE_MASQUERADE_IPV4
-#  define IPV4_SNAT_EXCLUSION_DST_CIDR 0xffff0000
-#  define IPV4_SNAT_EXCLUSION_DST_CIDR_LEN 16
-# endif /* ENABLE_MASQUERADE_IPV4 */

This removes the static, compile-time CIDR exclusion.

2. Define a new runtime struct for SNAT exclusion in node config.

A new struct is introduced for the exclusion prefix:

+struct ipv4_snat_exclusion_prefix {
+	union v4addr dst_addr;
+	__u8 bits;
+	bool enabled;
+};
+
+NODE_CONFIG(struct ipv4_snat_exclusion_prefix, ipv4_snat_exclusion,
+	    "IPv4 destination prefix excluded from SNAT")

This allows the destination address, prefix length, and enablement to be set at runtime.

3. Move union v4addr to a dedicated header for type availability.

Previously, union v4addr was defined in common.h, causing include order issues. It's now in its own header:

+#include <lib/ipv4_core.h>

And the new header:

+union v4addr {
+	__u8 addr[4];
+	__be32 be32;
+};

This makes the type available wherever needed, including in struct field declarations.

4. Update BPF logic to use the new config field.

The SNAT exclusion logic now checks the runtime config instead of the macro:

-#ifdef IPV4_SNAT_EXCLUSION_DST_CIDR
-	if (ipv4_is_in_subnet(tuple->daddr, IPV4_SNAT_EXCLUSION_DST_CIDR,
-			      IPV4_SNAT_EXCLUSION_DST_CIDR_LEN))
+	if (CONFIG(ipv4_snat_exclusion).enabled &&
+	    ipv4_is_in_subnet(tuple->daddr,
+			      CONFIG(ipv4_snat_exclusion).dst_addr.be32,
+			      CONFIG(ipv4_snat_exclusion).bits))
 		return NAT_PUNT_TO_STACK;
-#endif

This makes the exclusion dynamic and node-specific.

5. Wire up the Go side to populate the new config.

The Go node config struct is extended:

+	// IPv4 destination prefix excluded from SNAT.
+	IPv4SNATExclusion types.IPv4SNATExclusionPrefix `config:"ipv4_snat_exclusion"`

And the population logic:

+	if option.Config.EnableBPFMasquerade && option.Config.EnableIPv4Masquerade {
+		excludeCIDR := lnc.NativeRoutingCIDRIPv4
+		if option.Config.EnableIPMasqAgent {
+			excludeCIDR = option.Config.IPv4NativeRoutingCIDR
+		}
+
+		if excludeCIDR.IsValid() {
+			node.IPv4SNATExclusion.DstAddr.Addr = excludeCIDR.Addr().As4()
+			node.IPv4SNATExclusion.Bits = uint8(excludeCIDR.Bits())
+			node.IPv4SNATExclusion.Enabled = true
+		}
+	}

This ensures the runtime config is set from the correct source.

6. Remove macro-based config generation in Go.

The code that wrote the macro definitions to the BPF C header is removed:

-			if excludeCIDR.IsValid() {
-				cDefinesMap["IPV4_SNAT_EXCLUSION_DST_CIDR"] = fmt.Sprintf("%#x", byteorder.NetIPAddrToHost32(excludeCIDR.Addr()))
-				cDefinesMap["IPV4_SNAT_EXCLUSION_DST_CIDR_LEN"] = fmt.Sprintf("%d", excludeCIDR.Bits())
-			}

Now, exclusion is handled through the runtime config.

7. Update tests and generated types.

Test helpers and generated Go types are updated to reflect the new config struct:

+	t.IPv4SNATExclusion.DstAddr.Addr = [4]byte{0xff, 0xff, 0x00, 0x00}
+	t.IPv4SNATExclusion.Bits = 16
+	t.IPv4SNATExclusion.Enabled = true

And:

+// IPv4SNATExclusionPrefix is generated from the BPF C type ipv4_snat_exclusion_prefix.
+type IPv4SNATExclusionPrefix struct {
+	_       structs.HostLayout
+	DstAddr struct {
+		_    structs.HostLayout
+		Addr [4]uint8
+	}
+	Bits    uint8
+	Enabled bool
+	_       [2]byte
+}

From the review thread

tklauser: Why not make this a union v4addr? That way also the host endianness conversion won't need to happen in NodeConfig like with other runtime config IPv4 addresses.

viktor-kurchenko: Good point, but the problem with this is that union v4addr is currently incomplete when config/node.h is parsed because it is defined later in lib/common.h.
Does it make sense to extract the address type into a lightweight shared header?

tklauser: Huh, but I see other NODE_CONFIG declarations using union v4addr in config/node.h, e.g. here:
https://github.com/cilium/cilium/blob/90c05a6a583dc1f03bf745c6bee41461d00cae44/bpf/include/bpf/config/node.h#L24
But yeah, it might make sense to move it to a separate dedicated header, like we already do for union v6addr in bpf/lib/ipv6_core.h.

viktor-kurchenko:

Huh, but I see other NODE_CONFIG declarations using union v4addr in config/node.h
Yeah, that's because compiler translates macro to a tentative file-scope definition which type can be resolved later. A struct field type must be resolved immediately.
But yeah, it might make sense to move it to a separate dedicated header, like we already do for union v6addr in bpf/lib/ipv6_core.h.
Done, thanks!

Takeaways

• Prefer runtime configuration over compile-time macros for operational flexibility and per-node customization.
• When sharing types between C and Go (or across multiple C headers), isolate them in dedicated headers to avoid include order and definition issues.
• Refactoring configuration logic requires updating all code paths—data structures, initialization, and tests—to ensure consistency and correctness.