[X86] Match ENDBR immediates in little-endian order

Project: llvm/llvm-project | Language: C++ | Concepts: security, error-handling | Difficulty: mid
Link: https://github.com/llvm/llvm-project/pull/208754

The problem

The X86 backend in LLVM includes logic to prevent emitting immediate values in code that, when interpreted as bytes, match the ENDBR32 or ENDBR64 instruction sequences. These sequences are used for Intel Control-Flow Enforcement Technology (CET) branch protection. However, the check was only matching the opcode bytes in display order, not as they appear in little-endian memory layout (which is how immediates are actually encoded). As a result, some problematic byte sequences could slip through and appear in binaries, potentially creating security gadgets.

Why it's non-trivial

A naive fix might just compare integer values against the ENDBR opcode as a 32-bit or 64-bit integer in display order. But on x86, immediates are encoded in little-endian order, so the actual byte pattern in the binary is the reverse of the display order. Furthermore, optional x86 prefix bytes can be present between the leading 0xF3 and the rest of the opcode. The check must scan the immediate byte-by-byte, accounting for these prefixes, and do so for immediates of varying sizes.

The approach

1. Replace the old check with a byte-wise scan in little-endian order

The previous function only matched the opcode at fixed positions and in display order:

-static bool isEndbrImm64(uint64_t Imm) {
-// There may be some other prefix bytes between 0xF3 and 0x0F1EFA.
-// i.g: 0xF3660F1EFA, 0xF3670F1EFA
-  if ((Imm & 0x00FFFFFF) != 0x0F1EFA)
+static bool isEndbrImm(uint64_t Imm, unsigned BitWidth) {
+  assert(BitWidth <= 64 && BitWidth % 8 == 0 && "Unexpected immediate size");
+
+  const unsigned NumBytes = BitWidth / 8;
+  if (NumBytes < 4)
     return false;

This new function asserts the immediate is a multiple of bytes, and skips checks for too-small values.

2. Extract bytes and scan for the ENDBR pattern, allowing for optional prefixes

+  const uint8_t OptionalPrefixBytes[] = {0x26, 0x2e, 0x36, 0x3e, 0x64,
+                                         0x65, 0x66, 0x67, 0xf0, 0xf2};
+  uint8_t Bytes[8];
+  for (unsigned I = 0; I != NumBytes; ++I)
+    Bytes[I] = (Imm >> (I * 8)) & 0xFF;
+
+  for (unsigned I = 0; I + 3 < NumBytes; ++I) {
+    if (Bytes[I] != 0xf3)
+      continue;
+
+    unsigned J = I + 1;
+    while (J < NumBytes && llvm::is_contained(OptionalPrefixBytes, Bytes[J]))
+      ++J;
+
+    if (J + 2 < NumBytes && Bytes[J] == 0x0f && Bytes[J + 1] == 0x1e &&
+        (Bytes[J + 2] == 0xfa || Bytes[J + 2] == 0xfb))
+      return true;
+  }

This loop walks through the immediate byte-by-byte (little-endian order), looking for a 0xF3 byte, then skips any optional prefix bytes, and finally checks for the ENDBR opcode sequence.

3. Mask the complement value to the correct integer width

When replacing the offending constant, the code now ensures the complement is masked to the original bit width:

+          uint64_t ComplementImm =
+              (~Imm) & maskTrailingOnes<uint64_t>(VT.getSizeInBits());
+          SDValue Complement =
+              CurDAG->getConstant(ComplementImm, dl, VT, false, true);
+          Complement = CurDAG->getNOT(dl, Complement, VT);

This prevents accidental sign-extension or width mismatches when constructing the replacement value.

4. Update tests to match new logic and endianness

The test cases now use immediates in little-endian order and check for the correct masking and splitting of these values:

-; a = 0xF30F1EFA
+; a = 0xFA1E0FF3

Test comments and values are updated to match the real encoding order.

From the review thread

No notable review discussion on this PR.

Takeaways

• When matching instruction or data byte patterns for security or correctness, always match the actual in-memory encoding, not just display order.
• For variable-width values, write pattern-matching logic that works byte-wise and is robust to optional or variable-length prefixes.
• When constructing replacement values for a specific bit width, always mask or truncate to avoid accidental sign extension or overflow.