[SelectionDAG] Fix load/store legalization for softened x86_fp80

Project: llvm/llvm-project | Language: C++ | Concepts: error-handling, refactoring, testing-strategy | Difficulty: senior
Link: https://github.com/llvm/llvm-project/pull/209991

The problem

When compiling for x86 targets with x87 floating-point disabled, the x86_fp80 type (80-bit extended precision float) is "softened" to a wider integer type (e.g., i96). The SelectionDAG type legalizer would then emit regular loads and stores using this widened type. This caused a "Size mismatch" assertion failure, because the memory width (80 bits) did not match the register width (96 bits) of the softened type.

Why it's non-trivial

A naive fix might simply use the widened integer type for loads and stores, but this ignores that the actual memory layout is still 80 bits. Directly loading or storing a 96-bit value from/to 80 bits of memory is incorrect and triggers assertions. The solution needs to handle the width mismatch explicitly and safely, ensuring correctness regardless of the softening.

The approach

1. Use extending loads for widened softened types

Instead of always loading the widened type, the code now checks if the softened type (NVT) is wider than the original memory type (MemVT). If so, it performs an extending load from the original memory width to the softened type:

+    EVT MemVT = EVT::getIntegerVT(*DAG.getContext(), VT.getSizeInBits());
+    if (NVT.bitsGT(MemVT))
+      NewL = DAG.getLoad(L->getAddressingMode(), ISD::EXTLOAD, NVT, dl,
+                         L->getChain(), L->getBasePtr(), L->getOffset(),
+                         L->getPointerInfo(), MemVT, L->getBaseAlign(),
+                         MMOFlags, L->getAAInfo());
+    else
+      NewL = DAG.getLoad(L->getAddressingMode(), L->getExtensionType(), NVT, dl,
+                         L->getChain(), L->getBasePtr(), L->getOffset(),
+                         L->getPointerInfo(), NVT, L->getBaseAlign(), MMOFlags,
+                         L->getAAInfo());

What this does:

• If the softened type is wider than the memory type, it loads the original (narrower) memory width and extends it to the register width, avoiding size mismatches.
• Otherwise, it performs a regular load.

2. Use truncating stores for widened softened types

Similarly, when storing a softened value, the code checks if the value is wider than the memory type. If so, it truncates the value before storing:

+  EVT MemVT =
+      EVT::getIntegerVT(*DAG.getContext(), ST->getMemoryVT().getSizeInBits());
+  if (Val.getValueType().bitsGT(MemVT))
+    return DAG.getTruncStore(ST->getChain(), dl, Val, ST->getBasePtr(), MemVT,
+                             ST->getMemOperand());
   return DAG.getStore(ST->getChain(), dl, Val, ST->getBasePtr(),
                       ST->getMemOperand());

What this does:

• If the value's type is wider than the memory type, it truncates the value to the correct width before storing, ensuring the store matches the memory layout.

3. Add regression tests

A new test file verifies correct code generation for x86_fp80 loads and stores with x87 disabled:

; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=-x87 | FileCheck %s --check-prefix=X64
; RUN: llc < %s -mtriple=i686-unknown-linux-gnu -mattr=-x87 | FileCheck %s --check-prefix=X86

define void @test_load_store_f80(ptr %p, ptr %q) #0 {
  %v = load x86_fp80, ptr %p, align 16
  store x86_fp80 %v, ptr %q, align 16
  ret void
}

What this does:

• Ensures that the compiler generates correct instructions for loads and stores of x86_fp80 when the type is softened and x87 is disabled.

From the review thread

No notable review discussion on this PR.

Takeaways

• When transforming types, always ensure that memory operations (loads/stores) match the actual memory layout, not just the register representation.
• Use extending loads and truncating stores to safely bridge width mismatches between softened types and memory types.
• Add targeted regression tests whenever fixing a crash or assertion, especially for platform-specific or rarely used configurations.