sea: keep ELF segments on separate pages in --build-sea output

Project: nodejs/node | Language: JavaScript, C++ | Concepts: memory-management, testing-strategy | Difficulty: mid
Link: https://github.com/nodejs/node/pull/65564

The problem

On certain Linux distributions (notably RHEL 8), executables built with Node.js's --build-sea option could fail to run with a segmentation fault. The kernel error message indicated that ELF segments overlapped in memory ("elf segment at ... requested but the memory is mapped already"). This happened because the ELF loader tried to map two PT_LOAD segments onto the same memory page, which is not allowed by kernels that use MAP_FIXED_NOREPLACE (Linux < 5.4, RHEL 8).

Why it's non-trivial

A naive fix might simply try to realign or reorder segments, but the issue is subtle:

• The layout of ELF segments is determined by the linker and can change depending on build options or section sizes.
• The LIEF library (used for ELF manipulation) would, by default, move the program header into the largest gap between segments, potentially causing two segments to share a page if that gap is not page-aligned.
• Only non-PIE (Position-Dependent Executable) binaries are affected, and the problem can come and go depending on the exact binary layout, making it hard to test and verify.

The approach

1. Change the ELF program header relocation strategy for non-PIE executables

+  if (binary->header().file_type() == LIEF::ELF::Header::FILE_TYPE::EXEC &&
+      binary->relocate_phdr_table(LIEF::ELF::Binary::PHDR_RELOC::BSS_END) ==
+          0) {
+    return {InjectResult::kError, {}, "Failed to relocate ELF program headers"};
+  }

• What it does:
  • Checks if the ELF file is a non-PIE executable (ET_EXEC).
  • Calls relocate_phdr_table with PHDR_RELOC::BSS_END, which moves the program header after the .bss section, ensuring it gets its own set of pages.
  • If relocation fails, returns an error.
• Why:
  • This avoids overlapping segment mappings by ensuring every segment, including the program header, resides on its own page(s). This matches what the postject tool does and sidesteps the problematic default LIEF behavior.

2. Add a regression test to verify that no two PT_LOAD segments share a page

+const elf = readFileSync(buildSEA(fixtures.path('sea', 'basic')));
+assert.strictEqual(elf[4], 2);  // ELFCLASS64
+const phoff = Number(elf.readBigUInt64LE(0x20));
+const phentsize = elf.readUInt16LE(0x36);
+const phnum = elf.readUInt16LE(0x38);
+const loads = [];
+for (let i = 0; i < phnum; i++) {
+  const at = phoff + i * phentsize;
+  if (elf.readUInt32LE(at) !== 1) continue;  // PT_LOAD
+  const vaddr = elf.readBigUInt64LE(at + 0x10);
+  const memsz = elf.readBigUInt64LE(at + 0x28);
+  if (memsz > 0n) loads.push({ first: vaddr >> 12n, last: (vaddr + memsz - 1n) >> 12n });
+}
+loads.sort((a, b) => (a.first < b.first ? -1 : 1));
+for (let i = 1; i < loads.length; i++) {
+  assert(loads[i].first > loads[i - 1].last,
+         `PT_LOAD ${i} starts on page 0x${loads[i].first.toString(16)}, ` +
+         `which PT_LOAD ${i - 1} already covers`);
+}

• What it does:
  • Reads the built ELF binary.
  • Parses the program headers to find all PT_LOAD segments.
  • Calculates the page ranges for each segment.
  • Asserts that no two segments overlap on the same page.
• Why:
  • This directly tests for the kernel-level problem: overlapping segment mappings. It ensures the fix is robust and future changes don't reintroduce the bug.

From the review thread

No notable review discussion on this PR.

Takeaways

• Always consider platform-specific loader behavior when manipulating binary formats. What works on one kernel or toolchain may fail on another, especially with low-level details like memory mapping.
• Regression tests for binary layout bugs should check invariants at the byte level. Automated tests that parse and validate ELF headers or segment layouts can catch subtle issues that only manifest on certain systems.
• When using third-party libraries for binary manipulation, understand and, if needed, override their default strategies. Defaults may not be safe for all use cases or platforms.