rustdoc: Take into account edition information for keyword highlighting

Project: rust-lang/rust | Language: Rust | Concepts: api-design, backwards-compatibility, testing-strategy | Difficulty: mid
Link: https://github.com/rust-lang/rust/pull/161944

The problem

Rust syntax highlighting in rustdoc did not correctly account for the edition of the code being rendered. This led to situations where keywords introduced in newer editions (like async in 2018/2021/2024) were highlighted as keywords even in code blocks or macro expansions from older editions (like 2015), or vice versa. This was especially problematic for documentation and macro expansion output, where the edition context matters for correct syntax highlighting.

Why it's non-trivial

A naive fix would just hardcode the latest edition for highlighting, but this would be incorrect for code blocks or macro expansions written in older editions. The edition context can vary per code block, macro expansion, or even within a single file, so the highlighter must be able to accept and use the correct edition dynamically. Additionally, the codebase had multiple entry points for highlighting, all of which needed to be updated to pass the edition through.

The approach

1. Pass edition information through the highlighting pipeline

The main change is to thread an Edition parameter through all relevant functions, replacing the previous hardcoded edition logic.

-pub(crate) fn render_example_with_highlighting(
-    tooltip: Option<&Tooltip>,
-    playground_button: Option<&str>,
-    extra_classes: &[String],
-) -> impl Display {
+pub(crate) fn render_example_with_highlighting(
+    tooltip: Option<&Tooltip>,
+    playground_button: Option<&str>,
+    extra_classes: &[String],
+    edition: Edition,
+) -> impl Display {

This change adds an edition parameter to the main entry point for rendering code examples with highlighting. This allows the caller to specify the correct edition for each code block.

2. Propagate edition to the actual code highlighter

-        write_code(f, src, None, None, None);
+        let edition = match tooltip {
+            Some(Tooltip::Edition(edition)) => *edition,
+            _ => edition,
+        };
+        write_code(f, src, None, None, edition, None);

This ensures that if the tooltip specifies an edition (e.g., from a code block attribute), it is used; otherwise, the default is passed through.

3. Update the signature of write_code and all its callers to require the edition

-pub(super) fn write_code(
-    src: &str,
-    href_context: Option<HrefContext<'_, '_>>,
-    decoration_info: Option<&DecorationInfo>,
-    line_info: Option<LineInfo>,
-) {
+pub(super) fn write_code(
+    src: &str,
+    href_context: Option<HrefContext<'_, '_>>,
+    decoration_info: Option<&DecorationInfo>,
+    edition: Edition,
+    line_info: Option<LineInfo>,
+) {

This makes the edition parameter required, removing the previous defaulting behavior.

4. Remove the hardcoded edition logic

-fn is_keyword(symbol: Symbol) -> bool {
-    // FIXME(#148221): Don't hard-code the edition. The classifier should take it as an argument.
-    symbol.is_reserved(|| Edition::Edition2024)
-}

The old function that always treated keywords as if the code was Edition 2024 is removed. Instead, edition is now tracked per classifier instance:

+fn is_keyword(&self, symbol: Symbol) -> bool {
+    symbol.is_reserved(|| self.edition)
+}

This allows correct edition-aware keyword detection.

5. Update all tests and macro expansion code to pass the correct edition

For example, in test code:

-            write_code(&mut out, src, None, None, None);
+            write_code(&mut out, src, None, None, Edition::Edition2024, None);

All test invocations now specify the edition explicitly.

And in macro expansion rendering:

-                super::highlight::write_code(&mut out, &code, None, None, None);
+                super::highlight::write_code(
+                    &mut out,
+                    &code,
+                    None,
+                    None,
+                    // NOTE: This is only "an approximation" or "best effort" since the edition of
+                    // individual tokens contained in the expansion can differ from the the edition
+                    // of the entire expansion. And we can't fix that since code is just a `String`
+                    // that was produced by `rustc_ast_pretty` meaning more precise edition
+                    // information has been lost.
+                    original_span.edition(),
+                    None,
+                );

A note is added to explain that this is a best-effort approximation, since macro expansion can cross edition boundaries.

6. Add regression tests for edition-aware highlighting

New test files were added to ensure that code blocks and macro expansions are highlighted according to their edition context, not just the latest edition.

From the review thread

fmease: Could you add a // NOTE: here that states that this is only "an approximation" or "best effort" since the edition of individual tokens contained in the expansion can differ from the "overarching edition" / the edition of the entire expansion (original_span.edition()).

And we can't fix that since code is just a String that was produced by rustc_ast_pretty meaning more precise edition information has been lost.

To give an example:
...
Here, your PR doesn't highlight the async in the rendered expansion … let _ = async {} … since it uses the edition of the entire expansion (which is Rust 2015) but the async in the Rust 2015 expansion does actually refer to Rust 2024 keyword async and thus contains an async block, not a struct expression! That's because the keyword async originates from a Rust 2024 crate (root expansion).[^1]

I think that's perfectly acceptable for now but a NOTE would be nice.

GuillaumeGomez: Fair enough!

Takeaways

• Always propagate contextual information (like language edition) through all relevant layers of an API, rather than hardcoding or defaulting at one point.
• When code generation or transformation can cross boundaries (such as macro expansion across editions), document any best-effort or approximate behaviors and their limitations.
• Update and extend tests to cover all variants of context-sensitive behavior, especially when fixing subtle bugs related to language evolution.