[clang] Don't add documentation comments to the AST if not requested

Project: llvm/llvm-project | Language: C++ | Concepts: memory-management, refactoring | Difficulty: mid
Link: https://github.com/llvm/llvm-project/pull/206363

The problem

Clang was always collecting and retaining documentation comments (like Doxygen comments) in the AST, even if nothing in the compilation would ever use them. This consumed unnecessary memory and processing time for most builds that don't need comment information.

Why it's non-trivial

A naive fix would simply stop collecting comments, but that would break tools and workflows that do need them (such as documentation extraction, code completion, or when certain warnings are enabled). The solution must only collect comments when they're actually needed, based on various flags, diagnostics, or tool integrations, without breaking existing consumers.

The approach

Only install the comment handler when needed

-  CommentSemaHandler.reset(new ActionCommentHandler(actions));
-  PP.addCommentHandler(CommentSemaHandler.get());
+  // Only install the comment handler when some consumer may read documentation
+  // comments back.
+  if (actions.shouldRetainCommentsFromLexer(SourceLocation())) {
+    CommentSemaHandler.reset(new ActionCommentHandler(actions));
+    PP.addCommentHandler(CommentSemaHandler.get());
+  }

Annotation:
This change in Parser.cpp ensures the comment handler is only added if comments are actually needed. The decision is delegated to a new function, shouldRetainCommentsFromLexer, which checks if any consumer will use the comments.

Add a function to decide when to retain comments

+bool Sema::shouldRetainCommentsFromLexer(SourceLocation Loc) const {
+  if (LangOpts.CommentOpts.ParseAllComments)
+    return true;
+
+  if (PP.getPreprocessorOpts().RetainComments)
+    return true;
+
+  // When building a PCH the comments are serialized into the AST file
+  // so downstream consumers like clangd) can retrieve documentation, and the
+  // incremental/REPL front end may query them interactively.
+  if (TUKind != TU_Complete)
+    return true;
+
+  if (PP.isCodeCompletionEnabled())
+    return true;
+
+  // Keep the comment if -Wdocumentation is enabled at its location (checking
+  // the location handles warnings turned on by `#pragma clang diagnostic`).
+  if (!Diags.isIgnored(diag::warn_doc_param_not_found, Loc) ||
+      !Diags.isIgnored(diag::warn_unknown_comment_command_name, Loc))
+    return true;
+
+  return false;
+}

Annotation:
This function in Sema.cpp checks various conditions—such as whether all comments should be parsed, whether code completion is active, if a precompiled header is being built, or if documentation warnings are enabled—to decide if comments should be kept.

Actually skip adding comments if not needed

 void Sema::ActOnComment(SourceRange Comment) {
   if (!LangOpts.RetainCommentsFromSystemHeaders &&
       SourceMgr.isInSystemHeader(Comment.getBegin()))
     return;
+  if (!shouldRetainCommentsFromLexer(Comment.getBegin()))
+    return;
   RawComment RC(SourceMgr, Comment, LangOpts.CommentOpts, false);
   ...
 }

Annotation:
Now, when a comment is encountered, it is only added to the AST if shouldRetainCommentsFromLexer returns true for its location.

Ensure tools that need comments still get them

+  // libclang and other ASTUnit clients query documentation comments after
+  // parsing, so keep them in the AST.
+  CI->getPreprocessorOpts().RetainComments = true;

Annotation:
In ASTUnit.cpp, the RetainComments option is set for libclang and similar tools, so they continue to see documentation comments as before.

Add new options and propagate them

+  /// Force the front end to retain all documentation comments in the AST, even
+  /// when no comment consuming diagnostic or language option is enabled. Tools
+  /// that query comments after parsing set this
+  bool RetainComments = false;

Annotation:
A new RetainComments flag is added to PreprocessorOptions and CommentOptions, allowing both command-line flags and internal tools to explicitly request comment retention.

From the review thread

No notable review discussion on this PR.

Takeaways

• Only collect and retain data in memory if there is a clear consumer for it; default to not collecting to save resources.
• When changing default behavior to reduce overhead, provide explicit flags or options for tools and workflows that still require the previous behavior.
• Centralize decision logic (e.g., in a helper function) for when to retain or skip processing, so that all code paths are consistent and easy to maintain.