gh-118150: difflib: expose autojunk flag from SequenceMatcher to public methods and functions

Project: python/cpython | Language: Python | Concepts: api-design, backwards-compatibility | Difficulty: mid
Link: https://github.com/python/cpython/pull/153959

The problem

The difflib module in Python uses a class called SequenceMatcher to compare sequences. By default, SequenceMatcher uses an "autojunk" heuristic to speed up comparisons by ignoring certain elements in large sequences. However, the "autojunk" parameter was not exposed in the higher-level public functions and classes (like ndiff, unified_diff, get_close_matches, etc.), so users could not control this behavior without dropping down to the lower-level API.

Why it's non-trivial

A naive fix would just add an autojunk parameter to one or two functions, but difflib has several public functions and classes that internally use SequenceMatcher. To provide a consistent and complete API, the parameter needs to be threaded through all relevant functions and classes, with careful attention to defaults and documentation. Backwards compatibility must also be preserved, so new parameters must be keyword-only and default to the current behavior.

The approach

Expose autojunk in all relevant public APIs

• The autojunk parameter is added as a keyword-only argument (with default True) to all relevant functions and classes in difflib. For example:

-def get_close_matches(word, possibilities, n=3, cutoff=0.6):
+def get_close_matches(word, possibilities, n=3, cutoff=0.6, *, autojunk=True):

This makes autojunk available to users, while keeping the default behavior unchanged.

• The parameter is then passed to the underlying SequenceMatcher instance:

-    s = SequenceMatcher()
+    s = SequenceMatcher(autojunk=autojunk)

This ensures that the user's choice for autojunk is respected.

• The same pattern is applied to other functions and classes, such as unified_diff, context_diff, ndiff, and the Differ and HtmlDiff classes. For example, in unified_diff:

-def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',
-                 tofiledate='', n=3, lineterm='\n', *, color=False):
+def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',
+                 tofiledate='', n=3, lineterm='\n', *, autojunk=True, color=False):

and

-    for group in SequenceMatcher(None,a,b).get_grouped_opcodes(n):
+    for group in SequenceMatcher(None, a, b, autojunk=autojunk).get_grouped_opcodes(n):

• The Differ class is updated to accept and store an autojunk parameter, and to pass it to all internal SequenceMatcher uses:

-    def __init__(self, linejunk=None, charjunk=None):
+    def __init__(self, linejunk=None, charjunk=None, *, autojunk=True):
         ...
+        self.autojunk = autojunk

and

-        cruncher = SequenceMatcher(self.linejunk, a, b)
+        cruncher = SequenceMatcher(self.linejunk, a, b, autojunk=self.autojunk)

Update documentation and changelogs

• All affected functions and classes are updated in the documentation (Doc/library/difflib.rst) to mention the new parameter, its default, and its effect:

-   .. method:: __init__(linejunk=None, charjunk=None)
+   .. method:: __init__(linejunk=None, charjunk=None, autojunk=True)

with added explanation:

+      Setting the optional *autojunk* argument to ``False`` will turn
+      :ref:`automatic junk heuristic <difflib-junk>` off.

• The whatsnew entry for Python 3.16 is updated to mention the new parameter and its effect.

From the review thread

Lenormju: I think the Documentation (Doc/library/difflib.rst) should get updated too to include this new parameter.

faithlesstomas: Quick check if this works for the original issue test:

 for autojunk in (True, False):
     old_new = list(difflib.unified_diff(
         get_lines('small.external.old.json'),
         get_lines('small.external.new.json'),
         autojunk=autojunk
     ))
     new_old = list(difflib.unified_diff(
         get_lines('small.external.new.json'),
         get_lines('small.external.old.json'),
         autojunk=autojunk
     ))
     print('Autojunk flag:', autojunk)
     print('diff external.old external.new.json | wc -l')
     print(len(old_new))
     print('diff external.new external.old.json | wc -l')
     print(len(new_old))

Autojunk flag: True
diff external.old external.new.json | wc -l
90
diff -u external.new external.old.json | wc -l
25
Autojunk flag: False
diff external.old external.new.json | wc -l
25
diff -u external.new external.old.json | wc -l
25

Takeaways

• When exposing a previously hidden or internal parameter, ensure it is consistently available across all relevant public APIs, and maintain backwards compatibility by making it keyword-only with a sensible default.
• Always update documentation and changelogs when public APIs change, so users are aware of new capabilities and how to use them.
• When a function or class wraps another, user-facing parameters should be threaded through all layers to give users full control over underlying behavior.