gh-152204: Validate date and time fields in `_pydatetime.date{time}.fromisoformat`

Project: python/cpython | Language: Python | Concepts: error-handling, testing-strategy, backwards-compatibility | Difficulty: mid
Link: https://github.com/python/cpython/pull/152205

The problem

The pure-Python implementation of datetime.date.fromisoformat and related methods accepted malformed ISO 8601 date/time strings that the C implementation correctly rejected. Specifically, fields like year, month, day, etc., were parsed with int() on string slices, which allowed non-digit characters (like +, -, spaces, or Unicode digits) and incorrect field lengths. This led to silent acceptance of invalid inputs and inconsistent behavior between the C and Python versions.

Why it's non-trivial

A naive fix might just check for errors after converting with int(), but this wouldn't catch all malformed cases—int() will happily parse strings like '+12', ' 12', or even Unicode digits, and will not enforce the required field width. Ensuring strict ISO 8601 compliance requires checking both that the field is exactly the expected length and that all characters are ASCII digits, matching the C accelerator's behavior.

The approach

1. Centralized field validation in a helper function

A new helper function is introduced to validate both length and digit-only content for each field:

def _read_isoformat_component(s, n):
    # The caller has verified the string is ASCII, so isdigit() matches only
    # the ASCII digits accepted by the C parser.
    if len(s) != n or not s.isdigit():
        raise ValueError("Invalid isoformat string")
    return int(s)

This function ensures that a string slice is exactly n ASCII digits before converting to int, raising ValueError otherwise.

2. Enforcing ASCII-only input up front

Before parsing, the code checks that the entire string is ASCII:

if not dtstr.isascii():
    raise ValueError("Invalid isoformat string")

This prevents Unicode digits or other non-ASCII characters from being accepted.

3. Using the helper for all fixed-width fields

All date and time fields are now parsed using the new helper, guaranteeing strict format:

year = _read_isoformat_component(dtstr[0:4], 4)
...
weekno = _read_isoformat_component(dtstr[pos:pos + 2], 2)
...
dayno = _read_isoformat_component(dtstr[pos:pos + 1], 1)
...
month = _read_isoformat_component(dtstr[pos:pos + 2], 2)
...
day = _read_isoformat_component(dtstr[pos:pos + 2], 2)
...
time_comps[comp] = _read_isoformat_component(tstr[pos:pos+2], 2)

Each field is now strictly validated for length and digit content.

4. Consistent digit check for fractions

The fraction part of the time is now also validated with isdigit():

if not tstr[pos:].isdigit():
    raise ValueError("Non-digit values in fraction")

This ensures the fractional seconds are composed only of ASCII digits.

5. Expanded test coverage

Tests are added for all the problematic cases, ensuring both C and Python implementations now reject malformed inputs:

'2020+12',          # '+' in a basic-format field
'2020 12',          # space in a basic-format field
'+020-06-15',       # leading sign in the year
'202012+9',         # '+' in the day field
'2020-W 5',         # space in the week number
'2020061',          # 7 chars: day slice reads a 1-character tail
'2020-W2',          # 1-digit week number
...
'2020-12-12T0٥:02:03',          # Unicode digit in the hour
'2020-12-12T01:0٥:03',          # Unicode digit in the minute
'2020-12-12T01:02:0٥',          # Unicode digit in the second
'2020-12-12T01:02:03.٥',        # Unicode digit in the fraction
'2020-12-12T01:02:03.4_6',      # underscore in the fraction
'2020-12-12T01:02:03+0٥:00',    # Unicode digit in the tz hour
'2020-12-12T01:02:03+01:0٥',    # Unicode digit in the tz minute
'20201212T0102٣٤',              # Unicode digits in the basic-format time

These tests ensure that all previously accepted malformed inputs are now rejected.

From the review thread

Done in 46f6881: one dtstr.isascii() up front (also covers the datetime path, which lacked it) + s.isdigit() per fragment — ~10x faster than the per-character check, and the same rejection.

Done in a025467: moved the reader to a module-level _read_isoformat_component, added the same isascii()+isdigit() guard to the time path (fixes the T0102٣٤ case), and fixed the missing comma. Kept isdigit() per Serhiy.

Takeaways

Always validate both the length and character content of fixed-width fields when parsing structured text formats.
Use helper functions to centralize and enforce strict input validation, reducing code duplication and the risk of inconsistent checks.
Match error-handling and validation logic between different implementations (e.g., C and Python) to ensure consistent behavior and avoid subtle bugs.
Expand test coverage to include edge cases and malformed inputs that could previously slip through parsing logic.

gh-152204: Validate date and time fields in _pydatetime.date{time}.fromisoformat