unlock unlimited code runs for $79, limited time →
NOSSLETTER.techLIVE
back to OSSpedia
python/cpython
mid8/10
securityapi-designbackwards-compatibility

gh-155694: Scope HTTPPasswordMgr credentials by URL scheme

Project: python/cpython | Language: Python | Concepts: security, api-design, backwards-compatibility | Difficulty: mid
Link: https://github.com/python/cpython/pull/155696

The problem

The HTTPPasswordMgr in urllib.request did not scope stored credentials by URL scheme. This meant that credentials saved for an HTTPS URL (e.g., https://example.com/) would also be used for the same host over HTTP (http://example.com/). This could accidentally leak credentials to insecure connections, which is a security risk.

Why it's non-trivial

A naive fix might simply check the scheme when matching credentials, but this could break existing code that intentionally omits schemes (e.g., using just example.com). The fix needs to:

  • Prevent credential leakage from HTTPS to HTTP.
  • Preserve existing behavior for URIs without a scheme (they should still match any scheme).
  • Not break backwards compatibility for users who rely on the old matching logic for schemeless URIs.

The approach

1. Update the documentation to specify new scheme-aware behavior:

+   authentication tokens when authentication for *realm* and a super-URI of any
+   of the given URIs is given. If a URI includes a scheme, its credentials only
+   match authentication URIs with the same scheme or no scheme. A URI without a
+   scheme matches authentication URIs with any scheme.
+
+   .. versionchanged:: next
+      Authentication credentials for URIs with a scheme are now scoped by
+      that scheme.

This clarifies for users that credentials are now scheme-scoped, and documents the backwards-compatible behavior for schemeless URIs.

2. Store and match credentials using both scheme and authority/path:

-                self.reduce_uri(u, default_port) for u in uri)
+                self._reduce_uri_with_scheme(u, default_port) for u in uri)

In add_password, credentials are now stored using a tuple that includes the scheme (or None if absent), not just the authority/path.

3. When searching for credentials, match scheme as well:

-            reduced_authuri = self.reduce_uri(authuri, default_port)
+            reduced_authuri = self._reduce_uri_with_scheme(
+                authuri, default_port)

In find_user_password, the lookup key now includes the scheme.

4. Implement helpers to extract and compare scheme-aware URIs:

+    def _reduce_uri_with_scheme(self, uri, default_port=True):
+        parts = urlsplit(uri)
+        scheme = parts[0] if parts[1] else None
+        return (scheme or None, *self.reduce_uri(uri, default_port))
+
+    def _is_suburi_with_scheme(self, base, test):
+        if (base[0] is not None and test[0] is not None and
+                base[0] != test[0]):
+            return False
+        return self.is_suburi(base[1:], test[1:])

_reduce_uri_with_scheme returns a tuple of (scheme, authority, path). _is_suburi_with_scheme only matches if schemes are equal (unless either is None, in which case it matches any scheme). This preserves backwards compatibility for schemeless URIs.

5. Update credential tracking for prior authentication:

-                reduced_uri = self.reduce_uri(u, default_port)
+                reduced_uri = self._reduce_uri_with_scheme(u, default_port)
...
-            reduced_authuri = self.reduce_uri(authuri, default_port)
+            reduced_authuri = self._reduce_uri_with_scheme(
+                authuri, default_port)
-                if self.is_suburi(uri, reduced_authuri):
+                if self._is_suburi_with_scheme(uri, reduced_authuri):

All credential tracking now uses the scheme-aware helpers.

6. Add comprehensive tests for the new behavior:

+    def test_password_manager_scheme(self):
+        mgr = urllib.request.HTTPPasswordMgr()
+        mgr.add_password(
+            "realm", "https://example.com/", "user", "password")
+
+        self.assertEqual(
+            mgr.find_user_password("realm", "https://example.com/"),
+            ("user", "password"))
+        self.assertEqual(
+            mgr.find_user_password("realm", "http://example.com/"),
+            (None, None))
+        # Support an authority without a scheme.
+        self.assertEqual(
+            mgr.find_user_password("realm", "example.com"),
+            ("user", "password"))
+        # An authority without a scheme continues to match any scheme.
+        mgr.add_password(
+            "realm", "schemeless.example.com", "user", "password")
+        for scheme in "http", "https":
+            with self.subTest(scheme=scheme):
+                self.assertEqual(
+                    mgr.find_user_password(
+                        "realm", f"{scheme}://schemeless.example.com/"),
+                    ("user", "password"))

These tests verify that credentials for HTTPS are not used for HTTP, and that schemeless URIs still match any scheme.

From the review thread

No notable review discussion on this PR.

Takeaways

  • Credentials should be scoped as narrowly as possible to avoid accidental leakage, especially between secure and insecure protocols.
  • When changing matching logic for security, preserve backwards compatibility for legacy usage where possible, especially for intentionally ambiguous cases (like schemeless URIs).
  • Tests are critical when tightening security boundaries, to ensure both the new restrictions and the old, intentional flexibilities are preserved.