3. Storage

Packfile Delta Compression

midversion-control-internalsstorage

Git packs near-duplicate objects (say, two commits' versions of a large file with one small edit) as a DELTA against a similar BASE object, instead of storing the full content twice -- this is most of what keeps a repo's .git directory from growing linearly with history. A delta is a short sequence of instructions:

  • ("copy", offset, length) -- take length bytes from base starting at offset, reusing bytes that already exist elsewhere.
  • ("insert", data) -- literal bytes not found in base, stored in full.

Implement:

  • encode_delta(base, target, min_match=4) -- walk target left to right. At each position, find the single LONGEST run of bytes starting there that also occurs SOMEWHERE in base (ties broken by the EARLIEST offset in base). If that longest match is at least min_match bytes, emit ("copy", offset, length) and jump length bytes ahead. Otherwise, accumulate this one byte into a pending literal run and advance by 1 byte. Flush any pending literal run into a SINGLE ("insert", data) op whenever a copy is about to be emitted (so a run of consecutive unmatched bytes becomes one insert op, not one per byte) -- and once more at the very end if anything's left.
  • decode_delta(base, ops) -- apply the ops against base in order to reconstruct target: ("copy", offset, length) appends base[offset:offset+length]; ("insert", data) appends data literally.

Example

base   = b"the quick brown fox jumps over the lazy dog"
target = b"the quick brown fox leaps over the lazy dog"

encode_delta(base, target)
# -> [("copy", 0, 20), ("insert", b"lea"), ("copy", 23, 20)]
#     ^ "the quick brown fox "   ^ new    ^ "s over the lazy dog"

decode_delta(base, encode_delta(base, target)) == target   # True

Constraints

  • encode_delta must never emit two consecutive "insert" ops -- a run of unmatched bytes is always ONE insert op covering the whole run.
  • A match shorter than min_match is never worth a copy op's overhead -- treat it as ordinary unmatched bytes (add to the literal run), even if a match of length 1 or 2 technically exists somewhere in base.
  • decode_delta(base, encode_delta(base, target)) must equal target exactly, for any base/target pair (including empty ones).

Real-world grounding

  • Git's delta encoder (finds and emits copy/insert instructions against a base, via a hash-table-indexed search rather than this exercise's simpler O(len(base)) scan per position): diff-delta.c
  • Git's delta APPLIER (the decode side -- walks the same copy/insert instruction stream to reconstruct the target): patch-delta.c

Real Git indexes base with a rolling hash over fixed-size blocks so it can find candidate matches in roughly constant time instead of this exercise's linear scan per position, and packs the ops themselves into a compact binary instruction encoding -- but "longest match against a base, copy what matches, insert what doesn't" is the exact idea underneath both.

hints
Loading...
test result

This is a preview. Sign in to run this exercise against the real tests.

Packfile Delta Compression — nossletter.tech