TCP delivers a stream of bytes with no message boundaries of its own -- a
single recv() call might return part of a message, exactly one message,
or several messages all concatenated together. Every database wire
protocol has to solve this with FRAMING: embed each message's length in the
message itself, then parse by reading that length and slicing out exactly
that many bytes.
This exercise uses Postgres's actual framing shape. Each message:
- 1 byte: message type (e.g.
b"Q"for a simple query) - 4 bytes: big-endian length, INCLUDING these 4 length bytes themselves
(but NOT the 1-byte type) -- so payload length is
length - 4 - payload:
length - 4bytes
Implement:
build_message(msg_type, payload): the ENCODE direction -- return the bytes for one correctly-framed message.parse_messages(buffer):bufferis bytes, possibly containing multiple complete messages and/or a trailing partial message. Return(messages, remaining):messages: a list of(msg_type_byte, payload_bytes)for every COMPLETE message found, in order.remaining: whatever bytes are left over -- a partial message, orb""if the buffer ended exactly on a boundary -- meant to be prepended to the next chunk of data that arrives.
The missing link: a length prefix is only as safe as its bound
A length-prefixed framing scheme looks complete once you handle partial
reads, but there's a second failure mode that has nothing to do with slow
networks: what if the length field itself is wrong? Corruption, a buggy
client, a version mismatch, or an actively malicious peer can all produce
a message that claims to be, say, 2 GB long. A parser that just does "wait
until I have length bytes" will wait -- forever, if that many bytes never
arrive -- quietly consuming memory and a connection slot for a message
that was never real. That's not a parsing bug you'll catch by testing
normal traffic; it only shows up under corruption or attack, which is
exactly why it needs to be handled explicitly rather than discovered in
production. MAX_MESSAGE_SIZE plus an immediate ProtocolError the moment
a declared length exceeds it closes this off: reject up front, before ever
entering the "wait for more bytes" state.
Example
build_message(b"Q", b"SELECT 1")
# -> b"Q" + length(4 bytes) + b"SELECT 1"
parse_messages(build_message(b"Q", b"SELECT 1"))
# -> ([(b"Q", b"SELECT 1")], b"")
Constraints
- Never consume bytes for a message you can't yet prove is complete -- if
the buffer doesn't have enough bytes for the full declared length yet
(even just the 4-byte length field itself being incomplete), stop and
return whatever's left as
remaining, untouched. - Check the declared length against
MAX_MESSAGE_SIZEas soon as the header is parsed -- BEFORE checking whether the full message has arrived, so an oversized claim is rejected even with zero payload bytes present yet. struct.unpack(">I", ...)reads a big-endian unsigned 4-byte integer -- that's the length field's format, andstruct.packis its inverse forbuild_message.
Real-world grounding
- Postgres's actual wire protocol message framing (this exact 1-byte-type +
4-byte-length shape):
src/interfaces/libpq/fe-protocol3.c(searchPqMsg_length)
Every network protocol with variable-length messages solves this same framing problem -- HTTP/2's frame header, gRPC's length-prefixed messages, Kafka's protocol -- the specific byte layout differs, but "read enough to know the length, bound it, wait for that many bytes, repeat" is universal.