NOSSLETTER.techLIVE
back to archive

enhance: add the primary key index core storage library

Language: Go+5551 -11feature
concepts demonstrated
LSM-tree storage structureOrder-preserving encodingFile-based SSTable managementInterface-based design with read-write and lifecycle abstractions
The Issue & Context

Before this PR, the primary key index lacked a shared, core storage library, resulting in duplicated or fragmented storage logic across components and no unified implementation for handling primary key data storage and retrieval.

Root Cause Analysis

The absence of a dedicated, shared storage library meant that primary key indexing logic was either embedded within nodes or test-only memory engines, preventing consistent, efficient management of LSM-tree based storage layers and lifecycle transitions.

Solution Strategy & Architectural Pattern

This PR adds `internal/pkindex`, a core library implementing order-preserving encoding for primary keys, SSTable read/write via Pebble over local files, and an engine managing write paths and lifecycle transitions of LSM generations without write pauses. It includes error categorization and integrates Pebble v1.1.5 for storage, while maintaining backward-compatible error message formatting.

Key Takeaway for Contributors

Separating core storage logic into a shared library enables consistent handling of complex storage lifecycles and facilitates incremental adoption across components without disrupting existing behavior.