NOSSLETTER.techLIVE
back to archive

feat: add remote TransformLog subscriptions and MVCC barriers

Language: Go+12144 -3126feature
concepts demonstrated
gRPC remote procedure callsMulti-Version Concurrency Control (MVCC)Backpressure and cache eviction strategiesOrdered event delivery and resumable cursors
The Issue & Context

Before this PR, the system lacked remote TransformLog subscriptions with ordered, resumable delivery and MVCC barriers, causing inefficient recovery and inconsistent query progress tracking during streaming replication.

Root Cause Analysis

The previous implementation did not support remote subscription routing or resumable cursors for TransformLog entries, lacked MVCC barrier coordination to preserve query snapshot isolation, and used simplistic backpressure based on scanner mode rather than cache occupancy, leading to unreliable delivery and checkpoint synchronization.

Solution Strategy & Architectural Pattern

This PR introduces SubscribeTransform RPCs with local/remote owner routing, handshake protocols for creation and closure, ordered Entry/SyncUp delivery, and resumable cursors. It aligns recovery with the WAL Consumer by filtering replay before data-only handlers, implements backoff for failed attempts, suspends reconnects without subscriptions, and reclaims idle streams. MVCC barriers are preserved and scoped progress is published in WAL order without advancing durable checkpoints, queuing volatile barriers until RecoveryStorage confirms prefix completeness. The WAB cache uses capacity and TTL eviction with RecoveryStorage consumption constraints, replacing scanner mode backpressure with cache-occupancy-based admission control using configurable slowdown and rejection thresholds. WALSummary reads reuse verified chunk footers and prevent repopulation of retired entries while supporting independent reader cancellation and GC snapshot protection.

Key Takeaway for Contributors

When implementing distributed streaming subscriptions with strong consistency, combine resumable cursors and handshake protocols with MVCC barriers and cache-aware backpressure to ensure ordered delivery, efficient recovery, and consistent query progress without advancing durable checkpoints prematurely.