unlock unlimited code runs for $79, limited time →
NOSSLETTER.techLIVE
back to archive

[vllm-bench, feature] Added support for Mooncake-style, timed-traces replay.

Language: Python+794 -90feature
concepts demonstrated
Command-line argument parsingFile I/O with JSONL formatScheduling and timing controlCross-language feature porting
The Issue & Context

Previously, the vllm benchmark tool's timed-trace replay feature was only available in the Python client, limiting performance testing fidelity and consistency across implementations.

Root Cause Analysis

The Rust-based vllm-bench client lacked support for the Mooncake-style timed-trace dataset format and the self-timed scheduling needed to faithfully reproduce recorded request arrival schedules.

Solution Strategy & Architectural Pattern

Ported the timed-trace replay functionality to the Rust client by implementing parsing of the Mooncake JSONL trace format and the self-timed scheduling logic, removing eight flags previously restricted to Python only, and validating correctness with local and live A/B tests against the Python client.

Key Takeaway for Contributors

This enhancement enables high-fidelity, schedule-accurate benchmarking in Rust, ensuring consistent load replay across client implementations and improving confidence in performance evaluations.