NOSSLETTER.techLIVE
back to archive

[Model] Add Dory support

Language: Python+1058 -1feature
concepts demonstrated
Class inheritance and extensionModel architecture integrationAttention mechanisms in neural networksConfiguration and parameter management
The Issue & Context

Before this PR, vLLM did not support the Dory1.0-4B model, a latent-recurrent LLM with a unique looped architecture and hybrid sliding window attention, limiting users from running inference on this NVIDIA model.

Root Cause Analysis

The lack of a dedicated model implementation and configuration for Dory1.0-4B meant vLLM's model executor and transformers utilities could not correctly load or run this model, which requires handling loop counts for repeated application of a shared block and managing its interleaved attention layers.

Solution Strategy & Architectural Pattern

This PR introduces a new model class in vllm/model_executor/models/dory.py that implements Dory's normalized-GPT architecture with looped layers and hybrid sliding window attention. It adds corresponding configuration files in vllm/transformers_utils/configs/dory.py and integrates the model into the registry. Extensive tests were added to validate correct loading, configuration, and inference behavior, ensuring minimal impact on existing core components by reusing existing infrastructure where possible.

Key Takeaway for Contributors

When integrating models with non-standard architectures like looped layers and hybrid attention, encapsulate their unique execution logic in dedicated classes and configurations to maintain modularity and reuse existing infrastructure, enabling scalable support for diverse model designs.