NOSSLETTER.techLIVE
back to archive

ggml : bump version to v.28.0

Language: C+++694 -86feature
concepts demonstrated
Fused operationsSparse attention mechanismsBackend hardware accelerationAPI design for precision control
The Issue & Context

Before this release, ggml lacked a fused sparse-attention mask operation and had limited precision introspection on compute nodes; also, skipping matrix slots required out-of-range index hacks.

Root Cause Analysis

The previous design did not support a dedicated fused top-k sparse-attention mask op, and the matrix ID indexing treated invalid indices as out-of-range patches rather than explicit skip signals, complicating backend implementations and reducing efficiency.

Solution Strategy & Architectural Pattern

This release introduces the GGML_OP_TOP_K_MASK operation with CPU, CUDA, and Metal implementations to efficiently fuse sparse-attention masking. It adds ggml_prec_get_acc() and ggml_prec_get_src() APIs for querying compute node precisions. The MUL_MAT_ID and ADD_ID ops now accept -1 as a skip index, allowing backends to zero output rows and skip computation cleanly. Additionally, scale and sum ops were relaxed to support non-contiguous inputs, and several backend throughput improvements and bug fixes were included.

Key Takeaway for Contributors

Explicit sentinel values for skipping computation slots simplify backend logic and improve correctness, and providing fused operations for common sparse patterns can significantly boost performance and maintainability in ML compute libraries.