AI/LLM Internals
How a large language model actually turns text into tokens and logits into words: byte-pair encoding and sampling strategies, grounded in how llama.cpp, vLLM, and transformers implement the same mechanisms.
7 exercises
Start with "Byte-Pair Encoding Tokenizer"Tap the menu button in the corner to browse all exercises.