top_p_filter ("nucleus sampling") is supposed to return how many top-ranked
tokens you need to keep before their cumulative probability reaches p. Right
now it returns one too few -- it finds the exact token whose addition pushes
the running total past the threshold, and then excludes that very token
from the count instead of including it.
Your task: fix the off-by-one in top_p_filter. top_k_filter is correct
and given as a model for how top_p_filter should be structured -- don't
change it.
Example that currently fails
top_p_filter([0.9, 0.05, 0.05], p=0.5)
# should be: 1 (the first token alone, at 0.9, already exceeds p=0.5)
# currently: 0 -- an empty nucleus, which is meaningless: you can't sample
# from zero tokens.
Real-world grounding
- llama.cpp's sampler chain (top-k, top-p, and other sampling strategies):
src/llama-sampler.cpp - transformers' generation-time logits processors (
TopPLogitsWarperand friends):src/transformers/generation/utils.py
Getting sampling boundaries exactly right matters in practice: an off-by-one here either silently produces an empty candidate set (a crash or an always-picks-nothing bug) or systematically excludes a token that should have been eligible, subtly skewing what the model is allowed to say.