2. Decoding & Sampling

Fix the Bug: Nucleus Sampling

midai-mldebugging

top_p_filter ("nucleus sampling") is supposed to return how many top-ranked tokens you need to keep before their cumulative probability reaches p. Right now it returns one too few -- it finds the exact token whose addition pushes the running total past the threshold, and then excludes that very token from the count instead of including it.

Your task: fix the off-by-one in top_p_filter. top_k_filter is correct and given as a model for how top_p_filter should be structured -- don't change it.

Example that currently fails

top_p_filter([0.9, 0.05, 0.05], p=0.5)
# should be: 1   (the first token alone, at 0.9, already exceeds p=0.5)
# currently: 0   -- an empty nucleus, which is meaningless: you can't sample
#                    from zero tokens.

Real-world grounding

Getting sampling boundaries exactly right matters in practice: an off-by-one here either silently produces an empty candidate set (a crash or an always-picks-nothing bug) or systematically excludes a token that should have been eligible, subtly skewing what the model is allowed to say.

hints
sampling.pysign in to run
Loading...
test result

This is a preview. Sign in to run this exercise against the real tests.