A Kubernetes controller gets a stream of change notifications, and often several of them arrive for the same object before it's even had a chance to process the first one. Naively queuing every notification would mean reprocessing the same object repeatedly for no reason; client-go's workqueue instead deduplicates -- multiple pending notifications for the same key collapse into a single queued item. It also bounds retries: a handler that keeps failing (a permanently broken object, a persistent bug) shouldn't retry forever.

add() and get() are done for you (dedup via a queued set, FIFO order; get() also marks the item as processing, in-flight, not yet finished). Implement:

• done(item): the worker has finished with item (called exactly once, after whatever retry()/forget() call it makes). Remove it from processing. If it was marked dirty while it was being processed, clear that and requeue it.
• retry(item): record a failure. If item has now failed max_retries times or more, give up permanently -- return False. Otherwise, request a requeue (via add()) and return True.
• forget(item): clear item's failure count.

The missing link: a naive requeue lets the same key run twice at once

The obvious version of retry() just calls add() directly and stops there. That's wrong the moment you consider WHEN retry()/forget() get called: while item is still checked out by the current worker, before that worker has finished handling it. If add() requeued it immediately, get() could hand the SAME key to a second worker while the first one is still mid-flight -- exactly the concurrent double-processing bug a work queue exists to prevent.

The fix is the processing / dirty split already wired into add(): an add() for an item that's currently processing doesn't requeue it, it just marks it dirty -- a note that says "reprocess this once you're done with the copy you're currently holding." Only done() -- called once the current worker is actually finished -- checks that note and performs the deferred requeue. This guarantees a key is worked on by at most one "in-flight" instance at a time, no matter how many change events (or retries) pile up for it while it's being processed.

Example

wq = WorkQueue(max_retries=2)
wq.add("a"); wq.get()      # "a" checked out, now processing
wq.add("a")                # a second change event arrives mid-processing
wq.get()                   # -> None: "a" is still in-flight, not re-dispatched
wq.done("a")                # worker finished -- the pending change is now applied
wq.get()                   # -> "a": requeued because of the add() during processing

Constraints

• retry()/forget() reuse add()'s existing dedup/in-flight logic when requesting a requeue -- don't duplicate that logic.
• Failure counts are per-item and persist across separate retry() calls until forget() clears them.
• done() must be safe to call even if nothing was marked dirty (no requeue in that case) -- most successful processing runs end this way.

Real-world grounding

• client-go's rate-limiting workqueue, including the exact processing/dirty mechanism this exercise implements: staging/src/k8s.io/client-go/util/workqueue/queue.go

Real controllers call AddRateLimited (which also computes a growing backoff delay, similar to the Retry with Exponential Backoff exercise) or Forget, then always call Done -- this exercise implements the bounded-retry and in-flight dedup accounting underneath that pattern, without the timing/delay part.