This circuit breaker is supposed to reopen IMMEDIATELY the moment a HALF_OPEN
trial request fails -- that's the entire point of the trial: it's a single
probe, and one failure means the dependency clearly hasn't recovered yet. Right
now, a failed trial just quietly increments a counter instead, so the breaker
can stay stuck in HALF_OPEN -- letting requests through to a
dependency that just proved it's still broken.
Your task: find the bug in record_failure and fix it. allow_request and
record_success are already correct -- don't change them.
Reminder of the three states:
CLOSED-- normal operation, requests pass throughOPEN-- tripped afterfailure_thresholdconsecutive failures, requests rejected immediatelyHALF_OPEN-- entered oncereset_timeouthas elapsed since opening; exactly one trial request is let through to test recovery
Example that currently fails
cb = CircuitBreaker(failure_threshold=3, reset_timeout=10)
cb.record_failure(0); cb.record_failure(0); cb.record_failure(0) # opens
cb.allow_request(now=10) # True -- half-open trial allowed
cb.record_failure(now=10) # the trial failed
cb.state # should be "OPEN" immediately
# currently: still "HALF_OPEN" (needs 3 MORE failures
# to reopen, because the counter was reset to 0 when
# it originally opened)
Real-world grounding
- Envoy's outlier detection (the mechanism circuit breaking is built on in a
proxy):
source/common/upstream/outlier_detection_impl.cc
Real circuit breakers often track a failure rate over a rolling window rather
than a simple consecutive-failure count, and may allow more than one trial
request in HALF_OPEN -- but "one bad probe result should immediately reflect
reality" is the rule this bug violates.