I read "Release It" recently, not a bad book, some good stories, I wish more people would pay attention to things like Stability and Capacity.
Anyway I liked the Circuit Breaker pattern, as I've seen a number of apps with highly coupled systems that barf once latency increases in the back ends then you get "Cascading Failures" and the webapp's request handler thread pool is exhausted and they all wait their mandatory 30 seconds for a response. Eventually the app either dies, or gets taken out of a load balance farm causing a "Chain Reaction".
Here is an Experimental Circuit Breaker Implementation (circuit_breaker.tgz) , NOT built or tested FOR PRODUCTION use in anyway. There is explicity no thread safety no the actual CircuitBreakerSimple class.
download, extract and run ant.
Play around with the parameters to see the affect. Generally it seems to thrash more with increased concurrent threads and interestingly in jdk1.6 sometimes reverts to the default timeout instead of the instance value.
Based roughly on the following logic:
When Circuit is Closed:
on call = pass through
call succeeds = reset count
call fails = count failure
threshold reached = trip breaker. Open State
when Circuit is Half-Open
on call = pass through
call succeeds = reset go. Close State
call fails = trip breaker. Open State
when Circuit is Open
on call = return/fail
on timeout = attempt reset. Half-Open State
4 Responses
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.
I was in a rush to write it up, hence the warning and the lack of a real supported project.
I left out explicit thread safety so you get to choose how much safety and liveness you want/need.
The other impl looks ok, but it is lacking in some areas, biggest one for me is: tests and logging.
Continuing the Discussion