Implementing The Circuit Breaker Pattern In C# – Part 2

In my previous post, I discussed an implementation of The Circuit Breaker Pattern as described in Michael T. Nygard’s book, Release It! Design and Deploy Production-Ready Software. In this post, I will talk about several additions and improvements I have made to the initial implementation.

Service Level

The Circuit Breaker Pattern contains a failure count that keeps track of the number of consecutive exceptions thrown by an operation. When the failure count reaches a threshold, the circuit breaker trips. If an operation succeeds before the failure threshold is reached, the failure count is reset to zero. This works well if the service outage causes multiple consecutive failures, but if the threshold is set to 100 and the service fails 99 times, then one operation succeeds, the failure count is reset to 0, even though there is obviously a problem with the service that should be handled.

To deal with intermittent service failures, I have implemented a “service level” calculation. This indicates the ratio of successful operations to failed operations, expressed as a percentage. For example, if the circuit breaker has a threshold of 100 and an operation fails 50 times, then the current service level is 50%. If the service recovers and 25 operations succeed, then the service level will be 75%. The circuit breaker will not completely reset after a single successful operation. Each successful operation increments the service level, while each failed operation decrements the service level. Once the service level reaches 0%, i.e. the ratio of failed operations over successful ones have reached the threshold, the circuit trips.

A ServiceLevelChanged event allows the client application to be notified of service level changes. This could be used for monitoring performance, or for tracking service levels against a Service Level Agreement (SLA).

The threshold value determines the circuit breaker’s resistance to failures. If a client makes a lot of calls to a service, a higher threshold will allow it more time to recover from failures before tripping. If the client makes fewer calls, but the calls are expensive to the service, a lower threshold will allow the circuit breaker to trip more easily.

Ignored Exception Types

Sometimes a service will throw an exception as part of the service logic. We might not want these exceptions to affect the circuit breaker service level. I have added an IgnoredExceptionTypes property, which holds a list of exception types for the circuit breaker to ignore. If an operation throws one of these exceptions, the exception is thrown back to the caller and is not logged as a failure.

 CircuitBreaker cb = new CircuitBreaker();
 cb.IgnoredExceptionTypes.Add(typeof(AuthorizationException));

Invoker Exceptions

If the operation invoker throws an exception that was not caused by the operation itself, then the exception is thrown back to the caller and is not logged as a failure.

Threading

As mentioned in a comment by Søren on my last post, it is likely a circuit breaker would be used in a multi-threaded environment, therefore it should be able to function property when multiple threads are executing operations.

The failure count is now updated atomically using the System.Threading.Interlocked.Increment and System.Threading.Interlocked.Decrement methods. This ensures the failure count variable is locked while being modified by a thread. Other threads wanting to update the failure count must wait until it is released.

While this does not guarantee the circuit breaker is completely thread-safe, it does prevent problems with multiple threads executing operations and tracking failures. I have to confess I’m not an expert at multi-threaded application designs, so if anyone has any further suggestions on how to make the circuit breaker more thread-safe, I would love to hear them!

For more information implementing threading, see the .NET Framework Threading Design Guidelines.

Download

Download the circuit breaker code and unit tests (VS 2008).

I hope you find these enhancements helpful. Does providing a service level make sense? How can I improve multi-threading support? If you have any comments or suggestions, please let me know your thoughts.

About these ads

11 Responses to “Implementing The Circuit Breaker Pattern In C# – Part 2”


  1. 1 Søren February 18, 2008 at 8:42 am

    Looks great. I’ll have a look at the threading part, and give you some feedback later this week (hopefully).

  2. 2 Stefan August 21, 2008 at 7:24 am

    I downloaded your implementation and it looks great. I’m not an expert on threading as well, but I miss that “Interlocked.Increment/.Decrement” stuff in your CircuitBreaker class. Is it not implemented yet or do I overlook something?

  3. 3 timross August 26, 2008 at 9:59 pm

    Hi Stefan, sorry for the late reply – I have had problems with my FTP and was unable to get a fix up. I have now added the Interlocked.Increment/Decrement calls to the CircuitBreaker code. I’m not sure why I left this out, but thanks for pointing it out!

  4. 4 Rinat Abdullin December 1, 2008 at 8:04 pm

    Hi!

    Thanks for the inspiration and the post!

    I’ve leveraged slightly different concurrency support while implementing my own Circuit Breaker. The post is here, if you are interested:

    http://rabdullin.com/journal/2008/12/1/net-exception-handling-action-policies-application-block.html

    Best regards, Rinat Abdullin

  5. 5 Willie Wheeler March 9, 2010 at 10:54 pm

    Good stuff. I also have a Java/Spring implementation at

    http://code.google.com/p/kite-lib/.

    It includes the template method piece, along with support for AOP and annotation-driven configuration.

  6. 6 Willie Wheeler March 9, 2010 at 11:02 pm

    Also, just a quick comment regarding concurrency and circuit breakers. I’m not a C# guy but I suspect the concepts are analogous in Java and C#. The basic point is that the failure count and state variables should be declared as volatile (i.e., push to and pull from main memory, not just local cache). Otherwise it’s possible for different threads not to see each others’ updates to those variables. Locking is too heavyweight for something like a circuit breaker since it essentially creates a blocking bottleneck at your guarded integration points.

  7. 7 Willie Wheeler March 10, 2010 at 12:56 am

    Ugh, I wasn’t paying attention when I posted the last one. :-)

    The state variable should be volatile.

    The failure count should be an atomic variable (not merely volatile) to get lock-free but atomic increments and sets.

  8. 8 ffalardeau June 28, 2012 at 6:41 pm

    Really neat pattern ! Exactly what I was looking for. I am having problem running the CanExecuteUsingMultipleThreads test. There seems to be race conditions. Anyone encountered this ?

  9. 9 guillaume singer (@guillaumesinger) May 22, 2014 at 3:40 pm

    Perhaps a problem in the implementation :

    in the following extract of your code

    if (this.failureCount > 0)
    {
    // Decrement failure count to improve service level
    Interlocked.Decrement(ref this.failureCount);
    OnServiceLevelChanged(new EventArgs());
    }

    let say that failureCount=1 at the beginning of the scenario
    -thread1 performs the if and is about to decrement the failureCount
    -but at this moment, thread2 preempt the cpu and performs the same “if”
    =>both thread will decrements the value

    in summary : the if and the decrementation must be atomic

    solution => protect the entire “if” block by a lock
    (perhaps interlocked.compareexchange can be used)


  1. 1 Implementing The Circuit Breaker Pattern In C# « Tim Ross - .NET Developer Trackback on February 17, 2008 at 5:07 pm
  2. 2 CircuitBreaker Interceptor « Yoot Trackback on February 24, 2009 at 12:42 am

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





Follow

Get every new post delivered to your Inbox.

%d bloggers like this: