Circuit Breaker

Background

• There are problems when calls to remote service failed usually because of slow network, timeout, etc.
• This problem usually fixed by using retry pattern strategies where the caller will retry to call remote service (can add backoff too here).
• But sometimes it failed due to something that is take longer to fix.
• Hence retry pattern here will be pointless because it will exhaust the caller resources.
• Additionally if the service is very busy this could create cascading failures.
• Example:
  • Request comes in from user to our service.
  • Then our service need to do external request to 3rd party API.
  • But because of some issue, the request is timed out after 10s.
  • The response is timeout and we have retry patter that will retry request at max 5 times.
  • This will make the user request stuck for at least 50s.
  • Imagine if our service is very busy and handler thousand request each second.
  • This could lead to resource locked by at leash 50,000 request that could lead to cascading failure.

Solution

Circuit Breaker act as a proxy for application that might fail. The proxy should monitor the number of recent failures that occurred. And use that information to decide wether to allow operation or simply return error immediately.

3 States of circuit breaker patter:

• Closed: The normal state, where all requests are allowed to pass through to the service.
• Open: The circuit breaker rejects all requests to the service to prevent further failures
• Half-open The circuit breaker allows a limited number of requests to pass through to the service to test if it's working again

Benefits:

• Can prevent application from repeatedly trying to execute and operation thats likely to fail.
• Allow to continue without waiting.
• Prevent wasting CPU and Memory resource.

Example

Using Sony's Library

Here are example of circuit breaker usage from the popular sony/gobreaker library.

package main

import (
    "fmt"
    "log"
    "time"

    "github.com/sony/gobreaker/v2"
)

// Simulate an external service
func callExternalService() (string, error) {
    // Simulate random failure
    if time.Now().Unix()%2 == 0 {
        return "", fmt.Errorf("service unavailable")
    }
    return "Success: Data retrieved", nil
}

// Wrap the external service call with the Circuit Breaker
func externalServiceWithCircuitBreaker(cb *gobreaker.CircuitBreaker[string]) (string, error) {
    // Execute the circuit breaker-protected function
    result, err := cb.Execute(func() (string, error) {
        return callExternalService()
    })
    if err != nil {
        return "", err
    }
    return result, nil
}

func main() {
    // Configure the Circuit Breaker
    cbSettings := gobreaker.Settings{
        Name:        "ExternalServiceCB",
        MaxRequests: 3,                // Allow 3 requests in half-open state
        Interval:    10 * time.Second, // Reset state after 10 seconds
        Timeout:     5 * time.Second,  // Time to wait before transitioning to half-open
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            // Open the circuit when failures exceed 50% of total requests
            failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
            return counts.Requests >= 5 && failureRatio >= 0.5
        },
    }
    circuitBreaker := gobreaker.NewCircuitBreaker[string](cbSettings)

    // Simulate periodic calls to the external service
    for i := 0; i < 10; i++ {
        result, err := externalServiceWithCircuitBreaker(circuitBreaker)
        if err != nil {
            log.Printf("Request %d: Circuit Breaker Triggered - %v", i+1, err)
        } else {
            log.Printf("Request %d: %s", i+1, result)
        }
        time.Sleep(1 * time.Second)
    }
}