Architecting Resilient .NET Core Applications: Handling Transient Failures.

 Transient Failures happen when server is temporarily overloaded. We need to handle this instead of sending response back to client and say like that “Server is not responding”

In the world of Cloud there are temporary failures that may occur in a system and can be recovered from by retrying the operation. These failures typically occur due to network issues, temporary unavailability of external services, timeouts, or rate limiting, and they often resolve themselves after a brief period.

Possible Reasons for Transient Failures

1. Resource overloaded: For example, in high-traffic scenarios, limited system resources (like memory or CPU) can lead to failures when trying to access those resources.
2. Network Issues: Temporary issues like DNS resolution errors, dropped connections, or network congestion can cause operations to fail.
3. Service Unavailability: External services or databases might be down or unavailable temporarily. For example, a database connection might fail because the server is temporarily overloaded.
4. Timeouts: A request might take longer than expected to respond due to server overload or high network latency.
5. Rate Limiting: Some external APIs or services may impose rate limits, causing requests to be temporarily rejected if too many requests are made in a short time.

Fixing Transient Failures in .NET Core

We’ll use the Polly library, a .NET library for resilience and transient-fault handling, to implement these patterns.

1. Retry Pattern

The retry pattern automatically retries a failed operation that might succeed if attempted again.

Example: Retrying a failed HTTP request due to a transient network issue.

Below will be steps of implementation

We can Create Policy and configure how many times we want to retry and after how much seconds

// POLICY Declaration 
    private static readonly AsyncRetryPolicy _retryPolicy = Policy
        .Handle<HttpRequestException>()
        .Or<TaskCanceledException>()
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

Complete Code Example

using Polly;
using Polly.Retry;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class RetryExample
{
    private static readonly HttpClient _httpClient = new HttpClient();

   // POLICY Declaration 
    private static readonly AsyncRetryPolicy _retryPolicy = Policy
        .Handle<HttpRequestException>()
        .Or<TaskCanceledException>()
        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

    public static async Task Main(string[] args)
    {
        try
        {
            var response = await _retryPolicy.ExecuteAsync(() => MakeRequest());
            Console.WriteLine($"Response: {response}");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Request failed: {ex.Message}");
        }
    }

    private static async Task<string> MakeRequest()
    {
        var response = await _httpClient.GetAsync("https://example.com");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Details

• The retry policy is configured to handle HttpRequestException and TaskCanceledException.
• The operation is retried up to 3 times with an exponential backoff (2, 4, and 8 seconds).
• If the operation succeeds within the retries, the response is processed; otherwise, an exception is logged.

2. Circuit Breaker Pattern

The circuit breaker pattern prevents an application from repeatedly trying to execute an operation likely to fail, allowing the system to recover.

Steps of Implementation

Policy Code Example

private static readonly AsyncCircuitBreakerPolicy _circuitBreakerPolicy = Policy
        .Handle<HttpRequestException>()
        .CircuitBreakerAsync(2, TimeSpan.FromMinutes(1));

Complete Code Example

using Polly;
using Polly.CircuitBreaker;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class CircuitBreakerExample
{
    private static readonly HttpClient _httpClient = new HttpClient();

// POLICY Declaration 
    private static readonly AsyncCircuitBreakerPolicy _circuitBreakerPolicy = Policy
        .Handle<HttpRequestException>()
        .CircuitBreakerAsync(2, TimeSpan.FromMinutes(1));

    public static async Task Main(string[] args)
    {
        try
        {
            var response = await _circuitBreakerPolicy.ExecuteAsync(() => MakeRequest());
            Console.WriteLine($"Response: {response}");
        }
        catch (BrokenCircuitException)
        {
            Console.WriteLine("Circuit is open; requests are blocked.");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Request failed: {ex.Message}");
        }
    }

    private static async Task<string> MakeRequest()
    {
        var response = await _httpClient.GetAsync("https://medium.com");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Detailed Explanation

• The circuit breaker policy is configured to handle HttpRequestException.

private static readonly AsyncCircuitBreakerPolicy _circuitBreakerPolicy = Policy
        .Handle<HttpRequestException>()
        .CircuitBreakerAsync(2, TimeSpan.FromMinutes(1));

• If two consecutive requests fail, the circuit breaker opens for one minute, during which no requests are attempted.
• When the circuit is open, a BrokenCircuitException is thrown, indicating that requests are blocked to allow the system to recover.

3. Fallback Pattern

The fallback pattern provides an alternative action when a transient failure occurs.

The Fallback pattern is a resilience design pattern used to provide an alternative or backup solution when the primary service or component fails. It helps ensure that the application continues to function and provide some level of service, even if certain parts are not working as expected.

Details are already added on below post

Fallback Pattern in .NET Core: Handling Service Failures Gracefully | by .Net Labs | Nov, 2024 | Medium

4. Bulkhead Isolation Pattern

The bulkhead isolation pattern limits the number of concurrent calls to a particular resource, preventing the entire system from being affected by failures in one part.

Steps of implementation

In Policy we can define policy like more than 2 or 3 concurrent request should not reach to server.

Code Example

using Polly;
using Polly.Bulkhead;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class BulkheadExample
{
    private static readonly HttpClient _httpClient = new HttpClient();
    private static readonly AsyncBulkheadPolicy _bulkheadPolicy = Policy
        .BulkheadAsync(2, int.MaxValue);

    public static async Task Main(string[] args)
    {
        try
        {
            var response = await _bulkheadPolicy.ExecuteAsync(() => MakeRequest());
            Console.WriteLine($"Response: {response}");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Request failed: {ex.Message}");
        }
    }

    private static async Task<string> MakeRequest()
    {
        var response = await _httpClient.GetAsync("https://example.com");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Explanation:

• The bulkhead policy limits the number of concurrent requests to 2.
• This prevents the system from being overwhelmed by too many concurrent requests to the same resource.
• Requests exceeding the limit are queued, ensuring that the resource is not overloaded.

5. Timeout Pattern

The timeout pattern specifies a time limit for an operation, preventing it from running indefinitely.

Approach 1 : Using middleware and details are below .

Increase UI Performance using Timeout Middleware in .NET Core | by .Net Labs | Nov, 2024 | Medium

Approach 2: Same steps as above

using Polly;
using Polly.Timeout;
using System;
using System.Net.Http;
using System.Threading.Tasks;

public class TimeoutExample
{
    private static readonly HttpClient _httpClient = new HttpClient();
    private static readonly AsyncTimeoutPolicy _timeoutPolicy = Policy
        .TimeoutAsync(5, TimeoutStrategy.Pessimistic);

    public static async Task Main(string[] args)
    {
        try
        {
            var response = await _timeoutPolicy.ExecuteAsync(() => MakeRequest());
            Console.WriteLine($"Response: {response}");
        }
        catch (TimeoutRejectedException)
        {
            Console.WriteLine("The operation timed out.");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Request failed: {ex.Message}");
        }
    }

    private static async Task<string> MakeRequest()
    {
        var response = await _httpClient.GetAsync("https://example.com");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Detailed Explanation:

• The timeout policy sets a time limit of 5 seconds for the operation.
• If the operation does not complete within the specified time, a TimeoutRejectedException is thrown.
• This prevents operations from hanging indefinitely, allowing the system to recover more quickly.

Best Practices

1. Use Resilient Libraries: Utilize libraries like Polly for implementing resilience strategies.
2. Combine Patterns: Use a combination of patterns (e.g., Retry with Circuit Breaker) for better resilience.
3. Configure Policies Appropriately: Set sensible values for retries, circuit breakers, and timeouts based on your application’s requirements.
4. Monitor and Log: Implement logging to monitor transient failures and understand their frequency and nature.
5. Graceful Degradation: Ensure your application can degrade gracefully by providing fallback mechanisms.
6. Regular Testing: Regularly test your application’s resilience strategies by simulating transient faults.

Thanks

Please clap