Build Resilient Microservices (Web API) using Polly in ASP.NET Core

In this article, we will learn about how to implement Fault Tolerance in Microservices i.e. build resilient Microservices (Web API) using Polly in ASP.NET Core. By implementing fault tolerance in Microservices we ensure that the entire system is not affected in case of any failure in one of the services.

In this article, I will not cover how to build a microservice in ASP.NET Core as I have already covered that in detail in my other article on Microservices with ASP.NET Core. Here we will see how to implement fault tolerance in Microservices using Polly in ASP.NET Core.

Whether you are working on Microservices or Monolithic applications there are high chances that there will be a need to call external third-party or internal API so you need to build your code in such a way that it can handle failures of that API as your application flow is dependent on the response from that API.

To build resilience in the application or resilient web services means we need to ensure that the web service is always available with acceptable functionality even in conditions like a high load on the service, network failures, other services on which our service is dependent are failing, etc.

What is Polly and why do we need it?

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner.

We will be totally wrong if we say that we have thoroughly tested our application and there won’t be any outages in the production environment. There will be applications failures due to application crashes, slow response, excessive load on the system, hardware failures, network issues, and many more.

To handle these failures in our application first we will have to admit that these failures will happen and secondly we will have to incorporate fault tolerance in our application i.e. we ensure that the entire system does not fail due to one or more service failures.

For example, Microservices is a design where one large application is developed as a set of small independent services with their own datastore. By building fault tolerance in Microservices we design it in such a way that failure of one service does not affect the working of other services i.e. If a service related to profile update is down then users should not be able to update the profile but other transactions like order entry\inquiry should work fine.

Also building resilient services is not about avoiding failures but it’s about the ability to recover from failures and perform its functions in a way that avoids downtime and data loss. Microservices should be designed to handle partial failures. If you do not design and implement techniques to ensure fault tolerance, even partial failures can be amplified.

Using Polly in ASP.NET Core with just a few lines of code we can build resilient applications that function smoothly in spite of partial failure that occur in complex microservices or cloud-based deployments. After implementing the Fault Tolerance in Microservices using Polly in ASP.NET Core, we ensure that the entire system will not be down if a service fails or goes down.

Using policies of Polly in ASP.NET Core we can design our applications to respond in a specified way in case of failures.

Design Principles for handling partial failures

Here is the list of some of the design principles that are recommended for handling partial failures in your Microservices

It is highly recommended to use asynchronous communication instead of a long chain of synchronous HTTP calls across the internal Microservices. The only synchronous call should be the front-end call between client applications and entry-level microservice or API Gateway.

There can be intermittent network or channel failure which can be avoided by implementing retries in the service calls. These retries should be for a limited number of times and cannot be infinite.

Always implements timeouts for each and every network call. The calling client should not wait endlessly for the response from any service instead it should wait for a predefined time limit and once that time is elapsed then it should fail the call.

Use circuit breaker pattern where a retry is made to the failing service and after some fixed some of retries if the service is still failing then the circuit breaker is tripped so that further attempts fail immediately i.e. no new call to failing service will be made instead it will be assumed that its failing or is down. There is a time limit for which new calls to failing service will not be made and once that is elapsed then-new calls will go to the failing service to verify if the service is up & running again or not. If new requests are successful then the circuit breaker will be closed and requests will be forwarded to the service.

Provide some fallback or default behaviour for failing service i.e. if service request fails then provide some fallback logic like return cached data or default data. This can be worked out for querries is difficult to implement for inserts & updates.

For communication between two microservices the calling (client), microservice should implement some limit on the number of requests which are pending from a particular service i.e. if the limit has been reached then it might be pointless to send additional requests to the same service and instead additional requests should fail immediately.

Resilience Policies Supported in Polly

Here is the list of resilience policies supported by Polly in ASP.NET Core

Retry

This policy of Polly in ASP.NET Core allows us to configure automatic retries while calling a service.

Suppose we have an order service that makes calls to product service to obtain details of items being ordered. Now if a product service has random behaviour that works most of the time but fails sometimes.

Now in this case, if the order service receives a failure response from the product service then retrying the request might fetch the results from the product service.

Polly helps us to implement this retry policy with a limit on the maximum number of retries from order service to product service.

Circuit-breaker

This Policy of Polly in ASP.NET Core helps us to break the circuit i.e. block execution of a service request for a configured time period when the service request failure count exceeds some pre-configured threshold.

We will take the same example of order service making a request to product service for item details. Now assume that request from order service to product service fails continuously even on retries then in this case we block calling the product service and provide either cached or default data.

This design of not calling the service in case service fails for the configured number of times and relying on the fallback mechanism is called the Circuit Breaker. When order service calls product service continuously with success then we say that circuit is closed (close state). But when order service does not call product service and relies on fallback mechanism then in that case we say that the circuit is open (open state).

Timeout

This policy of Polly in ASP.NET Core allows us to implement a timeout during HTTP requests to another service which ensures that the caller service won’t have to wait beyond the timeout.

When order service is calling product service for item details and if the response from product service is delayed (product service might be waiting for a response from slow/hang database) then order service assumes that beyond timeout period success result from product service is unlikely. So beyond the timeout period order service assumes that there is some problem with product service and it will stop waiting for a response from the product service and take appropriate action.

Bulkhead Isolation

This policy of Polly in ASP.NET Core allows us to limit the total amount of resources any part of our application can consume so that a failing part of the application does not cause a cascading failure also bringing down other parts of the application.

When order service calls the product service to get item details and if due to some reasons if product service is unavailable then requests will start to back up on the order service and can cause the order service to degrade performance or can even crash the order service.

Bulkhead Isolation helps to isolate part of the application and controls the usage of Memory, CPU, Sockets, Threads, etc so that if one part of your application is not working smoothly this policy will prevent this part from impacting or stopping the complete application.

It also allows you to specify how many concurrent requests can execute and how many requests can be queued for execution so that once concurrent & queue slots are full then new requests are failed immediately.

Cache

This policy in Polly in ASP.NET Core allows storing the responses automatically in the cache (in memory or distributed cache) when they are retrieved for the first time so that subsequent requests for the same resource can be returned from the cache.

When the order service makes a call to the product service for the item details then item details can be stored in the cache by the order service so that the next request for the same product can be fetched from the cache instead of calling the product service again for the same product.

Fallback

This policy in Polly in ASP.NET Core allows us to provide an alternative path i.e. value that can be returned or action that can be taken in case if the service being called is down i.e. is returning an error or timeouts are occurring.

When the order service makes a call to the product service for the item details and if a request to product service fails then fallback configured will allow the order service to decide what do to in case of failure from the product service. Order service can return the default data or take some action based on failure.

Failures are bound to happen no matter how many times you retry so you need to plan what should be done in events of failure. Fallbacks are generally used in combination with other policies like retry, circuit breaker, etc.

Policy Wrap

This policy in Polly in ASP.NET Core allows any of the supported policies in Polly to be combined flexibly to be able to combine resilience strategies. There will be different types of failures that will require different strategies and we can apply a combination of policies based on the failure type.

In short when you want to use more than one policy together then you use Policy Wrap

Now let’s look at how to implement these policies which are supported by Polly in ASP.NET Core

Implement Policies of Polly in ASP.NET Core

Overall Approach for demonstration

Here are the details of the complete approach that has been taken for this demonstration

1. We will create the first ASP.NET Core Web API project for Customer Microservice that contains a Get action method to return the Customer Name for the given Customer Code
2. We will add a second ASP.NET Core Web API project for Order Microservice that contains a Get action method to return order details for the customer.
3. Along with Order details, this Order service also returns the customer name. To get this customer name order service makes a call to customer service get method.
4. We have implemented this HTTP call from order service to customer service to get the customer name.
5. We will implement & test various Polly policies in the order service while making an HTTP request to customer service.
6. We will simulate failures for customer service and see how we can our order service fault tolerant by using policies of Polly in ASP.NET Core.

Here is a short & quick video on the implementation of Polly in ASP.NET Core

Let’s first create the required projects of type ASP.NET Core Web API that will be used to demonstrate how to use policies of Polly in ASP.NET Core

Create ASP.NET Core Web API Project

To demonstrate the implementation of policies of Polly in ASP.NET Core we will create a couple of ASP.NET Core Web API projects and configure them as per the details specified below

Create Customer Service

Create a new project of type ASP.NET Core Web API with the name as ProCodeGuide.Polly.Customer

After creating the project the default WeatherForecast Controller has been deleted as it is not required for the demonstration.

Add Customer Controller

We need to add a Customer controller which will have a get action method that returns the customer name based on the customer code entered. We will add Controllers\CustomerController.cs as shown below

[Route("api/[controller]")]
[ApiController]
public class CustomerController : ControllerBase
{
    private Dictionary<int, string> _customerNameDict = null;

    public CustomerController()
    {
        if(_customerNameDict == null)
        {
            _customerNameDict = new Dictionary<int, string>();
            _customerNameDict.Add(1, "Pro Code Guide");
            _customerNameDict.Add(2, "Support - Pro Code Guide");
            _customerNameDict.Add(3, "Sanjay");
            _customerNameDict.Add(4, "Sanjay - Pro Code Guide");
        }
    }

    [HttpGet]
    [Route("GetCustomerName/{customerCode}")]
    public ActionResult<string> GetCustomerName(int customerCode)
    {
        if (_customerNameDict != null && _customerNameDict.ContainsKey(customerCode))
        {
            return _customerNameDict[customerCode];
        }
        return "Customer Not Found";
    }
}

For demo purposes, I have hardcoded the customer code and name list in the controller itself but ideally, this data should come from a database using an entity framework.

Run and test the customer service

You should see the below screen from swagger (OpenAPI) after building & running the application from visual studio.

On executing the Get action /api/Customer/GetCustomerName/2 you should get the below response from the action method.

Create Order Service

Create the second project of type ASP.NET Core Web API in the same solution with the name as ProCodeGuide.Polly.Order

After creating the project the default WeatherForecast Controller has been deleted as it is not required for the demonstration.

Add Models

Let first add the required models for Order details as shown below in Models\Item.cs & Models\OrderDetails.cs

public class Item
{
    public int Id { get; set; }
    public string Name { get; set; }
}

public class OrderDetails
{
    public int Id { get; set; }
    public string CustomerName { get; set; }
    public DateTime SetupDate { get; set; }
    public List<Item> Items { get; set; }
}

Add Order Controller

We need to add an Order controller which will have a get action method that returns the order detailed based on the customer code entered. This method will also make an HTTP call to the customer service to get the customer name for the customer code.

Let’s first add the httpclient service in the dependency container so that we can get that object httpclient in the order controller to make an HTTP call to the customer service. To add httpclient service in dependency container add the below line to the ConfigureServices method in Startup.cs

services.AddHttpClient();

We will add Controllers\OrderController.cs as shown below

[Route("api/[controller]")]
[ApiController]
public class OrderController : ControllerBase
{
    private readonly ILogger<OrderController> _logger;
    private readonly IHttpClientFactory _httpClientFactory;
    private HttpClient _httpClient;
    private string apiurl = @"http://localhost:23833/";

    private OrderDetails _orderDetails = null;
    public OrderController(ILogger<OrderController> logger, IHttpClientFactory httpClientFactory)
    {
        _logger = logger;
        _httpClientFactory = httpClientFactory;

        if (_orderDetails == null)
        {
            _orderDetails = new OrderDetails
            {
                Id = 7261,
                SetupDate = DateTime.Now.AddDays(-10),
                Items = new List<Item>()
            };
            _orderDetails.Items.Add(new Item
            {
                Id = 6514,
                Name = ".NET Core Book"
            });
        }
    }

    [HttpGet]
    [Route("GetOrderByCustomer/{customerCode}")]
    public OrderDetails GetOrderByCustomer(int customerCode)
    {
        _httpClient = _httpClientFactory.CreateClient();
        _httpClient.BaseAddress = new Uri(apiurl);
        var uri = "/api/Customer/GetCustomerName/" + customerCode;
        var result = _httpClient.GetStringAsync(uri).Result;

        _orderDetails.CustomerName = result;

        return _orderDetails;
    }
}

apiurl – is the URL (Host & port no) for the customer service

For demo purposes, I have hardcoded the order details i.e. same order details for all customers but ideally, this data should come from a database using an entity framework.

Enable File Logging using Serilog

Next to check the behaviour of the code after adding Polly policies we will add support for Serilog Logging to log to a file in the code.

Install following Packages to the project using Package Manager Console

Install-Package Serilog.AspNetCore
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.File

Add configuration for Serilog to appsettings.json file as shown below

"Serilog": {
  "MinimumLevel": "Information",
  "Override": {
    "Microsoft.AspNetCore": "Information"
  },
  "WriteTo": [
    {
      "Name": "File",
      "Args": {
        "path": "Serilogs\\AppLogs.log"

      }
    }
  ]
}

Configure Serilog in the method CreateHostBuilder in Program.cs file as shown in below code

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .UseSerilog()
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });

Configure Serilog in Startup Constructor in Startup.cs file as shown in below code

public Startup(IConfiguration configuration)
{
    Configuration = configuration;
    Log.Logger = new LoggerConfiguration()
                    .ReadFrom.Configuration(configuration)
                    .CreateLogger();
}

The above configuration will generate logs to the files under path {Project Path}\Serilogs\AppLogs.log

If you want to further read in detail on how to add Serilog Logging to the project then you check my detailed article on the same here

Now that we have added the required projects and configured the project let’s run and check the project. As this order service is dependent on customer service so we need to ensure that while testing both the projects are up and running. To start both the project together from Visual studio we will make changes to Startup Project.

Right-click on the Solution file in the Solution Explorer and select properties that will load the property screen where you can configure to start both the projects together by selecting the Multiple startup projects option as shown below

Now when you will run the projects from the visual studio both the order and customer service projects will be started.

Run and test the order service

You should see the below screen from swagger (OpenAPI) after building & running the application from visual studio.

On executing the Get action /api/Order/GetOrderByCustomer/2 you should get the below response from the action method.

Now let’s see what happens when customer service is not available i.e. there is no problem with order service but customer service is not up and running. To simulate this condition I have just started the Order service but not started the customer service so customer service is not up and running.

As we can see above that when customer service is not up and running then order service as well starts throwing an error. From Serilog you would be able to see that the order service made a request to customer service which returned an exception so in cascading effect order service also returned 500

Let’s explore how we can avoid this behaviour using policies of Polly in ASP.NET Core

Configure Policies of Polly in ASP.NET Core in Order Service

To configure policies of Polly in ASP.NET Core you need to install the Polly package in the project. You can add the Polly package by running the below-mentioned command in the Package Manager Console window

Install-Package Polly

Now that we installed Polly package binaries in our Order service project let see how we can use policies of Polly in our ASP.NET Core Web API (Order Service) project to make our order service fault-tolerant in spite of customer service not running or failing.

There is more than one way to declare Polly policies i.e. either using registries or adding them via Startup. However, to keeps things simple in this introduction article, we will create Polly policies directly in our controller class in the constructor.

Retry Policy

As per the definition of name this policy suggests that you need to retry the request in case of failure of the request during 1st attempt. Now, these retries have to be for a fixed number of times as this retry business cannot go on forever. This retry policy lets you configure the number of retries you want to make.

This retry policy allows for both to add a delay before retry or also don’t wait before making a retry call for the failed service so if you expect that problem in the service returning error will be corrected immediately then only you should implement retry logic without any delay.

Consider a scenario where HTTP request from order service to customer service is failing. This error from customer service can be either permanent or temporary. To handle temporary failures you want to add the logic to retry the request to customer service a minimum of 2 more times to ensure that temporary failures from customer service are handled using retries.

As per this retry logic, order service will make a request to customer service for customer name and if customer service returns an exception then order service will still retry the request to customer service for 2 more times before concluding that now it is now not possible to get a success response from the customer service.

To simulate random failures from customer service add the below action method to customer service. This method is randomly returning data or an error. To implement such random behaviour we are generating a number between 1 to 10 and if this generated number is even then we are returning a server error with an HTTP status code 500 and if the generated number is not even i.e. it is odd then we are returning the success response with customer name as per customer code.

So this customer service action method GetCustomerNameWithTempFailure will behave randomly i.e. sometimes will return an error or in some cases, it will return a successful response

[HttpGet]
[Route("GetCustomerNameWithTempFailure/{customerCode}")]
public ActionResult<string> GetCustomerNameWithTempFailure(int customerCode)
{
    try
    {
        Random rnd = new Random();
        int randomError = rnd.Next(1, 11);  // creates a number between 1 and 10

        if (randomError % 2 == 0)
            throw new Exception();

        if (_customerNameDict != null && _customerNameDict.ContainsKey(customerCode))
        {
            return _customerNameDict[customerCode];
        }
        return "Customer Not Found";
    }
    catch
    {
        //Log Error
        return StatusCode(StatusCodes.Status500InternalServerError);
    }
}

To implement retry logic using Polly in ASP.NET Core we need to declare the object of type RetryPolicy and define the policy as shown in the code below

//Remaining Code has been removed for readability

private readonly RetryPolicy _retryPolicy;

public OrderController(ILogger<OrderController> logger, IHttpClientFactory httpClientFactory)
{

    //Remaining Code has been removed for readability

    _retryPolicy = Policy
        .Handle<Exception>()
        .Retry(2);
}

The above code example will create a retry policy which will retry up to two times if an HTTP service call fails with an exception handled by the Policy. Here we have specified that the retry policy handles generic Exceptions so it will retry for all types of exceptions but you can even configure the retry policy for more specific exceptions like HttpRequestException then it will retry only for the exception of type HttpRequestException.

Next, we will add a new action method in order service which will make use of the RetryPolicy object to make an HTTP request to the new action method of customer service (GetCustomerNameWithTempFailure) which is returning an error on a random basis. Retry policy is being used to handle random failures from customer service.

[HttpGet]
[Route("GetOrderByCustomerWithRetry/{customerCode}")]
public OrderDetails GetOrderByCustomerWithRetry(int customerCode)
{
    _httpClient = _httpClientFactory.CreateClient();
    _httpClient.BaseAddress = new Uri(apiurl);
    var uri = "/api/Customer/GetCustomerNameWithTempFailure/" + customerCode;
    var result = _retryPolicy.Execute(() => _httpClient.GetStringAsync(uri).Result);

    _orderDetails.CustomerName = result;

    return _orderDetails;
}

The RetryPolicy object uses the delegate to execute the required HTTP call to customer service in the Execute() delegate. If the HTTP call throws an exception that is being handled by the retry policy then the HTTP call will be retried for the configured number of times.

Let’s run & test the retry policy of Polly in ASP.NET Core. After running the solution in the visual studio both the projects i.e. customer & order should start. After both services start goto to order service and you should see below screen from swagger (OpenAPI)

On above screen select action /api/Order/GetOrderByCustomerWithRetry/(customerCode) it should expand and then click on Try it out button. After that, you should see the below screen where you need to enter a value for customer code and click on execute button.

As shown above, after clicking on execute we got a successful response with the proper customer name as per the value entered for the customer code.

But the action GetOrderByCustomerWithRetry in Order Service is making an HTTP call to the Customer service which is returning an error on a random basis so let’s check logs and see what happened during the HTTP call to GetCustomerNameWithTempFailure in customer service

As we can see in the screenshot of the above log that when we called customer service from order service the first call returned an error but since we had configured retry policy and it was retried and on first retry customer service return success response with proper customer name as per the value of customer code. So with the use of a retry policy in order service we were able to handle temporary failures in customer service.

Timeout Policy

As per the definition of name this policy suggests that you need to terminate the request in case of no response from another service within the set time limit.

Consider a scenario where HTTP request from order service to customer service is being delayed. This error from customer service can be never-ending as customer service might be waiting for either response from slow/hang database or response from third party service and customer service has not implemented timeout for these calls.

To handle delayed responses you want to add the logic to timeout the request to customer service after a set time limit has elapsed to ensure that order service doesn’t wait endlessly from the response from customer service as it will keep the thread busy forever. This waiting endlessly can have a cascading effect on order service as well and may exhaust all the resources available on the order service server.

As per this timeout logic, order service will make a request to customer service for customer name and if customer service doesn’t get a response within the set time limit then order service assumes that now there is no chance to get a successful response from customer service so it terminates or timeout the request and takes appropriate action & returns the response.

To simulate a delay in response from customer service add the below action method to customer service. This method is returning a response after a delay of 2 minutes. To implement such behaviour we make use of the sleep method in the Thread class which stops the thread execution for the specified time.

So this customer service action method GetCustomerNameWithDelay will delay the response for 2 minutes to order service.

[HttpGet]
[Route("GetCustomerNameWithDelay/{customerCode}")]
public ActionResult<string> GetCustomerNameWithDelay(int customerCode)
{
    Thread.Sleep(new TimeSpan(0, 2, 0));
    if (_customerNameDict != null && _customerNameDict.ContainsKey(customerCode))
    {
        return _customerNameDict[customerCode];
    }
    return "Customer Not Found";
}

To implement timeout logic using Polly in ASP.NET Core we need to declare the object of type TimeoutPolicy and define the policy as shown in the code below

private static TimeoutPolicy _timeoutPolicy;

public OrderController(ILogger<OrderController> logger, IHttpClientFactory httpClientFactory)
{
    _timeoutPolicy = Policy.Timeout(20, TimeoutStrategy.Pessimistic);
}

The above code example will create a timeout policy that will wait for a response for 20 seconds and after 20 seconds it will assume that no success response is not possible and will timeout the request i.e. execute delegate or function should be abandoned.

Timeout policy in Polly in ASP.NET Core supports optimistic and pessimistic timeout. Optimistic timeout is recommended wherever possible, as it consumes fewer resources.

Optimistic – assumes that delegates you execute support cancellation and that the delegates express that timeout by throwing Exception

Pessimistic – recognises that there are cases where you may need to execute delegates that have no in-built timeout, and do not honour cancellation i.e. caller stops waiting for the underlying delegate to complete

Next, we will add a new action method in order service which will make use of the TimeoutPolicy object to make an HTTP request to the new action method of customer service (GetCustomerNameWithDelay) which is returning a delayed response. The timeout policy is being used to handle delays from customer service.

[HttpGet]
[Route("GetOrderByCustomerWithTimeout/{customerCode}")]
public OrderDetails GetOrderByCustomerWithTimeout(int customerCode)
{
    try
    {
        _httpClient = _httpClientFactory.CreateClient();
        _httpClient.BaseAddress = new Uri(apiurl);
        var uri = "/api/Customer/GetCustomerNameWithDelay/" + customerCode;
        var result = _timeoutPolicy.Execute(() => _httpClient.GetStringAsync(uri).Result);

        _orderDetails.CustomerName = result;

        return _orderDetails;
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Excpetion Occurred");
        _orderDetails.CustomerName = "Customer Name Not Available as of Now";
        return _orderDetails;
    }
}

The TimeoutPolicy object uses the delegate to execute the required HTTP call to customer service in the Execute() delegate. If the HTTP call doesn’t return a response within 20 seconds i.e. as per the time set in timeout policy then the HTTP call will be terminate and it will raise a timeout – operationcancelledexception and in catch block, we have set the customer name as ‘Customer Name Not Available as of Now’. This is for demo purposes only in practice you will return an error to the user and at the same time inform the admin about the error so that this can be fixed.

Let’s run & test the timeout policy of Polly in ASP.NET Core. After running the solution in the visual studio both the projects i.e. customer & order should start. After both services start goto to order service and you should see below screen from swagger (OpenAPI)

On above screen select action /api/Order/GetOrderByCustomerWithTimeout/(customerCode) it should expand and then click on Try it out button. After that, you should see the below screen where you need to enter a value for customer code and click on execute button.

As shown above, after clicking execute we got a successful response with customer name as ‘Customer Name Not Available as of Now’ as per our handling of timeout event.

But the action GetOrderByCustomerWithTimeout in Order Service is making an HTTP call to the Customer service which is returning delayed response so let’s check logs and see what happened during the HTTP call to GetCustomerNameWithDelay in customer service

As we can see in the screenshot of the above log that when we call customer service from order service then due to a delay in response from customer service as timeout event is raised by order service timeout policy and exception of type Polly.Timeout.TimeoutRejectedException is raised and operation is cancelled. In the catch block, we have added code to return success but with a custom customer name. So with the use of a timeout policy in order service we were able to handle delays from customer service & avoid endless waits for order service.

Fallback Policy

As per the definition of name this policy suggests that you need to have some fallback (plan B) in case of failure of the request being called. Now, here you can implement first the retry policy to rule out the temporary failure of the service being called and after all retries to the service also fail then you can have some fallback mechanism i.e. what to do in case of failure. This fallback policy lets you provide the substitute value (or substitute action to be actioned) for the response in case of failure from the service being called.

Consider a scenario where HTTP request from order service to customer service is failing. This error from customer service can be either permanent or temporary. Now request has failed even during retries so instead of causing the order service to fail due to the failure of customer service you want to provide some substitute value for the response so that the order service can take that as a response (instead of failure) and execute remaining code based on that response.

As per this fallback logic, order service will make a request to customer service for customer name and if customer service returns an exception then order service will use the substitute value configured in fallback policy as the final response from customer service and process that response.

To simulate permanent failures from customer service add the below action method to customer service. This method is always returning an error.

So this customer service action method GetCustomerNameWithPermFailure will throw an exception and always return an error in all the cases

[HttpGet]
[Route("GetCustomerNameWithPermFailure/{customerCode}")]
public ActionResult<string> GetCustomerNameWithPermFailure(int customerCode)
{
    try
    {
        throw new Exception("Database Not Available");
    }
    catch
    {
        //Log Error
        return StatusCode(StatusCodes.Status500InternalServerError);
    }
}

To implement Fallback logic using Polly in ASP.NET Core we need to declare the object of type FallbackPolicy and define the policy as shown in the code below

private readonly FallbackPolicy<string> _fallbackPolicy;

public OrderController(ILogger<OrderController> logger, IHttpClientFactory httpClientFactory)
{
    _fallbackPolicy = Policy<string>
                        .Handle<Exception>()
                        .Fallback("Customer Name Not Available - Please retry later");
}

The above code example will create a fallback policy that will substitute the response value as ‘Customer Name Not Available – Please retry later’ if an HTTP service call fails with an exception handled by the Policy. A fallback policy with datatype string (TResult) is being used as customer service action return a string (customer name) as a response.

You can even use the fallback policy for void-returning calls. In case of the void, it specifies an alternate Action to be run if the policy handles a fault (rather than substitute return value) .Fallback(() => DoSomeFallbackAction())

Also in the above code, we have specified that the fallback policy handles generic Exceptions so it will provide substitute value for all types of exceptions but you can even configure the fallback policy for more specific exceptions like HttpRequestException then it will provide fallback value only for the exception of type HttpRequestException.

Next, we will add a new action method in order service which will make use of the FallbackPolicy object to make an HTTP request to the new action method of customer service (GetCustomerNameWithPermFailure) which is returning an error. The fallback policy is being used to handle failures from customer service by providing fallback or substitute value for response in case of failure.

[HttpGet]
[Route("GetOrderByCustomerWithFallback/{customerCode}")]
public OrderDetails GetOrderByCustomerWithFallback(int customerCode)
{
    _httpClient = _httpClientFactory.CreateClient();
    _httpClient.BaseAddress = new Uri(apiurl);
    var uri = "/api/Customer/GetCustomerNameWithPermFailure/" + customerCode;
    var result = _fallbackPolicy.Execute(() => _httpClient.GetStringAsync(uri).Result);

    _orderDetails.CustomerName = result;

    return _orderDetails;
}

The FallbackPolicy object uses the delegate to execute the required HTTP call to customer service in the Execute() delegate. If the HTTP call throws an exception that is being handled by the fallback policy then provide a substitute value in the event of failure.

Let’s run & test the Fallback policy of Polly in ASP.NET Core. After running the solution in the visual studio both the projects i.e. customer & order should start. After both services start goto to order service and you should see below screen from swagger (OpenAPI)

On above screen select action /api/Order/GetOrderByCustomerWithFallback/(customerCode) it should expand and then click on Try it out button. After that, you should see the below screen where you need to enter a value for customer code and click on execute button.

As shown above, after clicking on execute we got a successful response (even though customer service is failing permanently) with the customer name as per the fallback substitute value configured for the customer code.

But the action GetOrderByCustomerWithFallback in Order Service is making an HTTP call to the Customer service which is returning an error on a so let’s check logs and see what happened during the HTTP call to GetCustomerNameWithPermFailure in customer service.

As we can see in the screenshot of the above log that when we called customer service from order service and it returned an error but still order service was successful and fallback value was used for the response. So with the use of a fallback policy in order service we were able to handle failures in customer service.

Circuit Breaker Policy

This circuit breaker policy suggests that you need to have some mechanism or logic to not call the particular service in case that service has been failing permanently for the previous few requests. This circuit breaker policy lets you let configure to block HTTP requests to a particular failing service for a configured time period when the service request failure count exceeds some pre-configured threshold.

Consider a scenario where HTTP request from order service to customer service is failing. This error from customer service can be either permanent or temporary. Now request has failed even during retries so you have provided some substitute value for the response using the fallback policy. But now since few consecutive calls to customer service has failed so for some time (say few minutes) you don’t want to waste time in calling customer service instead assume that it will return an error and use the alternate response for processing the request to order service.

This logic assumes that if the service has failed a few consecutive times then there is some permanent issue with that service and it might need some time to rectify the issue. So let’s not waste the time in calling or making retries to the failing service instead take an alternate fallback path to provide some time to the service to recover.

As per this circuit breaker logic, order service will make a request to customer service for customer name and if customer service returns an exception for 2 consecutive times then the circuit will break (i.e. circuit will open) for 1 minute and for this 1-minute order service will not make any call to customer service instead of on it own assume that customer service will return an error.

To simulate permanent failures from customer service we will make use of the action GetCustomerNameWithPermFailure in customer service that we used for demonstration of fallback policy.

To implement Circuit Breaker logic using Polly in ASP.NET Core we need to declare the object of type CircuitBreakerPolicy and define the policy as shown in the code below

private static CircuitBreakerPolicy _circuitBreakerPolicy;

public OrderController(ILogger<OrderController> logger, IHttpClientFactory httpClientFactory)
{
    if (_circuitBreakerPolicy == null)
    {
        _circuitBreakerPolicy = Policy.Handle<Exception>()
                                        .CircuitBreaker(2, TimeSpan.FromMinutes(1));
    }
}

The above code example will create a Circuit Breaker policy that defines that while calling service if there is an exception for 2 consecutive times then the circuit will break (calls to service will be blocked) for a time span of 2 minutes.

Also in the above code, we have specified that the circuit breaker policy handles generic Exceptions so it will break for all types of exceptions but you can even configure the circuit breaker policy for more specific exceptions like HttpRequestException then it will break only for the exception of type HttpRequestException.

Next, we will add a new action method in order service which will make use of the Circuit Breaker Policy object to make an HTTP request to the action method of customer service (GetCustomerNameWithPermFailure) which is returning an error. Circuit Breaker policy is being used to handle failures from customer service by not making any calls to the customer service for 1 minute after it has failed for 2 consecutive times.

[HttpGet]
[Route("GetOrderByCustomerWithCircuitBreaker/{customerCode}")]
public OrderDetails GetOrderByCustomerWithCircuitBreaker(int customerCode)
{
    try
    {
        _httpClient = _httpClientFactory.CreateClient();
        _httpClient.BaseAddress = new Uri(apiurl);
        var uri = "/api/Customer/GetCustomerNameWithPermFailure/" + customerCode;
        var result = _circuitBreakerPolicy.Execute(() => _httpClient.GetStringAsync(uri).Result);

        _orderDetails.CustomerName = result;
        return _orderDetails;
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Excpetion Occurred");
        _orderDetails.CustomerName = "Customer Name Not Available as of Now";
        return _orderDetails;
    }
}

The circuit breaker policy object uses the delegate to execute the required HTTP call to customer service in the Execute() delegate. If the HTTP call throws an exception that is being handled by the catch block to provide an alternate value for the customer name.

Let’s run & test the circuit breaker policy of Polly in ASP.NET Core. After running the solution in the visual studio both the projects i.e. customer & order should start. After both services start goto to order service and you should see below screen from swagger (OpenAPI)

On above screen select action /api/Order/GetOrderByCustomerWithCircuitBreaker/(customerCode) it should expand and then click on Try it out button. After that, you should see the below screen where you need to enter a value for customer code and click on execute button.

As shown above, after clicking on execute we got a successful response (even though customer service is failing permanently) with the customer name as per the fallback value configured in the catch block.

But the action GetOrderByCustomerWithCircuitBreaker in Order Service is making an HTTP call to the Customer service which is returning an error so let’s check logs and see what happened during the HTTP call to GetCustomerNameWithPermFailure in customer service

As we can see in the screenshot of the above log that when we called customer service from order service it returned an error and order service used an alternate value for customer name from the catch block. Also, we can see from logs that when we tried to make calls to customer service when the circuit was open then Polly didn’t make calls to customer service instead provided an exception for that Policy.CircuitBreaker.BrokenCircuitException – The circuit is now open and is not allowing calls.

Bulkhead Isolation Policy

To implement bulkhead isolation logic using Polly in ASP.NET Core we need to declare the object of type BulkheadPolicy and define the policy as shown in the code below

private static BulkheadPolicy _bulkheadPolicy;

public OrderController(ILogger<OrderController> logger, IHttpClientFactory httpClientFactory)
{
    _bulkheadPolicy = Policy.Bulkhead(3, 6);
}

The above code example will create a bulkhead isolation policy that defines that while calling service limit the number of resources to call the service i.e. max 3 parallelizations of executions through the bulkhead & max 6 number of requests that may be queuing (waiting to acquire an execution slot) at any time.

Next, we will add a new action method in order service which will make use of the Bulkhead Isolation Policy object to make an HTTP request to the action method of customer service (GetCustomerName). Bulkhead Isolation policy is being used to limit the resources used to call the customer service i.e. at any given time there will 3 parallel requests execution and another 6 requests can be in the queue. So that if the response from customer service is delayed or blocked then we don’t use too many resources on order service and cause a cascading failure in order service as well.

[HttpGet]
[Route("GetOrderByCustomerWithBulkHead/{customerCode}")]
public OrderDetails GetOrderByCustomerWithBulkHead(int customerCode)
{
    _httpClient = _httpClientFactory.CreateClient();
    _httpClient.BaseAddress = new Uri(apiurl);
    var uri = "/api/Customer/GetCustomerName/" + customerCode;
    var result = _bulkheadPolicy.Execute(() => _httpClient.GetStringAsync(uri).Result);

    _orderDetails.CustomerName = result;

    return _orderDetails;
}

The bulkhead isolation policy object uses the delegate to execute the required HTTP call to customer service in the Execute() delegate.

Bulkhead Isolation policy works on the policy that one fault shouldn’t bring down the whole ship! i.e. when service begins to fail then it can build up a large number of requests that all are failing slowly in parallel and this can lead to utilization of resources (CPU/threads/memory) in the order service thus degrading capability or causing failure of the order service.

For Cache Policy I will suggest that don’t implement a logic to cache data based on exceptions instead design caching logic based on the data i.e. static/dynamic data, frequently used data, etc. You can read my detailed article on caching in ASP.NET Core here

So far we looked at the important policies of Polly in ASP.NET Core. Also, it is possible to combine multiple policies of Polly in ASP.NET Core for a single service call like

fallback.Wrap(waitAndRetry).Wrap(breaker).Execute(action);

fallback.Execute(() => waitAndRetry.Execute(() => breaker.Execute(action)));

Summary

We learned about various policies of Polly in ASP.NET Core Web API. What we saw as part of this article is just the tip of the iceberg i.e. we just got started and looked at the basic implementation of the policies.

You can even try the combination of multiple policies know as policy wrap where you can combine retry policy with fallback policy or circuit breaker policy.

We saw the implementation of Polly in ASP.NET Core with sync methods of the policies, comparable async method also exists for all the policies.

Please provide your suggestions & questions in the comments section below

Download Source Code

Here you can download complete source code developed as part of this article i.e. Polly in ASP.NET Core Web API

https://github.com/procodeguide/ProCodeGuide.Sample.Polly

Hope you found this article useful. Please support the Author