Custom RetryPolicy class for Windows Azure transient error handling
According to definition, the Cloud it’s “A set of interconnected servers located in one or more data centres”. As per definition this kind of environment is naturally prone to network latency and other related environmental issues. This is especially true if we are communicating with on premises applications to synchronize and load the data over public network.
In order to ensure that our systems are reliable and functioning correctly within such a environment, we should use RetryPolicy to retry error prone operation when an transient error occurs.
In this article I will show you how to use default RetryPolicy with configuration stored in web.config file. You will also learn how to create custom retry policy class to be used throughout your application for the specific error types you define.
Using retry policy objects simply allows us to retry an operation multiple times at the intervals we configure. Image below shows debugging information when retrying operations that cause the errors:
In our example we will user Microsoft Enterprise Library 5.0 (updated version). To create instances we will use RetryPolicyFactory that can create retry policy objects of following types: AzureCachingRetryPolicy, AzureServiceBusRetryPolicy, AzureStorageRetryPolicy, SqlCommandRetryPolicy, SqlConnectionRetryPolicy. Each type handles specific error types.
The configuration also includes ErrorDetectionStrategy object that is being configured in web.config file:
<configSections>
<section name="RetryPolicyConfiguration" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling.Configuration.RetryPolicyConfigurationSettings, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling" requirePermission="true"/>
<section name="typeRegistrationProvidersConfiguration" type="Microsoft.Practices.EnterpriseLibrary.Common.Configuration.TypeRegistrationProvidersConfigurationSection, Microsoft.Practices.EnterpriseLibrary.Common"/>
</configSections>
<RetryPolicyConfiguration defaultRetryStrategy="Fixed Interval Retry Strategy" defaultSqlConnectionRetryStrategy="Incremental Retry Strategy">
<incremental name="Incremental Retry Strategy" retryIncrement="00:00:01" initialInterval="00:00:01" maxRetryCount="10"/>
<fixedInterval name="Fixed Interval Retry Strategy" retryInterval="00:00:01" maxRetryCount="10"/>
<exponentialBackoff name="Backoff Retry Strategy" minBackoff="00:00:01" maxBackoff="00:00:30" deltaBackoff="00:00:10" maxRetryCount="10" firstFastRetry="false"/>
</RetryPolicyConfiguration>
<typeRegistrationProvidersConfiguration>
<clear/>
<add name="Caching" sectionName="cachingConfiguration"/>
<add name="Cryptography" sectionName="securityCryptographyConfiguration"/>
<add name="Exception Handling" sectionName="exceptionHandling"/>
<add name="Instrumentation" sectionName="instrumentationConfiguration"/>
<add name="Logging" sectionName="loggingConfiguration"/>
<add name="Policy Injection" sectionName="policyInjection"/>
<add name="Security" sectionName="securityConfiguration"/>
<add name="Data Access" providerType="Microsoft.Practices.EnterpriseLibrary.Data.Configuration.DatabaseSyntheticConfigSettings, Microsoft.Practices.EnterpriseLibrary.Data"/>
<add name="Validation" providerType="Microsoft.Practices.EnterpriseLibrary.Validation.Configuration.ValidationTypeRegistrationProvider, Microsoft.Practices.EnterpriseLibrary.Validation"/>
<add sectionName="RetryPolicyConfiguration" name="RetryPolicyConfiguration"/>
</typeRegistrationProvidersConfiguration>
Let’s create helper class to handle creation of retry policy objects. The first method retrieves the default policy with the configuration stored in web.config file.
We will also attach an Retrying event to log the retry operations.
public static RetryPolicy GetDefaultPolicy(string name)
{
RetryPolicy retryPolicy;
try
{
retryPolicy = RetryPolicyFactory.GetRetryPolicy<StorageTransientErrorDetectionStrategy>(name);
retryPolicy.Retrying += retryPolicy_Retrying;
}
catch (NullReferenceException)
{
throw new Exception("Unable to read transient fault handling behaviour from web.config file - section for TransientFaultHandling could be missing.");
}
return retryPolicy;
}
static void retryPolicy_Retrying(object sender, RetryingEventArgs e)
{
var message = string.Format(
"Retry - Count: {0}, Delay: {1}, Exception: {2}",
e.CurrentRetryCount,
e.Delay,
e.LastException.Message);
Trace.TraceEvent(TraceEventType.Information, 0, message);// write to log
}
Our next helper function will create custom retry policy. In the constructor we will pass in error types we want to be included in retry operations. Ideally this could be stored in config file.
public static RetryPolicy GetCustomRetryPolicy()
{
var retryPolicy = new RetryPolicy(
new CustomTransientErrorDetectionStrategy(
new List<Type>() {
typeof(DivideByZeroException),
typeof(IndexOutOfRangeException),
}),
new CustomRetryStrategy());
retryPolicy.Retrying += retryPolicy_Retrying;
return retryPolicy;
}
Now let’s create custom retry policy class. The most important part of it is ShouldRetry method which simply returns the delegate allowing to evaluate whether to continue retrying operation or not. You can also apply your own logic in this place based on your requirements.
public class CustomRetryStrategy : RetryStrategy
{
private readonly int retryCount = 3;
private readonly TimeSpan retryInterval = TimeSpan.FromMilliseconds(1000);
public CustomRetryStrategy()
: base("customRetryStrategy", true)
{
//default values
}
public CustomRetryStrategy(int retryCount, TimeSpan retryInterval)
: base("customRetryStrategy", true)
{
this.retryCount = retryCount;
this.retryInterval = retryInterval;
}
public override ShouldRetry GetShouldRetry()
{
if (this.retryCount == 0)
{
return delegate(int currentRetryCount, Exception lastException, out TimeSpan interval)
{
interval = TimeSpan.Zero;
return false;
};
}
return delegate(int currentRetryCount, Exception lastException, out TimeSpan interval)
{
if (currentRetryCount < this.retryCount)
{
var random = new Random();
//set random interval within the threshold
interval = TimeSpan.FromMilliseconds(random.Next((int)(this.retryInterval.TotalMilliseconds * 0.8), (int)(this.retryInterval.TotalMilliseconds * 1.2)));
//use your logic here
//....
return true;
}
interval = TimeSpan.Zero;
return false;
};
}
}
When creating custom retry policy we also need to create CustomTransientErrorDetectionStrategy class inheriting from ITransientErrorDetectionStrategy interface. In this class we simply evaluate the error type that is currently occurring and need to decide whether retry policy object will attempt to handle it. In order to do that we will pass in our error types to the class constructor. Next, we will check the error type within the IsTransient method to return true if error must cause the retry operation or false otherwise.
public class CustomTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
{
List<Type> exceptionTypesToRetry;
public CustomTransientErrorDetectionStrategy(List<Type> exceptionType)
{
exceptionTypesToRetry = exceptionType;
}
public bool IsTransient(Exception ex)
{
if (exceptionTypesToRetry.Contains(ex.GetType()))
{
return true;
}
return false;
}
}
Finally in our controller we can test it as follows:
RetryPolicyHelper.GetDefaultPolicy("Incremental Retry Strategy").ExecuteAction(() =>
{
//index out of range exception - it won't be retried
//as it defaults to sql connection errors
var array = new int[] { 1, 2, 3 };
var a = array[5];
});
RetryPolicyHelper.GetCustomRetryPolicy().ExecuteAction(() =>
{
//divide by zero exception - will be retried
//as we defined exception types to:
//DivideByZeroException, IndexOutOfRangeException
var a = 0;
var b = 10 / a;
});
I have included project files below for your tests 🙂
(No Ratings Yet)
Loading...
