Deferring Processing of Azure Service Bus Messages

Sometimes when you’re handling a message from a message queue, you realise that you can’t currently process it, but might be able to at some time in the future. What would be nice is to delay or defer processing of the message for a set amount of time.

Unfortunately, with brokered messages in  Azure Service Bus, there is no built-in feature to do this simply, but there are a few workarounds. In this post, we’ll look at four separate techniques: let the lock time out, sleep and abandon, defer the message, and resubmit the message.

Let the Lock Time Out

The simplest option is doing nothing. When you get your BrokeredMessage, don’t call Complete or Abandon. This will mean that the lock on the message will eventually time out, and it will become available for processing again once that happens. By default the lock duration for a message is 1 minute, but this can be configured for a queue by using the QueueDescription.LockDuration property.

The advantage is that this is a very simple way of deferring re-processing the message for about a minute. The main disadvantage is that the time is not so easy to control as the lock duration is a property of the queue, not the message being received.

In the following simple example, we create a queue with a lock duration of 30 seconds, send a message, but then never actually complete or abandon it in the handler. This results in us seeing the same message getting retried with an incrementing Delivery Count until eventually it is dead-lettered automatically on the 10th attempt.

string connectionString = // some connection string
const string queueName = "TestQueue";

// PART 1 - CREATE THE QUEUE
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);

// ensure it is empty
if (namespaceManager.QueueExists(queueName))
{
    namespaceManager.DeleteQueue(queueName);
}
var queueDescription = new QueueDescription(queueName);
queueDescription.LockDuration = TimeSpan.FromSeconds(30);
namespaceManager.CreateQueue(queueDescription);

// PART 2 - SEND A MESSAGE
var body = "Hello World";
var message = new BrokeredMessage(body);
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
client.Send(message);

// PART 3 - RECEIVE MESSAGES
// Configure the callback options.
var options = new OnMessageOptions();
options.AutoComplete = false; // we will call complete ourself
options.AutoRenewTimeout = TimeSpan.FromMinutes(1); 

// Callback to handle received messages.
client.OnMessage(m =>
{
    // Process message from queue.
    Console.WriteLine("-----------------------------------");
    Console.WriteLine($"RX: {DateTime.UtcNow.TimeOfDay} - {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}");

    // Don't abandon, don't complete - let the lock timeout
    // m.Abandon();

}, options);

Sleep and Abandon

If we want greater control of how long we will wait before resubmitting the message, we can explicitly call abandon after sleeping for the required duration. Sadly there is no AbandonAfter method on brokered message. But it’s very easy to wait and then call Abandon. Here we wait for two minutes before abandoning the message:

client.OnMessage(m =>
{
    Console.WriteLine("-----------------------------------");
    Console.WriteLine($"RX: {DateTime.UtcNow.TimeOfDay} - {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}");

    // optional - sleep until we want to retry
    Thread.Sleep(TimeSpan.FromMinutes(2));

    Console.WriteLine("Abandoning...");
    m.Abandon();

}, options);

Interestingly, I thought I might need to periodically call RenewLock on the brokered message during the two minute sleep, but it appears that the Azure SDK OnMessage function is doing this automatically for us. The down-side of this approach is of course that our handler is now in charge of marking time, and so if we wanted to hold off for an hour or longer, then this would tie up resources in the handling process, and wouldn’t work if the computer running the handler were to fail. So this is not ideal.

Defer the Message

It turns out that BrokeredMessage has a Defer method whose name suggests it can do exactly what we want – put this message aside for processing later. But, we can’t specify how long we want to defer it for, and when you defer it, it will not be retrieved again by the OnMessage function we’ve been using in our demos.

So how do you get a deferred message back? Well, you must remember it’s sequence number, and then use a special overload of QueueClient.Receive that will retrieve a message by sequence number.

This ends up getting a little bit complicated as now we need to remember the sequence number somehow. What you could do is post another message to yourself, setting the ScheduledEnqueueTimeUtc to the appropriate time, and that message simply contains the sequence number of the deferred message. When you get that message you can call Receive passing in that sequence number and try to process the message again.

This approach does work, but as I said, it seems over-complicated, so let’s look at one final approach.

Resubmit Message

The final approach is simply to Complete the original message and resubmit a clone of that message scheduled to be handled at a set time in the future. The Clone method on BrokeredMessage makes this easy to do. Let’s look at an example:

client.OnMessage(m =>
{

    Console.WriteLine("----------------------------------------------------");
    Console.WriteLine($"RX: {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}");

    // Send a clone with a deferred wait of 5 seconds
    var clone = m.Clone();
    clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(5);
    client.Send(clone);

    // Remove original message from queue.
    m.Complete();
}, options);

Here we simply clone the original message, set up the scheduled enqueue time, send the clone and complete the original. Are there any downsides here?

Well, it’s a shame that sending the clone and completing the original are not an atomic operation, so there is a very slim chance of us seeing the original again should the handling process crash at just the wrong moment.

And the other issue is that DeliveryCount on the clone will always be 1, because this is a brand new message. So we could infinitely resubmit and never get round to dead-lettering this message.

Fortunately, that can be fixed by adding our own resubmit count as a property of the message:

client.OnMessage(m =>
{
    int resubmitCount = m.Properties.ContainsKey("ResubmitCount") ?  (int)m.Properties["ResubmitCount"] : 0;

    Console.WriteLine("----------------------------------------------------");
    Console.WriteLine($"RX: {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}, ResubmitCount: {resubmitCount}");

    if (resubmitCount > 5)
    {
        Console.WriteLine("DEAD-LETTERING");
        m.DeadLetter("Too many retries", $"ResubmitCount is {resubmitCount}");
    }
    else
    {
        // Send a clone with a deferred wait of 5 seconds
        var clone = m.Clone();
        clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(5);
        clone.Properties["ResubmitCount"] = resubmitCount + 1;
        client.Send(clone);

        // Remove message from queue.
        m.Complete();
    }
}, options);

Summary

It is a shame that there isn’t an overload of Abandon that specifies a time to wait before re-attempting processing of the message. But there are several ways you can work around this limitation as we’ve seen in this post. Of course you may know of a better way to tackle this problem. If so, please let me know in the comments.