Posted in:

Sometimes when you’re handling a message from a message queue, you realise that you can’t currently process it, but might be able to at some time in the future. What would be nice is to delay or defer processing of the message for a set amount of time.

Unfortunately, with brokered messages in  Azure Service Bus, there is no built-in feature to do this simply, but there are a few workarounds. In this post, we’ll look at four separate techniques: let the lock time out, sleep and abandon, defer the message, and resubmit the message.

Let the Lock Time Out

The simplest option is doing nothing. When you get your BrokeredMessage, don’t call Complete or Abandon. This will mean that the lock on the message will eventually time out, and it will become available for processing again once that happens. By default the lock duration for a message is 1 minute, but this can be configured for a queue by using the QueueDescription.LockDuration property.

The advantage is that this is a very simple way of deferring re-processing the message for about a minute. The main disadvantage is that the time is not so easy to control as the lock duration is a property of the queue, not the message being received.

In the following simple example, we create a queue with a lock duration of 30 seconds, send a message, but then never actually complete or abandon it in the handler. This results in us seeing the same message getting retried with an incrementing Delivery Count until eventually it is dead-lettered automatically on the 10th attempt.

string connectionString = // some connection string
const string queueName = "TestQueue";

// PART 1 - CREATE THE QUEUE
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);

// ensure it is empty
if (namespaceManager.QueueExists(queueName))
{
    namespaceManager.DeleteQueue(queueName);
}
var queueDescription = new QueueDescription(queueName);
queueDescription.LockDuration = TimeSpan.FromSeconds(30);
namespaceManager.CreateQueue(queueDescription);

// PART 2 - SEND A MESSAGE
var body = "Hello World";
var message = new BrokeredMessage(body);
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
client.Send(message);

// PART 3 - RECEIVE MESSAGES
// Configure the callback options.
var options = new OnMessageOptions();
options.AutoComplete = false; // we will call complete ourself
options.AutoRenewTimeout = TimeSpan.FromMinutes(1); 

// Callback to handle received messages.
client.OnMessage(m =>
{
    // Process message from queue.
    Console.WriteLine("-----------------------------------");
    Console.WriteLine($"RX: {DateTime.UtcNow.TimeOfDay} - {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}");

    // Don't abandon, don't complete - let the lock timeout
    // m.Abandon();

}, options);

Sleep and Abandon

If we want greater control of how long we will wait before resubmitting the message, we can explicitly call abandon after sleeping for the required duration. Sadly there is no AbandonAfter method on brokered message. But it’s very easy to wait and then call Abandon. Here we wait for two minutes before abandoning the message:

client.OnMessage(m =>
{
    Console.WriteLine("-----------------------------------");
    Console.WriteLine($"RX: {DateTime.UtcNow.TimeOfDay} - {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}");

    // optional - sleep until we want to retry
    Thread.Sleep(TimeSpan.FromMinutes(2));

    Console.WriteLine("Abandoning...");
    m.Abandon();

}, options);

Interestingly, I thought I might need to periodically call RenewLock on the brokered message during the two minute sleep, but it appears that the Azure SDK OnMessage function is doing this automatically for us. The down-side of this approach is of course that our handler is now in charge of marking time, and so if we wanted to hold off for an hour or longer, then this would tie up resources in the handling process, and wouldn’t work if the computer running the handler were to fail. So this is not ideal.

Defer the Message

It turns out that BrokeredMessage has a Defer method whose name suggests it can do exactly what we want – put this message aside for processing later. But, we can’t specify how long we want to defer it for, and when you defer it, it will not be retrieved again by the OnMessage function we’ve been using in our demos.

So how do you get a deferred message back? Well, you must remember it’s sequence number, and then use a special overload of QueueClient.Receive that will retrieve a message by sequence number.

This ends up getting a little bit complicated as now we need to remember the sequence number somehow. What you could do is post another message to yourself, setting the ScheduledEnqueueTimeUtc to the appropriate time, and that message simply contains the sequence number of the deferred message. When you get that message you can call Receive passing in that sequence number and try to process the message again.

This approach does work, but as I said, it seems over-complicated, so let’s look at one final approach.

Resubmit Message

The final approach is simply to Complete the original message and resubmit a clone of that message scheduled to be handled at a set time in the future. The Clone method on BrokeredMessage makes this easy to do. Let’s look at an example:

client.OnMessage(m =>
{

    Console.WriteLine("----------------------------------------------------");
    Console.WriteLine($"RX: {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}");

    // Send a clone with a deferred wait of 5 seconds
    var clone = m.Clone();
    clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(5);
    client.Send(clone);

    // Remove original message from queue.
    m.Complete();
}, options);

Here we simply clone the original message, set up the scheduled enqueue time, send the clone and complete the original. Are there any downsides here?

Well, it’s a shame that sending the clone and completing the original are not an atomic operation, so there is a very slim chance of us seeing the original again should the handling process crash at just the wrong moment.

And the other issue is that DeliveryCount on the clone will always be 1, because this is a brand new message. So we could infinitely resubmit and never get round to dead-lettering this message.

Fortunately, that can be fixed by adding our own resubmit count as a property of the message:

client.OnMessage(m =>
{
    int resubmitCount = m.Properties.ContainsKey("ResubmitCount") ?  (int)m.Properties["ResubmitCount"] : 0;

    Console.WriteLine("----------------------------------------------------");
    Console.WriteLine($"RX: {m.MessageId} - '{m.GetBody<string>()}'");
    Console.WriteLine($"DeliveryCount: {m.DeliveryCount}, ResubmitCount: {resubmitCount}");

    if (resubmitCount > 5)
    {
        Console.WriteLine("DEAD-LETTERING");
        m.DeadLetter("Too many retries", $"ResubmitCount is {resubmitCount}");
    }
    else
    {
        // Send a clone with a deferred wait of 5 seconds
        var clone = m.Clone();
        clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(5);
        clone.Properties["ResubmitCount"] = resubmitCount + 1;
        client.Send(clone);

        // Remove message from queue.
        m.Complete();
    }
}, options);

Summary

It is a shame that there isn’t an overload of Abandon that specifies a time to wait before re-attempting processing of the message. But there are several ways you can work around this limitation as we’ve seen in this post. Of course you may know of a better way to tackle this problem. If so, please let me know in the comments.

Comments

Comment by Conor Mccarthy

> "there is a very slim chance of us seeing the original again should the handling process crash at just the wrong moment"
If you have a long running task (which can often happen unintentionally if your database or network plays up) then your message lock will expire and `m.Complete()` will fail. So it's quite likely that you'll see the original again.
If you're using a queue simply to schedule a job, and expect to have one message only in the queue, then a solution here is to get the queue MessageCount and if it's not 1 then Complete() the message without adding a clone to the queue.
```
NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(ServiceBusConnectionString);
long count = namespaceManager.GetQueue("queueemcqueueface").MessageCount;
if (count != 1)
{
// don't process
message.Complete();
return;
}
```

Conor Mccarthy
Comment by Conor Mccarthy

Awesome post by the way - helped me a lot!

Conor Mccarthy
Comment by DisqusTech

Thanks for the post. I came up with the same Clone and Resubmit with a custom Property as well (to track "how manies"), before stumbling onto this article. Like you said, small chance of losing it since its not atomic. :( But I think its the best one can do.

DisqusTech
Comment by DisqusTech

Great write up on the different options. Thanks! PS Too bad they sh0t down this guy's idea. https://feedback.azure.com/...

DisqusTech
Comment by Mark Heath

yes, it's a shame. there are a lot of people who would find this useful

Mark Heath
Comment by Sean Feldman

Good write up! In regards to delay in your callback, you might want to swap Thread.Sleep with Task.Sleep() to prevent thread blocking.

Sean Feldman
Comment by Mark Heath

thanks for the reminder. Old habits die hard!

Mark Heath
Comment by Steve Culshaw

Excellent and very useful post ... thanks for sharing

Steve Culshaw
Comment by DisqusTech

As seen here http://stackoverflow.com/qu... .Defer is also incrementing the DeliveryCount on you, so the DeliveryCount will max out on you and go to dead letter. The clone/complete/sendClone seems to be the best approach. (aka, what is called "Resubmit Message" here)

DisqusTech
Comment by Dion Olsthoorn

Thanks for this excellent solution, Mark.
Note that duplicate-detection (a feature of Service Bus queues and topics) can cause cloned messages to get suppressed/ignored when resubmitting to the service bus.
In my solution I therefore added the line `clone.MessageId = Guid.NewGuid().ToString("N");` right after the Clone() command.

Dion Olsthoorn
Comment by Dion Olsthoorn

When your task is long running, you should renew the message lock in a async thread just before it expires.
Or, when using the OnMessage pump, you can use the AutoRenewTimeout message option for this.

Dion Olsthoorn
Comment by Sean Feldman

@dionolsthoorn:disqus I'd be careful with "should". Renewing a lock as an operation can fail. Hence why I recommend people to avoid it rather than heavily rely on it. Just like native deduplication. It's working for a defined period of N, but not N+1 which always can happen.

Sean Feldman
Comment by rakesh agarwal

Does this AutoRenewTimeout renew the lock once, or keeps renewing again and again until the message complete

rakesh agarwal
Comment by Mark Heath

it keeps renewing as many times as necessary but after AutoRenewTimeout expires it won't renew any more

Mark Heath
Comment by rakesh agarwal

If it keeps renewing then how it could expire before the message completes

rakesh agarwal
Comment by Mark Heath

say AutoRenewTimeout is 20 minutes. It will keep renewing the lock for 20 minutes and then the message can expire

Mark Heath
Comment by rakesh agarwal

Got it, thanks :)

rakesh agarwal
Comment by Sean Feldman

It is a DateTime value @Rakesh. That's how 😉

Sean Feldman
Comment by Glen Elder

This is exactly what I was looking for. I was running circles trying to figure out the best way to use the Defer method, but your final solution is much nicer. Thanks for the great post!

Glen Elder
Comment by Jim Aho

Great article!

Jim Aho
Comment by Catalin Moldovan

Excellent explanation, thanks

Catalin Moldovan
Comment by Nestor Orest Plysyuk

This is what I was looking for. Thank you. I could add this code to yours to don't increment database counter if you insert something.
using (var transaction = unitOfWork.BeginTransaction())
{
try {
// works
transaction.Commit();
}
catch (Exception e) {
// fails
transaction.Rollback();
}
}

Nestor Orest Plysyuk