0 Comments Posted in:

One of the challenges of using Durable Functions is how to handle upgrading a workflow to a new version without breaking any in-progress orchestrations.

Example 1 - inserting a new step

For example, suppose you've created a media processing workflow for uploaded videos. Step [a] performs a virus scan, step [b] performs a transcode, and step [c] extracts a thumbnail image. So far nice and simple.

[a]-->[b]-->[c]

But what if I have a new requirement that I want to perform two transcodes, at different bitrates to create a high and low resolution copy?

I could simply add the new step [d] onto the end of the workflow. This is a relatively safe change and any workflows that were started in the previous version should just be able to carry on with the new orchestrator and will successfully call the new step.

[a]-->[b]-->[c]-->[d]

But maybe I want to add the new step in parallel with the original transcode to create a workflow like this:

     +-->[d]---+
     |         |
[a]--+-->[b]---+-->[c]

Now we are going to confuse Durable Functions if we upgrade while an orchestration is mid-progress. The order of operations has fundamentally changed, and the event sourced history of the in-progress orchestration won't map onto the new orchestrator function.

Example - modifying an activity

There are other more subtle ways which we might make breaking changes even if we don't touch our orchestrator function.

Perhaps we modify activity [a] to write data into a database that a new version of activity [c] needs to make use of. This means that if we upgraded mid-orchestration it is possible for the workflow to have run v1 of activity [a], and v2 of activity [c] which would result in an error.

This means you need to ensure you have a clearly articulated strategy for upgrades to durable workflows, that all developers understand.

Importance of idempotency

Before looking at four possible approaches to handling upgrades to workflows, it's important to realise that one of the biggest keys to success is making sure you write "idempotent" code wherever possible.

If a method is "idempotent" then running it twice has the same effect as running it once.

A classic example is charging a credit card in an ecommerce workflow. If I place an order, and something goes wrong midway through handling that order, the order processing pipeline might need to be restarted. But I don't want my credit card to get charged twice for the same order, and the vendor doesn't want to ship the same order twice.

Achieving idempotency usually involves being able to check "have I already done this?" Of course, that adds complexity, so for some activities you might decide that it doesn't really hurt if it happens twice. Maybe if you send a status update email twice it's not a big deal.

If you have a workflow where each activity is either idempotent or safe to run multiple times, then you're in a much better position to support upgrading to new versions of your workflow code.

Let's consider a few different strategies for handling upgrading to a new version of a Durable Orchestration.

Strategy 0 - Don't make breaking changes!

The first thing to say is that it is sometimes possible to make changes to a workflow that will not break in-progress orchestrations. Knowing what are and aren't breaking changes to a workflow will help you to identify what modifications can be made safely.

Strategy 1 - Upgrade with no workflows in progress

The simplest approach when you do have breaking changes, is to ensure that no workflows are currently in progress when you upgrade to a new version of your orchestration.

How easy that is depends on how frequently your workflows are called and how long they take to complete. If you trigger your orchestrations via a queue message, that gives you an easy way to disable starting new orchestrations temporarily, allowing all in-progress ones to finish. Then, after upgrading, re-enable the queue processor to start working through the queued workflows.

Strategy 2 - Just let them fail

The second approach for breaking changes is simply to allow in-progress orchestrations to fail. This might sound like a crazy idea at first, but if you've taken the trouble to ensure that the activities in your workflow are idempotent, then you can simply detect failed orchestrations and resubmit them.

You can even forcibly stop all in-flight instance using the technique described here. Obviously you'll also need a way to track which ones need to be resubmitted after the upgrade.

Strategy 3 - Separate task hubs

The versioning approach recommended in the official Azure Functions documentation is referred to as "side by side deployments". There are a few variations on how exactly you implement this, but the main way suggested is to deploy an entirely separate Function App containing the new version of your workflow.

That Function App could use its own storage account, or a different "task hub" within the same storage account to keep the Durable Functions orchestration state separate.

The trouble with doing this is that often a Function App contains more than just orchestrators and activity functions. For example if there is a "starter" function that is triggered by a HTTP request or a queue message, then the calling code now needs to know how to direct new requests to the updated Function App.

Strategy 4 - Separate orchestrators and activities

The final strategy is to create alternative orchestrator and activity functions within the same Function App. For example you could create an OrchestratorV2 function, leaving the original orchestrator unchanged to finish off any in-flight orchestrations.

With this approach any starter functions you have can simply be updated to point to the new orchestrator function, and you can eventually retire the code for the original orchestrator once all in-progress workflows have completed.

There's actually another page in the Durable Functions documentation that shows an example of how this setup can be achieved. It claims that every function needs to be branched (e.g. You create a V1 and a V2 of every activity and orchestrator function).

I'm not sure that is necessary. It's perfectly fine for the same activity function to be used in more than one orchestration. But I suppose by branching everything, it makes the code a bit easier to reason about. And hopefully its not too long before you can retire the V1 orchestrator and activity functions.

Summary

If you're using Durable Functions, you do need to think about how you want to version your workflows. Fortunately if you follow good practices of avoiding breaking changes and writing idempotent activity functions, many pitfalls can be avoided. And even when you do need to make breaking changes, there are a variety of strategies you can adopt to ensure all in-progress orchestrations complete successfully.

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

0 Comments Posted in:

One of the things I love about Azure Durable Functions is how easy it makes it to implement tricky workflow patterns like fan out and fan in (running workflow steps in parallel), and waiting for notifications from an external system with timeout support.

And in typical enterprise applications there are lots of workflows that would benefit from being written as Durable Orchestrations. The trouble is, it's not always a trivial task to take an existing complex workflow and rewrite the whole thing in Durable Functions.

Fortunately, that's not necessary. It's possible to simply use Durable Functions for the orchestration part of the code, and leave the implementation of the individual steps in the workflow where they were.

And you don't even necessarily need to convert the entire workflow in one hit. It can be done incrementally, with a Durable Functions orchestration managing part of the workflow and then handing back over to the legacy implementation. You might just want to start off by moving the complex stuff like fan out fan in into a Durable Functions orchestration initially.

How does this work?

A Durable Functions orchestrator function defines the steps (or "activities") in a workflow - starting with what to do first, and then what should happen next when each individual step completes. The orchestrator doesn't perform any of the actual activities itself. Those are normally implemented as "activity functions".

However, those activity functions can simply trigger behaviour implemented elsewhere. This might be by making a HTTP request, or posting a message to a queue. So it's pretty straightforward to trigger your legacy workflow step implementations.

Waiting for those steps to finish is a little bit trickier. How does the legacy code inform our orchestrator that it has completed the step and report back the result? There are a few different options.

Option 1 - Polling

Your external workflow step implementation might support polling for completion. That can be a bit cumbersome to write in Durable Functions, since it's not a good practice for an Activity Function to sleep (since they are billed by the second and are intended to be relatively short-running).

This means that after kicking off the potentially long-running action, you'd need to return to your orchestrator, which then sleeps for a bit (with CreateTimer). When it wakes up it call another activity function which makes the HTTP request to poll for progress. That activity function then tells the orchestrator whether the operation has completed or not.

However, thanks to the Durable HTTP requests feature I wrote about recently, that whole process is simpler, so long as there is a suitable endpoint that can be polled for progress returning 202 until the operation has completed. This API isn't currently quite as flexible as I'd like, but if you can use it, it will greatly simplify your overall orchestrator code.

Option 2 - External Events

The second option is for the workflow step to notify you when it's completed. This could be by calling a web-hook, or posting a message onto a queue. Then all you need is a regular Azure function that receives that notification, and passes it on to your durable orchestration by means of RaiseEventAsync. The orchestrator then simply needs to call WaitForExternalEvent

Of course the external activity could post the event directly to the target orchestration because Durable Functions exposes an API that you can call to trigger external events. However, I prefer to leave a level of abstraction in between, as I don't think that the code that implements a step in a workflow should be tightly coupled to the specific technology that orchestrates it.

Migrating slowly

With these two approaches to triggering activities implemented outside your Durable Function app, it is possible to incrementally migrate large and complex workflows into Durable Functions. By doing so you'll end up with multiple benefits of using Durable Functions.

Of course, if you are planning to slowly evolve these workflows over time, you do need to make sure you've thought about a good versioning strategy for your orchestrators, as changing an orchestrator function can cause in-progress orchestrations to fail.

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

0 Comments Posted in:

My usual approach for creating NuGet packages is to have a Git repository per NuGet package. This is a relatively simple and straightforward way of working, but when you're creating with multiple NuGet packages that all depend on each other, it's not always ideal.

For example, with NAudio, I've often thought about splitting it up into several smaller packages, e.g. NAudio.Core, NAudio.Asio, NAudio.MediaFoundation etc. However, I'd want to keep all the source in the same Git repository, and use local project references between them while developing to make debugging easier. At publish time, though, they'd need to reference each other as NuGet packages.

I'd assumed for ages that this required some complex build script syntax to achieve. I'd even done loads of web searches to try to find out how other people were achieving this setup, but came up blank every time.

However, it turns out that this capability is built right in with the SDK style csproj files, and really easy to achieve! Just create a couple of class libraries, add project references between them, and when you run dotnet pack, it just does the right thing and creates NuGet packages that depend on each other properly.

How to set this up

To explain what I mean and how to achieve this setup, here's some very simple instructions. We're going to create two NuGet packages - LibraryA and LibraryB, with LibraryA depending on LibraryB. We can actually set up the entire project structure including creating a .sln file and setting up the project references, using various dotnet CLI commands.

Let's start by creating the two projects (class libraries), a .sln file that includes both projects, and then ask for LibraryA to take a dependency on LibraryB:

md MultiPackageLibrary
cd MultiPackageLibrary
dotnet new classlib -o LibraryA
dotnet new classlib -o LibraryB
dotnet new solution
dotnet sln add LibraryA
dotnet sln add LibraryB
dotnet add LibraryA reference LibraryB

If we then look at LibraryA.csproj, we see it's generated a project reference from LibraryA to LibraryB:

  <ItemGroup>
    <ProjectReference Include="..\LibraryB\LibraryB.csproj" />
  </ItemGroup>

Let's create an interface in LibraryB...

namespace LibraryB
{
    public class IMyInterface
    {
    }
}

...and a concrete implementation of it in LibraryA:

using LibraryB;
namespace LibraryA
{
    public class MyClass : IMyInterface
    {
    }
}

Now we're ready to actually create our NuGet packages, which we can do with dotnet pack (and I want a release build):

dotnet pack -c Release

This will create LibraryA.1.0.0.nupkg and LibraryB.1.0.0.nupkg. And just to be sure that the dependencies are set up correctly, we can use the excellent free NuGet Package Explorer utility to look inside the LibraryA NuGet package and ensure that it correctly depends on LibraryB:

NuGet Package Explorer

Of course, one challenge with this approach is that if you have an automated CI server that publishes the packages to a NuGet feed, how does it know which NuGet package has changed? Probably the simplest option is to always up-version all the NuGet packages in the repo and republish them. But that might mean you sometimes unnecessarily publish new versions of a package without any underlying changes.

Anyway, hope someone else finds this useful - I wish I'd known about this a long time ago. Big thanks to John Wostenberg for alerting me to this capability.