0 Comments Posted in:

One of the workflow patterns that Azure Durable Functions supports is waiting for an external event. This is ideal for scenarios like waiting for manual approval before a process can continue, or waiting for an external system to perform a task.

Whenever you implement this pattern, it's very likely that you will also want to put a timeout into the workflow, so if the external event is not received, you can respond in some way.

Orchestrator code - waiting for events

Say for example, that we are implementing an order processing pipeline, and for orders over a certain value, we need to get manual approval.

Here's an example orchestrator function that requires a manual approval process for orders over $1000.

[FunctionName("O_ProcessOrder")]
public static async Task<object> ProcessOrder([OrchestrationTrigger] DurableOrchestrationContext ctx, TraceWriter log)
{
    var order = ctx.GetInput<Order>();

    if (order.Amount > 1000)
    {
        await ctx.CallActivityAsync("A_RequestOrderApproval", order);
        using (var cts = new CancellationTokenSource())
        {
            var timeoutAt = ctx.CurrentUtcDateTime.AddHours(24);
            var timeoutTask = ctx.CreateTimer(timeoutAt, cts.Token);
            var approvalTask = ctx.WaitForExternalEvent<string>("OrderApprovalResult");

            var winner = await Task.WhenAny(approvalTask, timeoutTask);
            if (winner == approvalTask)
            {
                cts.Cancel(); // we should cancel the timeout task
            }
            else
            {
                // timed out
                await ctx.CallActivityAsync("A_SendNotApprovedEmail", order);
                return "Order not approved";
            }
        }

First of all, if approval is needed we call the A_RequestOrderApproval activity function. This might send an email to someone who can review and approve the order.

Then we need a new CancellationTokenSource to support cancelling the task that's waiting for an external event. Next we use DurableOrchestrationContext.CreateTimer to create a task that will complete at a certain time. We might be tempted to use DateTime.UtcNow here, but that's against the rules for an orchestrator function, which must be strictly deterministic. So instead we use DurableOrchestrationContext.CurrentUtcDateTime to calcuate the timeout end time. In this example I've set it to 24 hours.

Notice we don't await the return of ctx.CreateTimer. That's because we're also going to call ctx.WaitForExternalEvent in parallel. This is the task that's waiting for the external event to be sent to the workflow.

Next, we use Task.WhenAny to see which task completes first. If it's the approvalTask that means we got an external event before the timeout completed. We should cancel the timeout task because we don't need it. If the timeout task is the winner, then we can proceed with whatever action we want to take in the case of a timeout.

Notice that there is no way to cancel the WaitForExternalEvent task. Should the event we were waiting for turn up in the future after the timeout has fired, then our orchestrator function will not take action on it, because the event sourcing history has already recorded that the timeout task won.

Simplifying waiting for timeout

One of the downsides of the orchestrator code shown above is that it is quite verbose, and (inspired by this tweet from Mikhail Shilkov) I wondered if it might be possible to create a helper overload of DurableOrchestrationContext.WaitForExternalEvent that took a timeout.

Imagine if we could simplify the orchestrator code down to something like this:

await ctx.CallActivityAsync("A_RequestOrderApproval", order);

var approvalResult = await ctx.WaitForExternalEvent<string>(
    "OrderApprovalResult", TimeSpan.FromHours(24));

if (approvalResult == null)
{
    // we timed out
    await ctx.CallActivityAsync("A_SendNotApprovedEmail", order);
    return "Order not approved";
}

This has the advantage not only of being more succinct and readable, but also protecting developers from implementing this pattern incorrectly.

I wondered if this could be implemented as an extension method on DurableOrchestrationContext, but my initial attempts failed. It's harder to implement than you might think as the Durable Task framework (which is what Durable Functions is built on) places very strict constraints on your use of the await keyword. However, thanks to some expert advice from Chris Gillum I was able to create an extension method that gave me the behaviour I wanted.

The only question was how this method should signal a timeout. Chris rightly suggested that throwing a TaskCanceledException was the proper way to do it, but that does mean the orchestrator function needs a try catch block. I was happy for my purposes for it to return null to simplify the orchestrator code (if only C# came with a built-in Option<T> type!). Either approach is possible, and in my code example below I show both ways:

public static class DurableOrchestrationContextExtensions
{
    public static Task<T> WaitForExternalEvent<T>(
        this DurableOrchestrationContext ctx, string name, TimeSpan timeout)
    {
        var tcs = new TaskCompletionSource<T>();
        var cts = new CancellationTokenSource();

        var timeoutAt = ctx.CurrentUtcDateTime + timeout;
        var timeoutTask = ctx.CreateTimer(timeoutAt, cts.Token);
        var waitForEventTask = ctx.WaitForExternalEvent<T>(name);

        waitForEventTask.ContinueWith(t =>
        {
            using (cts)
            {
                if (t.Exception != null)
                {
                    tcs.TrySetException(t.Exception);
                }
                else
                {
                    tcs.TrySetResult(t.Result);
                }
                cts.Cancel();
            }
        }, TaskContinuationOptions.ExecuteSynchronously);

        timeoutTask.ContinueWith(t =>
        {
            using (cts)
            {
                //tcs.TrySetCanceled(); - if you'd prefer to throw a TaskCanceled exception
                tcs.TrySetResult(default(T));
            }
        }, TaskContinuationOptions.ExecuteSynchronously);

        return tcs.Task;
    }
}

Hopefully something like this will become officially part of the Durable Functions, extension in the future, but in the mean-time it's very easy to use this extension method in your function app.

Sending external events to orchestrations

Durable Functions has a REST API that can be used to send an external event to an orchestration. You need to know the orchestration id, the name of the event, and the secure key to authorize the call. Then you post to the raiseEvent endpoint, and pass whatever JSON payload you want as the event body.

However, if you're waiting for manual approval, you're hardly going to expect the approver to crack open Postman and call the REST API directly. And most external systems will have their own way of communicating back to you, whether by a webhook, or posting a message to a queue, or some other mechanism.

So usually, you create another Azure Function that will be triggered by the external event, and from within that function, pass on the message to the orchestration. Within that function, you will discover the orchestration id you need to send the event to, and then use DurableOrchestrationClient.RaiseEventAsync to send the event to the orchestration.

In the following example, I have a regular HTTP triggered Azure Function, whose route contains the order id of the order to be approved (note this is not the same as the orchestration id). I've used a table storage binding to look up the orchestration id that relates to this order (The orchestrator has already written details of this order to table storage before it started waiting for this external event). And then I simply pass on whatever was in the body of the HTTP request as the event data for an external event, using DurableOrchestrationClient.RaiseEventAsync.

[FunctionName("ApproveOrderById")]
public static async Task<IActionResult> ApproveOrderById(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = "approve/{id}")]HttpRequest req,
    [OrchestrationClient] DurableOrchestrationClient client,
    [Table("Orders", "ORDER", "{id}", Connection = "AzureWebJobsStorage")] OrderEntity order,
    TraceWriter log, string id)
{
    log.Info($"Setting approval status of order {id}");

    if (order == null)
    {
        return new NotFoundResult();
    }
    var requestBody = await new StreamReader(req.Body).ReadToEndAsync();
    await client.RaiseEventAsync(order.OrchestrationId, "OrderApprovalResult", requestBody);
    return new OkResult();
}

As you can see, the RaiseEventAsync method makes it really straightfoward to pass on an event to an orchestration, whatever mechanism the external system you are waiting on actually uses to report back to you.

Postscript 1 - Durable Functions presentation

I'm really pleased to announce that I'm going to be speaking about Durable Functions at ProgNET London on September 12th 2018. I'd love to see you there. Let me know if you're planning on attending.

Postscript 2 - MVP Award

I should also take this opportunity to say how grateful and honoured I am to have been awarded the Microsoft MVP award for the second time. Last year was an amazing experience, especially making my first ever trip to the USA to attend the MVP Summit.

My award category is now Azure, which makes sense as that's been my main focus over the past few years. So I want to thank everyone at Microsoft for building awesome cloud based products and developer tools which I'm having loads of fun learning and teaching. Also, thanks to everyone who has followed my blog, watched my Pluralsight courses, and offered support and encouragement along the way - it means a lot to me. Finally, a huge thank you to everyone who is working hard at building developer community and sharing knowledge through blogs, books, videos, talks, and open source projects. Whether or not you've been recognized for your contribution with an award, it really is appreciated.

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

0 Comments Posted in:

It's only been a month since I released my Durable Functions Fundamentals course on Pluralsight, but it's great to see that the platform is continuing to evolve and pick up new features.

A new v1.5 was released yesterday, and so I thought I'd highlight some of my favourite updates since I released my course.

Durable Functions and Azure Functions v2

Azure Functions version 2 has been in development for some time, and is hopefully close to being ready to go live, but isn't quite finished yet, so for my course I created a sample app with Azure Functions v1 and then ported it to Azure Functions v2

The changes between the two were relatively minor, but one gotcha with Durable Functions and Azure Functions V2 revolves around using the CreateCheckStatusResponse method of DurableOrchestrationClient.

Here's my starter function that kicks off a new orchestration. Notice that it takes a HttpRequestMessage parameter and returns a HttpResponseMessage:

[FunctionName("ProcessVideoStarter")]
public static async Task<HttpResponseMessage> Run(
    [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)]
    HttpRequestMessage req,
    [OrchestrationClient] DurableOrchestrationClient starter,
    TraceWriter log)
{
    string video = req.RequestUri.ParseQueryString()["video"];

    if (video == null)
    {
        return req.CreateResponse(HttpStatusCode.BadRequest,
            "Please pass the video location the query string");
    }

    log.Info($"About to start orchestration for {video}");

    var orchestrationId = await starter.StartNewAsync("O_ProcessVideo", video);

    return starter.CreateCheckStatusResponse(req, orchestrationId);
}

This compiles just fine in Azure Functions V2, but if you use the templates to create a new HTTP triggered function, then you'll end up with a function that takes a HttpRequest and returns an IActionResult. This prevents you from using the CreateCheckStatusResponse method in these new-style HTTP triggered functions.

The good news is that there is now a new CreateHttpManagementPayload method that generates a JSON payload containing the correct URIs for the management APIs for this orchestration that you can use for the response. So here's an updated version of my starter function that uses the new-style function signature.

[FunctionName("ProcessVideoStarter")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)]
    HttpRequest req,
    [OrchestrationClient] DurableOrchestrationClient starter,
    TraceWriter log)
{
    string video = req.GetQueryParameterDictionary()["video"];

    if (video == null)
    {
        return new BadRequestObjectResult(
            "Please pass the video location the query string");
    }

    log.Info($"About to start orchestration for {video}");

    var orchestrationId = await starter.StartNewAsync("O_ProcessVideo", video);
    var payload = starter.CreateHttpManagementPayload(orchestrationId);
    return new OkObjectResult(payload);
}

Custom Orchestration Status

Another great recently added feature is the ability to store a custom orchestration status against any running orchestration. The orchestrator function can set this value by calling SetCustomStatus on the DurableOrchestrationContext.

When you use the REST API to query the status of a running orchestration, you get to see the value of this custom status, making it great for tracking how far through the workflow you are.

For example, you might just update it with a simple string just before calling an activity or sub-orchestration:

ctx.SetCustomStatus("sending approval request email");

This would result in the following when we queried the status for this instance:

{
  instanceId: "741b87080ed74430a17863d9ee437101",
  runtimeStatus: "Running",
  input: "example.mp4",
  customStatus: "sending approval request email",
  output: null,
  createdTime: "2018-06-22T10:54:50Z",
  lastUpdatedTime: "2018-06-22T10:54:58Z"
}

But you're not limited to strings. You can store any serializable object in there, so it could include the IDs of sub-orchestrations that have been kicked off, or URLs of files that have been created in blob storage or IDs of rows that have been added to a database. It's a simple feature, but very powerful if you want to make your workflows easier to diagnose.

Enumerating Instances

Another great new feature in 1.5, is that now we can call an API (or use the GetStatusAsync method on the DurableOrchestrationClient) to retrieve details of all orchestrations stored in the task hub. This allows you to discover the orchestration ids of any running orchestrations, if you'd failed to keep track of them.

I think that management APIs like this will be vital in persuading developers to be willing to give Durable Functions a go in production. However, I think it needs to go further and give us support for paging and filtering the orchestrations (as there could be a vast number of them if Durable Functions is being used heavily), and it would also be good to be able to purge the event sourcing history of old Durable Functions from the task hub, either to simply clean up and save space, or as part of a required data retention cleanup.

I've submitted a GitHub issue with my desired enhancements to this feature, so do get involved in the discussion if you think this is something that would be useful to you as well.

JavaScript support

Finally, I have to mention that it seems like great progress is being made on adding JavaScript support to Durable Functions. It's not a feature I've tried myself yet, but there are some sample apps which give you a good feel for the required syntax in a JavaScript orchestrator function. Hopefully supporting more languages will result in Durable Functions being embraced by a wider pool of developers.

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

0 Comments Posted in:

I started creating NAudio back in 2002, using v1.0 of the .NET Framework and developing on the open source SharpDevelop IDE.

Of course, a huge amount has changed in the .NET world since then. However, since NAudio was heavily used in commercial applications running on Windows XP, I was always reluctant to depend on newer .NET features that would cause us to have to stop supporting legacy versions of Windows, or to maintain two different versions of NAudio.

So NAudio remained for a long time on .NET 2.0, before eventually upgrading to .NET 3.5. This means it has not yet taken advantage of Task and async / await, and a .NET Standard version has not been built, meaning it can't be used from .NET Core.

Part of the reason for this is simply that NAudio is very Windows-centric. A large part of the codebase consists of P/Invoke or COM interop wrappers around the various Windows audio APIs. So even if a .NET Standard build were to be created, much of the functionality would fail to work if you tried to use it in a .NET Core app running on Linux.

Having said that, there are a fair amount of general purpose utility classes in NAudio that would be usable cross-platform, and it seems a shame to block .NET Core apps from using it, especially if they are being run on Windows. So over my Easter holidays I started to see what it would take to make a .NET Standard version of NAudio.

New csproj format

The first step was moving to the new csproj file format. I really love this new format, and it brings a lot of benefits. You don't need to explicitly include source files - they are picked up automatically. It's got a nicer way of specifying NuGet dependencies, and it can also contain all the metadata needed to define the project as a NuGet package. I also need to support unsafe code blocks.

Here's the first part of my csproj file, specifying three target frameworks (more on that later!), and the metadata for the NuGet package;

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netstandard2.0;net35;uap10.0.10240</TargetFrameworks>
    <Version>1.9.0-preview1</Version>
    <Authors>Mark Heath &amp; Contributors</Authors>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <Description>NAudio, an audio library for .NET</Description>
    <PackageLicenseUrl>https://github.com/naudio/NAudio/blob/master/license.txt</PackageLicenseUrl>
    <PackageProjectUrl>https://github.com/naudio/NAudio</PackageProjectUrl>
    <PackageTags>C# .NET audio sound</PackageTags>
    <RepositoryUrl>https://github.com/naudio/NAudio</RepositoryUrl>
    <Copyright>© Mark Heath 2018</Copyright>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <GenerateDocumentationFile Condition=" '$(Configuration)' == 'Release' ">true</GenerateDocumentationFile>
  </PropertyGroup>

Target Frameworks

I started off by trying to target the lowest version of .NET Standard I could get away with - .NET Standard 1.6. I'd had some success with a previous proof of concept attempt for that in the past. But I soon found there was just too much to fix up, and that .NET Standard 2.0 was going to be much easier.

But I also wanted to see if I could still produce a .NET 3.5 build, so I added the net35 target framework moniker. This resulted in some code that could compile for .NET 3.5 and not for .NET Standard 2.0 and vice versa.

There were two ways of handling this. First, I could exclude whole C# files that weren't going to work on that target framework. So for example, in .NET Standard, the WinForms user interface components were never going to work, but also a number of the input and output device implementations used things like Windows Forms objects or the Windows Registry and so couldn't easily be made to compile for .NET Standard 2.0. Here's how to exclude certain files from a particular framework.

  <ItemGroup Condition=" '$(TargetFramework)' == 'netstandard2.0' ">
    <Compile Remove="Utils\ProgressLog*.*" />
    <Compile Remove="Gui\*.*" />
    <Compile Remove="Wave\MmeInterop\WaveWindow.cs" />
    <Compile Remove="Wave\MmeInterop\WaveCallbackInfo.cs" />
    <Compile Remove="Wave\WaveInputs\WaveIn.cs" />
    <Compile Remove="Wave\WaveOutputs\WaveOut.cs" />
    <Compile Remove="Wave\WaveOutputs\AsioOut.cs" />
    <Compile Remove="Wave\WaveOutputs\AsioAudioAvailableEventArgs.cs" />
    <Compile Remove="Wave\WaveFormats\WaveFormatCustomMarshaler.cs" />
  </ItemGroup>

I also needed to reference Windows.Forms for the .NET 3.5 target framework only, which can be done like this:

  <ItemGroup Condition=" '$(TargetFramework)' == 'net35' ">
    <Reference Include="System.Windows.Forms" />
  </ItemGroup>

There were still some individual bits of code I needed to exclude depending on what target framework I was building. For this, I needed to use the built-in #define symbols, which are NET35 and NETSTANDARD2_0.

A fair amount of marshaling code needed to switch based on the target. Here's a simple example:

public static int SizeOf<T>()
{
#if NET35
    return Marshal.SizeOf(typeof (T));
#else
    return Marshal.SizeOf<T>();
#endif
}

One of the biggest challenges at this point was making sense of the thousands of error messages I was getting. Unfortunately, Visual Studio doesn't seem to offer any way to just build for a specific target framework, so you end up with errors mixed together for different target frameworks and it can be confusing to know which is for which. Add that to the fact that ReSharper also struggles with multi-targetting, and it was quite a frustrating experience trying to chase all the compile errors out.

There is a dropdown in the top-left of the VS 2017 code editor window that lets you see the syntax highlighting depending on what target you chose, which is handy to see which code surrounded by #if blocks is valid.

UWP

Over the past few years I've maintained a rather hacky and incomplete build of NAudio for Windows 8, WinRT, Universal Windows apps, and I wanted to see if I could target that as well. The framework moniker is uap10.0, which I'm sure I had compiling successfully at Easter, but for whatever reason on my new dev machine I can't make that compile at all, and so for now I've specifically targetted uap10.0.10240 which is the first version of Windows 10.

Obviously lots of NAudio has to be excluded to target UWP, and I also had to reference the Microsoft.NETCore.UniversalWindowsPlatform NuGet package, as well as MSBuild.Sdk.Extras which I then imported as a project. To be honest, I don't really understand exactly what this does or why it was necessary, but I was really floundering at this point until I found a project by Oren Novotny, and copied the csproj from there (which has since changed to be .NET Standard 2.0 only).

Here's my UWP specific parts of the project file:

  <ItemGroup Condition=" '$(TargetFramework)' == 'uap10.0.10240' ">
    <Compile Remove="Utils\ProgressLog*.*" />
    <Compile Remove="Gui\*.*" />
    <Compile Remove="Wave\Compression\*.cs" />
    <Compile Remove="Wave\Asio\*.cs" />
    <Compile Remove="Wave\MmeInterop\WaveWindow.cs" />
    <Compile Remove="Wave\MmeInterop\WaveCallbackInfo.cs" />
    <Compile Remove="Wave\Midi\MidiInterop.cs" />
    <Compile Remove="Wave\WaveInputs\WaveIn.cs" />
    <Compile Remove="Wave\WaveInputs\WaveInEvent.cs" />
    <Compile Remove="Wave\WaveInputs\WasapiCapture.cs" />
    <Compile Remove="Wave\WaveInputs\WasapiLoopbackCapture.cs" />
    <Compile Remove="Wave\WaveOutputs\WaveOut.cs" />
    <Compile Remove="Wave\WaveOutputs\WaveOutEvent.cs" />
    <Compile Remove="Wave\WaveOutputs\DirectSoundOut.cs" />
    <Compile Remove="Wave\WaveOutputs\WasapiOut.cs" />
    <Compile Remove="Wave\WaveOutputs\MediaFoundationEncoder.cs" />
    <Compile Remove="Wave\WaveOutputs\AsioOut.cs" />
    <Compile Remove="Wave\WaveOutputs\AsioAudioAvailableEventArgs.cs" />
    <Compile Remove="Wave\WaveStreams\AudioFileReader.cs" />
    <Compile Remove="Wave\WaveStreams\Mp3FileReader.cs" />
    <Compile Remove="Wave\WaveStreams\WaveFormatConversionProvider.cs" />
    <Compile Remove="Wave\WaveStreams\WaveFormatConversionStream.cs" />
    <Compile Remove="Wave\WaveFormats\WaveFormatCustomMarshaler.cs" />
    <Compile Remove="Wave\WaveProviders\MediaFoundationResampler.cs" />
    <Compile Remove="FileFormats\Mp3\Mp3FrameDecompressor.cs" />
  </ItemGroup>

  <ItemGroup Condition=" '$(TargetFramework)' == 'uap10.0' or '$(TargetFramework)' == 'uap10.0.10240'">
    <PackageReference Include="Microsoft.NETCore.UniversalWindowsPlatform" Version="6.0.8" />
    <PackageReference Include="MSBuild.Sdk.Extras" Version="1.5.4" PrivateAssets="All" />
  </ItemGroup>
  <Import Project="$(MSBuildSDKExtrasTargets)" Condition="Exists('$(MSBuildSDKExtrasTargets)')" />

I also had to add in a lot more code exclusions as in UWP there are lots of small missing attributes that were part of my interop signatures. For example:

    [ComImport,
#if !WINDOWS_UWP
    System.Security.SuppressUnmanagedCodeSecurity,
#endif
    Guid("59eff8b9-938c-4a26-82f2-95cb84cdc837"),
    InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

Success?

With this in place, I was eventually able to get NAudio building, outputting a single NuGet package containing a .NET 3.5 dll, a .NET Standard DLL, and a UAP DLL.

A quick test revealed that this worked great - I could add it to regular .NET Framework apps, and use all the WinForms stuff if I wanted. I could also use it from a .NET Core console app and play audio with the WaveOutEvent class (on Windows of course)! And I could use it from a UWP application as well.

So mission accomplished perhaps? Well, I have one reservation about what I've done so far, and that's whether it's worth having the UWP target framework. With this target, it's not possible to build the NAudio solution on Windows 7 or without the UWP 10240 SDK installed. This would be a pain for contributors who want to make pull requests or experiment locally.

I could of course go back to separate solutions and a nuspec file to piece it all together, but given how experimental the UWP code still is, and the fact that newer versions of UWP now support .NET Standard 2.0 anyway, I'm wondering whether I'd be better off just targetting .NET Standard 2.0 and .NET 3.5, and producing an entirely separate assembly for the few UWP specific classes I have, and requiring that you use it on versions of Windows 10 that support .NET Standard 2.0. Let me know in the comments if you have a strong preference either way.

Try it out!

If you want to be nosey and see what I'm doing, the code is available on GitHub in the netstandard branch and I've pushed a 1.9.0.preview1 package to NuGet which you're encouraged to try and report your results.

What's Next?

My idea is that NAudio 1.9 will support both .NET Standard 2.0 and .NET 3.5 (not sure about UWP), but that it will mark the final version that I avoid making any breaking changes to the public API.

For NAudio 2.0, I'd like to modernize the interfaces for playback and recording, taking advantage of Span<T> and async/await for a much more modern coding style. I'd also split the behaviour out into more packages. The core NAudio 2.0 package would have contain .NET Standard cross-platform friendly code, but you'd then add additional NuGet packages for WaveIn/WaveOut, or WASAPI, or ASIO. There'd be another for UWP.

Of course, this assumes I manage to find some free time to work on NAudio 2.0, which realistically there won't be much of, but that's at least an idea of where I'd like to go next with the project.

Want to get up to speed with the the fundamentals principles of digital audio and how to got about writing audio applications with NAudio? Be sure to check out my Pluralsight courses, Digital Audio Fundamentals, and Audio Programming with NAudio.