Posted in:

This is the fifth (and final) part in a series about how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. Today we're finally ready to create our workflow with Durable Functions. The workflow has the following steps:

  • A HTTP triggered starter function starts off a new Durable Functions orchestration
  • The orchestrator function calls an activity function that uses the Azure .NET SDK to create the new ACI container group
  • It then waits for that ACI container group to finish exiting using a "sub-orchestration"
    • The sub-orchestrator function repeatedly calls an activity function that polls for the status of the ACI container group
  • The orchestrator then calls a final activity function that deletes the ACI container group

Table of contents:

Starter Function

First of all, we need a function to start off our durable orchestration. Since this is just a proof of concept application, I'm using a HTTP triggered function that you can post an instance of my ContainerGroupDefinition class to. This means that the caller has complete freedom to ask for whatever container image they want and customize the ACI container settings. Obviously for the real-world scenarios I want to use this in, the "starter" function would have a more focused scope - e.g. it might simply let you submit the URI of a video file you want to be processed. But this starter function gives us the flexibility to try out different ideas with ACI container groups.

Here's a simplified version of my starter function. You can see that I return the durable functions "HTTP management payload" which contains the URLs needed to check on the progress of the Durable Functions orchestration, and to cancel it if necessary. This is really convenient for test purposes, but again, in a real-world application we may not expose low-level details like this to the end users of our workflow.

[FunctionName("AciCreate")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequest req,
    [OrchestrationClient] DurableOrchestrationClientBase client)
{
    // deserialize the HTTP request body into a ContainerGroupDefinition
    var body = await req.ReadAsStringAsync();
    var def = JsonConvert.DeserializeObject<ContainerGroupDefinition>(body);
    // start the new orchestration
    var orchestrationId = await client.StartNewAsync(nameof(AciCreateOrchestrator), def);
    // return some useful information about the orchestration
    var payload = client.CreateHttpManagementPayload(orchestrationId);
    return new OkObjectResult(payload);
}

Orchestrator function

The starter function used StartNewAsync to call our orchestrator function, and passed it a ContainerGroupDefinition. The orchestrator function retrieves this input data (with GetInput), calls the AciCreateActivity activity function which creates our ACI container group, and then starts off a sub-orchestrator that will wait up to 30 minutes for that ACI container group to finish running. Finally, whether the container finished or not within the time limit, we call our third activity function that deletes the container group.

[FunctionName(nameof(AciCreateOrchestrator))]
public static async Task AciCreateOrchestrator(
    [OrchestrationTrigger] DurableOrchestrationContextBase ctx)
{
    // get the orchestration input data
    var definition = ctx.GetInput<ContainerGroupDefinition>();

    // call an activity function to create our ACI container group
    await ctx.CallActivityAsync(nameof(AciCreateActivity), definition);
    
    // start a sub-orchestration to wait for the ACI container group to exit
    var subOrchestrationId = $"{ctx.InstanceId}-1";
    var maximumRunDuration = ctx.CurrentUtcDateTime.AddMinutes(30);
    await ctx.CallSubOrchestratorAsync(nameof(AciWaitForExitOrchestrator), subOrchestrationId, (definition, maximumRunDuration));

    // call an activity function to delete the ACI container group
    await ctx.CallActivityAsync(nameof(AciDeleteContainerGroupActivity), definition);
}

Wait for exit sub-orchestrator function

Let's look at the "wait for exit" sub-orchestrator function next. In an ideal world we wouldn't need this. I'd like to see Azure Container Instances automatically publishing events to Event Grid when a container instance or container group stops running. That way, we wouldn't need this sub-orchestrator, and could use WaitForExternalEvent (with a timeout) instead, with the AciMonitor function we looked at earlier passing on the Event Grid notification to the Durable Functions orchestration. But for now we are required to poll.

I've implemented the polling using an "eternal orchestration" pattern, where we use ContinueAsNew to loop back round and run the same orchestrator function again. This is because the underlying event-sourcing implementation of Durable Functions means that an orchestrator function should avoid looping many times like ours potentially will.

In this example, we start by calling an activity function that can get the status of our container group, and if that indicates that the first (and only in our example) container instance has terminated, that means the container group's work is complete and we can continue our workflow.

In a real-world application we'd also want to check the exit code of the container instance, and put in some exception handling in case we fail to retrieve the container group status for any reason.

Then we sleep for 30 seconds with a call to await ctx.CreateTimer, and so long as we've not been going longer than our maximum wait time, we'll loop back round with ContinueAsNew. Notice that the current time check needs to use DurableOrchestrationContextBase.CurrentUtcDateTime to function correctly. Orchestrator functions should never access DateTime.Now directly as it makes them non-deterministic.

[FunctionName(nameof(AciWaitForExitOrchestrator))]
public static async Task AciWaitForExitOrchestrator(
    [OrchestrationTrigger] DurableOrchestrationContextBase ctx)
{
    // get the input data for this sub-orchestration
    var (definition,maximumRunDuration) = 
        ctx.GetInput<(ContainerGroupDefinition,DateTime)>();

    // call an activity function to get the ACI container status
    var containerGroupStatus = await ctx.CallActivityAsync<ContainerGroupStatus>
        (nameof(AciGetContainerGroupStatusActivity), definition);

    // if the container group has finished we're done
    if (containerGroupStatus.Containers[0]?.CurrentState?.State == "Terminated")
    {
        return;
    }

    // the container group has not finished - sleep for 30 seconds
    using(var cts = new CancellationTokenSource())
    {
        await ctx.CreateTimer(ctx.CurrentUtcDateTime.AddSeconds(30), cts.Token);
    }
    
    // abort if we've been waiting too long
    if (ctx.CurrentUtcDateTime > maximumRunDuration)
    {
        return;
    }

    // container group is still working, restart this sub-orchestration with
    // the same input data
    ctx.ContinueAsNew(definition);
}

Activity functions

Our three activity functions are extremely simple. They all just pass through to the methods on AciHelpers that we discussed in part 4. Here's the AciCreateActivity function. If you're wondering why we even need activity functions and couldn't call the AciHelpers directly from the orchestrator function, that's because the orchestrator function must not be used for long-running or non-deterministic tasks. So we have to do this from an activity function.

[FunctionName(nameof(AciCreateActivity))]
public static async Task AciCreateActivity(
    [ActivityTrigger] ContainerGroupDefinition definition)
{
    await AciHelpers.RunTaskBasedContainer(logger, definition);
}

Testing the workflow

With our Durable Functions orchestration in place, we're finally ready to test this thing. For my test scenario, I'm going to upload a video to our Azure Storage File Share, and then ask for a container running FFMPEG with that file share attached to extract a thumbnail image.

First, let's upload a randomly selected video from the excellent Channel 9 website to our file share to use as the input file. Note that the PowerShell commands I'm showing here assume that we still have access to the various PowerShell variables we retrieved in part 2 of this series.

# download our test video from channel 9
$testVideoFilename = "azfr536_mid.mp4"
$testVideo = "https://sec.ch9.ms/ch9/6fde/5c47fd06-e7d0-40d8-ab88-fe25edd66fde/$testVideoFilename"
Invoke-WebRequest -Uri $testVideo -OutFile $testVideoFilename
# upload to Azure Storage File share
az storage file upload -s $shareName --source "$testVideoFilename" `
        --account-key $storageAccountKey `
        --account-name $storageAccountName

Now I'm going to create my ContainerGroupDefinition JSON to pass to our starter function. You can see that the main things I'm customizing are the container image, the command line (FFMEG), and the file share volume to mount.

$mountPath ="/mnt/azfile"
$commandLine = "ffmpeg -i $mountPath/$testVideoFilename -vf" + `
    " ""thumbnail,scale=640:360"" -frames:v 1 $mountPath/thumb.png"
$ffmpeg = @{
    ResourceGroupName=$aciResourceGroup
    ContainerGroupName=$containerGroupName
    ContainerImage="jrottenberg/ffmpeg"
    CommandLine=$commandLine
    AzureFileShareVolumes=@(
        @{
            StorageAccountName=$storageAccountName
            StorageAccountKey=$storageAccountKey
            ShareName=$shareName
            VolumeName="vol-1"
            MountPath=$mountPath
        })
}

$json = $ffmpeg | ConvertTo-Json

Once I've created the JSON, I can call my Azure Function with Invoke-RestMethod, passing in the JSON as the HTTP request body, and making sure I provide the function's authorization code.

$orchestrationInfo = Invoke-RestMethod -Method POST `
                  -Uri "https://$hostName/api/AciCreate?code=$functionCode" `
                  -Body $json `
                  -Headers @{ "Content-Type"="application/json" }

Write-Output "Started orchestration $($orchestrationInfo.id)"

This will return very quickly, as it's not waiting for the whole process to complete (or even for the ACI container group to be created). It has simply started off the Durable Functions orchestration.

Monitor orchestration progress

When you enable Durable Functions for an Azure Functions app, it exposes some additional APIs that can be used to query the status of orchestrations and cancel them. Since we returned these URLs from our starter function, we can use the statusQueryGetUri to request the current status of the Durable Functions orchestration. And we can use the terminatePostUri to cancel our workflow if we want. (Note that this won't cancel the sub-orchestrator, but you'll notice I created that with a predictable orchestration ID, so we could send a termination request through for that as well if we wanted).

# check the orchestration status
Invoke-RestMethod $orchestrationInfo.statusQueryGetUri

# to cancel the orchestration
Invoke-RestMethod -Method Post -Uri $orchestrationInfo.terminatePostUri.Replace("{text}","cancelled")

And we can of course also use the Azure CLI to see if the container group has been created yet, and if so, whether it is still running or not by calling az container show. We can also use az storage file list to see if the thumbnail we generate from the container has appeared on our File Share yet.

# see if the container exists yet:
az resource list -g $aciResourceGroup -o table
# see detailed information about the container
az container show -g $aciResourceGroup -n $containerGroupName
# look to see if it has finished
az container show -g $aciResourceGroup -n $containerGroupName `
        --query "containers[0].instanceView.currentState"

# check the contents of the file share
az storage file list -s $shareName  `
        --account-key $storageAccountKey `
        --account-name $storageAccountName -o table

Cleaning up

Although our Durable Functions orchestrator function deletes the container once it's done, here's how you can delete all the Azure resources we created in this demo application with the Azure CLI:

# to delete the thumbnail
az storage file delete -s $shareName `
        -p "thumb.png" `
        --account-key $storageAccountKey `
        --account-name $storageAccountName

# clean up the container
az container delete -g $aciResourceGroup -n $containerGroupName -y

# to delete the event grid subscription
az eventgrid event-subscription delete `
        --name "AciEvents" --source-resource-id $resourceId

# delete the main resource group containing the file share and function app
az group delete -n $resourceGroup -y --no-wait

# delete the ACI resource group
az group delete -n $aciResourceGroup -y --no-wait

Summary

This series has been quite a long journey, and it was more complicated than I hoped to get all this working. I certainly learned a lot in the process. But we achieved the objective: we've got an Azure Durable Functions orchestration that is able to make use of Azure Container Instances with Azure File Shares to implement tasks that could not easily be performed by the Function App itself. From my limited testing it seems very quick: the whole container can spin up and run to completion in under a minute, so costs should be very reasonable.

There are of course a lot of ways in which my sample app could be improved. I only supported customizing the specific features of ACI that I needed to modify for my demo. My orchestration relies on a sub-orchestrator to perform polling to wait for container exit when an event driven approach powered by Event Grid would be much nicer.

It might be possible to take this code and turn it into a generic helper or extension for Durable Functions to simplify the process of running a containerized "activity" in ACI. While I've been working on this series, I've actually come across several people who are attempting similar things, such as this great example from Anthony Chu) who uses PowerShell functions to create the containers (which is actually quite a bit simpler compared to using the Azure SDK)! So I'm certainly not alone in seeing the potential of combining ACI with Azure Functions.

All the sample code and PowerShell automation scripts for my sample app are available here on GitHub.

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

Comments

Comment by aregaz

Just read the whole series. Looks very promising. I had plans to try exactly the same combination - Azure Durable Functions with Azure Container Instances - and exactly for the same thing - video transcoding using ffmpeg. So your article series is very helpful. I have not so much experience with PowerShell, but everything you showed looks pretty clear to me. Thanks a lot!

aregaz