0 Comments Posted in:

Back in January 2017 I released my Azure Functions Fundamentals Pluralsight Course. But of course a lot has happened in the world of Azure Functions in the last two years, and so I'm pleased to announce that I have just released a completely updated version of my course that covers Azure Functions v2.

It is a complete re-record of the entire course, and is now just over three and a half hours long. Some of the new things that I've been able to include this time round are:

I've also put the demo code from the course up on GitHub, allowing me to keep it updated to reflect the latest changes to Azure Functions.

Azure Functions remains one my favourite Azure services. It's great to see just how far it's come in the past two years, and the amazing team behind Azure Functions is continuing to improve it with all kinds of new capabilities, so I'm sure that it won't be too long before another round of updates to my course are required.

Anyway I hope you enjoy the updated course!


0 Comments Posted in:

This is the fifth (and final) part in a series about how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. Today we're finally ready to create our workflow with Durable Functions. The workflow has the following steps:

  • A HTTP triggered starter function starts off a new Durable Functions orchestration
  • The orchestrator function calls an activity function that uses the Azure .NET SDK to create the new ACI container group
  • It then waits for that ACI container group to finish exiting using a "sub-orchestration"
    • The sub-orchestrator function repeatedly calls an activity function that polls for the status of the ACI container group
  • The orchestrator then calls a final activity function that deletes the ACI container group

Table of contents:

Starter Function

First of all, we need a function to start off our durable orchestration. Since this is just a proof of concept application, I'm using a HTTP triggered function that you can post an instance of my ContainerGroupDefinition class to. This means that the caller has complete freedom to ask for whatever container image they want and customize the ACI container settings. Obviously for the real-world scenarios I want to use this in, the "starter" function would have a more focused scope - e.g. it might simply let you submit the URI of a video file you want to be processed. But this starter function gives us the flexibility to try out different ideas with ACI container groups.

Here's a simplified version of my starter function. You can see that I return the durable functions "HTTP management payload" which contains the URLs needed to check on the progress of the Durable Functions orchestration, and to cancel it if necessary. This is really convenient for test purposes, but again, in a real-world application we may not expose low-level details like this to the end users of our workflow.

[FunctionName("AciCreate")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequest req,
    [OrchestrationClient] DurableOrchestrationClientBase client)
{
    // deserialize the HTTP request body into a ContainerGroupDefinition
    var body = await req.ReadAsStringAsync();
    var def = JsonConvert.DeserializeObject<ContainerGroupDefinition>(body);
    // start the new orchestration
    var orchestrationId = await client.StartNewAsync(nameof(AciCreateOrchestrator), def);
    // return some useful information about the orchestration
    var payload = client.CreateHttpManagementPayload(orchestrationId);
    return new OkObjectResult(payload);
}

Orchestrator function

The starter function used StartNewAsync to call our orchestrator function, and passed it a ContainerGroupDefinition. The orchestrator function retrieves this input data (with GetInput), calls the AciCreateActivity activity function which creates our ACI container group, and then starts off a sub-orchestrator that will wait up to 30 minutes for that ACI container group to finish running. Finally, whether the container finished or not within the time limit, we call our third activity function that deletes the container group.

[FunctionName(nameof(AciCreateOrchestrator))]
public static async Task AciCreateOrchestrator(
    [OrchestrationTrigger] DurableOrchestrationContextBase ctx)
{
    // get the orchestration input data
    var definition = ctx.GetInput<ContainerGroupDefinition>();

    // call an activity function to create our ACI container group
    await ctx.CallActivityAsync(nameof(AciCreateActivity), definition);
    
    // start a sub-orchestration to wait for the ACI container group to exit
    var subOrchestrationId = $"{ctx.InstanceId}-1";
    var maximumRunDuration = ctx.CurrentUtcDateTime.AddMinutes(30);
    await ctx.CallSubOrchestratorAsync(nameof(AciWaitForExitOrchestrator), subOrchestrationId, (definition, maximumRunDuration));

    // call an activity function to delete the ACI container group
    await ctx.CallActivityAsync(nameof(AciDeleteContainerGroupActivity), definition);
}

Wait for exit sub-orchestrator function

Let's look at the "wait for exit" sub-orchestrator function next. In an ideal world we wouldn't need this. I'd like to see Azure Container Instances automatically publishing events to Event Grid when a container instance or container group stops running. That way, we wouldn't need this sub-orchestrator, and could use WaitForExternalEvent (with a timeout) instead, with the AciMonitor function we looked at earlier passing on the Event Grid notification to the Durable Functions orchestration. But for now we are required to poll.

I've implemented the polling using an "eternal orchestration" pattern, where we use ContinueAsNew to loop back round and run the same orchestrator function again. This is because the underlying event-sourcing implementation of Durable Functions means that an orchestrator function should avoid looping many times like ours potentially will.

In this example, we start by calling an activity function that can get the status of our container group, and if that indicates that the first (and only in our example) container instance has terminated, that means the container group's work is complete and we can continue our workflow.

In a real-world application we'd also want to check the exit code of the container instance, and put in some exception handling in case we fail to retrieve the container group status for any reason.

Then we sleep for 30 seconds with a call to await ctx.CreateTimer, and so long as we've not been going longer than our maximum wait time, we'll loop back round with ContinueAsNew. Notice that the current time check needs to use DurableOrchestrationContextBase.CurrentUtcDateTime to function correctly. Orchestrator functions should never access DateTime.Now directly as it makes them non-deterministic.

[FunctionName(nameof(AciWaitForExitOrchestrator))]
public static async Task AciWaitForExitOrchestrator(
    [OrchestrationTrigger] DurableOrchestrationContextBase ctx)
{
    // get the input data for this sub-orchestration
    var (definition,maximumRunDuration) = 
        ctx.GetInput<(ContainerGroupDefinition,DateTime)>();

    // call an activity function to get the ACI container status
    var containerGroupStatus = await ctx.CallActivityAsync<ContainerGroupStatus>
        (nameof(AciGetContainerGroupStatusActivity), definition);

    // if the container group has finished we're done
    if (containerGroupStatus.Containers[0]?.CurrentState?.State == "Terminated")
    {
        return;
    }

    // the container group has not finished - sleep for 30 seconds
    using(var cts = new CancellationTokenSource())
    {
        await ctx.CreateTimer(ctx.CurrentUtcDateTime.AddSeconds(30), cts.Token);
    }
    
    // abort if we've been waiting too long
    if (ctx.CurrentUtcDateTime > maximumRunDuration)
    {
        return;
    }

    // container group is still working, restart this sub-orchestration with
    // the same input data
    ctx.ContinueAsNew(definition);
}

Activity functions

Our three activity functions are extremely simple. They all just pass through to the methods on AciHelpers that we discussed in part 4. Here's the AciCreateActivity function. If you're wondering why we even need activity functions and couldn't call the AciHelpers directly from the orchestrator function, that's because the orchestrator function must not be used for long-running or non-deterministic tasks. So we have to do this from an activity function.

[FunctionName(nameof(AciCreateActivity))]
public static async Task AciCreateActivity(
    [ActivityTrigger] ContainerGroupDefinition definition)
{
    await AciHelpers.RunTaskBasedContainer(logger, definition);
}

Testing the workflow

With our Durable Functions orchestration in place, we're finally ready to test this thing. For my test scenario, I'm going to upload a video to our Azure Storage File Share, and then ask for a container running FFMPEG with that file share attached to extract a thumbnail image.

First, let's upload a randomly selected video from the excellent Channel 9 website to our file share to use as the input file. Note that the PowerShell commands I'm showing here assume that we still have access to the various PowerShell variables we retrieved in part 2 of this series.

# download our test video from channel 9
$testVideoFilename = "azfr536_mid.mp4"
$testVideo = "https://sec.ch9.ms/ch9/6fde/5c47fd06-e7d0-40d8-ab88-fe25edd66fde/$testVideoFilename"
Invoke-WebRequest -Uri $testVideo -OutFile $testVideoFilename
# upload to Azure Storage File share
az storage file upload -s $shareName --source "$testVideoFilename" `
        --account-key $storageAccountKey `
        --account-name $storageAccountName

Now I'm going to create my ContainerGroupDefinition JSON to pass to our starter function. You can see that the main things I'm customizing are the container image, the command line (FFMEG), and the file share volume to mount.

$mountPath ="/mnt/azfile"
$commandLine = "ffmpeg -i $mountPath/$testVideoFilename -vf" + `
    " ""thumbnail,scale=640:360"" -frames:v 1 $mountPath/thumb.png"
$ffmpeg = @{
    ResourceGroupName=$aciResourceGroup
    ContainerGroupName=$containerGroupName
    ContainerImage="jrottenberg/ffmpeg"
    CommandLine=$commandLine
    [email protected](
        @{
            StorageAccountName=$storageAccountName
            StorageAccountKey=$storageAccountKey
            ShareName=$shareName
            VolumeName="vol-1"
            MountPath=$mountPath
        })
}

$json = $ffmpeg | ConvertTo-Json

Once I've created the JSON, I can call my Azure Function with Invoke-RestMethod, passing in the JSON as the HTTP request body, and making sure I provide the function's authorization code.

$orchestrationInfo = Invoke-RestMethod -Method POST `
                  -Uri "https://$hostName/api/AciCreate?code=$functionCode" `
                  -Body $json `
                  -Headers @{ "Content-Type"="application/json" }

Write-Output "Started orchestration $($orchestrationInfo.id)"

This will return very quickly, as it's not waiting for the whole process to complete (or even for the ACI container group to be created). It has simply started off the Durable Functions orchestration.

Monitor orchestration progress

When you enable Durable Functions for an Azure Functions app, it exposes some additional APIs that can be used to query the status of orchestrations and cancel them. Since we returned these URLs from our starter function, we can use the statusQueryGetUri to request the current status of the Durable Functions orchestration. And we can use the terminatePostUri to cancel our workflow if we want. (Note that this won't cancel the sub-orchestrator, but you'll notice I created that with a predictable orchestration ID, so we could send a termination request through for that as well if we wanted).

# check the orchestration status
Invoke-RestMethod $orchestrationInfo.statusQueryGetUri

# to cancel the orchestration
Invoke-RestMethod -Method Post -Uri $orchestrationInfo.terminatePostUri.Replace("{text}","cancelled")

And we can of course also use the Azure CLI to see if the container group has been created yet, and if so, whether it is still running or not by calling az container show. We can also use az storage file list to see if the thumbnail we generate from the container has appeared on our File Share yet.

# see if the container exists yet:
az resource list -g $aciResourceGroup -o table
# see detailed information about the container
az container show -g $aciResourceGroup -n $containerGroupName
# look to see if it has finished
az container show -g $aciResourceGroup -n $containerGroupName `
        --query "containers[0].instanceView.currentState"

# check the contents of the file share
az storage file list -s $shareName  `
        --account-key $storageAccountKey `
        --account-name $storageAccountName -o table

Cleaning up

Although our Durable Functions orchestrator function deletes the container once it's done, here's how you can delete all the Azure resources we created in this demo application with the Azure CLI:

# to delete the thumbnail
az storage file delete -s $shareName `
        -p "thumb.png" `
        --account-key $storageAccountKey `
        --account-name $storageAccountName

# clean up the container
az container delete -g $aciResourceGroup -n $containerGroupName -y

# to delete the event grid subscription
az eventgrid event-subscription delete `
        --name "AciEvents" --source-resource-id $resourceId

# delete the main resource group containing the file share and function app
az group delete -n $resourceGroup -y --no-wait

# delete the ACI resource group
az group delete -n $aciResourceGroup -y --no-wait

Summary

This series has been quite a long journey, and it was more complicated than I hoped to get all this working. I certainly learned a lot in the process. But we achieved the objective: we've got an Azure Durable Functions orchestration that is able to make use of Azure Container Instances with Azure File Shares to implement tasks that could not easily be performed by the Function App itself. From my limited testing it seems very quick: the whole container can spin up and run to completion in under a minute, so costs should be very reasonable.

There are of course a lot of ways in which my sample app could be improved. I only supported customizing the specific features of ACI that I needed to modify for my demo. My orchestration relies on a sub-orchestrator to perform polling to wait for container exit when an event driven approach powered by Event Grid would be much nicer.

It might be possible to take this code and turn it into a generic helper or extension for Durable Functions to simplify the process of running a containerized "activity" in ACI. While I've been working on this series, I've actually come across several people who are attempting similar things, such as this great example from Anthony Chu) who uses PowerShell functions to create the containers (which is actually quite a bit simpler compared to using the Azure SDK)! So I'm certainly not alone in seeing the potential of combining ACI with Azure Functions.

All the sample code and PowerShell automation scripts for my sample app are available here on GitHub.

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

0 Comments Posted in:

This is the fourth part in a series about how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. Today we'll look at using the Azure .NET SDK to automate the creation of ACI Container Groups from C# using the Azure .NET SDK. We'll also see how to retrieve the status of an existing container group and delete one.

Current table of contents:

Reference the Azure .NET SDK

For this demo, I'm going to use the fluent Azure management APIs available in the Microsoft.Azure.Management.Fluent NuGet Package

We can add that to our .csproj file with:

<PackageReference Include="Microsoft.Azure.Management.Fluent" Version="1.20.0" />

Authenticate with Azure

Next, we need to authenticate with Azure using the managed identity we created in part 2. I'd also like us to be able to optionally use a regular service principal with a client id and secret, which we might want to use for testing locally, or if we ever needed this code from a place that couldn't use managed identities.

In the GetAzure function below we either use environment variables to create a ServicePrincipalLoginInformation object to log in directly with a tenant id, client id and secret, or (preferably) we use managed identities with MSILoginInformation (MSI was the old name for managed identities). Once we've generated the credentials object we want to use, we log in with the Authenticate fluent method, and then I optionally select a subscription to use if a subscription id has been provided as an environment variable.

private static readonly IAzure azure = GetAzure();

private static IAzure GetAzure()
{
    var tenantId = Environment.GetEnvironmentVariable("TenantId");
    var clientId = Environment.GetEnvironmentVariable("ClientId");
    var clientSecret = Environment.GetEnvironmentVariable("ClientSecret");
    AzureCredentials credentials;

    if (!string.IsNullOrEmpty(tenantId) && 
        !string.IsNullOrEmpty(clientId) &&
        !string.IsNullOrEmpty(clientSecret))
    {
        var sp = new ServicePrincipalLoginInformation
        {
            ClientId = clientId,
            ClientSecret = clientSecret
        };
        credentials = new AzureCredentials(sp, tenantId, AzureEnvironment.AzureGlobalCloud);
    }
    else
    {
        credentials = SdkContext
            .AzureCredentialsFactory
            .FromMSI(new MSILoginInformation(MSIResourceType.AppService), 
                     AzureEnvironment.AzureGlobalCloud);
    }
    var authenticatedAzure = Azure
        .Configure()
        .WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic)
        .Authenticate(credentials);
    var subscriptionId = System.Environment.GetEnvironmentVariable("SubscriptionId");
    if (!string.IsNullOrEmpty(subscriptionId))
        return authenticatedAzure.WithSubscription(subscriptionId);
    return authenticatedAzure.WithDefaultSubscription();
}

Create an ACI Container Group

Next, let's create an ACI container group. Although there is a fluent API for managing container instances, it can be a bit of a pain to work with if you want run-time flexibility over the exact configuration of your container group. I think there are some lower-level APIs that I could have used instead, but for now I've stuck with the fluent API.

I started by creating my own simple class to define the parameters for my container group. There's a bunch of stuff missing (like operating system, CPU and memory choice, credentials if you're using an image with ACR), so I'll probably update this sample code in the future with more capabilities, but this was the minimum feature set I needed for my experiment.

public class ContainerGroupDefinition
{
    public string ResourceGroupName { get; set; }
    public string ContainerGroupName { get; set; }
    public string ContainerImage { get; set;}
    public string CommandLine { get; set; }
    public List<AzureFileShareVolumeDefinition> AzureFileShareVolumes { get; set; }
    public Dictionary<string, string> EnvironmentVariables { get; set; }
}

Next, I actually create the "container group" (in ACI, you always create a "container group" even if it will often only have a single container in it - it's like a "pod" in Kubernetes).

The first step is to get the region to put the container group in - I'm asking for the region of the resource group. Then I use the fluent API to specify all the characteristics of the container group I want, such as the operating system (Linux at the moment) and details of the container instances in the container group. We're only supporting a single container instance, and I've hard-coded it to expose port 80, have 1 CPU and 1GB RAM, and use a public (i.e. Docker hub) image. So there's plenty more work needed to make this completely generic.

However, I have added some of my own "optional" customizations. You can optionally mount a volume, and you can optionally override the container instance startup command line. I achieved that by creating some additional fluent extension methods of my own, because fluent APIs are very bad at supporting optional steps.

Once everything has been defined about the container group, we create it with CreateAsync.

public static async Task RunTaskBasedContainer(
            ContainerGroupDefinition cg)
{
    // Get the resource group's region
    IResourceGroup resGroup = await azure.ResourceGroups.GetByNameAsync(cg.ResourceGroupName);
    Region azureRegion = resGroup.Region;
    // Create the container group
    var vol = cg.AzureFileShareVolumes?.FirstOrDefault();
    var containerGroupDefinition = azure.ContainerGroups.Define(cg.ContainerGroupName)
        .WithRegion(azureRegion)
        .WithExistingResourceGroup(cg.ResourceGroupName)
        .WithLinux()
        .WithPublicImageRegistryOnly()
        .WithOptionalVolume(vol)
        .DefineContainerInstance(cg.ContainerGroupName + "-1")
            .WithImage(cg.ContainerImage)
            .WithExternalTcpPort(80)
            .WithCpuCoreCount(1.0)
            .WithMemorySizeInGB(1)
            .WithOptionalVolumeMount(vol)
            .WithOptionalCommandLine(log, cg.CommandLine)
            .WithEnvironmentVariables(cg.EnvironmentVariables)
            .Attach()
        .WithDnsPrefix(cg.ContainerGroupName)
        .WithRestartPolicy(ContainerGroupRestartPolicy.Never);

    var containerGroup = await containerGroupDefinition.CreateAsync();
}

Here's an example of one of my "optional" extension methods. You can find them in the AciHelpers class in my demo GitHub repo.

private static IWithContainerInstanceAttach<IWithNextContainerInstance> WithOptionalVolumeMount(this IWithContainerInstanceAttach<IWithNextContainerInstance> baseDefinition, AzureFileShareVolumeDefinition vol)
{
    if (vol != null)
        baseDefinition = baseDefinition.WithVolumeMountSetting(vol.VolumeName, vol.MountPath);
    return baseDefinition;
}

This extension uses another simple DTO I created to define the settings for a file share:

public class AzureFileShareVolumeDefinition
{
    public string StorageAccountName { get; set; }
    public string StorageAccountKey { get; set; }
    public string ShareName { get; set; }
    public string VolumeName { get; set; }
    public string MountPath { get; set; }
}

One other thing worth pointing out is that the CreateAsync method will succeed if the container group already exists. I haven't experimented too much yet with what happens if it already exists but with different configuration, so it's probably worth putting in additional checks for existence before you create the container group.

Retrieving Container Group Status

Another important requirement for my application is to be able to monitor a container group to see when the container instances have finished.

Again I defined my own simplified DTOs to store information about the current state of a container group. These are not strictly necessary as you may be happy to use the types in the Azure SDK as is.

public class ContainerGroupStatus
{
    public ContainerInstanceStatus[] Containers { get; set;}
    public string State { get; set; }
    public string Id { get; set; }
    public string Name { get; set; }
    public string ResourceGroupName { get; set; }
}

public class ContainerInstanceStatus
{
    public string Name { get; set; }
    public string Image { get; set; }
    public IList<string> Command { get; set; }
    public ContainerState CurrentState { get; set; }
    public int? RestartCount { get; set; }
}

And we can then use ContainerGroups.GetByResourceGroupAsync to get information about that container group and I just project the returned information into my own custom type.

public static async Task<ContainerGroupStatus> GetContainerGroupStatus(
                ContainerGroupDefinition cg)
{
    var containerGroup = await azure.ContainerGroups
        .GetByResourceGroupAsync(cg.ResourceGroupName, cg.ContainerGroupName);

    var status = new ContainerGroupStatus() {
        State = containerGroup.State,
        Id = containerGroup.Id,
        Name = containerGroup.Name,
        ResourceGroupName = containerGroup.ResourceGroupName,
        Containers = containerGroup.Containers.Values.Select(c => 
            new ContainerInstanceStatus() {
                Name = c.Name,
                Image = c.Image,
                Command = c.Command,
                CurrentState = c.InstanceView.CurrentState,
                RestartCount = c.InstanceView.RestartCount,
            }).ToArray()
    };
    return status;
}

I can then check if a specific container instance has exited by looking for the value Terminated in the current state:

if(containerGroupStatus.Containers[0]?.CurrentState?.State == "Terminated")

Deleting Container Groups

Finally, I want to be able to delete container groups to save cost or kill them if they have hung while performing their task. That's super easy with the Azure management SDKs - we just need to call DeleteByResourceGroupAsync.

public static async Task DeleteContainerGroup(
    ContainerGroupDefinition cg)
{
    await azure.ContainerGroups.DeleteByResourceGroupAsync(
            cg.ResourceGroupName, cg.ContainerGroupName);
}

Summary

The Azure .NET management SDK makes it easy to create and manage Azure Container Instances from C#. We can authenticate either with a service principal id and secret directly, or preferably with a managed service identity if its available. The fluent nature of the API makes things a little tricky if we want to support optional steps, but we were able to work around that with some custom extension methods.

With these capabilities in place, we're now finally ready to create a Durable Functions workflow that creates, monitors and then deletes the container instances on demand. We'll see how to do that in the next part.

Want to learn more about how easy it is to get up and running with Azure Container Instances? Be sure to check out my Pluralsight course Azure Container Instances: Getting Started.