0 Comments Posted in:

This is the third part in a series about how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. Today we'll look at creating an Event Grid subscription and an Azure Function triggered by an event.

Current table of contents:

Azure Event Grid is a service that acts as a centralized hub that can route events raised by any Azure services (or your own custom events) to any number of subscribers. Many Azure services already publish events to Event Grid, so if you want to know when a deployment completes or when a new blob is uploaded to a storage container, you can use Event Grid to be notified. It uses a push model rather than polling like you'd need to with Service Bus, so Event Grid will call you when an interesting event happens (and can retry with back-off if necessary).

Creating an Event Grid Triggered Azure Function

To support Event Grid triggered functions in Azure Functions, we need to reference the Microsoft.Azure.WebJobs.Extensions.EventGrid NuGet package. Simply add this line to your .csproj file.

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid" 
    Version="2.0.0" />

Then creating an Event Grid triggered function is really easy. You just need to apply the EventGridTrigger attribute to an EventGridEvent parameter and it, and it will be called whenever an event fires. In this example, I'm simply logging out interesting information about the events received. In my case I'm particularly interested in events with the Microsoft.ContainerInstance resource provider. Subscriptions can be filtered so you only receive event types you are interested in.

[FunctionName("AciMonitor")]
public static void Run([EventGridTrigger]EventGridEvent eventGridEvent, 
                       ILogger log)
{
    log.LogInformation($"EVENT: {eventGridEvent.EventType}-{eventGridEvent.Subject}-{eventGridEvent.Topic}");
    log.LogInformation(eventGridEvent.Data.ToString());
    
    // some example properties on data:
    // "resourceProvider": "Microsoft.ContainerInstance"
    // "status": "Succeeded"
    // "resourceUri": "/subscriptions/my-sub-id/resourceGroups/DurableFunctionsAciContainers/providers/Microsoft.ContainerInstance/containerGroups/markacitest1",
    dynamic data = eventGridEvent.Data;
    if (data.operationName == "Microsoft.ContainerInstance/containerGroups/delete")
    {
        log.LogInformation($"Deleted container group {data.resourceUri} with status {data.status}");
    }
    else if (data.operationName == "Microsoft.ContainerInstance/containerGroups/write")
    {
        log.LogInformation($"Created or updated container group {data.resourceUri} with status {data.status}");
    }
}

Generate the endpoint address

Now you might be wondering how we connect up an Event Grid subscription to this function. Well, we simply need to create an Event Grid subscription, specifying what we are interested in receiving, and giving it the URL of this function. Where it gets complicated is that the URL needs to include a secret code that is astonishingly hard to automate the retrieval of. Hopefully this is something the Azure Functions team will improve in the future (it does seem like some work is being done in this area)

We need to get several different keys, so we'll use some PowerShell functions I discuss in this article

First a function that can get the Kudu credentials for a Function App

function getKuduCreds($appName, $resourceGroup)
{
    $user = az webapp deployment list-publishing-profiles `
            -n $appName -g $resourceGroup `
            --query "[?publishMethod=='MSDeploy'].userName" -o tsv

    $pass = az webapp deployment list-publishing-profiles `
            -n $appName -g $resourceGroup `
            --query "[?publishMethod=='MSDeploy'].userPWD" -o tsv

    $pair = "$($user):$($pass)"
    $bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
    $encodedCreds = [System.Convert]::ToBase64String($bytes)
    return $encodedCreds
}

Second, we need a function that can get the "master" function key given the Kudu creds

function getMasterFunctionKey([string]$appName, [string]$encodedCreds)
{
    $jwt = Invoke-RestMethod `
            -Uri "https://$appName.scm.azurewebsites.net/api/functions/admin/token" `
            -Headers @{Authorization=("Basic {0}" -f $encodedCreds)} -Method GET

    $keys = Invoke-RestMethod -Method GET `
            -Headers @{Authorization=("Bearer {0}" -f $jwt)} `
            -Uri "https://$appName.azurewebsites.net/admin/host/systemkeys/_master" 

    # n.b. Key Management API documentation currently doesn't explain how to get master key correctly
    # https://github.com/Azure/azure-functions-host/wiki/Key-management-API
    # https://$appName.azurewebsites.net/admin/host/keys/_master = does NOT return master key
    # https://$appName.azurewebsites.net/admin/host/systemkeys/_master = does return master key

    return $keys.value
}

Now let's use these functions to get the master key for our Function App:

$kuduCreds = getKuduCreds $functionAppName $resourceGroup
$masterKey = getMasterFunctionKey $functionAppName $kuduCreds

But we're still not done! Now we need to get yet another key - this one is the Event Grid "extension key". We can use the master key we just retrieved to get the extension key:

$extensionKeyUri = "https://$functionAppName.azurewebsites.net/admin/host/systemkeys/eventgrid_extension?code=$masterKey"
$extensionKey = (Invoke-RestMethod -Method GET -Uri $extensionKeyUri).value

Next, we generate the URL of the webhook that Event Grid will use to trigger our Azure Function. You can read about the format of that URL here, but basically we need the name of our function ("AciMonitor" in our case) and the extension key that we just retrieved:

$functionName = "AciMonitor"
$functionUrl = "https://$hostName/runtime/webhooks/EventGrid?functionName=$functionName" + "&code=$extensionKey" 

This function URL is the one that Event Grid will use to report events, but it also uses it to confirm the subscription. That's needed because Event Grid is a push model - it calls your API, and so it needs to check that the endpoint it is calling really does want to receive events. The Azure Functions Event Grid extension already knows how to respond to the validation handshake so you don't need to do anything additional on the Function App side.

However, there is one more pitfall here, and I think it's to do with using the Azure CLI from PowerShell. And that is that we need to escape the & character in the URL to successfully pass it to the Azure CLI command that creates the Event Grid subscription. We do that like this:

$functionUrlEscaped = $functionUrl.Replace("&", "^^^&")

Create the Event Grid Subscription

Finally, we're ready to actually create our subscription. We need to ensure we've registered the Microsoft.EventGrid provider for this subscription, which we can do with the following command:

az provider register -n "Microsoft.EventGrid" 

Now we just need to get hold of the subscription id and the resource id of the resource group we are monitoring for events, and we can create an Event Grid subscription with az eventgrid event-subscription create method, passing in the (escaped) endpoint URL we just constructed. I've asked for all event types in this example, as I want to explore what events are available, but you can filter it down to just the event types you're interested in if you want

$subscriptionId = az account show --query id -o tsv
$resourceId = "/subscriptions/$subscriptionId/resourcegroups/$aciResourceGroup"
az eventgrid event-subscription create --name "AciEvents" `
    --source-resource-id $resourceId `
    --endpoint-type "WebHook" --included-event-types "All" `
    --endpoint $functionUrlEscaped

Note that for this command to execute successfully our Azure Function App needs to be running so it can respond to the validation handshake.

By the way, the Event Grid subscription does not live in a resource group, so if you want to delete it, then you can use the following command:

az eventgrid event-subscription delete --name "AciEvents" `
    --source-resource-id $resourceId

Summary

In this post we saw how we can create an Event Grid subscription that sends events about a particular resource group to an Azure Function. There were a lot of hoops to jump through to get the URL we needed to subscribe - I'm hoping that becomes much easier to automate in the future. But next up in part 4, let's learn how we can use the fluent Azure C# SDK to create ACI container groups from inside our Azure Function App.

Want to learn more about the Azure CLI? Be sure to check out my Pluralsight course Azure CLI: Getting Started.

0 Comments Posted in:

This is the second part in a series about how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. Today we'll look at creating the necessary infrastructure using the Azure CLI.

Current table of contents:

We'll create a resource group for the Function App containing:

  • A Storage Account for Durable Functions to use as well as a File Share for our containers to use
  • An Application Insights instance for monitoring and diagnostics
  • A Function App (running on the consumption plan) with a system assigned managed identity

We'll also create another resource group that will be used to host our ACI containers. Then we'll grant contributor access to our managed identity so that our Function App is allowed to create resources in that resource group.

Create Storage Account

As usual, I've chosen to automate this with the Azure CLI in PowerShell. First, we create a resource group for our Storage Account and Function App:

$resourceGroup = "DurableFunctionsAci"
$location = "westeurope"
az group create -n $resourceGroup -l $location

In my demo code, I'm using a random number as part of the Storage Account name, but this means my script isn't idempotent unless I first check whether the Storage Account already exists and use the same random number.

$existingName = az storage account list -g $resourceGroup `
  --query "[].name" -o tsv
$prefix = "durablefuncsaci"
if ($existingName) {
    $rand = $existingName.SubString($prefix.Length)
}
else
{ $rand = Get-Random -Minimum 10000 -Maximum 99999 }
$storageAccountName = "$prefix$rand"

Now we know the Storage Account name we want to use, we can create it with az storage account create which is idempotent if the Storage Account already exists. Then we can get the Storage Account key with az storage account keys list which we need for later.

az storage account create `
  -n $storageAccountName `
  -l $location `
  -g $resourceGroup `
  --sku Standard_LRS

$storageAccountKey = az storage account keys list -n $storageAccountName `
  --query [0].value -o tsv

Create a File Share

We want a File Share that we can mount to our ACI containers later. So I'll create one with az storage share create

$shareName = "acishare"
az storage share create `
  -n $shareName `
  --account-key $storageAccountKey `
  --account-name $storageAccountName

Create an Application Insights instance

I've blogged before about creating an Application Insights instance with the Azure CLI, but I've since found a slightly less cumbersome way to do it. Here's how to create the App Insights instance.

$appInsightsName = "$prefix$rand"
az resource create `
  -g $resourceGroup -n $appInsightsName `
  --resource-type "Microsoft.Insights/components" `
  --properties '{\"Application_Type\":\"web\"}'

Create a Function App

We also need a Function App to host our Durable Functions workflow, and connect it to the Storage Account and App Insights instance we just created. The az functionapp create command conveniently has flags that lets us connect everything together and with the --consumption-plan-location flag we can indicate that we want to use the consumption App Service Plan without explicitly needing to create one first.

$functionAppName = "$prefix$rand"
az functionapp create `
    -n $functionAppName `
    --storage-account $storageAccountName `
    --consumption-plan-location $location `
    --app-insights $appInsightsName `
    --runtime dotnet `
    -g $resourceGroup

Give the Function App a Managed Identity

We need to give our Function App a "system assigned" managed identity, and we can use az functionapp identity assign to do that, which again is idempotent, returning the existing identity if one has already been created. I also need to get hold of the principal and tenant ids for use later.

az functionapp identity assign -n $functionAppName -g $resourceGroup

$principalId = az functionapp identity show -n $functionAppName `
  -g $resourceGroup --query principalId -o tsv
$tenantId = az functionapp identity show -n $functionAppName `
  -g $resourceGroup --query tenantId -o tsv

Grant Contributor Rights to the Managed Identity

We want our Function App to have permissions to create new ACI container groups, so we're going to put the managed identity we just created into the "contributor" role for a resource group that will hold all the ACI container groups. Let's create the resource group first:

$aciResourceGroup = "DurableFunctionsAciContainers"
az group create -n $aciResourceGroup -l $location

Now we can assign the contributor role to the managed identity but scope it to only have rights to the resource group we just created.

$subscriptionId = az account show --query "id" -o tsv
az role assignment create --role "Contributor" `
    --assignee-object-id $principalId `
    --scope "/subscriptions/$subscriptionId/resourceGroups/$aciResourceGroup"

Summary

We've created most of the Azure resources we need for this demo. The only missing thing is the Event Grid subscription, which is a bit more complex to set up so we'll look at that in part 3

Want to learn more about the Azure CLI? Be sure to check out my Pluralsight course Azure CLI: Getting Started.

0 Comments Posted in:

In this series I want to show how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. This is something I promised to write about a long time ago but it turned out to be fairly complex to implement. I've finally got enough working to be able to share a basic demo app (special thanks to Noel Bundick for some awesome sample code that got me through some sticky bits).

In this first post I'll explain what the motivation for this is, and some of the technologies we'll be using. Then over the coming days I'll explain in more depth how each part of the demo application works.

Current table of contents:

What are we building?

The demo app we will build (the code is available on GitHub at markheath/durable-functions-aci) shows how we can have a step in an Azure Durable Functions workflow that is implemented as an ACI container. Basically, when the workflow starts, it calls an "activity function" that uses the Azure .NET SDK to create a new ACI "container group" running our container.

Next, our durable workflow needs to wait for that container to finish executing. Unfortunately, we currently have to do that by polling, but I've included code that listens for Event Grid events so in the future we might be able to simplify this part of the code.

Finally, once the container has finished its task, we want to delete the ACI container group. I don't think that ACI container groups are charged while they are stopped, but I'm not 100% sure, so I'd rather be on the safe side!

One of my ideas is that maybe in the future, the code in this application could be converted into a generic extension for Durable Functions to greatly simplify the work involved in implementing workflow steps as container instances.

Why do we need it?

Azure Durable Functions makes it really easy to create serverless workflows, but sometimes the steps in the workflow cannot be straightforwardly implemented as "activity functions" in the Azure Function App itself. This might be because it is a long-running process (Azure Functions are limited to 5 minutes execution time by default), or because it requires custom software to be installed that cannot easily run on Azure App Service, or because we want to mount an Azure File Share (which Azure Functions does not currently support).

By using Azure Container Instances to implement these long-running custom activities, we can still get the serverless benefits of paying only for the compute we need (i.e. avoiding having Virtual Machines on standby waiting to implement these tasks), and benefit from automatic scaleout - we can simply create as many ACI instances as we want (within the constraints of the ACI service) to manage demand. But we also get additional benefits of containers - running whatever code we like in a sandboxed environment, and the ability to specify exactly how much memory and CPU we require for the task at hand.

The example scenario that motivated this is my need to perform custom media transcoding on demand. In my job I often need to transcode and process video files in a variety of obscure CCTV formats, which require custom software that can't be installed onto App Service. Some of the media files are extremely large (multiple GB) and so the transcoding process can take several hours to complete (9 hours is the longest one so far). Also, there can be sudden influxes of very large amounts of media, which all need to be transcoded as quickly as possible, so I need rapid scaleout. The ability to mount Azure File Shares is also useful as often the same file goes through multiple processing stages, each of which might be performed by a different container.

What will we use?

We're going to be integrating several of my favourite Azure technologies in this demo, and quite a few of them I've created Pluralsight courses about, so here's the bits and pieces we'll be using including links to my courses if you're interested in diving in a bit deeper.

Read part 2 here

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.