0 Comments Posted in:

This is the second part in a series about how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. Today we'll look at creating the necessary infrastructure using the Azure CLI.

Current table of contents:

We'll create a resource group for the Function App containing:

  • A Storage Account for Durable Functions to use as well as a File Share for our containers to use
  • An Application Insights instance for monitoring and diagnostics
  • A Function App (running on the consumption plan) with a system assigned managed identity

We'll also create another resource group that will be used to host our ACI containers. Then we'll grant contributor access to our managed identity so that our Function App is allowed to create resources in that resource group.

Create Storage Account

As usual, I've chosen to automate this with the Azure CLI in PowerShell. First, we create a resource group for our Storage Account and Function App:

$resourceGroup = "DurableFunctionsAci"
$location = "westeurope"
az group create -n $resourceGroup -l $location

In my demo code, I'm using a random number as part of the Storage Account name, but this means my script isn't idempotent unless I first check whether the Storage Account already exists and use the same random number.

$existingName = az storage account list -g $resourceGroup `
  --query "[].name" -o tsv
$prefix = "durablefuncsaci"
if ($existingName) {
    $rand = $existingName.SubString($prefix.Length)
{ $rand = Get-Random -Minimum 10000 -Maximum 99999 }
$storageAccountName = "$prefix$rand"

Now we know the Storage Account name we want to use, we can create it with az storage account create which is idempotent if the Storage Account already exists. Then we can get the Storage Account key with az storage account keys list which we need for later.

az storage account create `
  -n $storageAccountName `
  -l $location `
  -g $resourceGroup `
  --sku Standard_LRS

$storageAccountKey = az storage account keys list -n $storageAccountName `
  --query [0].value -o tsv

Create a File Share

We want a File Share that we can mount to our ACI containers later. So I'll create one with az storage share create

$shareName = "acishare"
az storage share create `
  -n $shareName `
  --account-key $storageAccountKey `
  --account-name $storageAccountName

Create an Application Insights instance

I've blogged before about creating an Application Insights instance with the Azure CLI, but I've since found a slightly less cumbersome way to do it. Here's how to create the App Insights instance.

$appInsightsName = "$prefix$rand"
az resource create `
  -g $resourceGroup -n $appInsightsName `
  --resource-type "Microsoft.Insights/components" `
  --properties '{\"Application_Type\":\"web\"}'

Create a Function App

We also need a Function App to host our Durable Functions workflow, and connect it to the Storage Account and App Insights instance we just created. The az functionapp create command conveniently has flags that lets us connect everything together and with the --consumption-plan-location flag we can indicate that we want to use the consumption App Service Plan without explicitly needing to create one first.

$functionAppName = "$prefix$rand"
az functionapp create `
    -n $functionAppName `
    --storage-account $storageAccountName `
    --consumption-plan-location $location `
    --app-insights $appInsightsName `
    --runtime dotnet `
    -g $resourceGroup

Give the Function App a Managed Identity

We need to give our Function App a "system assigned" managed identity, and we can use az functionapp identity assign to do that, which again is idempotent, returning the existing identity if one has already been created. I also need to get hold of the principal and tenant ids for use later.

az functionapp identity assign -n $functionAppName -g $resourceGroup

$principalId = az functionapp identity show -n $functionAppName `
  -g $resourceGroup --query principalId -o tsv
$tenantId = az functionapp identity show -n $functionAppName `
  -g $resourceGroup --query tenantId -o tsv

Grant Contributor Rights to the Managed Identity

We want our Function App to have permissions to create new ACI container groups, so we're going to put the managed identity we just created into the "contributor" role for a resource group that will hold all the ACI container groups. Let's create the resource group first:

$aciResourceGroup = "DurableFunctionsAciContainers"
az group create -n $aciResourceGroup -l $location

Now we can assign the contributor role to the managed identity but scope it to only have rights to the resource group we just created.

$subscriptionId = az account show --query "id" -o tsv
az role assignment create --role "Contributor" `
    --assignee-object-id $principalId `
    --scope "/subscriptions/$subscriptionId/resourceGroups/$aciResourceGroup"


We've created most of the Azure resources we need for this demo. The only missing thing is the Event Grid subscription, which is a bit more complex to set up so we'll look at that in part 3

Want to learn more about the Azure CLI? Be sure to check out my Pluralsight course Azure CLI: Getting Started.

0 Comments Posted in:

In this series I want to show how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. This is something I promised to write about a long time ago but it turned out to be fairly complex to implement. I've finally got enough working to be able to share a basic demo app (special thanks to Noel Bundick for some awesome sample code that got me through some sticky bits).

In this first post I'll explain what the motivation for this is, and some of the technologies we'll be using. Then over the coming days I'll explain in more depth how each part of the demo application works.

Current table of contents:

What are we building?

The demo app we will build (the code is available on GitHub at markheath/durable-functions-aci) shows how we can have a step in an Azure Durable Functions workflow that is implemented as an ACI container. Basically, when the workflow starts, it calls an "activity function" that uses the Azure .NET SDK to create a new ACI "container group" running our container.

Next, our durable workflow needs to wait for that container to finish executing. Unfortunately, we currently have to do that by polling, but I've included code that listens for Event Grid events so in the future we might be able to simplify this part of the code.

Finally, once the container has finished its task, we want to delete the ACI container group. I don't think that ACI container groups are charged while they are stopped, but I'm not 100% sure, so I'd rather be on the safe side!

One of my ideas is that maybe in the future, the code in this application could be converted into a generic extension for Durable Functions to greatly simplify the work involved in implementing workflow steps as container instances.

Why do we need it?

Azure Durable Functions makes it really easy to create serverless workflows, but sometimes the steps in the workflow cannot be straightforwardly implemented as "activity functions" in the Azure Function App itself. This might be because it is a long-running process (Azure Functions are limited to 5 minutes execution time by default), or because it requires custom software to be installed that cannot easily run on Azure App Service, or because we want to mount an Azure File Share (which Azure Functions does not currently support).

By using Azure Container Instances to implement these long-running custom activities, we can still get the serverless benefits of paying only for the compute we need (i.e. avoiding having Virtual Machines on standby waiting to implement these tasks), and benefit from automatic scaleout - we can simply create as many ACI instances as we want (within the constraints of the ACI service) to manage demand. But we also get additional benefits of containers - running whatever code we like in a sandboxed environment, and the ability to specify exactly how much memory and CPU we require for the task at hand.

The example scenario that motivated this is my need to perform custom media transcoding on demand. In my job I often need to transcode and process video files in a variety of obscure CCTV formats, which require custom software that can't be installed onto App Service. Some of the media files are extremely large (multiple GB) and so the transcoding process can take several hours to complete (9 hours is the longest one so far). Also, there can be sudden influxes of very large amounts of media, which all need to be transcoded as quickly as possible, so I need rapid scaleout. The ability to mount Azure File Shares is also useful as often the same file goes through multiple processing stages, each of which might be performed by a different container.

What will we use?

We're going to be integrating several of my favourite Azure technologies in this demo, and quite a few of them I've created Pluralsight courses about, so here's the bits and pieces we'll be using including links to my courses if you're interested in diving in a bit deeper.

Read part 2 here

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

0 Comments Posted in:

In C# we work with collections of data all the time, and .NET provides several types for storing collections, such as Lists, Arrays, Dictionaries, and HashSets.

But one question that often comes up during code review, is "what types should we use to pass collections around in code"? When methods take collections as parameters, or return them, should we use concrete types like List<T>, or maybe everything should be IEmumerable<T>? And what about DTOs that get serialized into JSON or XML and back out again? How should we declare properties that contain collections on DTOs?

In this post we'll look at three scenarios: (1) passing collections into methods, (2) returning collections from methods (3) collection properties on DTOs.

Passing collections into methods

If your method needs to accept a collection as a parameter, then generally IEnumerable<T> is the best choice. This offers the most flexibility to the caller. It doesn't matter what concrete collection type they use, and it even allows them to pass lazily evaluated sequences in.

For example, in the following code snippet, ProcessOrders2 is a better choice as it allows the caller to pass in a lazily evaluated IEnumerable generated by the LINQ Where method.

void ProcessOrders1(List<Order> orders)
    // ...
void ProcessOrders2(IEnumerable<Order> orders)
    // ...

void Caller() 
    ProcessOrders1(myOrders.Where(o => !o.Shipped)); // won't compile - requires us to add a ToList
    ProcessOrders2(myOrders.Where(o => !o.Shipped)); // works

When might you use something other than IEnumerable<T>? Sometimes I've seen code where a List<T> is passed to a method because that method is going to add elements into the list. Generally I prefer to take a more functional programming approach where methods do not modify parameters they are passed. Methods that modify parameters makes it harder to reason about your code and test it, and introduces potential thread safety issues especially when you're dealing with async methods.

So avoid writing methods that look like this:

// avoid modifying collections passed into methods
Task AddCustomer(Guid id, List<Customers> customers) {
    var customer = await customers.GetById(id);

Another common scenario is if your method needs to iterate through the collection more than once, or access the count of items in the collection it's been passed. If the parameter type is IEnumerable<T> in this scenario, we can't know that it is safe to enumerate more than once - it could be an expensive operation that goes to a database, or it could even return different results the second time through. Here's a slightly contrived example of a method that enumerates the orders parameter up to three times:

// avoid: multiple enumeration
void ProcessOrders(IEnumerable<Order> orders)
    Console.WriteLine($"Processing {orders.Count()} orders");
    var allValid = true;
    foreach(var order in orders)
        if (!IsValid(order))
            Console.WriteLine($"Order {orders.Id} is invalid");
            allValid = false;
    if (allValid)
        foreach(var order in orders)

One way we could solve this is if ProcessOrders method simply performed a ToList on orders to get it in memory. That would allow it to enumerate the list multiple times. This approach is nicest from the caller's perspective: they can still provide an IEnumerable<T> if they want.

But suppose ProcessOrders and all its callers are under my control and I know I don't need the flexibility of passing an IEnumerable<T>. In that case I might simply choose to declare parameter type as an IReadOnlyCollection<T> instead. IReadOnlyCollection<T> allows us to be sure that all items are already available in memory so we can safely enumerate multiple times, and also exposes a Count property. So it's worth considering instead of IEnumerable<T> if you find you're adding unnecessary calls to .ToList on objects that are probably already lists in the first place.

In summary, my recommendations for passing collections into methods are:

  • Use IEnumerable<T> where possible
  • Avoid passing concrete collection types
  • Avoid modifying collections passed as parameters
  • Consider IReadOnlyCollection<T> if you need to enumerate multiple times and your callers are easily able to provide an in-memory list

Returning collections from methods

Because IEnumerable<T> is arguably the best type to use to pass a collection to a method, many developers assume that it is also the best type to return a collection from a method. So we might declare a method like this:

IEnumerable<Order> GetOrders(DateTime orderDate)

It's not a bad choice, but it does mean that the caller cannot make any assumptions about the collection they have been given. For example, they don't know if enumerating the return value will be an in-memory operation or a potentially expensive action. It could even throw an exception as they enumerate. They also don't know if they can safely enumerate it more than once. So often the caller ends up doing a .ToList or similar on the collection you passed, which is wasteful if it was already a List<T> already.

You end up seeing a lot of code like this:

var orders = GetOrders(orderDate).ToList(); // we want to multiply enumerate orders, so convert to List

This is actually another case in which IReadOnlyCollection<T> can be a good fit if you know your method is always going to return an in-memory collection. It gives your caller the ability to access the count of items and iterate through as many times as they like, but doesn't allow them to modify that collection, which might be important if you have a method that returns a cached list like this:

private List<string> validFileExtensions = ...;
// ensure that callers aren't able to modify this list
public IReadOnlyCollection<string> GetValidFileExtensions()
    return validFileExtensions;

So in summary, when a method returns a collection:

  • Consider IReadOnlyCollection<T> if you always return an in memory list, and your callers would benefit from having the whole collection already in memory.
  • Use IEnumerable<T> where you want to offer callers the ability to partially enumerate through, and potentially generate items in the sequence on the fly.

Collections as properties on DTOs

What about when we declare data transfer objects (DTOs) that are going to be serialized to JSON perhaps as part of a web API request or a queue message? I've seen lots of approaches here. Here's some common examples:

class Example1
    public IEnumerable<string> People { get; set; }

class Example2
    public IReadOnlyCollection<string> People { get; set; }

class Example3
    public ICollection<string> People { get; set; }

class Example4
    public IList<string> People { get; set; }

In the above code samples, Newtonsoft.Json has no problem serializing and deserializing them. It actually deserializes them all to a List<T> except for Example2 where it creates a ReadOnlyCollection. Example1 and Example2 require us to set the entire list ready populated when we create an instance in code, while Example3 and Example4 let us add or remove elements from the People collection after creating the DTO.

Personally, I would avoid IEnumerable<T> (Example1) as it seems unnecessarily restrictive given that we know all the items are available in memory for this type of object. I would prefer IReadOnlyCollection<T> here (Example2), allowing callers to access the Count of items easily.

One nice thing about Newtonsoft.Json is that it can successfully deserialize instances where we don't even put a public setter on the collection property, like this:

class Example5
    public Example5(IEnumerable<string> people)
        People = people.ToList();
    public IReadOnlyCollection<string> People { get; }

Personally I don't tend to bother with that as it's cumbersome to write on DTOs with many properties. Hopefully if C# 8 record types become a thing, it will be easy to declare an immutable DTO type that supports deserialization.


There are a huge variety of collection types in .NET and it's not always obvious what the most appropriate one to use when passing them around in method calls and DTOs. IEnumerable<T> is a good fit for many scenarios, but do consider that IReadOnlyCollection<T> might be a better fit in circumstances where the collection is always going to be fully available in memory. Avoid passing round mutable collection types as this can cause confusion about who owns the collection.

Of course, there is a lot more that could be said about this. I've not touched at all on the newer immutable collections, which are great for a more functional approach to collections, or on Span<T> which is a great fit for high performance mutable collections of primitives. Do feel free to let me know in the comments what approach you take.