0 Comments Posted in:

I wrote recently about why you should use Azure Durable Functions to implement your serverless workflows rather than just manually chaining together a bunch of functions with queues. There was great news recently that Durable Functions is now in "release candidate", and in this post I want to explore in a bit more detail how it can greatly improve your error handling within workflows.

Unhandled Exceptions

First of all, a quick reminder about how Durable Functions works. You create an "orchestrator function", which defines your workflow. And then create multiple "activity functions", one for each step in your workflow. The orchestrator can call these activities either in sequence or parallel.

In an unhandled exception is thrown by an activity function, it will propagate up to the orchestrator function. This is brilliant as it means the orchestrator can make intelligent decisions on what should happen to the workflow based on an activity failing. This might involve triggering a cleaning up activity, or retrying, or maybe the workflow can carry on regardless.

Of course if the orchestrator function doesn't catch these exceptions itself, then the orchestration will terminate. However, even in this case, we'll get some useful information from the Durable Functions runtime. If we query an orchestration that has failed using the Durable Functions REST API we'll see a runtimeStatus of Failed and in the output we'll get information about which activity function the exception occurred in, and the error message.

So in this example, my Activity2 activity function threw an unhandled exception that was also unhandled by the orchestrator function, resulting in the orchestration ending. Here's the output from the Durable Functions REST API showing the orchestration status:

{
    runtimeStatus: "Failed",
    input: "hello",
    output: "Orchestrator function 'ExceptionHandlingExample' failed: The activity function 'Activity2' failed: \"Failure in Activity 2\". See the function execution logs for additional details.",
    createdTime: "2018-04-30T11:48:28Z",
    lastUpdatedTime: "2018-04-30T11:48:31Z"
}

Catching Exceptions in Activity Functions

Of course, you don't need to let exceptions propagate from activity functions all the way through to the orchestrator. In some cases it might make sense to catch your exceptions in the activity function.

One example is if the activity function needs to perform some cleanup of its own in the case of failure - perhaps deleting a file from blob storage. But it might also be to simply send some more useful information back to the orchestrator so it can decide what to do next.

Here's an example activity function that returns an anonymous object with a Success flag plus some additional information depending on whether the function succeeded or not. Obviously you could return a strongly typed custom DTO instead. The orchestrator function can check the Success flag and use it to make a decision on whether the workflow can continue or not.

[FunctionName("Activity2")]
public static async Task<object> Activity2(
    [ActivityTrigger] string input,
    TraceWriter log)
{
    try
    {
        var myOutputData = await DoSomething(input);
        return new 
        {
            Success = true,
            Result = myOutputData
        };
    }
    catch (Exception e)
    {
        // optionally do some cleanup work ...
        DoCleanup();
        return new 
        {
            Success = false,
            ErrorMessage = e.Message
        };
    }
}

Catching Exceptions in Orchestrator Functions

The great thing about orchestrator functions being able to handle exceptions thrown from activity functions is that it allows you to centralize the error handling for the workflow as a whole. In the catch block you can call a cleanup activity function, and then either re-throw the exception to fail the orchestration, or you might prefer to let the orchestration complete "successfully", and just report the problem via some other mechanism.

Here's an example orchestrator function that has one cleanup activity it runs whichever of the three activity functions the problem was found in.

[FunctionName("ExceptionHandlingOrchestrator")]
public static async Task<string> ExceptionHandlingOrchestrator(
    [OrchestrationTrigger] DurableOrchestrationContext ctx,
    TraceWriter log)
{
    var inputData = ctx.GetInput<string>();
    try
    {
        var a1 = await ctx.CallActivityAsync<string>("Activity1", inputData);
        var a2 = await ctx.CallActivityAsync<ActivityResult>("Activity2", a1);
        var a3 = await ctx.CallActivityAsync<string>("Activity3", a2);
        return a3;
    }
    catch (Exception)
    {
        await ctx.CallActivityAsync<string>("CleanupActivity", inputData);
        // optionally rethrow the exception to fail the orchestration
        throw;
    }
}

Retrying Activities

Another brilliant thing about using Durable Functions for your workflows is that it includes support for retries. Again, at first glance that might not seem like something that's too difficult to implement with regular Azure Functions. You could just write a retry loop in your function code.

But what if you want to delay between retries? That's much more of a pain, as you pay for the total duration your Azure Functions run for, so you don't want to waste time sleeping. And Azure Functions in the consumption plan are limited to 5 minutes execution time anyway. So you end up needing to send yourself a future scheduled message. That's something I have implemented in Azure Function in the past (see my randomly scheduled tweets example), but its a bit cumbersome.

Thankfully, with Azure Functions, we can simply specify when we call an activity (or a sub-orchestration) that we want to retry a certain number of times, and customise the back-off strategy, thanks to the CallActivityWithRetryAsync method and the RetryOptions class.

In this simple example, we'll retry Activity1 up to a maximum of 4 attempts with a five second delay before retrying.

var a1 = await ctx.CallActivityWithRetryAsync<string>("Activity1", 
               new RetryOptions(TimeSpan.FromSeconds(5),4), inputData);

Even better, we can intelligently decide which exceptions we want to retry. This is important as in cloud deployed applications some exceptions will be due to "transient" problems that might be resolved by simply retrying, but others are not worth retrying.

When an activity function throws an exception, it will appear in the orchestrator as a FunctionFailedException, but the inner exception will contain the exception thrown from the activity function. However, currently the type of that inner exception seems to be just System.Exception rather than the actual type (e.g. InvalidOperationException) that was thrown, so if you're making retry decisions based on this exception, you might have to just use its Message, although the actual exception type can seen if you call ToString.

Here's a very simple example of only retrying if the inner exception message exactly matches a specific string:

var a1 = await ctx.CallActivityWithRetryAsync<string>("Activity1", 
    new RetryOptions(TimeSpan.FromSeconds(5),4)
    {
        Handle = ex => ex.InnerException.Message == "oops"
    }, 
    inputData);

Summary

Durable Functions not only makes it much easier to define your workflows, but to handle the errors that occur within them. Whether you want to respond to exceptions by retrying with backoff, or by performing a cleanup operation, or even by continuing regardless, Durable Functions makes it much easier to implement than trying to do the same thing with regular Azure Functions chained together by queue messages.

Want to learn more about how easy it is to get up and running with Durable Functions? Be sure to check out my Pluralsight course Azure Durable Functions Fundamentals.

0 Comments Posted in:

I recently wanted to try out the preview of .NET Core 2.1, so that I could experiment a bit with the new Span<T> functionality, which is something I'm very excited about.

Which SDK do I need?

But when I visited the .NET homepage (found at the easy to remember https://dot.net), and navigated to the all downloads page, I must confess to being a little confused by the experience...

image

That's right, there's two 2.1 SDK's available. The one called 2.1.300-preview1 is the one I wanted - that's the SDK for the .NET Core 2.1 preview runtime. Confusingly the 2.1.103 SDK actually contains the 2.0 runtime (not 2.1)! There's a new versioning strategy going forwards that will make the relationship between runtime and SDK easier to understand, but before that, the runtime and SDK version numbers did not necessarily correspond.

Will bad things happen if I install a preview SDK?

So I've worked out that I need 2.1.300-preview1, but like many developers, I've been burned in the past by installing bad preview bits that mess up my development machine. So I had two key questions before I was ready to pull the trigger. First, will installing a preview .NET Core SDK break something?, and second, can I easily switch back to the release version of the SDK?

There's good news on both fronts. First of all, .NET Core SDKs are installed side by side, so will not modify any previous SDK installations. So when I install the preview, all that happens is that I get a new folder under C:\Program Files\dotnet\sdk. You can see here all the various .NET Core SDK versions I've got installed on this PC:

image

And the same is true for the .NET Core runtime. Inside the C:\Program Files\dotnet\shared\Microsoft.NETCore.App folder we can see all the versions of the runtime I've installed, including the 2.1.0 preview that came with the SDK:

image

So that's great, I've got the 2.1 preview SDK and runtime available on my machine, and I can experiment with it. But now we need to know how to identify which one of these SDKs is currently active, and how to switch between them.

What version of the SDK am I currently running?

If I go to a command line and type dotnet --version it will tell me the SDK version I'm currently running, which will be the preview version I just installed:

image

The way this works is that I'm actually calling dotnet.exe which resides in C:\Program Files\dotnet. That is just a proxy which passes on my command to whatever the latest SDK is, which by default will just be the folder in C:\Program Files\dotnet\sdk with the highest version number.

What SDK and runtime versions are installed?

Two other handy commands to know are dotnet --list-sdks to see all installed SDKs:

image

and dotnet --list-runtimes to see all installed runtimes:

image

How can I switch between .NET Core SDK versions?

Since my current version is 2.1.300-preview1-008174, if I run dotnet new console it will create me a .csproj with that targets the .NET Core 2.1 runtime (which is of course still in preview at the time of writing):

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>
</Project>

That's great if I want to experiment with the .NET Core 2.1 preview, but what if I just want to get back to the version I was using before, and build .NET Core 2.0 apps? In other words, when I type the dotnet command in the console, how can I control which version of the SDK I am actually using?

The answer is that we can easily switch between versions of the SDK by creating a global.json file. It simply needs to reside in the folder you're working in (or any parent folder). So if there is a global.json file in my current folder or any parent folder with the following contents, I'll be back to using the 2.1.103 version of the SDK (which if you remember targets the 2.0 runtime).

{
  "sdk": {
    "version": "2.1.103"
  }
}

Having created this global.json file, running dotnet --version now returns 2.1.103:

image

And if I do a dotnet new console we'll see that the generated .csproj targets the 2.0 runtime, as we'd expect:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>
</Project>

Easily create a global.json file

One final tip - you can easily create a global.json file in the current working folder with the dotnet new globaljson command. This will create a global.json file and pre-populate it with the latest version of the SDK available. You can also control the version that gets written to the file with the --sdk-version argument. For example, dotnet new globaljson --sdk-version 2.0.2 generates the following global.json:

{
  "sdk": {
    "version": "2.0.2"
  }
}

Summary

It's always daunting to install preview bits on your developer machine as you don't know what might break, but with .NET Core, SDKs and runtimes are installed side by side, and so long as you know about global.json it's very easy to control which version of the SDK you're using. And the version of the runtime can also be controlled using the TargetFramework (and, if necessary the RuntimeFrameworkVersion) properties in your .csproj file.


0 Comments Posted in:

Azure Container Instances combine the benefits of serverless and Docker containers, and I wrote recently about one compelling use case - using Azure Container Instances to implement Media Processing tasks.

Today I want to look at how we could use Azure Container Instances as a cost-effective option for CI builds. We'll actually be combining a whole host of cool technologies:

Building ASP.NET Core with Cake and deploying to Azure with Run From Zip

For the demo application that we'll be using, I've created a very simple ASP.NET Core website, and created a Cake build script for it. If you've not tried Cake yet - I can highly recommend it - it's a great build automation tool that is a great fit for C# projects as it uses C# as the basis for its DSL.

Once it's built, we are going to push the site live to Azure using a brand new deployment technique called Run-from-Zip. This was made easy thanks to a great article from Mattias Karlsson showing how to use "Run-From-Zip" with the Cake Kudu Client.

Here's my Cake file, with the test step removed for brevity as my demo app doesn't have any unit tests, and the main focus here is on building and deploying to Azure.

As you can see, I need the Cake.Kudu.Client addin. This does the zipping for us, so all we need to do is ensure that the three Kudu environment variables are set up with the site address and the deployment credentials.

#addin nuget:?package=Cake.Kudu.Client&version=0.5.0

// Target - The task you want to start. Runs the Default task if not specified.
var target = Argument("Target", "Default");  
var configuration = Argument("Configuration", "Release");

Information($"Running target {target} in configuration {configuration}");

var distDirectory = Directory("./dist");

// Deletes the contents of the Artifacts folder if it contains anything from a previous build.
Task("Clean")  
    .Does(() =>
    {
        CleanDirectory(distDirectory);
    });

// Run dotnet restore to restore all package references.
Task("Restore")  
    .Does(() =>
    {
        DotNetCoreRestore();
    });

// Build using the build configuration specified as an argument.
 Task("Build")
    .Does(() =>
    {
        DotNetCoreBuild(".",
            new DotNetCoreBuildSettings()
            {
                Configuration = configuration,
                ArgumentCustomization = args => args.Append("--no-restore"),
            });
    });

// Publish the app to the /dist folder
Task("PublishWeb")  
    .Does(() =>
    {
        DotNetCorePublish(
            "./coreapp.csproj",
            new DotNetCorePublishSettings()
            {
                Configuration = configuration,
                OutputDirectory = distDirectory,
                ArgumentCustomization = args => args.Append("--no-restore"),
            });
    });

Task("DeployToAzure")
    .Description("Deploy to Azure ")
    .Does(() =>
    {
        // https://hackernoon.com/run-from-zip-with-cake-kudu-client-5c063cd72b37
        string baseUri  = EnvironmentVariable("KUDU_CLIENT_BASEURI"),
               userName = EnvironmentVariable("KUDU_CLIENT_USERNAME"),
               password = EnvironmentVariable("KUDU_CLIENT_PASSWORD");
        IKuduClient kuduClient = KuduClient(
            baseUri,
            userName,
            password);
        var skipPostDeploymentValidation = true; // .NET core apps don't report their version number
        FilePath deployFilePath = kuduClient.ZipRunFromDirectory(distDirectory, skipPostDeploymentValidation);
        Information("Deployed to {0}", deployFilePath);
    });

// A meta-task that runs all the steps to Build and Test the app
Task("BuildAndTest")  
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build");

// The default task to run if none is explicitly specified. In this case, we want
// to run everything starting from Clean, all the way up to Publish.
Task("Default")  
    .IsDependentOn("BuildAndTest")
    .IsDependentOn("PublishWeb")
    .IsDependentOn("DeployToAzure");

// Executes the task specified in the target argument.
RunTarget(target);

The container image - Cake builder

The next piece of the puzzle was to create a Docker image that would be able to run my Cake script. Azure Container Instances does support Windows containers, but currently many of the surrounding features are not supported (such as mounting volumes), and since .NET Core and Cake are cross-platform anyway, I decided to make my Cake builder Docker image a Linux container.

The dockerfile is quite straightforward with the exception of the fact that we need Mono installed to run Cake. A very helpful article from Andrew Lock pointed me in the right direction for how to build ASP.NET Core Apps using Cake in Docker

Notice that my Docker file does not copy the source in. It simply assumes that a Cake build.sh script already exists in the /src folder (which is what our gitRepo volume mount will do). The --settings_skipverification=true build argument helped me work around a versioning issue, but probably isn't needed any more.

FROM microsoft/aspnetcore-build:2.0 AS build

# Install mono for Cake (https://andrewlock.net/building-asp-net-core-apps-using-cake-in-docker/)
ENV MONO_VERSION 5.4.1.6

RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF

RUN echo "deb http://download.mono-project.com/repo/debian stretch/snapshots/$MONO_VERSION main" > /etc/apt/sources.list.d/mono-official.list \  
  && apt-get update \
  && apt-get install -y mono-runtime \
  && rm -rf /var/lib/apt/lists/* /tmp/*

RUN apt-get update \  
  && apt-get install -y binutils curl mono-devel ca-certificates-mono fsharp mono-vbnc nuget referenceassemblies-pcl \
  && rm -rf /var/lib/apt/lists/* /tmp/*

WORKDIR /src
CMD ./build.sh -Target=Default --settings_skipverification=true

Using Azure Container Instances "gitRepo" Volume Share

One of the volume types you can mount with Azure Container Instances is a "gitRepo" volume. This basically clones a git repository into a folder of your choosing.

You simply give it the URL of the Git repository and optionally specify the SHA of the commit you want to clone. It doesn't appear that there is any provision to provide credentials to a private repository, so hopefully that's something that will get added in the future.

Another limitation is that the Azure CLI doesn't currently support mounting "gitRepo" volumes yet. That's unfortunate, as if it did we could kick off a build with one simple command like this (Don't try this - the --gitrepo-repository and --git-repo-mount-path arguments are made up as examples of what I hope is coming to the Azure CLI in the future):

# IMPORTANT! This doesn't currenly work - there aren't actually any --gitrepo-* arguments 
# for mounting gitRepo volumes yet
az container create `
    -g $resourceGroup `
    -n $containerGroupName `
    --image markheath/cakebuilder `
    --gitrepo-repository https://github.com/markheath/aspnet-core-cake `
    --gitrepo-mount-path "/src" `
    -e KUDU_CLIENT_BASEURI=https://$appName.scm.azurewebsites.net KUDU_CLIENT_USERNAME=$user KUDU_CLIENT_PASSWORD=$pass `
    --restart-policy never `
    --command-line "./build.sh -Target=Default --settings_skipverification=true"

So how can we deploy this? Well, to get at the gitRepo volume mounting capabilities we need to use ARM templates. In my ARM template you can see I've parameterized the environment variables allowing us to pass in the Kudu deployment URI and credentials.

You can also see I've hard-coded the repository we're mounting as https://github.com/markheath/aspnet-core-cake, but obviously that could be parameterized as well.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "containerGroupName": {
            "type": "string",
            "defaultValue": "myContainerGroup",
            "metadata": {
                "description": "Name for the container group"
            }
        },
        "KUDU_CLIENT_BASEURI": {
            "type": "string",
            "metadata": {
                "description": "Base URI for Kudu deployment"
            }
        },
        "KUDU_CLIENT_USERNAME": {
            "type": "string",
            "metadata": {
                "description": "Username for Kudu deployment"
            }
        },
        "KUDU_CLIENT_PASSWORD": {
            "type": "securestring",
            "metadata": {
                "description": "Password for Kudu deployment."
            }
        },
        "commandLine": {
            "type": "string",
            "defaultValue": "chmod 755 ./build.sh && ./build.sh -Target=Default --settings_skipverification=true",
            "metadata": {
                "description": "Command line to run on container start."
            }

        }
    },
    "variables": {
      "container1name": "cakebuilder",
      "container1image": "markheath/cakebuilder:0.1"
    },
    "resources": [
      {
        "name": "[parameters('containerGroupName')]",
        "type": "Microsoft.ContainerInstance/containerGroups",
        "apiVersion": "2018-02-01-preview",
        "location": "[resourceGroup().location]",
        "properties": {
          "containers": [
            {
              "name": "[variables('container1name')]",
              "properties": {
                "image": "[variables('container1image')]",
                "command": [
                    "/bin/bash",
                    "-c",
                    "[parameters('commandLine')]"
                ],
                "resources": {
                  "requests": {
                    "cpu": 1,
                    "memoryInGb": 2
                  }
                },
                "volumeMounts": [
                  {
                    "name": "gitrepo1",
                    "mountPath": "/src"
                  }
                ],
                "environmentVariables": [
                    {
                        "name": "KUDU_CLIENT_BASEURI",
                        "value": "[parameters('KUDU_CLIENT_BASEURI')]"
                    },
                    {
                        "name": "KUDU_CLIENT_USERNAME",
                        "value": "[parameters('KUDU_CLIENT_USERNAME')]"
                    },
                    {
                        "name": "KUDU_CLIENT_PASSWORD",
                        "value": "[parameters('KUDU_CLIENT_PASSWORD')]"
                    }
                ]
              }
            }
          ],
          "osType": "Linux",
          "restartPolicy": "Never",
          "volumes": [
            {
              "name": "gitrepo1",
              "gitRepo": {
                "repository": "https://github.com/markheath/aspnet-core-cake",
                "directory": "."
              }
            }
          ]
        }
      }
    ]
}

You might also notice that the command we run isn't quite as simple as ./build.sh. I'd been hoping I could do that as the dockerfile sets the working directory to /src which is where the gitRepo is mounted. However, there was a permissions issue with running the script from the gitRepo mounted volume directly, requiring me to jump through a few hoops.

I changed the build command line to chmod 755 ./build.sh && ./build.sh -Target=Default --settings_skipverification=true to make the build script executable before running it. And the container startup script itself calls /bin/bash -c passing in the build command as an argument.

I've also chosen to set the restartPolicy to never. I don't want to get into an infinite loop of continually trying to rebuild my app if there is a build failure. Instead I'd rather check why it has failed and kick off another build manually.

Launching the container and checking progress

To actually deploy our ARM template, we can use the Azure CLI. I'm assuming that we've already created an Azure App Service web app and stored its name and deployment credentials in the $appName, $user and $pass variables. You can see how I do that in my full PowerShell script that creates the app service plan, web app, enables Kudu run-from-zip and then launches the CI Cake builder container instance to build and deploy.

$containerGroupName = "cakebuilder"
az group deployment create `
    -n TestDeployment -g $resourceGroup `
    --template-file "cake-builder.json" `
    --parameters "KUDU_CLIENT_BASEURI=https://$appName.scm.azurewebsites.net" `
    --parameters "KUDU_CLIENT_USERNAME=$user" `
    --parameters "KUDU_CLIENT_PASSWORD=$pass" `
    --parameters "containerGroupName=$containerGroupName"

Once this is running, we can use the following two Azure CLI commands to check up on progress.

az container show -n $containerGroupName -g $resourceGroup
az container logs -n $containerGroupName -g $resourceGroup

az container show will tell us whether the container has started up yet, and if it has finished. Remember that it needs to download our Cake builder Docker image and clone the GitHub repository before our container will start running, so there will be a small delay.

And the az container logs command will let us see the actual output from our Cake build script. Obviously its possible that the container has started successfully but the build or deploy has failed, and so its these logs that will help us discover what exactly has gone wrong if we get a build failure.

Are Azure Container Instances cheaper than VMs?

Obviously, this all raises the question of why bother? Why not just have a Virtual Machine as a Docker host running your builds in containers? Can we save any money with this approach?

Azure Container Instances uses a consumption based pricing model where you only pay for exactly what you use. There's a very small charge for each container instance that's created ($.0025), and then for every second your container instance is running you pay $0.000012 per GB of RAM plus another $0.000012 for every CPU core.

Let's imagine you have a CI server you use for a project you're working on in a small team. Let's say each build takes 10 minutes, and in a typical day there are 15 builds. If there are 22 working days in a month, and we need a build agent with 2GB RAM and 1 CPU core, then to perform all these builds would cost us $7.95.

By comparison, the VM with the comparable spec (1 core and 2GB RAM) costs $26.28 per month. So the ACI option is much cheaper, but that's by virtue of the fact that our VM would be sitting idle for over 90% of the time.

If we wanted 2 Cores and 4GB RAM, the ACI cost goes up to about $15.08 a month, compared to paying $55.48 for the equivalent VM. And the containerized version doesn't actually have as great need for RAM as it's not running the base operating system, so the cost could be reduced further by reducing the memory allocation.

However, there will be a break even point. Let's say you're actually doing 50 builds a day on 2 cores with 4GB RAM. Now the ACI cost is $50.27 - pretty much the same as the VM. Another thing to bear in mind is that the VM is a fixed cost - it won't go up if you have a busy month - you'll just have to potentially wait longer for it to complete builds. But if someone introduces something that doubles the length of each build, then your ACI bill would double so you'd need to monitor for slow builds that could produce a nasty billing surprise.

So whether this technique is cheaper or not, really depends on how heavy the utilization of your existing resources is. As I mentioned in my post on media processing with ACI, it may be that a hybrid approach is best if you have a high CI workload. You can pay for one CI server that does the bulk of the work, but let ACI instances take up the strain when there's a backlog of work that the CI server can't keep up with. That way you keep costs under control and reduce latency of build throughput as ACI allows us to perform several builds in parallel rather than queuing them.

Summary

The demo I built out in this post is really just a proof of concept, showing how all these technologies can be brought together. I know that most CI servers already support using containers as build agents, and if they don't already give you the option to use ACI to perform the builds, it's the type of feature I can see coming along very soon. In fact it's not hard to imagine a completely serverless CI approach where Azure Functions are responding to GitHub webhooks and kicking off builds in ACI containers.

If you want to examine the code in more detail for this demo it's all up on GitHub:

Want to learn more about how to build serverless applications in Azure? Be sure to check out my Pluralsight course Building Serverless Applications in Azure.