Posted in:

Azure Container Instances combine the benefits of serverless and Docker containers, and I wrote recently about one compelling use case - using Azure Container Instances to implement Media Processing tasks.

Today I want to look at how we could use Azure Container Instances as a cost-effective option for CI builds. We'll actually be combining a whole host of cool technologies:

Building ASP.NET Core with Cake and deploying to Azure with Run From Zip

For the demo application that we'll be using, I've created a very simple ASP.NET Core website, and created a Cake build script for it. If you've not tried Cake yet - I can highly recommend it - it's a great build automation tool that is a great fit for C# projects as it uses C# as the basis for its DSL.

Once it's built, we are going to push the site live to Azure using a brand new deployment technique called Run-from-Zip. This was made easy thanks to a great article from Mattias Karlsson showing how to use "Run-From-Zip" with the Cake Kudu Client.

Here's my Cake file, with the test step removed for brevity as my demo app doesn't have any unit tests, and the main focus here is on building and deploying to Azure.

As you can see, I need the Cake.Kudu.Client addin. This does the zipping for us, so all we need to do is ensure that the three Kudu environment variables are set up with the site address and the deployment credentials.

#addin nuget:?package=Cake.Kudu.Client&version=0.5.0

// Target - The task you want to start. Runs the Default task if not specified.
var target = Argument("Target", "Default");  
var configuration = Argument("Configuration", "Release");

Information($"Running target {target} in configuration {configuration}");

var distDirectory = Directory("./dist");

// Deletes the contents of the Artifacts folder if it contains anything from a previous build.
Task("Clean")  
    .Does(() =>
    {
        CleanDirectory(distDirectory);
    });

// Run dotnet restore to restore all package references.
Task("Restore")  
    .Does(() =>
    {
        DotNetCoreRestore();
    });

// Build using the build configuration specified as an argument.
 Task("Build")
    .Does(() =>
    {
        DotNetCoreBuild(".",
            new DotNetCoreBuildSettings()
            {
                Configuration = configuration,
                ArgumentCustomization = args => args.Append("--no-restore"),
            });
    });

// Publish the app to the /dist folder
Task("PublishWeb")  
    .Does(() =>
    {
        DotNetCorePublish(
            "./coreapp.csproj",
            new DotNetCorePublishSettings()
            {
                Configuration = configuration,
                OutputDirectory = distDirectory,
                ArgumentCustomization = args => args.Append("--no-restore"),
            });
    });

Task("DeployToAzure")
    .Description("Deploy to Azure ")
    .Does(() =>
    {
        // https://hackernoon.com/run-from-zip-with-cake-kudu-client-5c063cd72b37
        string baseUri  = EnvironmentVariable("KUDU_CLIENT_BASEURI"),
               userName = EnvironmentVariable("KUDU_CLIENT_USERNAME"),
               password = EnvironmentVariable("KUDU_CLIENT_PASSWORD");
        IKuduClient kuduClient = KuduClient(
            baseUri,
            userName,
            password);
        var skipPostDeploymentValidation = true; // .NET core apps don't report their version number
        FilePath deployFilePath = kuduClient.ZipRunFromDirectory(distDirectory, skipPostDeploymentValidation);
        Information("Deployed to {0}", deployFilePath);
    });

// A meta-task that runs all the steps to Build and Test the app
Task("BuildAndTest")  
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build");

// The default task to run if none is explicitly specified. In this case, we want
// to run everything starting from Clean, all the way up to Publish.
Task("Default")  
    .IsDependentOn("BuildAndTest")
    .IsDependentOn("PublishWeb")
    .IsDependentOn("DeployToAzure");

// Executes the task specified in the target argument.
RunTarget(target);

The container image - Cake builder

The next piece of the puzzle was to create a Docker image that would be able to run my Cake script. Azure Container Instances does support Windows containers, but currently many of the surrounding features are not supported (such as mounting volumes), and since .NET Core and Cake are cross-platform anyway, I decided to make my Cake builder Docker image a Linux container.

The dockerfile is quite straightforward with the exception of the fact that we need Mono installed to run Cake. A very helpful article from Andrew Lock pointed me in the right direction for how to build ASP.NET Core Apps using Cake in Docker

Notice that my Docker file does not copy the source in. It simply assumes that a Cake build.sh script already exists in the /src folder (which is what our gitRepo volume mount will do). The --settings_skipverification=true build argument helped me work around a versioning issue, but probably isn't needed any more.

FROM microsoft/aspnetcore-build:2.0 AS build

# Install mono for Cake (https://andrewlock.net/building-asp-net-core-apps-using-cake-in-docker/)
ENV MONO_VERSION 5.4.1.6

RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF

RUN echo "deb http://download.mono-project.com/repo/debian stretch/snapshots/$MONO_VERSION main" > /etc/apt/sources.list.d/mono-official.list \  
  && apt-get update \
  && apt-get install -y mono-runtime \
  && rm -rf /var/lib/apt/lists/* /tmp/*

RUN apt-get update \  
  && apt-get install -y binutils curl mono-devel ca-certificates-mono fsharp mono-vbnc nuget referenceassemblies-pcl \
  && rm -rf /var/lib/apt/lists/* /tmp/*

WORKDIR /src
CMD ./build.sh -Target=Default --settings_skipverification=true

Using Azure Container Instances "gitRepo" Volume Share

One of the volume types you can mount with Azure Container Instances is a "gitRepo" volume. This basically clones a git repository into a folder of your choosing.

You simply give it the URL of the Git repository and optionally specify the SHA of the commit you want to clone. It doesn't appear that there is any provision to provide credentials to a private repository, so hopefully that's something that will get added in the future.

Another limitation is that the Azure CLI doesn't currently support mounting "gitRepo" volumes yet. That's unfortunate, as if it did we could kick off a build with one simple command like this (Don't try this - the --gitrepo-repository and --git-repo-mount-path arguments are made up as examples of what I hope is coming to the Azure CLI in the future):

# IMPORTANT! This doesn't currenly work - there aren't actually any --gitrepo-* arguments 
# for mounting gitRepo volumes yet
az container create `
    -g $resourceGroup `
    -n $containerGroupName `
    --image markheath/cakebuilder `
    --gitrepo-repository https://github.com/markheath/aspnet-core-cake `
    --gitrepo-mount-path "/src" `
    -e KUDU_CLIENT_BASEURI=https://$appName.scm.azurewebsites.net KUDU_CLIENT_USERNAME=$user KUDU_CLIENT_PASSWORD=$pass `
    --restart-policy never `
    --command-line "./build.sh -Target=Default --settings_skipverification=true"

So how can we deploy this? Well, to get at the gitRepo volume mounting capabilities we need to use ARM templates. In my ARM template you can see I've parameterized the environment variables allowing us to pass in the Kudu deployment URI and credentials.

You can also see I've hard-coded the repository we're mounting as https://github.com/markheath/aspnet-core-cake, but obviously that could be parameterized as well.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "containerGroupName": {
            "type": "string",
            "defaultValue": "myContainerGroup",
            "metadata": {
                "description": "Name for the container group"
            }
        },
        "KUDU_CLIENT_BASEURI": {
            "type": "string",
            "metadata": {
                "description": "Base URI for Kudu deployment"
            }
        },
        "KUDU_CLIENT_USERNAME": {
            "type": "string",
            "metadata": {
                "description": "Username for Kudu deployment"
            }
        },
        "KUDU_CLIENT_PASSWORD": {
            "type": "securestring",
            "metadata": {
                "description": "Password for Kudu deployment."
            }
        },
        "commandLine": {
            "type": "string",
            "defaultValue": "chmod 755 ./build.sh && ./build.sh -Target=Default --settings_skipverification=true",
            "metadata": {
                "description": "Command line to run on container start."
            }

        }
    },
    "variables": {
      "container1name": "cakebuilder",
      "container1image": "markheath/cakebuilder:0.1"
    },
    "resources": [
      {
        "name": "[parameters('containerGroupName')]",
        "type": "Microsoft.ContainerInstance/containerGroups",
        "apiVersion": "2018-02-01-preview",
        "location": "[resourceGroup().location]",
        "properties": {
          "containers": [
            {
              "name": "[variables('container1name')]",
              "properties": {
                "image": "[variables('container1image')]",
                "command": [
                    "/bin/bash",
                    "-c",
                    "[parameters('commandLine')]"
                ],
                "resources": {
                  "requests": {
                    "cpu": 1,
                    "memoryInGb": 2
                  }
                },
                "volumeMounts": [
                  {
                    "name": "gitrepo1",
                    "mountPath": "/src"
                  }
                ],
                "environmentVariables": [
                    {
                        "name": "KUDU_CLIENT_BASEURI",
                        "value": "[parameters('KUDU_CLIENT_BASEURI')]"
                    },
                    {
                        "name": "KUDU_CLIENT_USERNAME",
                        "value": "[parameters('KUDU_CLIENT_USERNAME')]"
                    },
                    {
                        "name": "KUDU_CLIENT_PASSWORD",
                        "value": "[parameters('KUDU_CLIENT_PASSWORD')]"
                    }
                ]
              }
            }
          ],
          "osType": "Linux",
          "restartPolicy": "Never",
          "volumes": [
            {
              "name": "gitrepo1",
              "gitRepo": {
                "repository": "https://github.com/markheath/aspnet-core-cake",
                "directory": "."
              }
            }
          ]
        }
      }
    ]
}

You might also notice that the command we run isn't quite as simple as ./build.sh. I'd been hoping I could do that as the dockerfile sets the working directory to /src which is where the gitRepo is mounted. However, there was a permissions issue with running the script from the gitRepo mounted volume directly, requiring me to jump through a few hoops.

I changed the build command line to chmod 755 ./build.sh && ./build.sh -Target=Default --settings_skipverification=true to make the build script executable before running it. And the container startup script itself calls /bin/bash -c passing in the build command as an argument.

I've also chosen to set the restartPolicy to never. I don't want to get into an infinite loop of continually trying to rebuild my app if there is a build failure. Instead I'd rather check why it has failed and kick off another build manually.

Launching the container and checking progress

To actually deploy our ARM template, we can use the Azure CLI. I'm assuming that we've already created an Azure App Service web app and stored its name and deployment credentials in the $appName, $user and $pass variables. You can see how I do that in my full PowerShell script that creates the app service plan, web app, enables Kudu run-from-zip and then launches the CI Cake builder container instance to build and deploy.

$containerGroupName = "cakebuilder"
az group deployment create `
    -n TestDeployment -g $resourceGroup `
    --template-file "cake-builder.json" `
    --parameters "KUDU_CLIENT_BASEURI=https://$appName.scm.azurewebsites.net" `
    --parameters "KUDU_CLIENT_USERNAME=$user" `
    --parameters "KUDU_CLIENT_PASSWORD=$pass" `
    --parameters "containerGroupName=$containerGroupName"

Once this is running, we can use the following two Azure CLI commands to check up on progress.

az container show -n $containerGroupName -g $resourceGroup
az container logs -n $containerGroupName -g $resourceGroup

az container show will tell us whether the container has started up yet, and if it has finished. Remember that it needs to download our Cake builder Docker image and clone the GitHub repository before our container will start running, so there will be a small delay.

And the az container logs command will let us see the actual output from our Cake build script. Obviously its possible that the container has started successfully but the build or deploy has failed, and so its these logs that will help us discover what exactly has gone wrong if we get a build failure.

Are Azure Container Instances cheaper than VMs?

Obviously, this all raises the question of why bother? Why not just have a Virtual Machine as a Docker host running your builds in containers? Can we save any money with this approach?

Azure Container Instances uses a consumption based pricing model where you only pay for exactly what you use. There's a very small charge for each container instance that's created ($.0025), and then for every second your container instance is running you pay $0.000012 per GB of RAM plus another $0.000012 for every CPU core.

Let's imagine you have a CI server you use for a project you're working on in a small team. Let's say each build takes 10 minutes, and in a typical day there are 15 builds. If there are 22 working days in a month, and we need a build agent with 2GB RAM and 1 CPU core, then to perform all these builds would cost us $7.95.

By comparison, the VM with the comparable spec (1 core and 2GB RAM) costs $26.28 per month. So the ACI option is much cheaper, but that's by virtue of the fact that our VM would be sitting idle for over 90% of the time.

If we wanted 2 Cores and 4GB RAM, the ACI cost goes up to about $15.08 a month, compared to paying $55.48 for the equivalent VM. And the containerized version doesn't actually have as great need for RAM as it's not running the base operating system, so the cost could be reduced further by reducing the memory allocation.

However, there will be a break even point. Let's say you're actually doing 50 builds a day on 2 cores with 4GB RAM. Now the ACI cost is $50.27 - pretty much the same as the VM. Another thing to bear in mind is that the VM is a fixed cost - it won't go up if you have a busy month - you'll just have to potentially wait longer for it to complete builds. But if someone introduces something that doubles the length of each build, then your ACI bill would double so you'd need to monitor for slow builds that could produce a nasty billing surprise.

So whether this technique is cheaper or not, really depends on how heavy the utilization of your existing resources is. As I mentioned in my post on media processing with ACI, it may be that a hybrid approach is best if you have a high CI workload. You can pay for one CI server that does the bulk of the work, but let ACI instances take up the strain when there's a backlog of work that the CI server can't keep up with. That way you keep costs under control and reduce latency of build throughput as ACI allows us to perform several builds in parallel rather than queuing them.

Summary

The demo I built out in this post is really just a proof of concept, showing how all these technologies can be brought together. I know that most CI servers already support using containers as build agents, and if they don't already give you the option to use ACI to perform the builds, it's the type of feature I can see coming along very soon. In fact it's not hard to imagine a completely serverless CI approach where Azure Functions are responding to GitHub webhooks and kicking off builds in ACI containers.

If you want to examine the code in more detail for this demo it's all up on GitHub:

Want to learn more about how to build serverless applications in Azure? Be sure to check out my Pluralsight courses Building Serverless Applications in Azure and Microsoft Azure Developer: Create Serverless Functions