0 Comments Posted in:

Last week I was tasked with tracking down a perplexing problem with an API - every call was returning a 500 error, but there was nothing in the logs that gave us any clue why. The problem was not reproducible locally, and with firewall restrictions getting in the way of remote debugging, it took me a while to find the root cause.

Once I had worked out what the problem was, it was an easy fix. But the real issue, was the fact that we had some codepaths where exceptions could go unlogged. After adding exception logging to some places that had been missed, it became immediately obvious what was going wrong. Had this issue manifest itself on a production system, we could have been looking at prolonged periods of down-time, simply because we weren't logging these exceptions.

So this post is a quick reminder to check all your services - are there any places where exceptions can be thrown that don't end up in your logs? Here's three places to check.

Application startup

When an application starts up, one of the first things you should do is create a logger, and log a "Starting up" message at informational level. This is invaluable in providing a quick sanity check that at your application code did in fact start running and that it is correctly configured for logging.

I also like to log an additional message once all the startup code has completed. This alerts you to any problems with your service only managing to get half-way through initialization, or if there is a long-running operation hidden in the start-up code (which is usually a design flaw).

Of course, you should also wrap the whole startup code in an exception handler, so that any failures to start the service are easy to diagnose. Something like this is a good approach:

public void Main()
{
    var logger = CreateLogger();
    try 
    {
        logger.Information("Starting up");
        Startup();
        logger.Information("Started up");
    }
    catch (Exception ex)
    {
        logger.Error(ex, "Startup error");
    }
}

Middleware

In our particular case, the issue was in the middleware of our web API. This meant the exception wasn't technically "unhandled" - a lower level of the middleware was already catching the exception and turning it into a 500 response. It just wasn't getting logged.

Pretty much all web API frameworks provide ways for you to hook into unhandled exceptions, and perform your own custom logic. ASP.NET Core has exception middleware that you can customize, and the previous ASP.NET MVC allows you to implement a custom IExceptionHandler or IExceptionLogger. Make sure you know how to do this for the web framework you're using.

Long-running threads

Another place where logging can be forgotten is in a long-running thread such as a message pump, that's reading messages from a queue and processing them. In this scenario, you probably have an exception handler around the handling of each message, but additionally you need to log any exceptions at the pump level - e.g. if it loses connection to the message broker, you don't want to die silently and end up no longer processing messages.

In this next sample, we've remembered to log exceptions handling a message, but not exceptions fetching the next message.

while(true)
{
    // don't forget to handle exception that happen here too!
    var message = FetchNextMessage(); 
    try
    {
        Handle(message);
    }
    catch(Exception ex)
    {
        logger.Error(ex, "Failed to handle message");
        // don't throw, we want to keep processing messages
    }
}

You might already have this

Of course, some programming frameworks and hosting platforms have good out-of-the box logging baked in, which saves you the effort of writing this yourself. But it is worth double-checking that you have sufficient logging of all exceptions at whatever point they are thrown. An easy way to do this is to just throw a few deliberate exceptions in various places in your code (e.g. MVC controller constructor, middleware, application startup, etc), and double-check that they find their way into the logs. You'll be glad you did so when something weird happens in production.

In a world of microservices, observability is more critical than ever, and ensuring that all exceptions are adequately logged is a small time investment that can pay big dividends.


0 Comments Posted in:

Recently I've been having a lot of discussions with teams wanting to move towards a cloud-based microservices architecture. And inevitably the question arises whether the best choice would be to go with containers, or a serverless "Functions as a Service" (FaaS) approach.

To keep this discussion from becoming too abstract, let's imagine we're planning to host our application in Azure. Should we create an AKS (Azure Kubernetes Service) cluster and implement each microservice as a container? Or should we use Azure Functions, and implement each microservice as a Function App?

And to keep this article from becoming too long, I'm going to restrict myself to making just a few key points in favour of both approaches.

It's not either/or

First, it's important to point out that hybrid architectures are possible. There is no rule preventing you from using both AKS and Azure Functions, playing to the strengths of each platform. And if you're migrating from a monolith, you may well be running alongside some legacy Virtual Machines anyway.

Also, if you like the Azure Functions programming model, it's quite possible to host Azure Functions in a container. And if you like the consumption-based pricing model and elastic scale associated with serverless, then technologies like Azure Container Instances can be combined with AKS to essentially give you serverless containers.

And while serverless essentially forces you in the direction of PaaS for your databases, event brokers, identity providers etc, you can do exactly the same with containers - there's no reason why they can't reach out to PaaS services for these concerns rather than containerizing everything.

A few strengths of containers

What factors might cause us to favour containers?

Containers are particularly good for migrating legacy services. If you've already implemented a batch process, or web API, then getting that running in a container is much easier than rewriting it for serverless.

Containers make it trivial for us to adopt third party dependencies that aren't easily available (or cost-effective) as PaaS. There's a wealth of open source containerized services you can easily make use of such as Redis, RabbitMQ, MongoDb, and Elasticsearch. You have freedom choose when and if it makes sense to switch to PaaS versions of these services (one nice pattern is to use containerized databases for dev/test environments, but a PaaS database like Azure SQL Database in production).

Containers have a particularly good story for local development. If I have 20 microservices, I can bundle them all into a Docker compose file, and start them all up in an instant. With serverless, you need to come up with your own strategy for how developers can test a microservice in the context of the overall application.

A containerized approach can also simplify the security story. With serverless, you're typically exposing each microservice with a HTTP endpoint publicly on the internet. That means each service could potentially be attacked, and great care must be taken to ensure only trusted clients can call each service. With a Kubernetes cluster, you don't need to expose all your microservices outside the cluster - only certain services are exposed by an ingress controller.

A few strengths of serverless

What are some key strengths of serverless platforms like Azure Functions?

Serverless promotes rapid development by providing a simplified programming model that integrates easily with a selection of external services. For example, with Azure Functions, makes it trivial to connect to many Azure services such as Azure Service Bus, Cosmos DB and Key Vault.

Serverless encourages an event-driven nanoservice model. Although containers place no constraints on what programming models you use, they make it easy to perpetuate older development paradigms involving large heavyweight services. Serverless platforms strongly push us in the direction of event-driven approaches which are inherently more scalable, and promote light-weight small "nanoservices" that can be easily discarded and rewritten to adapt to changing business requirements (a key driver behind the idea of "microservices").

Serverless can offer extremely low cost systems, by supporting a "scale to zero" approach. This is extremely compelling for startups, who want to keep their initial costs to a minimum during a proof of concept phase, and also allows lots of dev/test service deployments in the cloud without worrying about cost. By contrast, with containers, you would almost always have a core number of nodes in your cluster that were always running (so with containers you might control cost either by running locally, or by sharing a Kubernetes cluster).

Serverless also excels in supporting rapid scale out. Azure Functions very quickly scales from 0 to dozens of servers under heavy load, and you're still only paying for the time your functions are actually running. Achieving this kind of scale out is more work to configure with containerized platforms, but on the flip side, with container orchestrators you will have much more control over the exact rules governing scale out.

Summary

Both containerized and serverless are excellent approaches to building microservices, and are constantly borrowing each other's best ideas, so the difference isn't huge (and maybe this question won't even be meaningful in 5-10 years).

Which one would I pick? Well, I think for a more "startupy" application, where it's greenfield development with a small number of developers trying to prove out a business idea, I think serverless really shines, whereas for more "enterprisey" applications, with a lot more components, development teams and maybe some legacy components involved, I think containerized approaches are more promising. In fact, most systems I work on are essentially "hybrid" - combining aspects of serverless, containers and plain old virtual machines.

Finally, for an amusing take on the topic, make sure you check out this genius serverless vs containers rap battle from the Think FaaS podcast.

Want to learn more about how easy it is to get up and running with Azure Container Instances? Be sure to check out my Pluralsight course Azure Container Instances: Getting Started.

0 Comments Posted in:

The dotnet new command

One of my favourite things about .NET Core is the dotnet command line tool. With dotnet new, you can quickly scaffold a new project of various types. For example dotnet new webapp creates an ASP.NET Core web app. And if you simply type dotnet new you can see a list of all of the available templates.

Templates               Short Name         Language        Tags
-------------------------------------------------------------------------
Console Application     console            [C#], F#, VB    Common/Console
Class library           classlib           [C#], F#, VB    Common/Library
Unit Test Project       mstest             [C#], F#, VB    Test/MSTest
NUnit 3 Test Project    nunit              [C#], F#, VB    Test/NUnit
NUnit 3 Test Item       nunit-test         [C#], F#, VB    Test/NUnit
xUnit Test Project      xunit              [C#], F#, VB    Test/xUnit
Razor Component         razorcomponent     [C#]            Web/ASP.NET
Razor Page              page               [C#]            Web/ASP.NET
MVC ViewImports         viewimports        [C#]            Web/ASP.NET

Installing templates

Of course, there may not be templates available out of the box that meet your needs, but it's very easy to install additional templates. For example, if you want to create a Vue.js project, you can install a new template pack with dotnet new --install "Microsoft.AspNetCore.SpaTemplates" and then create a new project with dotnet new vue.

Questions

Now as cool as this feature is, it left me with a bunch of questions. Where are all these templates coming from? If I install a template pack, how do I keep it up to date? How do I find out what other template packs are available? If I wanted to make my own template, how would I do that? So I did a bit of digging, and here's what I found.

Templates are stored in NuGet packages

Templates are distributed as NuGet packages (.nupkg), typically hosted on NuGet.org, but you can install them from any NuGet server. The Vue.js template pack I mentioned earlier can be found here. Knowing this is very handy as it enables you to see whether the package is still being actively maintained, and whether there have been recent updates. Looks like this particular template pack hasn't been updated in a while.

How do I know what's available?

How can you find out what template packs are available? There are two main ways I know of.

First, there's this list maintained on GitHub containing many packages.

Second, there's a great searchable website at dotnetnew.azurewebsites.net. So if I'm looking for more up to date Vue.js templates, I can see that there is a very wide choice available.

How do I know what template versions I have?

This one took me a while to find, but I discovered that if you type dotnet new -u (the uninstall command) it gives you a really nice summary of each package installed, in this kind of format.

  Microsoft.AspNetCore.Blazor.Templates
    Details:
      NuGetPackageId: Microsoft.AspNetCore.Blazor.Templates
      Version: 0.7.0
      Author: Microsoft
    Templates:
      Blazor (hosted in ASP.NET server) (blazorhosted) C#
      Blazor Library (blazorlib) C#
      Blazor (Server-side in ASP.NET Core) (blazorserverside) C#
      Blazor (standalone) (blazor) C#
    Uninstall Command:
      dotnet new -u Microsoft.AspNetCore.Blazor.Templates

This command also conveniently shows the syntax to uninstall a template package.

How do I know when updates are available?

Of course, you don't want to have to constantly visit NuGet.org to check up on new versions of template packs, so how do you know when something updated is available? Well the good news is that there are a couple of helpful commands here for you.

First, to update a package to it's latest version, you can always simply install it again. So if I say dotnet new -i Microsoft.AspNetCore.Blazor.Templates, then I'll either install the Blazor templates, or update to the latest (non-prelease) version if they are already installed.

There are also a couple of new commands that perform an update check for you. dotnet new --update-check will check to see if there are new versions of any installed templates, and dotnet new --update-apply also updates them for you.

Note: I attempted to use this feature by deliberately installing an older version of a template, and then running the update check, but it reported that no updates were available. I don't know if that was because by explicitly specifying a version I had perhaps "pinned" to that version, or whether it was just a temporary glitch with the tool.

How do I install a specific template version?

Because templates are stored in NuGet packages, you might want to install a specific version (maybe a pre-release). For example, at the moment, to play with the new WebAssembly Blazor features, you need to install a pre-release of the Microsoft.AspNetCore.Blazor.Templates. That can easily be done by appending :: and the package number after the package name:

dotnet new -i Microsoft.AspNetCore.Blazor.Templates::3.0.0-preview9.19465.2

How can I create my own templates?

Finally, you might be wondering what it takes to create your own template. I'm not going to go into detail here, as there's a helpful tutorial on the Microsoft docs site. But it's relatively straightforward. You create the source files for the template, and a template.json that contains template metadata.

A great way to get a feel for what's possible is to use the excellent NuGet Package Explorer utility to take a look inside the contents of existing NuGet package templates.

Summary

dotnet new is a great productivity tool and once you know a little bit more about what's going on behind the scenes, you can confidently install and update additional template packs to greatly speed up your daily development tasks.