0 Comments Posted in:

Note: this is an updated version of my earlier post Managing Azure Functions Keys to use the new Functions ARM APIs

Azure Functions allows you to protect access to your HTTP triggered functions by means of authorization keys. For each function you can choose an "authorization level":

  • anonymous means no API key is required,
  • function means an API key is required.
    • This can be a function-specific key (each function can have its own collection of keys)
    • Or it can be one of the "host" keys which are keys that can be used for all functions with this authorization level
  • admin means you are required to provide the special "master" host key, which is a single key that can be used to call any function in your function app.

To call a protected function you either provide the key as a query string parameter (in the form code=<API_KEY>) or you can provide it as a HTTP x-functions-key header.

Accessing and managing keys in the portal

The Azure portal makes it nice and simple to discover the values these keys. First of all, if we navigate to any HTTP-triggered function in the portal, you'll see a "Get Function URL" link:

image

When we click it, it constructs the URL we need to call including the code query string parameter. This dialog also lets us access values for both types of key - the "function" keys specific to this function, and the "host" keys that can be used on all functions, including the special "_master" host key. You can read more about these key types here.

image

We can manage the keys for an individual function by heading into the "manage" tab for that function:

image

In here we get the ability to view, renew or revoke each individual function key as well as the host keys. You can create multiple function or host keys, which is great as it allows you to provide separate keys to every client you want to grant access to your function, or to implement key cycling.

image

Using the Functions ARM APIs to manage keys

Although its very convenient to manage keys in the portal, before long you'll probably want to manage these values programatically. One way to do that is with the key management API which is what I showed how to use in the original version of this tutorial.

However, it was quite tricky to use, and recently some new Functions ARM APIs were added that be used to do all the same things, and are easier to use.

Let's see how we can get the values of function keys, as well as generate our own, and update them with new values.

Listing keys for a specific function

To use these APIs you need to know how to generate an ARM resource id for your Function App. This requires the Function App name, the Resource Group name and the Azure subscription id.

Here's how to build the resource id for a Function App, using the Azure CLI's az account show method to look up the subscription id from the subscription name.

$subscriptionName = "My Azure Subscription"
$subscriptionId = az account show -s "$subscriptionName" --query id -o tsv
$resourceGroup = "MyResourceGroup"
$webAppName = "myfunctionapp"
$resourceId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroup/providers/Microsoft.Web/sites/$webAppName"

Next, we're going to call the listKeys endpoint to list the keys for a function. To start with, I'll show how we can do this with PowerShell's Invoke-RestMethod.

First, we need to get hold of a valid access token to call the ARM APIs, which we can get with Azure CLI's very convenient az account get-access-token. Then we can use that to POST to the listKeys endpoint for the specific function we want.

# get an access token
$accessToken = az account get-access-token --query accessToken -o tsv
$functionName = "MyFunctionName"
$listFunctionKeysUrl = "https://management.azure.com$resourceId/functions/$functionName/listKeys?api-version=2018-02-01"
$functionKeys = Invoke-RestMethod -Method Post -Uri $listFunctionKeysUrl `
    -Headers @{ Authorization="Bearer $accessToken"; "Content-Type"="application/json" }

This will return us a JSON object containing all the keys defined for your function. Typically there will just be one called default, so you can access the value with $functionKeys.default.

There is, however, an easier way for users of the Azure CLI. You can use the new preview az rest command which will get hold of the access token for you. So we can get the keys for a function with a single line:

az rest --method post --uri "https://management.azure.com$resourceId/functions/$functionName/listKeys?api-version=2018-02-01"

Listing host keys

You can also ask for the "host keys", with a similar technique:

az rest --method post --uri "$resourceId/host/default/listKeys?api-version=2018-11-01"

This will return something looking like this (no these aren't my actual keys!):

{
  "functionKeys": {
    "default": "VyEOe9oIHkIz6sj+Are3ffHcP7ptHKAidUAxgDMkdSSaYuGDraek5Q=="
  },
  "masterKey": "N9scClcIIRhRYGJN2JHfJatPbkmjvb08QZNixfwuoJrRIrj12iryKA==",
  "systemKeys": {
    "durabletask_extension": "vlNYs6GwWntJ7GXb9/MJ4A36XzM4a+06GFWZrPUqpQL5EBhYA0cHUg==",
    "eventgrid_extension": "UYe7XF4rpC0yhokdOIWEsMWfWQFlpLtf4usgZJq1tC6RHzr0BUKCrw=="
  }
}

Let me briefly explain what these all are:

  • functionKeys are the "host" keys shown in the portal. They can be used to call any HTTP triggered function. There will be a default key out of the box, and you can add your own.
  • masterKey is a special key (shown as _master in the portal) that can be used to access admin authorization level functions. This key should be guarded very carefully.
  • systemKeys contains keys used by any extensions you have installed. In this Function App I have the Durable Functions and Event Grid extensions installed, and they both add their own keys. It's really useful to access these if you want to call the Durable Functions REST API, or if you want to create a new Event Grid subscription.

By the way, if you're wondering how I knew what values are valid for the api-version (which is required on all REST API calls), I found this helpful article from Tobias Zimmergren, showing how to list the available versions with Azure PowerShell. I was using the newer Az PowerShell module, so I got the list of API versions like this:

((Get-AzResourceProvider -ProviderNamespace Microsoft.Web).ResourceTypes `
    | Where-Object ResourceTypeName -eq sites).ApiVersions

Adding and removing a function key

To add a new function key, we perform a HTTP PUT to the key we want to add (or update). You pass in an object containing the key name and value to be updated.

$keyName = "MyFunctionKey"
$payload = (@{ [email protected]{ name=$keyName; value="abcd1234" } } `
    | ConvertTo-Json -Compress).Replace('"', '\"')
az rest --method put `
    --uri "$resourceId/functions/$functionName/keys/$($keyName)?api-version=2018-11-01" `
    --body "$payload"

To remove the key, it's just a HTTP DELETE method to the same endpoint:

az rest --method delete `
    --uri "$resourceId/functions/$functionName/keys/$($keyName)?api-version=2018-11-01"

Adding and removing a host key

The technique for replacing host keys is the same, just now we call the /host/default/functionkeys endpoint. Here I'm adding a new host key.

$keyName = "MyHostKey"
$payload = (@{ [email protected]{ name=$keyName; value="efgh5678" } } | ConvertTo-Json -Compress).Replace('"', '\"')
az rest --method put --uri "$resourceId/host/default/functionkeys/$($keyName)?api-version=2018-11-01" --body "$payload"

I assume you can also use this technique to update the "system keys" and master key although I've not tried that yet. Don't try to delete them though.

To delete the host key we added, it's again very straightforward

az rest --method delete `
    --uri "$resourceId/host/default/functionkeys/$($keyName)?api-version=2018-11-01"

Summary

It's now possible to completely automate the retrieval, creation and update of keys using the new Functions ARM APIs. This is a lot simpler than the previous key management API, and has the benefit of being usable from ARM templates as well. Check out this GitHub thread for some examples of how to use these APIs in your ARM templates.

Want to learn more about how easy it is to get up and running with Azure Functions? Be sure to check out my Pluralsight courses Azure Functions Fundamentals and Microsoft Azure Developer: Create Serverless Functions

0 Comments Posted in:

It's very common in software development to trade off short term gain against long-term pain. If there's a "quick way" to do get a bug fixed or a feature implemented, then more often than not, developers tend to choose it. This results, over time, in the accumulation of "technical debt", where the decisions made in the past are hindering our progress going forwards.

This is why "technical debt" is often such a controversial topic. Martin Fowler recognizes that not every decision made to introduce technical debt is necessarily a prudent one. Maybe our own naivety and inexperience leads to us "inadvertently" introducing technical debt. Or maybe we just capitulate to the business pressure to get the work done as soon as possible, long-term consequences be damned.

But I would argue, that more often than not, as developers, we believe we are making a justifiable decision. In over 20 years as a software developer I don't recall ever having a discussion saying "let's be lazy and irresponsible and get this finished as quickly as possible". Instead, there is a quite reasonable sounding rationalization every time we choose the quick way.

Here's my list of the top seven justifications for introducing technical debt.

1. "It's urgent!"

Sometimes in software development, it really is urgent. When I worked on embedded software in telecoms hardware, downtime could result in mobile phone networks losing money at an astonishing rate, and their customers potentially unable to communicate urgent messages. So if a critical bug can be fixed with three lines of slightly "hacky" code, then that's probably preferable to spending a week implementing it the "proper way".

2. "We need to be strategic"

Another common reason for rushing through some work quicker than would be ideal is the desire to be "first to market". This is one of the main scenarios Ward Cunningham envisioned when he coined the term "technical debt". Sometimes, the commercial advantage of beating your competitors to market is worth the pain of needing to rewrite large portions of your application further down the road.

3. "It's just an MVP"

Closely related is the fact that often in software development, we're creating a feature that may not end up being used long-term. We think it's what the customers want, but until we get it out there in front of them, we're not sure. So we don't want to waste vast amounts of time perfecting code that may not end up being used by anyone. When we're in this scenario, there's a lot of logic to the idea of creating a "minimum viable product" or a "proof of concept". Of course, we intend to revisit the code in the future and provide a fuller, more robust implementation if the feature proves a success. But it's not uncommon for these bare-bones initial implementations to live a lot longer than we intended.

4. "We need to be pragmatic"

Suppose you're asked to design a new feature and estimate how long it will take to implement. You produce a thorough design considering security, scalability, observability, etc, and estimate that it will take six months development time. This comes as a shock to the business. They wanted the feature, but not that much. You're asked to be "realistic" and "pragmatic" about it. You're reminded that "The perfect is the enemy of the good". And of course, it's true that there's always "more than one way to skin a cat". Maybe there's another approach to the problem that's "good enough" and can be done in half the time.

5. KISS

The "Keep it Simple, Stupid (KISS)" principle is well-known and an important reminder that often in software, the most simple solution is in fact the best one. Often by writing less code, we end up with a better, more maintainable, more performant codebase. Technical debt can be introduced not only by a careless developer overlooking important concerns, but by an over-zealous developer writing copious amounts of bloated and complex code to handle imagined problems that may never occur in real-world systems. Many developers have a tendency to "gold plate" or "over-engineer", but it requires plenty of wisdom and experience to make the right judgment calls to resist this temptation.

6. YAGNI

Another closely related reason for technical debt is the well-meaning tendency to write additional code because we think it will probably be needed in the future. At first glance, this seems to be a wise approach - surely the more we bear the future in mind when we write code, the less technical debt we will introduce. The trouble is, we are typically terrible at predicting the future. For example, I've seen complex localization frameworks added into software that was only ever delivered in English, and fragile configuration DSLs introduced for logic that never changed and would have been far simpler to hard-code. Again, striking the right balance between a strict YAGNI approach of only coding what is absolutely needed now, and strategically designing in accessibility points for features you know are coming soon, is hard to get right.

7. "It's too risky"

This one is perhaps the most common reason for introducing (or more commonly, compounding) technical debt. Say you're working on extending some existing code, and you recognize that it is in dire need of refactoring, or maybe needs an upgrade to use a newer framework version. When this happens you have a great opportunity to pay off some technical debt, improving the overall state of the existing code, before adding your new feature. The trouble is, wide-scale refactoring and upgrading to newer technology stacks are generally disruptive and risky changes. Many developers quite understandably take the safer route of maintaining the status quo, but this results in an even bigger job for whoever does finally attempt to sort out the mess in the future.

Are these valid reasons?

What do all the reasons I gave above have in common? They are all arguably the "right decision" in certain circumstances. But they may all also cause us considerable problems in the future. The challenge with technical debt is that, unlike real-world debt (where your creditors can be relied on to demand repayment), it's not at all obvious which instances "technical debt" will come back to bite you. Sometimes your sub-optimal changes sit harmlessly in source control for decades to come without getting in anyone's way. Whereas other quite seemingly reasonable and pragmatic compromises turn out to be the source of constant pain. It's only with the benefit of hindsight that it becomes clear what decision should have been made.

What can we do about this? Is it hopeless given that you can't know for sure whether these "reasons" are valid until it's too late? A few brief thoughts in conclusion.

  1. Whenever you find yourself citing one of the seven reasons I gave above, try to give at least some thought to how the technical debt you introduce as a result might pose problems in the future, and what you can do to minimise those risks.

  2. Cut the developers who went before you some slack when you're next about to rant about how "stupid" and shortsighted the existing code is. Most likely they were making what seemed at the time to be a sensible and pragmatic choice.

  3. Accept a certain amount of "technical debt" is inevitable on any large, and ensure you incorporate sufficient time to address it and keep it from spiraling out of control. This should be a regular part of all ongoing development, not a one off technical debt "blitz".

Anyway, I know the topic of technical debt always raises some interesting (and sometimes heated) discussion, so I'd love to hear your thoughts on these reasons. This is also a great opportunity for me to say that on October 2nd I'll be speaking about technical debt at Techorama Netherlands. It's a great conference with an excellent lineup again this year, so if you're able to make it

Want to learn more about the problem of technical debt and how you can reduce it? Be sure to check out my Pluralsight course Understanding and Eliminating Technical Debt.

0 Comments Posted in:

The Microsoft.Azure.Storage.Blob NuGet package makes it really easy to work with Azure Blobs in .NET. Recently I was troubleshooting some performance issues with copying very large blobs between containers, and discovered that we were not copying blobs in the optimal way.

To copy files with the Azure Blob storage SDK, you first get references to the source and destination blobs like this:

var storageAccount = CloudStorageAccount.Parse(connectionString);
var blobClient =  storageAccount.CreateCloudBlobClient();

// details of our source file
var sourceContainerName = "source";
var sourceFilePath = "folder/test.zip";

// details of where we want to copy to
var destContainerName = "dest";
var destFilePath = "somewhere/test.zip";

var sourceContainer = blobClient.GetContainerReference(sourceContainerName);
var destContainer = blobClient.GetContainerReference(destContainerName);

CloudBlockBlob sourceBlob = sourceContainer.GetBlockBlobReference(sourceFilePath);
CloudBlockBlob destBlob = destContainer.GetBlockBlobReference(destFilePath);

At this point, you might be tempted to copy the blob with code like this:

using (var sourceStream = await sourceBlob.OpenReadAsync())
using (var destStream = await destBlob.OpenWriteAsync())
{
    await sourceStream.CopyToAsync(destStream);
}

Or maybe with the convenient UploadFromStreamAsync method:

using (var sourceStream = await sourceBlob.OpenReadAsync())
{
    await destBlob.UploadFromStreamAsync(sourceStream);
}

However, what you are doing in both those examples is downloading the entire contents of the source blob and re-uploading them to the target blob. This was taking about 20 minutes for the files I was using.

Copying the quick way

Let's see how to copy the blob the quick way:

await destBlob.StartCopyAsync(sourceBlob);

Not only is it trivially simple, but it completes almost instantaneously. And that's because when you're copying a blob within a storage account, the underlying platform doesn't need to make a new copy - it can just update a reference internally.

The name of the method (StartCopyAsync) might make you feel a bit nervous. It implies that it will finish before the copy has been completed. And that can happen if you're copying between storage accounts.

Copying between storage accounts

To copy between storage accounts, you still use the StartCopyAsync method, but pass the Uri of the source blob. The documentation is a bit sparse, but here's how I was able to get it to work.

Notice in this example that we need a separate CloudStorageAccount and CloudBlobClient to the one for the source file. We then create a readable SAS token for the source blob with enough time for a copy to take place. And finally after calling StartCopyAsync, we do need to keep track of the copy progress by checking the CopyState of the blob (and getting the latest value with FetchAttributesAsync)

// create the blob client for the destination storage account
var destStorageAccount = CloudStorageAccount.Parse(destConnectionString);
var destClient = destStorageAccount.CreateCloudBlobClient();

// destination container now uses the destination blob client
destContainer = destClient.GetContainerReference(destContainerName);

// create a 2 hour SAS token for the source file
var sas = sourceBlob.GetSharedAccessSignature(new SharedAccessBlobPolicy() {
    Permissions = SharedAccessBlobPermissions.Read,
    SharedAccessStartTime=DateTimeOffset.Now.AddMinutes(-5),
    SharedAccessExpiryTime=DateTimeOffset.Now.AddHours(2)
});

// copy to the blob using the 
destBlob = destContainer.GetBlockBlobReference(dest);
var sourceUri = new Uri(sourceBlob.Uri + sas);
await destBlob.StartCopyAsync(sourceUri);

// copy may not be finished at this point, check on the status of the copy
while (destBlob.CopyState.Status == CopyStatus.Pending)
{
    await Task.Delay(1000);
    await destBlob.FetchAttributesAsync();
}

if (destBlob.CopyState.Status != CopyStatus.Success)
{
    throw new InvalidOperationException($"Copy failed: {destBlob.CopyState.Status}");
}

If you do need to cancel the copy for any reason, you can get hold of the CopyId from the blob's CopyState and pass that to the AbortCopyAsync method on the CloudBlockBlob.

Uploading local files

Obviously if you're copying blobs into blob storage from local files, then you can't use StartCopyAsync, but there is the convenient UploadFromFileAsync method that will upload from a local file.

await destBlob.UploadFromFileAsync("mylocalfile.zip");

Unfortunately, at the time of writing, the current version of the blob storage SDK (10.0.0.3) is susceptible to out of memory exceptions when uploading huge files. Hopefully that will get resolved soon.