0 Comments Posted in:

Just over a year ago, a new .NET SDK for Azure Service Bus was released. This replaces the old WindowsAzure.ServiceBus NuGet package with the Microsoft.Azure.ServiceBus NuGet package.

You're not forced to change over to the new SDK if you don't want to. The old one still works just fine, and even continues to get updates. However, there are some benefits to switching over, so in this post I'll highlight the key differences and some potential gotchas to take into account if you do want to make the switch.

Benefits of the new SDK

First of all, why did we even need a new SDK? Well, the old one supported .NET 4.6 only, while the new one is .NET Standard 2.0 compatible, making it usable cross-platform in .NET core applications. It's also open source, available at https://github.com/Azure/azure-service-bus-dotnet, meaning you can easily examine the code, submit issues and pull requests.

It has a plugin architecture, supporting custom plugins for things like message compression or attachments. There are a few useful plugins already available. We encrypt all our messages with Azure Key Vault before sending them to Service Bus, so I'm looking forward to using the plugin architecture to simplify that code.

On top of that, the API has generally been cleaned up and improved, and its very much the future of the Azure Service Bus SDK.

Default transport type

One of the first gotchas I ran into was that there is a new default "transport type". The old SDK by default used what it called "NetMessaging", a proprietary Azure Service Bus protocol, even though the recommended option was the industry standard AMQP.

The new SDK however defaults to AMQP over port 5671. This was blocked by my work firewall, so I had to switch to the other option of AMQP over WebSockets which uses port 443. If you need to configure this option, append ;TransportType=AmqpWebSockets to the end of your connection string.

One unfortunate side-effect of this switch from the NetMessaging protocol to AMQP is the performance of batching. I blogged a while back about the dramatic speed improvements available by sending and receiving messages in batches. Whilst sending batches of messages with AMQP seems to have similar performance, when you attempt to receive batches, with AMQP you may get batches significantly smaller than the batch size you request, which slows things down considerably. The explanation for this is here, and the issue can be mitigated somewhat by setting the MessageReceiver.PrefetchCount property to a suitably large value.

Here's some simple code you can use to check out the performance of batch sending/receiving with the new SQK. It also shows off the basic operation of the MessageSender and MessageReceiver classes in the new SDK, along with the ManagementClient which allows us to create and delete queues.

string connectionString = // your connection string - remember to add ;TransportType=AmqpWebSockets if port 5671 is blocked
const string queueName = "MarkHeathTestQueue";

// PART 1 - CREATE THE QUEUE
var managementClient = new ManagementClient(connectionString);

if (await managementClient.QueueExistsAsync(queueName))
{
    // ensure we start the test with an empty queue
    await managementClient.DeleteQueueAsync(queueName);
}
await managementClient.CreateQueueAsync(queueName);

// PART 2 - SEND A BATCH OF MESSAGES
const int messages = 1000;
var stopwatch = new Stopwatch();

var client = new QueueClient(connectionString, queueName);

stopwatch.Start();

await client.SendAsync(Enumerable.Range(0, messages).Select(n =>
{
    var body = $"Hello World, this is message {n}";
    var message = new Message(Encoding.UTF8.GetBytes(body));
    message.UserProperties["From"] = "Mark Heath";
    return message;
}).ToList());

Console.WriteLine($"{stopwatch.ElapsedMilliseconds}ms to send {messages} messages");
stopwatch.Reset();

// PART 3 - RECEIVE MESSAGES
stopwatch.Start();
int received = 0;
var receiver = new MessageReceiver(connectionString, queueName);
receiver.PrefetchCount = 1000; // https://github.com/Azure/azure-service-bus-dotnet/issues/441
while (received < messages)
{
    // unlike the old SDK which picked up the whole thing in 1 batch, this will typically pick up batches in the size range 50-200
    var rx = (await receiver.ReceiveAsync(messages, TimeSpan.FromSeconds(5)))?.ToList();
    Console.WriteLine($"Received a batch of {rx.Count}");
    if (rx?.Count > 0)
    {
        // complete a batch of messages using their lock tokens
        await receiver.CompleteAsync(rx.Select(m => m.SystemProperties.LockToken));
        received += rx.Count;
    }
}

Console.WriteLine($"{stopwatch.ElapsedMilliseconds}ms to receive {received} messages");

Management Client

Another change in the new SDK is that instead of the old NamespaceManager, we have ManagementClient. Many of the method names are the same or very similar so it isn't too hard to port code over.

One gotcha I ran into is that DeleteQueueAsync (and the equivalent topic and subscription methods) now throw MessagingEntityNotFoundException if you try to delete something that doesn't exist.

BrokeredMessage replaced by Message

The old SDK used a class called BrokeredMessage to represent a message, whereas now it's just Message.

It's had a bit of a reorganize, so things like DeliveryCount and LockToken are now found in Message.SystemProperties. Custom message metadata is stored in UserProperties instead of Properties. Also, instead of providing the message body as a Stream, it is now a byte[], which makes more sense.

Another significant change is that BrokeredMessage used to have convenience methods like CompleteAsync, AbandonAsync, RenewLockAsync and DeadLetterAsync. You now need to make use of the ClientEntity to perform these actions (with the exception of RenewLockAsync to be discussed shortly).

ClientEntity changes

The new SDK retains the concept of a base ClientEntity which has derived classes such as QueueClient, TopicClient, SubscriptionClient etc. It's here that you'll find the CompleteAsync, AbandonAsync, and DeadLetterAsync methods, but one conspicuous by its absence is RenewLockAsync.

This means that if you're using QueueClient.RegisterMessageHandler (previously called QueueClient.OnMessage) or similar to handle messages, you don't have a way of renewing the lock for longer than the MaxAutoRenewDuration duration specified in MessageHandlerOptions (which used to be called OnMessageOptions.AutoRenewTimeout). I know that is a little bit of an edge case, but we were relying on being able to call BrokeredMessage.RenewLockAsync in a few places to extend the timeout further. With the new SDK, the ability to renew a lock is only available if you are using MessageReceiver, which has a RenewLockAsync method.

A few other minor changes that required a bit of code re-organization were the fact that old Close methods are now CloseAsync, meaning that it is trickier to use the Dispose pattern. There is no longer a ClientEntity.Abort method - presumably you now just call CloseAsync to shut down the message handling pump. And when you create MessageHandlerOptions you are required to provide an exception handler.

Summary

The new Azure Service Bus SDK offers lots of improvements over the old one, and the transition isn't too difficult, but there are a few gotchas to be aware of and I've highlighted some of the ones that I ran into. Hopefully this will be of use to you if you're planning to upgrade.


0 Comments

At the end of each year I like to look back and take stock of what I've done in the previous year (I've done this a few times now: 2017,2016,2015, 2013). 2018 was an exciting year for me, featuring a number of "firsts".

I visited America!

Undoubtedly one of my highlights of the year was my first ever visit to the USA for the Microsoft MVP Summit. I had an amazing time there, making many new friends and learning loads about what's new in the world of developer tools and Azure. I'm very grateful to Microsoft for letting me be part of this program, to my employer NICE for letting me have the week off work and covering my flights, and to my wife for freeing me to go for a week while she looked after our five children! For a flavour of what goes on at the MVP Summit check out this great writeup from David Pine.

MVP Summit

I was also very privileged to be re-awarded as an MVP in July, which means I'm going to make my second visit to Redmond in 2019 which I'm really looking forward to.

I spoke at some conferences!

Another huge first for me was speaking at two developer conferences. I've always done lots of technical talks, but that has generally been in the context of my work, or local user groups. After being rejected from the first few conferences I submitted to, I found myself accepted at two in short proximity. First, I spoke on Azure Durable Functions at ProgNET in London, and then on LINQ at the inaugural Techorama Netherlands conference. Both conferences were really well run, and as always, were wonderful opportunities to make new friends.

Of course, I still gave a number of local user group talks, including

I've submitted some more conference talk proposals for 2019, and so hopefully I'll have some news to share in the near future. I'm also open to speaking at local user groups, especially in the south of England, so feel free to reach out to me if you'd like me to visit your user group.

I published four Azure related Pluralsight courses!

The two main technologies I've been focusing on this year are Azure Functions and Containers on Azure, and that's reflected in the four Pluralsight courses I released:

One great thing about the two "Microsoft Azure Developer" courses is that they are being made available to anyone to watch for free, even if they haven't got a Pluralsight subscription, as part of the excellent Microsoft Learn initiative.

I've not finalised my plans for Pluralsight courses in 2019 yet, but expect plenty more Azure content, and I'm also hoping to update my Azure Functions Fundamentals course to reflect the updates to the platform since I first released it.

I created lots of blogs, videos and GitHub projects!

Wherever possible I like to share the things I'm learning for free, and so this year my contribution to the community this year included:

The major topics I covered were of course Azure Functions (especially Durable Functions) and Containers on Azure, but also my Techorama talk made me revisit LINQ and produce a series of tutorials on MoreLINQ.

There were a few things I sadly didn't find as much time as I hoped for. Though I've done a lot of prototyping of a .NET Standard version of NAudio, there are a couple of tricky decisions around UWP support I haven't yet decided how to resolve. And although I made a good start on this year's Advent of Code, this December was a little bit too full for me to continue past day 14.

What else?

A few other things worth noting from this year. In my day job, working with NICE as a software architect, the software we're building to help police forces manage digital evidence is being really well received by several forces who are at various stages of adopting it. This project promises to keep me busy staying on top of best architectural practices for Azure, and will no doubt drive much of my learning focus for the next year.

And of course life is much more than programming. One of the reasons I try to keep travel to a minimum with work is to spend as much time as possible with my family. This year I've taken my eldest son around various university open days, as well as taught various children guitar, football, computer maintenance, bike riding, and (of course) programming (we're currently enjoying the Get Coding books).

I also love being part of my local church community, where I have been teaching my way through the minor prophets, as well as enjoying playing more electric guitar and piano with a great bunch of musicians. One of the things I appreciate most about my church is the great diversity: it's made up of people of all ages, nationalities, social and education backgrounds, and is intentional about serving and welcoming those who are disadvantaged in various ways. So it's been great to see a big focus in the world of programming over the last year at improving diversity and inclusion.

Finally, I want to say a big thank you to everyone who's supported me this year, watching my courses, attending my talks, reading my blog posts. It's been great to meet many of you in person and hopefully I'll connect with more of you in 2019.


0 Comments Posted in:

A while ago I wrote an article explaining how you can normalize audio using NAudio. Normalizing is a way of increasing the volume of an audio file by the largest possible amount without clipping.

However, I also mentioned in that article that in many cases, dynamic range compression is a better option. That's because in many audio files there are often a few stray peaks that are very loud which can make it impossible to bring up the overall gain without clipping. What we'd like to do is reduce the level of those peaks, giving us more headroom to increase the gain of the rest of the audio.

In this post, I'll show how we can implement a simple "limiter" which is essentially a compressor with a very steep compression ratio, that can be used on a variety of types of audio, including spoken word, to achieve a nice consistent volume level throughout.

An audio effects framework

NAudio has an interface called ISampleProvider which is ideal for our limiter effect, but I've created an Effect base class to make it simpler to implement effects.

The Effect class implements ISampleProvider, and uses the decorator pattern to take the source ISampleProvider in. It requires derived effect classes to implement a method called Sample which is called for every stereo or mono sample frame. There are also some optional methods you can implement, such as ParamsChanged which is called whenever the values of the effect parameters are modified to allow recalculation of any constants, and Block which is called before each block of samples is processed, for efficiency.

The design of this is inspired by the JSFX effects framework that is part of the REAPER DAW. It's essentially a DSL for implementing effects, which I've found very useful for implementing my own custom effects such as this trance gate. To make it easier to port a JSFX effect to C#, I've added some additional static helper methods to match the method names used by JSFX.

abstract class Effect : ISampleProvider
{
    private ISampleProvider source;
    private bool paramsChanged;
    public float SampleRate { get; set; }

    public Effect(ISampleProvider source)
    {
        this.source = source;
        SampleRate = source.WaveFormat.SampleRate;
    }

    protected void RegisterParameters(params EffectParameter[] parameters)
    {
        paramsChanged = true;
        foreach(var param in parameters)
        {
            param.ValueChanged += (s, a) => paramsChanged = true;
        }
    }

    protected abstract void ParamsChanged();

    public int Read(float[] samples, int offset, int count)
    {
        if (paramsChanged)
        {
            ParamsChanged();
            paramsChanged = false;
        }
        var samplesAvailable = source.Read(samples, offset, count);
        Block(samplesAvailable);
        if (WaveFormat.Channels == 1)
        {
            for (int n = 0; n < samplesAvailable; n++)
            {
                float right = 0.0f;
                Sample(ref samples[n], ref right);
            }
        }
        else if (WaveFormat.Channels == 2)
        {
            for (int n = 0; n < samplesAvailable; n+=2)
            {
                Sample(ref samples[n], ref samples[n+1]);
            }
        }
        return samplesAvailable;
    }

    public WaveFormat WaveFormat { get { return source.WaveFormat; } }
    public abstract string Name { get; }
    // helper base methods these are primarily to enable derived classes to use a similar
    // syntax to REAPER's JS effects
    protected const float log2db = 8.6858896380650365530225783783321f; // 20 / ln(10)
    protected const float db2log = 0.11512925464970228420089957273422f; // ln(10) / 20 
    protected static float min(float a, float b) { return Math.Min(a, b); }
    protected static float max(float a, float b) { return Math.Max(a, b); }
    protected static float abs(float a) { return Math.Abs(a); }
    protected static float exp(float a) { return (float)Math.Exp(a); }
    protected static float sqrt(float a) { return (float)Math.Sqrt(a); }
    protected static float sin(float a) { return (float)Math.Sin(a); }
    protected static float tan(float a) { return (float)Math.Tan(a); }
    protected static float cos(float a) { return (float)Math.Cos(a); }
    protected static float pow(float a, float b) { return (float)Math.Pow(a, b); }
    protected static float sign(float a) { return Math.Sign(a); }
    protected static float log(float a) { return (float)Math.Log(a); }
    protected static float PI { get { return (float)Math.PI; } }

    /// <summary>
    /// called before each block is processed
    /// </summary>
    /// <param name="samplesblock">number of samples in this block</param>
    public virtual void Block(int samplesblock)
    {
    }

    /// <summary>
    /// called for each sample
    /// </summary>
    protected abstract void Sample(ref float spl0, ref float spl1);

    public override string ToString()
    {
        return Name;
    }
}

You'll also notice that there is support for the concept of an EffectParameter. This is just a way of allowing users to adjust a parameter between a minimum and maximum and making sure that the effect is notified of any parameter changes.

class EffectParameter
{
    public float Min {get;}
    public float Max {get;}
    public string Description {get;}
    private float currentValue;
    public event EventHandler ValueChanged;
    public float CurrentValue 
    {
        get { return currentValue;}
        set 
        {
            if (value < Min || value > Max)
                throw new ArgumentOutOfRangeException(nameof(CurrentValue));
            if (currentValue != value)
                ValueChanged?.Invoke(this, EventArgs.Empty);
            currentValue = value;
        }
    }

    public EffectParameter(float defaultValue, float minimum, float maximum, string description)
    {
        Min = minimum;
        Max = maximum;
        Description = description;
        CurrentValue = defaultValue;
    }
}

A simple limiter

For this example, the limiter I've chosen to port is one shipped with REAPER and created by Schwa, who's the author of a whole host of super useful effects. This is nice and simple and the only modification I made was to allow larger gain boost values and set the brickwall default to -0.1dB.

class SoftLimiter : Effect
{
    public override string Name => "Soft Clipper/ Limiter";

    public EffectParameter Boost { get; } = new EffectParameter(0f, 0f, 18f, "Boost");
    public EffectParameter Brickwall { get; } = new EffectParameter(-0.1f, -3.0f, 1f, "Output Brickwall(dB)");
    public SoftLimiter(ISampleProvider source):base(source)
    {
        RegisterParameters(Boost, Brickwall);
    }

    private float amp_dB = 8.6562f;
    private float baseline_threshold_dB = -9f;
    private float a = 1.017f;
    private float b = -0.025f;
    private float boost_dB;
    private float limit_dB;
    private float threshold_dB;

    protected override void ParamsChanged()
    {
        boost_dB = Boost.CurrentValue;
        limit_dB = Brickwall.CurrentValue;
        threshold_dB = baseline_threshold_dB + limit_dB;
    }

    protected override void Sample(ref float spl0, ref float spl1)
    {
        var dB0 = amp_dB * log(abs(spl0)) + boost_dB;
        var dB1 = amp_dB * log(abs(spl1)) + boost_dB;

        if (dB0 > threshold_dB)
        {
            var over_dB = dB0 - threshold_dB;
            over_dB = a * over_dB + b * over_dB * over_dB;
            dB0 = min(threshold_dB + over_dB, limit_dB);
        }

        if (dB1 > threshold_dB)
        {
            var over_dB = dB1 - threshold_dB;
            over_dB = a * over_dB + b * over_dB * over_dB;
            dB1 = min(threshold_dB + over_dB, limit_dB);
        }

        spl0 = exp(dB0 / amp_dB) * sign(spl0);
        spl1 = exp(dB1 / amp_dB) * sign(spl1);
    }
}

Using the limiter

Using the limiter couldn't be easier. In this example we use an AudioFileReader to read the input file (this supports multiple file types including WAV, MP3 etc). Next we create an instance of SoftLimiter and set the Boost parameter to the amount of boost we want. Here I'm asking for 12dB of gain. Essentially this means that any audio below -12dB will be amplified without clipping, and the soft clipping will be applied to any audio above -12dB.

Finally we use WaveFileWriter.CreateWaveFile16 to write the limited audio into a 16 bit WAV file. Obviously you can use other NAudio supported output file formats if you want, such as using the MediaFoundationEncoder for MP3.

var inPath = @"C:\Users\mheath\Documents\my-input-file.wav";
var outPath =  @"C:\Users\mheath\Documents\my-output-file.wav";

using (var reader = new AudioFileReader(inPath))
{
    var limiter = new SoftLimiter(reader);
    limiter.Boost.CurrentValue = 12;
    
    WaveFileWriter.CreateWaveFile16(outPath, limiter);
}

Summary

With a basic effects framework in place, its not too hard to port an existing limiter algorithm from another language into C# and use it with NAudio. If you'd like to see more examples of effects ported to NAudio, take a look at these from an early version of my Skype Voice Changer application, where I took a bunch of JSFX effects and ported them to C#.

Want to get up to speed with the the fundamentals principles of digital audio and how to got about writing audio applications with NAudio? Be sure to check out my Pluralsight courses, Digital Audio Fundamentals, and Audio Programming with NAudio.