Passing images to LLMs in C# with Microsoft.Extensions.AI
I recently posted about how to get started calling LLMs with C# using the Microsoft.Extensions.AI NuGet package.
The model we chose to use was gpt-4o-mini, which has the advantage of being "multimodal", as it can accept not only text input, but images as well.
In this post I'll show how you can pass an image to this model using Microsoft.Extensions.AI
and ask it to describe the image. It's relatively straightforward, but as always seems to be the case with these things, there was a gotcha I ran into.
Adding a local image to a ChatMessage
To pass in an image, you still create an instance of ChatMessage
in the same way you do with text conversations, and you can add a text prompt as well. But then you can add the binary contents of an image file to the ChatMessage
by appending an ImageContent
object to the message contents.
var message = new ChatMessage(ChatRole.User,
"Can you provide a brief, one-sentence description of this image.");
var data = File.ReadAllBytes(@"C:\temp\example.jpeg");
message.Contents.Add(new ImageContent(data, "image/jpeg"));
Adding an online image to a ChatMessage
If instead your image is online, you still use ImageContent
, but point to a URL instead...
message.Contents.Add(
new ImageContent(new Uri("https://example.net/image.jpg")));
Submitting the message and testing it
Submitting the chat message follows the same pattern demonstrated in my previous post. We add the ChatMessage
to our history, and then use CompleteStreamingAsync
to print the response as it is generated.
chatHistory.Add(message);
await foreach (var item in
chatClient.CompleteStreamingAsync(chatHistory))
{
Console.Write(item.Text);
}
If all is configured correctly, then you'll get a response describing the image you submitted. I was very impressed with the small number of "tokens" required to pass the image in. My test photograph added only 70 tokens - the equivalent of a couple of paragraphs of text.
A band is performing on stage, featuring instruments like electric guitars, drums, and a keyboard, with a colorful patterned backdrop behind them.
Potential gotcha - rate limiting
Unfortunately, when I initially tested this, it simply didn't work at all. I got no response from the model - it just hung. Eventually the root cause became apparent when I tried using the "chat playground" available at ai.azure.com. This told me that I was being rate limited.
That was surprising as the default rate limit for a model is 10,000 tokens per minute, and 100 requests per minute, which I'm pretty sure I wasn't over. However, upping my limit to 40,000 tokens per minute in the Azure portal for my deployment enabled everything to start working again.
Hopefully the library will be improved to be able to report if you are being rate-limited in the future.
Summary
The Microsoft.Extensions.AI
package makes it very easy to attach an image to a chat message. And I recommend using ai.azure.com to test out your chats if you run into problems using these models programmatically.