Image Search in C#: Find Photos by Description with CLIP, Offline

Type "a dog on a beach" and get back the photos of a dog on a beach, from a folder nobody has tagged. In C#, on your own machine, with one NuGet package.

dotnet add package Kjarni

using Kjarni;

using var index = new ImageIndex();
index.AddDirectory("/home/me/Pictures");

foreach (var hit in index.Search("a dog on a beach", 5))
    Console.WriteLine($"{hit.Score:F3}  {hit.Path}");

The first run downloads the CLIP model (580 MB) and caches it. After that it works with no network. Every number in this post comes from running this code, Kjarni 0.4.2 from NuGet, on 1,000 photos from the COCO dataset.

Try it on your own photos

You need the .NET 8 SDK. Then:

dotnet new console -o PhotoSearch
cd PhotoSearch
dotnet add package Kjarni

Replace Program.cs with:

using Kjarni;

var folder = args.Length > 0 ? args[0] : Environment.GetFolderPath(Environment.SpecialFolder.MyPictures);
var indexFile = Path.Combine(folder, ".photos.kjimg");

using var index = new ImageIndex();
if (File.Exists(indexFile))
{
    Console.WriteLine($"Loaded {index.Load(indexFile)} photos from {indexFile}");
}
else
{
    Console.WriteLine($"Indexing {folder}, this takes a moment the first time...");
    var report = index.AddDirectory(folder);
    index.Save(indexFile);
    Console.WriteLine($"Indexed {report.Added} photos, skipped {report.Skipped}");
}

while (true)
{
    Console.Write("\nSearch: ");
    var query = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(query)) break;

    foreach (var hit in index.Search(query, 5))
        Console.WriteLine($"  {hit.Score:F3}  {hit.Path}");
}

And run it on a folder of photos, or with no argument to use your Pictures folder. Here it is on the COCO photos used in this post:

dotnet run -- ~/coco-photos

Indexing /home/me/coco-photos, this takes a moment the first time...
Indexed 1000 photos, skipped 0

Search: a dog on a beach
  0.311  /home/me/coco-photos/000000052891.jpg
  0.227  /home/me/coco-photos/000000050326.jpg
  ...

The index is saved beside the photos as .photos.kjimg, so the next run starts searching at once. Delete that file to index the folder again. An empty line quits.

The other ways to do it

• A cloud vision API tags or captions each photo for you. Every photo is uploaded, you pay per image, and the search only knows the tags the service chose.
• A Python service with PyTorch and Faiss next to your .NET app. It works, but you now deploy and run a second runtime.
• CLIP exported to ONNX, run with ONNX Runtime. You still write the query tokenizer, the image resize, crop and normalisation, and the search yourself, and they have to match how the model was trained.

ImageIndex does all of that in your process: it decodes JPEG and PNG, prepares each image the way CLIP expects, tokenizes the query and ranks by cosine similarity.

How CLIP finds a photo from a sentence

CLIP is two encoders trained together on hundreds of millions of image and caption pairs: one turns an image into a vector, the other turns text into a vector of the same size. They were trained so that a photo and a sentence describing it end up pointing in nearly the same direction.

photo  -> resize, crop 224x224 -> vision encoder -> 512 numbers ┐
                                                                ├─ cosine similarity
"a dog on a beach" -> tokenizer -> text encoder -> 512 numbers  ┘

So the photos are embedded once, when you index them, and only the query is embedded at search time. Nothing reads filenames, EXIF tags or captions. The pixels are all it uses.

Indexing a folder

using var index = new ImageIndex();

var report = index.AddDirectory(photosDir);
Console.WriteLine($"indexed {report.Added}, skipped {report.Skipped}");

index.Save("photos.kjimg");

AddDirectory walks subfolders, indexes every JPEG and PNG, and returns how many it added and how many it could not read. A corrupt file is skipped and counted, not thrown, so one broken download does not stop a scan of the whole library.

On an Intel i7-13700 (24 threads), with the model already cached:

Indexing uses every core: photos are decoded and embedded in parallel. Run it on a background thread in a desktop app, because a large library takes minutes.

Save writes the vectors and paths, not the images. Next time, Load the file instead of re-embedding everything:

using var index = new ImageIndex();
if (File.Exists("photos.kjimg"))
    index.Load("photos.kjimg");
else
    index.AddDirectory(photosDir);

To add one new photo, call index.Add(path). Adding the same path again replaces its vector rather than adding a duplicate.

Searching

foreach (var hit in index.Search("a red double decker bus", 3))
    Console.WriteLine($"{hit.Score:F3}  {Path.GetFileName(hit.Path)}");

Here are the top results for a few queries. The description column is what a human wrote about that photo in COCO's annotations. The search never saw those captions; they are here so you can tell whether the hit is right.

The last query shows what to expect when it misses. The top hit is a sleeping cat, but not on a laptop. The two right answers are one thousandth behind it. Show users several results, not just the first.

What the scores mean

The scores are cosine similarities, and with CLIP they are small even for a perfect match: the hits above are all between 0.26 and 0.36. The order matters, but a score does not tell you how confident to be. Do not set a cut-off like "only show results above 0.5"; nothing will pass. If you need to hide weak results, compare each score with the top score of the same query rather than with a fixed number.

How good is it?

Picking queries that work is easy. To measure it, I used COCO's own captions: for each of the 1,000 photos, search with its first human caption and see where that photo ranks among all 1,000 indexed.

This is a strict test. A caption like "A person that is going out in the water" fits dozens of photos in the set, and when another surfer outranks the one it was written about, the test counts a miss even though a user would be happy. On a larger library the photo you want drops further down for the same reason, so a results grid of 20 or so works better than a list of 3.

A search over the 1,000 photos took 30 ms on average across the 1,000 searches of this test. Most of it is encoding the query with CLIP's text encoder, so it grows slowly with the library: an index of 100,000 entries searched in about 60 ms.

What it does not do

• Only JPEG and PNG. HEIC from iPhones, WebP, RAW and GIF are not read, and AddDirectory leaves them out of Skipped too, which only counts JPEG and PNG files that failed. Compare Added with your own count of files, and convert the others first.
• Black-and-white JPEGs need 0.4.2 or newer. Earlier versions skip them and count them in Skipped.
• No "find photos like this one". Search takes text. There is no image-to-image search in the C# API yet.
• English queries. This CLIP model was trained on English captions, so write queries in English.
• No reading of text in images. CLIP sometimes matches a word written in a photo, but do not rely on it to find a receipt by its total or a sign by what it says.
• No removal. There is no method to take one photo out of the index. If photos are deleted, check File.Exists(hit.Path) before showing a hit, or rebuild the index.

Searching is safe from several threads at once. Adding is not: do not call Add or AddDirectory while another thread searches.

FAQ

Does it need an internet connection?

Only the first time, to download the model. After that, indexing and searching run offline, and no photo or query leaves the machine.

Does it need a GPU or Python?

No. It runs on the CPU, inside your .NET process. The NuGet package holds the native library for Windows x64, Linux x64 and arm64, and macOS on Apple Silicon.

Which .NET versions does it work with?

The package targets .NET 8 and .NET Standard 2.1. These numbers were measured on .NET 8.

How many photos can it handle?

I tested a saved index of 100,000 entries. It loaded in 1.4 s, a search took about 60 ms, and the process peaked at 1.5 GB of memory. The saved file is large, 690 MB, because the vectors are stored as text. Indexing is the slow part: at 19 ms per photo on a 24-thread desktop, 100,000 photos takes about 32 minutes, once.

Which CLIP model is it?

LAION's ViT-B/32, trained on LAION-2B (laion/CLIP-ViT-B-32-laion2B-s34B-b79K). Pass a directory to new ImageIndex(modelDir) to use another CLIP checkpoint in the same format.

About the test photos

The photos are the first 1,000 images, by ID, of the COCO 2017 validation set. The captions are COCO's. The images are Flickr photos under their owners' Creative Commons licences, which is why this post shows captions rather than the photos themselves. See cocodataset.org.

Getting it

NuGet:   dotnet add package Kjarni
Package: https://www.nuget.org/packages/Kjarni
GitHub:  https://github.com/olafurjohannsson/kjarni
Docs:    https://kjarni.ai/docs/

Kjarni is MIT licensed. Questions and bug reports go to GitHub issues.

Related

• Semantic Search in C#: the same idea for text
• Document Search in C# without Elasticsearch: keyword, semantic and hybrid search over files
• Local Embeddings for Microsoft.Extensions.AI: text embeddings behind IEmbeddingGenerator
• ImageIndex reference