Local AI in Unity: Embeddings, Moderation and LLM NPCs, Offline in C#

A game that reads what the player types has three usual options. Send it to a cloud API, and the game now needs a network connection, a key it cannot keep secret, and a bill that grows with the player count. Use a wrapper around llama.cpp, which is built around chat models. Or convert a model to ONNX for Unity's own inference engine, and then write the tokenizer, the pooling and the pre- and post-processing yourself.

Kjarni is a fourth: one native library that runs the models in your process, tokenizer included, with a C# binding built for Unity. This post does three things with it, each a few lines of C#, with numbers from a real run:

• turn free-text player commands into game actions by meaning, not keywords
• flag toxic chat messages
• give an NPC a voice with a local LLM

Setup

You need Unity 2021.2 or later, with Api Compatibility Level set to .NET Standard 2.1 under Project Settings, Player, Other Settings, and Kjarni 0.2.4 or later. From 0.2.4 the package has no dependencies, so installing it means copying two files out of the NuGet package, which is a zip archive:

lib/netstandard2.1/Kjarni.dll           ->  Assets/Plugins/Kjarni/Kjarni.dll
runtimes/win-x64/native/kjarni_ffi.dll  ->  Assets/Plugins/Kjarni/x86_64/kjarni_ffi.dll

Select kjarni_ffi.dll and enable it in the Inspector for the Editor and for your standalone target. NuGetForUnity works too. The package also carries native libraries for Linux x64 and arm64 and for macOS on Apple Silicon, under runtimes/. Kjarni on Windows covers every file and what it is for.

Keep it off the main thread

Loading a model reads tens or hundreds of megabytes from disk, and the first time, downloads them. Inference uses every physical core for the length of the call. Neither belongs in Update. Load on a background thread and come back to the main thread with await, which Unity's synchronisation context makes safe:

using System.Threading.Tasks;
using Kjarni;
using UnityEngine;

public class LanguageServices : MonoBehaviour
{
    public Embedder Embedder { get; private set; }

    private async void Awake()
    {
        var cache = Application.persistentDataPath + "/kjarni";
        Embedder = await Task.Run(() => new Embedder("minilm-l6-v2", cacheDir: cache, quiet: true));
    }

    private void OnDestroy() => Embedder?.Dispose();
}

A handle is not safe to use from two threads at once. One handle per system, used from one task at a time, is the simple rule.

Free-text commands by meaning

The player types unlock the gate. Your game has an action called open the door. They share no words, and a keyword parser misses it. A sentence embedding model turns each into a vector, and texts that mean similar things get vectors pointing in similar directions, so the match is one comparison.

The first attempt, one phrase per action, shows the catch:

unlock the gate                   -> open the door          0.5069
hit the goblin with my sword      -> attack the enemy       0.4391
grab that sword off the floor     -> pick up the item       0.4125
ask the shopkeeper what he sells  -> talk to the merchant   0.5538
get out of here                   -> open the door          0.4459
sing a song                       -> attack the enemy       0.2652

Four are right. Get out of here went to open the door, because a single phrase is a thin description of an intent and "out" sits close to doors. And sing a song, which is nothing the game supports, still matched something, because the best score always wins even when it is bad.

Both have the same fix: describe each action with a few phrasings and score against the closest one, and refuse anything below a threshold.

using System.Collections.Generic;
using System.Linq;
using Kjarni;

public class CommandParser
{
    private static readonly Dictionary<string, string[]> Examples = new()
    {
        ["open"]   = new[] { "open the door", "unlock the gate", "go through the doorway" },
        ["attack"] = new[] { "attack the enemy", "hit him", "fight the monster" },
        ["pickup"] = new[] { "pick up the item", "take that", "grab the loot" },
        ["talk"]   = new[] { "talk to the merchant", "speak with the shopkeeper", "ask what he sells" },
        ["flee"]   = new[] { "run away", "get out of here", "escape", "retreat" },
    };

    private readonly Embedder _embedder;
    private readonly string[] _actions;
    private readonly float[][] _vectors;

    public CommandParser(Embedder embedder)
    {
        _embedder = embedder;
        _actions = Examples.SelectMany(kv => kv.Value.Select(_ => kv.Key)).ToArray();
        // Encode every example once, at load time.
        _vectors = embedder.EncodeBatch(Examples.SelectMany(kv => kv.Value).ToArray());
    }

    /// <summary>The matching action, or null if nothing is close enough.</summary>
    public string Parse(string input, float threshold = 0.4f)
    {
        var q = _embedder.Encode(input);
        var best = _actions
            .Select((action, i) => (action, score: Embedder.CosineSimilarity(q, _vectors[i])))
            .OrderByDescending(x => x.score)
            .First();
        return best.score >= threshold ? best.action : null;
    }
}

None of the test inputs below are in the examples:

smash the lock and go in     -> open    0.5949
stab the goblin              -> attack  0.5027
loot the chest               -> pickup  0.6611
what do you have for sale?   -> talk    0.5131
let's get out of here        -> flee    0.7834
flee!                        -> flee    0.5802
sing a song                  -> (none)  0.2652
what's the weather like      -> (none)  0.1195

Everything lands, and the two things the game cannot do are refused instead of guessed at. When the parser returns null, the game says it did not understand, which is better than attacking a merchant.

On an i7-13700, encoding one command takes 2.4 ms and loading the model 52 ms once it is on disk. That is cheap enough for every command the player submits, and too expensive to run every frame, which is also never needed. minilm-l6-v2 is 88 MB.

Chat moderation

toxic-bert scores a message on six labels at once, so you can treat an insult differently from a threat:

using var toxic = new Classifier("toxic-bert", quiet: true);

var result = toxic.Classify("you are a worthless idiot, uninstall");
foreach (var (label, score) in result.AllScores)
    Debug.Log($"{label}: {score:F3}");

"gg, well played everyone"              toxic 0.002
"that boss fight was brutal"            toxic 0.015
"you are a worthless idiot, uninstall"  toxic 0.984, insult 0.944, obscene 0.665,
                                        severe_toxic 0.030, identity_hate 0.010, threat 0.001

Brutal in the context of a boss fight scores 0.015: the model reads the sentence, not a word list. It is 419 MB and loads in about 0.4 seconds.

Running moderation on the player's own machine has an obvious limit in multiplayer: the client is in the player's hands, so it cannot be the only line of defence. What it can do is warn before a message is sent, filter what each player sees on their own screen, and keep a single-player or local-network game working with no server at all.

An NPC with a local LLM

Chat runs an instruction-tuned model with a system prompt and streams tokens as they are produced:

using var npc = new Chat("qwen2.5-0.5b-instruct",
    systemPrompt: "You are Bram, a gruff blacksmith in a small mountain village. " +
                  "Answer in one or two short sentences, in character.",
    quiet: true);

npc.Stream("Can you repair my sword?", GenerationConfig.Greedy(), token =>
{
    pending.Enqueue(token);   // a ConcurrentQueue<string>, drained in Update
    return true;              // false stops generation
});

The callback runs on whichever thread is generating, never on Unity's main thread, so it cannot touch a TextMeshPro component directly. Put the tokens in a ConcurrentQueue<string> and append them to the dialogue box from Update. Run the Stream call itself inside Task.Run, as with loading.

Model size is the honest trade-off here. With qwen2.5-0.5b-instruct, a 469 MB download:

Bram: Certainly! I can help you with that. Just let me know what's broken and how it
affects your weapon's performance.

That is 25 tokens in 1.6 seconds, and it is not a gruff blacksmith. It is a help desk. qwen2.5-1.5b-instruct, at 1.1 GB, holds the voice better:

Bram: Certainly! I can mend the blade and sharpen it to its former glory if that's what
you need. But first let me see your weapon so we know exactly how much work needs doing.

It produces its first token after about a second and the full reply in 6.3. Streaming makes that feel like an NPC talking rather than a game freezing, but it is still the pace of speech, not of a menu. Small local models work best on short, constrained lines with a tight system prompt. Long open-ended conversation is where they show their size.

Shipping the models with the game

By default models download on first use. For a game that should work offline from the first launch, ship them. Copy each model's folder out of the Kjarni cache (%LOCALAPPDATA%\kjarni on Windows, ~/.cache/kjarni on Linux) into Assets/StreamingAssets/kjarni/ and point cacheDir there:

var cache = Path.Combine(Application.streamingAssetsPath, "kjarni");
var embedder = new Embedder("minilm-l6-v2", cacheDir: cache, quiet: true);

I checked this with the folder made read-only and the network blocked: the model loads and gives the same 0.5069 for unlock the gate against open the door, while the same code pointed at an empty folder fails, because it can no longer download. Nothing is written to the folder, so a read-only install location is fine.

What it does not do yet

• Phones, consoles and WebGL. The native library is built for Windows x64, Linux x64 and arm64, and macOS on Apple Silicon. Android, iOS and WebGL builds have no native library to load.
• Limiting cores. A call uses every physical core while it runs. That is right for a load screen or a typed command and worth knowing for anything that runs during gameplay. There is no setting for it in the C# binding yet.

FAQ

Which Unity versions does Kjarni support?

Unity 2021.2 and later, with Api Compatibility Level set to .NET Standard 2.1. Earlier versions only offer .NET Standard 2.0, which the package does not target.

Does it work with IL2CPP?

It is built for it. The binding is written for both the Mono and IL2CPP scripting backends. The places where native code calls back into C#, chat token streaming and indexing progress, use static methods marked [MonoPInvokeCallback], which is what IL2CPP requires. Indexing progress gained this in 0.2.4.

Does it need an internet connection?

Only to download a model the first time, and not at all if you ship the model files in StreamingAssets and point cacheDir at them. Inference never uses the network.

Does it need a GPU?

No. Everything in this post runs on CPU. GPU support exists for the encoder models, through DirectX 12, Vulkan or Metal, so it does not need CUDA either.

How is this different from Unity's own inference engine?

Unity's engine runs ONNX models, so you convert the model first and then supply the tokenizer and the processing around the model yourself. Kjarni loads HuggingFace models as published and includes the tokenizer, pooling and generation loop, so embedder.Encode("unlock the gate") is the whole call.

Can I use it on Android or iOS?

Not yet. There are no native builds for mobile platforms or WebGL.

Getting it

NuGet:    https://www.nuget.org/packages/Kjarni
Releases: https://github.com/olafurjohannsson/kjarni/releases
GitHub:   https://github.com/olafurjohannsson/kjarni

Related

• Kjarni on Windows: every file, what it is for and where it comes from
• Semantic Search in C#: how embeddings and similarity work
• Sentiment Analysis in C#: the classifiers in more depth
• Local LLM Chat in C#: streaming, history and sampling