GitHub Install

Rust API reference · Kjarni 0.2.8

kjarni::chat

High-level chat API for conversational AI.

This module provides the primary interface for interacting with chat/instruction-tuned language models.

Quick Start

use kjarni::chat;

// Ultra-simple one-liner
let response = chat::send("llama3.2-1b", "Hello!").await?;

// Or with more control
use kjarni::chat::Chat;

let chat = Chat::new("llama3.2-1b").await?;
let response = chat.send("Hello!").await?;

Multi-turn Conversations

use kjarni::chat::Chat;

let chat = Chat::new("llama3.2-1b").await?;

// Stateful conversation
let mut convo = chat.conversation();
convo.send("Hello!").await?;
convo.send("What did I just say?").await?;

// Or stateless with manual history
use kjarni::chat::History;

let mut history = History::with_system("You are helpful.");
history.push_user("Hello");
history.push_assistant("Hi!");

let response = chat.send_with_history(&history, "Continue").await?;

Using Presets

use kjarni::chat::{Chat, presets::CHAT_SMALL_V1};

let chat = Chat::from_preset(&CHAT_SMALL_V1)
    .system_prompt("You are a pirate.")
    .build()
    .await?;

Streaming

use futures::StreamExt;
use kjarni::chat::Chat;

let chat = Chat::new("llama3.2-1b").await?;
let mut stream = chat.stream("Tell me a story.").await?;

while let Some(token) = stream.next().await {
    print!("{}", token?);
}

Chat

pub struct Chat

High-level chat interface for conversational AI.

Chat wraps a Generator and adds chat-specific functionality:

  • Chat template formatting
  • System prompts
  • Conversation history management
  • Chat modes (default, creative, reasoning)

Example

use kjarni::chat::Chat;

// Simple one-shot message
let chat = Chat::new("llama3.2-1b-instruct").await?;
let response = chat.send("What is Rust?").await?;
println!("{}", response);

// With system prompt and history
let chat = Chat::builder("llama3.2-1b-instruct")
    .system("You are a helpful coding assistant.")
    .mode(ChatMode::Reasoning)
    .build()
    .await?;

// Multi-turn conversation
let mut convo = chat.conversation();
let r1 = convo.send("What is a closure in Rust?").await?;
let r2 = convo.send("Can you give me an example?").await?;

from_preset

pub fn from_preset(preset: &ChatPreset) -> ChatBuilder

Create a builder from a preset.

new

pub async fn new(model: &str) -> ChatResult<Self>

Create a Chat with default settings.

builder

pub fn builder(model: &str) -> ChatBuilder

Create a builder for custom configuration.

from_builder

pub async fn from_builder(builder: ChatBuilder) -> ChatResult<Self>

Internal: construct from builder.

format_prompt

pub fn format_prompt(&self, conversation: &Conversation) -> String

Format a conversation using the chat template.

history_to_conversation

pub fn history_to_conversation(&self, history: &History) -> Conversation

Convert History to Conversation.

send

pub async fn send(&self, message: &str) -> ChatResult<String>

Send a message and get a response.

send_with_history

pub async fn send_with_history(&self, history: &History, message: &str) -> ChatResult<String>

Send with conversation history.

The history is used for context but not modified.

send_with_config

pub async fn send_with_config(&self, message: &str, overrides: &GenerationOverrides) -> ChatResult<String>

Send with custom generation overrides.

stream

pub async fn stream(&self, message: &str) -> ChatResult<std::pin::Pin<Box<dyn Stream<Item = ChatResult<String>> + Send>>>

Stream a response token by token.

Example

use futures::StreamExt;

let mut stream = chat.stream("Tell me a story.").await?;
while let Some(token) = stream.next().await {
    print!("{}", token?);
}

stream_with_history

pub async fn stream_with_history(&self, history: &History, message: &str) -> ChatResult<std::pin::Pin<Box<dyn Stream<Item = ChatResult<String>> + Send>>>

Stream with conversation history.

stream_with_config

pub async fn stream_with_config(&self, message: &str, overrides: GenerationOverrides) -> ChatResult<std::pin::Pin<Box<dyn Stream<Item = ChatResult<String>> + Send>>>

Stream with custom overrides.

conversation

pub fn conversation(&self) -> ChatConversation<'_>

Create a stateful conversation.

The conversation maintains history automatically.

conversation_with_system

pub fn conversation_with_system(&self, system: impl Into<String>) -> ChatConversation<'_>

Create a stateful conversation with a custom system prompt.

generate

pub async fn generate(&self, prompt: &str, runtime_overrides: &GenerationOverrides) -> ChatResult<String>

Generate using the inner generator.

generate_stream

pub async fn generate_stream(&self, prompt: String, runtime_overrides: GenerationOverrides) -> ChatResult<std::pin::Pin<Box<dyn Stream<Item = ChatResult<String>> + Send>>>

Generate streaming using channel approach.

model_type

pub fn model_type(&self) -> ModelType

Get the model type.

model_name

pub fn model_name(&self) -> &str

Get the model's CLI name.

mode

pub fn mode(&self) -> ChatMode

Get the current chat mode.

device

pub fn device(&self) -> Device

Get the device the model is running on.

context_size

pub fn context_size(&self) -> usize

Get the context window size.

system_prompt

pub fn system_prompt(&self) -> Option<&str>

Get the system prompt.

generator

pub fn generator(&self) -> &Generator

Get a reference to the inner generator.

For advanced use cases that need direct access.

ChatBuilder

pub struct ChatBuilder

Builder for configuring a Chat instance.

Example

let chat = Chat::builder("llama3.2-1b-instruct")
    .system("You are a helpful assistant.")
    .mode(ChatMode::Creative)
    .temperature(0.8)
    .build()
    .await?;

new

pub fn new(model: impl Into<String>) -> Self

Create a new builder for the specified model.

from_preset

pub fn from_preset(preset: &ChatPreset) -> Self

Create a builder from a preset.

system

pub fn system(mut self, prompt: impl Into<String>) -> Self

Set the system prompt.

mode

pub fn mode(mut self, mode: ChatMode) -> Self

Set the chat mode.

creative

pub fn creative(self) -> Self

reasoning

pub fn reasoning(self) -> Self

device

pub fn device(mut self, device: KjarniDevice) -> Self

cpu

pub fn cpu(self) -> Self

gpu

pub fn gpu(self) -> Self

Use GPU for inference.

with_context

pub fn with_context(mut self, context: Arc<WgpuContext>) -> Self

Provide a pre-created GPU context.

draft

pub fn draft(mut self, name: impl Into<String>, num_tokens: usize) -> Self

Set the sampling temperature. Enables speculative decoding with name as the draft model.

The draft must share this model's vocabulary — a smaller model of the same family. Measured on Qwen2.5, 0.5B drafting for 1.5B is 3.5x faster at eight proposed tokens, because verifying a batch reads the target's weights once instead of once per token.

prefix_cache

pub fn prefix_cache(mut self, enabled: bool) -> Self

Reuses the KV cache across turns, so a conversation prefills only the new message instead of replaying its whole history.

Every turn re-sends the transcript, so turn N normally pays to prefill everything said before it. With this on, only the tail is new: on Qwen2.5-0.5B a 2048 token prompt costs 15.96s cold and 0.49s with 1984 tokens already cached, for identical logits.

Holds [DEFAULT_PREFIX_CACHE_TOKENS] tokens; use prefix_cache_tokens to choose. Longer conversations still work, they simply stop reusing once they outgrow it.

Off by default: the KV cache is eagerly allocated f32, roughly 24KB per token on Qwen2.5-0.5B, and one cache holds one conversation, so sends on a single instance serialise. Ignored while speculative decoding is active.

prefix_cache_tokens

pub fn prefix_cache_tokens(mut self, tokens: usize) -> Self

Enables prefix reuse with an explicit capacity in tokens.

Capacity is the longest transcript that can be reused, and what the memory cost scales with. Clamped to the model's context.

greedy

pub fn greedy(mut self) -> Self

Use greedy decoding (temperature = 0, deterministic).

Setting temperature(0.0) alone is not enough: without do_sample = false the sampler still runs, and two identical calls can differ.

temperature

pub fn temperature(mut self, temp: f32) -> Self

Set the sampling temperature.

max_tokens

pub fn max_tokens(mut self, max: usize) -> Self

Set the maximum number of tokens to generate.

top_p

pub fn top_p(mut self, p: f32) -> Self

Set top-p (nucleus) sampling threshold.

top_k

pub fn top_k(mut self, k: usize) -> Self

Set top-k sampling limit.

min_p

pub fn min_p(mut self, p: f32) -> Self

Set min-p sampling threshold.

repetition_penalty

pub fn repetition_penalty(mut self, penalty: f32) -> Self

Set repetition penalty.

generation_config

pub fn generation_config(mut self, overrides: GenerationOverrides) -> Self

Set all generation overrides at once.

with_load_config

pub fn with_load_config<F>(mut self, f: F) -> Self where F: FnOnce(LoadConfigBuilder) -> LoadConfigBuilder,

Configure model loading options.

cache_dir

pub fn cache_dir(mut self, path: impl Into<PathBuf>) -> Self

Set the cache directory for model files.

download_policy

pub fn download_policy(mut self, policy: DownloadPolicy) -> Self

Set the download policy.

offline

pub fn offline(mut self) -> Self

Never download models automatically.

quiet

pub fn quiet(mut self) -> Self

Suppress informational output.

allow_suboptimal

pub fn allow_suboptimal(mut self) -> Self

Allow suboptimal model choices without warnings.

build

pub async fn build(self) -> ChatResult<Chat>

Build the Chat instance.

ChatConversation

pub struct ChatConversation<'a>

A stateful conversation that maintains history automatically.

Example

let chat = Chat::new("llama3.2-1b-instruct").await?;
let mut convo = chat.conversation();

// Blocking send (adds to history automatically)
let response = convo.send("What is Rust?").await?;

// Streaming (must manually add response to history)
convo.push_user("Tell me more.");
let mut stream = convo.stream_next().await?;
let mut response = String::new();
while let Some(token) = stream.next().await {
    let text = token?;
    print!("{}", text);
    response.push_str(&text);
}
convo.push_assistant(&response);

send

pub async fn send(&mut self, message: &str) -> ChatResult<String>

Send a message and get a response.

Both the user message and assistant response are added to history.

send_with_config

pub async fn send_with_config(&mut self, message: &str, overrides: &GenerationOverrides) -> ChatResult<String>

Send with custom generation overrides.

push_user

pub fn push_user(&mut self, message: &str)

Add a user message to history (for streaming workflow).

push_assistant

pub fn push_assistant(&mut self, response: &str)

Add an assistant response to history (for streaming workflow).

stream_next

pub async fn stream_next(&self) -> ChatResult<std::pin::Pin<Box<dyn Stream<Item = ChatResult<String>> + Send>>>

Stream the next response based on current history.

Important: You must call push_user() before this, and push_assistant() after collecting the full response.

Example

convo.push_user("Hello!");
let mut stream = convo.stream_next().await?;
let mut response = String::new();
while let Some(token) = stream.next().await {
    response.push_str(&token?);
}
convo.push_assistant(&response);

stream

pub async fn stream(&mut self, message: &str) -> ChatResult<std::pin::Pin<Box<dyn Stream<Item = ChatResult<String>> + Send>>>

Convenience: Push user message and stream response.

Returns the stream. You must still call push_assistant() after.

history

pub fn history(&self) -> &History

Get the conversation history.

history_mut

pub fn history_mut(&mut self) -> &mut History

Get mutable access to history.

clear

pub fn clear(&mut self, keep_system: bool)

Clear the conversation history.

If keep_system is true, the system prompt is preserved.

len

pub fn len(&self) -> usize

Get the number of messages in history.

is_empty

pub fn is_empty(&self) -> bool

Check if history is empty.

ChatPreset

pub struct ChatPreset

Preset configuration for chat models.

Fields

  • pub model: &'static str,: Model name from registry.
  • pub recommended_device: KjarniDevice,: Recommended device.
  • pub mode: ChatMode,: Default chat mode.
  • pub system_prompt: Option<&'static str>,: Default system prompt (if any).
  • pub temperature: Option<f32>,: Recommended temperature.
  • pub max_tokens: Option<usize>,: Recommended max tokens.
  • pub description: &'static str,: Human-readable description.

ChatWarning

pub struct ChatWarning

Fields

  • pub message: String,
  • pub severity: WarningSeverity,

new

pub fn new(message: impl Into<String>) -> Self

with_severity

pub fn with_severity(message: impl Into<String>, severity: WarningSeverity) -> Self

History

pub struct History

new

pub fn new() -> Self

with_system

pub fn with_system(system: impl Into<String>) -> Self

push_user

pub fn push_user(&mut self, content: impl Into<String>)

push_assistant

pub fn push_assistant(&mut self, content: impl Into<String>)

messages

pub fn messages(&self) -> &[Message]

len

pub fn len(&self) -> usize

is_empty

pub fn is_empty(&self) -> bool

clear

pub fn clear(&mut self, keep_system: bool)

Message

pub struct Message

Fields

  • pub role: Role,
  • pub content: String,

system

pub fn system(content: impl Into<String>) -> Self

user

pub fn user(content: impl Into<String>) -> Self

assistant

pub fn assistant(content: impl Into<String>) -> Self

ChatDevice

pub enum ChatDevice

Variants

  • Cpu,
  • Gpu,
  • Auto,

resolve

pub fn resolve(&self) -> ChatDevice

resolve_to_device

pub fn resolve_to_device(&self) -> kjarni_transformers::traits::Device

ChatError

pub enum ChatError

Variants

  • UnknownModel(String),
  • ModelNotDownloaded(String),
  • DownloadFailed
  • LoadFailed
  • GpuUnavailable,
  • GenerationFailed(#[from] anyhow::Error),
  • NoChatTemplate(String),
  • IncompatibleModel
  • InvalidModel(String, String),
  • InvalidConfig(String),

ChatMode

pub enum ChatMode

Variants

  • Default,
  • Creative,
  • Reasoning,

default_temperature

pub fn default_temperature(&self) -> f32

default_max_tokens

pub fn default_max_tokens(&self) -> usize

ChatResult

pub type ChatResult<T> = Result<T, ChatError>

ChatTier

pub enum ChatTier

Variants

  • Fast,: Fastest, smallest model.
  • Balanced,: Balanced speed and quality.
  • Quality,: Best quality, may need GPU.

preset

pub fn preset(&self) -> &'static ChatPreset

Get the preset for this tier.

resolve

pub fn resolve(&self) -> &'static ChatPreset

DownloadPolicy

pub enum DownloadPolicy

Policy for downloading models.

Variants

  • IfMissing,: Download model if not present locally (default).
  • Never,: Never download, fail if model not present.
  • Eager,: Always check for updates and download if newer.

Role

pub enum Role

Variants

  • System,
  • User,
  • Assistant,

available_models

pub fn available_models() -> Vec<&'static str>

List all available chat-capable models.

Returns CLI names of models that can be used with Chat.

is_chat_model

pub fn is_chat_model(model: &str) -> ChatResult<()>

Check if a model is valid for chat.

Returns Ok(()) if valid, or an error describing why not.

Example

if kjarni::chat::is_chat_model("llama3.2-1b").is_ok() {
    println!("Model can be used for chat");
}

send

pub async fn send(model: &str, message: &str) -> ChatResult<String>

Send a single message and get a response.

Example

let response = kjarni::chat::send("llama3.2-1b", "What is Rust?").await?;
println!("{}", response);

send_preset

pub async fn send_preset(preset: &ChatPreset, message: &str) -> ChatResult<String>

Send a message using a preset.

Example

use kjarni::chat::presets::CHAT_SMALL_V1;

let response = kjarni::chat::send_preset(&CHAT_SMALL_V1, "Hello!").await?;

send_tier

pub async fn send_tier(tier: ChatTier, message: &str) -> ChatResult<String>

Send a message using a tier (auto-selects best model for that tier).

Example

use kjarni::chat::ChatTier;

let response = kjarni::chat::send_tier(ChatTier::Small, "Hello!").await?;

suggested_models

pub fn suggested_models() -> Vec<&'static str>

Get suggested models for chat.

chat::model

Core Chat implementation using Generator.

send

pub async fn send(model: &str, message: &str) -> ChatResult<String>

Send a single message with default settings.

Example

let response = kjarni::chat::send("llama3.2-1b-instruct", "Hello!").await?;

chat::types

Chat types and error definitions.

WarningSeverity

pub enum WarningSeverity

Variants

  • Info,
  • Warning,
  • Important,

chat::validation

Model validation for chat compatibility.

Provides tiered validation:

  • Hard errors for capability mismatches (encoder used for chat)
  • Warnings for suboptimal configurations (base model for chat)

ValidationResult

pub struct ValidationResult

Result of validating a model for chat use.

Fields

  • pub is_valid: bool,: Model passed validation (may still have warnings).
  • pub warnings: Vec<ChatWarning>,: Warnings about suboptimal configuration.

is_decoder_architecture

pub fn is_decoder_architecture(arch: ModelArchitecture) -> bool

Check if a model architecture supports text generation.

is_instruct_model

pub fn is_instruct_model(model_type: ModelType) -> bool

Check if a model is instruction-tuned.

suggest_chat_models

pub fn suggest_chat_models() -> Vec<&'static str>

Get suggested models for chat, smallest first.

A curated shortlist rather than everything [crate::chat::available_models] returns, since the point is to give someone a starting point rather than a catalogue.

Written as [ModelType] variants, not string literals, on purpose: the compiler rejects a rename, and cli_name() keeps the strings in sync by construction, so a suggestion never leads to a second "unknown model" error.

validate_for_chat

pub fn validate_for_chat(model_type: ModelType) -> ChatResult<ValidationResult>

Validate that a model can be used for chat.

Returns Ok(ValidationResult) if the model can be used (possibly with warnings), or Err(ChatError) if the model fundamentally cannot perform chat.

Kjarni runs the same engine on every platform:

GitHub NuGet (C#) npm (WebAssembly) Go module