kjarni::generation

Generation configuration and resolution.

This module handles the merging of generation parameters from multiple sources: model defaults, user preferences, and runtime overrides.

GenerationOverrides

pub struct GenerationOverrides

User- or runtime-provided overrides for text generation

Fields

• pub temperature: Option<f32>,: Sampling temperature (0.0 = deterministic, higher = more random).
• pub top_k: Option<usize>,: Limit sampling to top K tokens.
• pub top_p: Option<f32>,: Nucleus sampling: consider tokens with cumulative probability <= top_p.
• pub min_p: Option<f32>,: Minimum probability threshold for sampling.
• pub repetition_penalty: Option<f32>,: Penalty for repeating tokens (1.0 = no penalty).
• pub no_repeat_ngram_size: Option<usize>,: Prevent repeating n-grams of this size.
• pub max_new_tokens: Option<usize>,: Maximum number of new tokens to generate.
• pub do_sample: Option<bool>,: Sampling vs deterministic decoding. - Some(false) => Force greedy decoding - Some(true) => Force sampling - None => Keep model default
• pub num_beams: Option<usize>,: Number of beams for beam search (> 1 enables beam search).
• pub length_penalty: Option<f32>,: Length penalty for beam search (< 1 favors shorter, > 1 favors longer).

greedy

pub fn greedy() -> Self

Create overrides for greedy decoding.

creative

pub fn creative() -> Self

Create overrides for creative generation.

precise

pub fn precise() -> Self

Create overrides for precise/factual generation.

is_empty

pub fn is_empty(&self) -> bool

Check if any overrides are set.

ResolvedGenerationConfig

pub struct ResolvedGenerationConfig

Fully resolved generation configuration.

This wraps the engine's GenerationConfig after all overrides have been applied. It's immutable once created.

new

pub fn new(config: GenerationConfig) -> Self

Create from a resolved config.

into_inner

pub fn into_inner(self) -> GenerationConfig

Consume and return the inner config.

as_ref

pub fn as_ref(&self) -> &GenerationConfig

Get a reference to the inner config.

max_new_tokens

pub fn max_new_tokens(&self) -> Option<usize>

Get the max new tokens setting.

is_sampling

pub fn is_sampling(&self) -> bool

Check if using sampling strategy.

is_beam_search

pub fn is_beam_search(&self) -> bool

Check if using beam search.

is_greedy

pub fn is_greedy(&self) -> bool

Check if using greedy decoding.

GenConfig

pub type GenConfig = GenerationOverrides

resolve_generation_config

pub fn resolve_generation_config(model_defaults: GenerationConfig, user: &GenerationOverrides, runtime: &GenerationOverrides) -> ResolvedGenerationConfig

Merge model defaults