Models and formats

Kinetic reads GGUF files as llama.cpp and the Hugging Face GGUF tools write them, and its own .ktc files. It does not convert anything: you point it at a file on your disk.

The quickest check for a file is kinetic info:

kinetic info Qwen3.5-4B-MTP-Q4_K_M.gguf

Qwen3.5-4B-MTP-Q4_K_M.gguf
  name           Qwen3.5-4B
  architecture   qwen35
  size           2.64 GiB, 4.33 B parameters
  context        262144 tokens
  layers         33, width 2560
  prediction     yes (drafts for speculative decoding)
  weights        Q4_K    149 tensors    1478.0 MiB
                 Q6_K     35 tensors    1018.2 MiB
                 Q5_K     24 tensors     165.0 MiB
                 F32     232 tensors      18.7 MiB
                 Q8_0      1 tensors      13.3 MiB
  tokenizer      gpt2 (qwen35), 248320 tokens, BOS none, EOS <|im_end|>
  chat template  7992 bytes

An architecture Kinetic does not know is marked (not supported yet). A known architecture can still use a feature that is not supported yet, such as a kind of rotary scaling; kinetic run then names it when it loads the file.

Architectures

The names are the GGUF general.architecture values.

With --device auto, a model that cannot run on the GPU runs on the CPU and Kinetic prints why. Through Vulkan (--device vulkan), llama, qwen2, qwen3, lfm3 and qwen35 models run today; see Devices.

Embedding and ranking models

kinetic serve answers /v1/embeddings for these and /v1/rerank for rerankers. They run on the CPU.

A file is served as an embedding model when it is an encoder or its metadata sets a pooling type (<arch>.pooling_type).

Vision

With kinetic serve <model> --mmproj <projector>, chat requests may carry images. The projector file names its vision tower in clip.projector_type (or clip.vision.projector_type when it also holds an audio tower):

When the model runs on a CUDA GPU, images are encoded there too. A prompt with images runs on the GPU when every layer of the model is there, and on the CPU otherwise. More projectors are coming.

Adding a projector type takes one file in crates/kinetic-vision/src/:

1. A struct holding the weights, borrowed from the file through Loader.
2. A load function that reads the hyperparameters and tensors and returns the struct boxed as a Projector.
3. impl Projector: n_embd_out, prepare (resize and normalize the way the model's processor does) and encode (pixels to one row per image token, row-major over the token grid).
4. One line in PROJECTORS in lib.rs mapping the clip.projector_type string to load.

Shared pieces live in ops.rs (linear, layer and RMS norms, GELU, bidirectional attention, patch gathering) and preprocess.rs (decoding, the aspect-keeping resize). Check a new type against llama.cpp's llama-mtmd-debug --image rainbow -n <size> with cargo run --release -p kinetic-vision --example encode -- <mmproj> rainbow:<size>: the first and last values and the sum of the final embeddings should match to a few digits. On the text side, an image's rows enter the prompt as Input::Image; models with (time, row, column) rotary positions place them on the image grid, every other model gives them consecutive positions.

Tokenizers

The tokenizer is read from the file. Kinetic has byte-level BPE (tokenizer.ggml.model = gpt2), SentencePiece (llama) and WordPiece (bert). BPE models also name a pre-tokenizer (tokenizer.ggml.pre); these are supported: qwen35, qwen2, llama3, llama-v3, llama-bpe, lfm2, falcon-h1, dbrx, smollm, refact, gpt-2, exaone4 and hunyuan-dense. kinetic info names a missing one on its tokenizer line.

Chat templates and tool calls

The chat template is the file's own Jinja template, rendered as the model's authors wrote it, or the one --chat-template-file names. Tool calls are read in the formats of Qwen3.5, Qwen3, LFM3, LFM2, Llama 3 and Mistral.

Weight formats

These are the GGUF tensor types, by the names kinetic info prints. A file mixes several (Q4_K_M, for one, is mostly Q4_K with some Q6_K).

"Experts only" means the type runs on the GPU as the routed experts of a mixture-of-experts model; in any other tensor it keeps the model on the CPU. The format definitions follow ggml's, so a GGUF file means the same in Kinetic as in llama.cpp.

On the CPU, AVX2 kernels cover the common formats; any other type still runs, through a slower path that dequantizes each row.

Files that do not load

Split files (model-00001-of-00003.gguf) are not read yet; join them first with llama.cpp's llama-gguf-split --merge.

If a model you need is missing, open a "model not supported" issue with the output of kinetic info, or add it: see Adding an architecture.

.ktc files

A .ktc file holds everything needed to run a model, weights, tokenizer and chat template, with a manifest that records the BLAKE3 hash of every section, an optional Ed25519 signature over the manifest, and an execution profile tuned for a kind of hardware. Every byte of the file is either the header, the manifest, zero padding or a hashed section, so nothing can hide in it.

kinetic verify model.ktc

recomputes every hash and reports whether the file is signed; it exits with 1 when anything does not match. Every command that takes a .gguf file takes a .ktc file too. The format is specified in FORMAT.md and versioned on its own; see Versioning.