Kinetic

Kinetic: open engine for running AI locally. Runs your GGUF models today, and .ktc models verified and tuned for your hardware.

Kinetic runs language, embedding and reranking models on your own computer, from one binary with an OpenAI-compatible server. It is written in Rust, runs on Windows and Linux (x86-64), on the CPU, on NVIDIA GPUs through CUDA and on AMD and Intel GPUs through Vulkan. Kinetic never connects to the network, except when you run kinetic pull.

kinetic run Qwen3.5-4B-Q4_K_M.gguf "Explain recursion in two sentences."
kinetic serve Qwen3.5-4B-Q4_K_M.gguf     # OpenAI API at http://127.0.0.1:8080/v1

Why Kinetic

• Your files, unchanged. Kinetic reads GGUF files as llama.cpp writes them, from Qwen3.5, Gemma 4 and LFM3 to Llama, Gemma 3, Granite and Falcon-H1, in every common quantization.
• Faster without changing the answer. Speculative decoding, prompt caching and a shortcut through the output layer are on by default and exact: greedy output is token for token what plain decoding gives, and a seeded sampled answer is the same answer for the same seed. There is nothing to tune and no draft model to pick. See How Kinetic is fast.
• Large mixture-of-experts models on small GPUs. Experts that do not fit in video memory stay in system RAM, and Kinetic decides layer by layer where each one runs.
• Private by design. Kinetic never connects to the network, except when you run kinetic pull. No telemetry, no update checks, no accounts; run, chat and serve never download anything. The server listens on 127.0.0.1 and refuses requests from web pages.
• Drop-in for llama.cpp users. The common flags work as they do in llama.cpp, the ones Kinetic does not need are accepted, and the few with no equivalent stop with a line saying what to use instead. See llama.cpp flags.
• Verifiable models. A .ktc file carries weights, tokenizer and chat template with a hash of every byte and an optional signature, and kinetic verify checks them.

Where to go next

• Install Kinetic, then follow the quick start.
• Look up a command in the command line reference or an endpoint in the server API.
• Check whether your model runs: Models and formats.
• Use Kinetic from Python, C or Rust.
• Something wrong? FAQ and troubleshooting.

Kinetic is open source under the Apache License 2.0, made by Velocity. The source is at github.com/Veloresearch/kinetic.

Install

Kinetic runs on Windows 10 and 11 and on Linux, on x86-64. Pick one way to install it.

Install script

irm https://raw.githubusercontent.com/Veloresearch/kinetic/main/scripts/install.ps1 | iex

curl -fsSL https://raw.githubusercontent.com/Veloresearch/kinetic/main/scripts/install.sh | sh

The scripts download the latest release (the full build, which runs on NVIDIA GPUs through CUDA, on AMD and Intel GPUs through Vulkan and on the CPU), check it against the release's SHA256SUMS and install it for the current user: into %LOCALAPPDATA%\Programs\kinetic, put on the user PATH, on Windows, and into ~/.local/bin on Linux. -Version 0.1.0 / --version 0.1.0 picks a release and -Cpu / --cpu the smaller CPU-only build.

Release binaries

From the releases page, take the archive for your system:

The cuda archives also run on machines without a GPU, on the CPU; they are the right choice unless you want the smallest download. They need only the NVIDIA driver, not the CUDA toolkit.

Each archive holds the kinetic binary, the C library and header (lib/, include/), the license files and a SHA256SUMS of its contents. The release also has a SHA256SUMS for the archives themselves. Check the download before you unpack it:

# Linux
sha256sum -c SHA256SUMS --ignore-missing
tar -xzf kinetic-0.1.0-linux-x86_64-cuda.tar.gz
sudo install kinetic-0.1.0-linux-x86_64-cuda/kinetic /usr/local/bin/

# Windows (PowerShell): compare the hash with the line in SHA256SUMS
Get-FileHash .\kinetic-0.1.0-windows-x86_64-cuda.zip -Algorithm SHA256
Expand-Archive .\kinetic-0.1.0-windows-x86_64-cuda.zip -DestinationPath $env:LOCALAPPDATA\Programs

Then put the unpacked folder on your PATH and check:

kinetic --version

On Windows, SmartScreen may warn about a new download the first time you run it; see FAQ.

Python

pip install kinetic-engine

Wheels are built for Windows and Linux (manylinux 2.28) on x86-64, for Python 3.9 and newer. They run models on the CPU; to use an NVIDIA GPU from Python, build the package from source with the CUDA toolkit installed, or run kinetic serve and use any OpenAI client. See Python, C and Rust for the API.

Docker

The images run kinetic serve on port 8080 as an unprivileged user, with your models folder mounted read-only. Build them from a clone of the repository:

docker build -f docker/Dockerfile -t kinetic .
docker run --rm -p 8080:8080 -v "$PWD/models:/models:ro" kinetic /models/model.gguf

For NVIDIA GPUs, with the NVIDIA Container Toolkit on the host:

docker build -f docker/Dockerfile.cuda -t kinetic:cuda .
docker run --rm --gpus all -p 8080:8080 -v "$PWD/models:/models:ro" kinetic:cuda /models/model.gguf

Inside the container the server listens on all interfaces, so that -p 8080:8080 reaches it; -p 127.0.0.1:8080:8080 keeps it on your own machine. Arguments after the image name go to kinetic serve, so kinetic /models/model.gguf -c 8192 works as on the command line.

From source

With Rust 1.91 or newer:

git clone https://github.com/Veloresearch/kinetic
cargo install --path kinetic/crates/kinetic-cli

For GPU support, install the CUDA toolkit 12.8 or newer first. See Building from source.

Get a model

Any .gguf file of a supported architecture works; run, chat and serve only read files from your disk. Download one from Hugging Face with kinetic pull <org>/<repo> (see pull) or your browser. Models Ollama or LM Studio already downloaded run by name; see models. To try it out:

kinetic info model.gguf shows what a file contains and whether Kinetic supports it.

Quick start

This page assumes kinetic is installed and you have a model file; the examples use model.gguf.

Ask one question

kinetic run model.gguf "Explain recursion in two sentences."

The answer streams to the terminal as it is written. When it is done, a status line on stderr gives the speed and the device:

Recursion is a programming technique where a function calls itself ...
[45 tokens at 25.5 tok/s; prompt 37 tokens at 62.8 tok/s on the CPU]

Without a prompt, run reads it from stdin, so files and pipes work:

kinetic run model.gguf < prompt.txt
git diff | kinetic run model.gguf --system "Write a commit message for this diff." -n 100

Chat

kinetic chat model.gguf --system "You are a concise assistant."

Type a message and press Enter. /clear starts a new conversation and /exit (or Ctrl+D) quits. Each turn reuses the work of the previous ones, so only your new message is processed.

Pick the sampling

By default Kinetic uses the sampling settings the model file recommends. Override them as in llama.cpp:

kinetic run model.gguf "Write a haiku about rivers." --temp 0.8 --top-p 0.95 --seed 42
kinetic run model.gguf "List the planets." --temp 0 -n 64

--temp 0 always takes the likeliest token. With a seed, the same command gives the same answer every time; speculative decoding never changes that.

Serve an OpenAI-compatible API

kinetic serve model.gguf

Serving model on the CPU at http://127.0.0.1:8080/v1

Any OpenAI client works against this address; the API key can be anything.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Name three rivers."}]}'

On Windows PowerShell, where curl means something else:

$body = @{ messages = @(@{ role = "user"; content = "Name three rivers." }) } | ConvertTo-Json -Depth 5
Invoke-RestMethod http://127.0.0.1:8080/v1/chat/completions -Method Post -ContentType application/json -Body $body

With the official openai package for Python:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
stream = client.chat.completions.create(
    model="model",
    messages=[{"role": "user", "content": "Write a haiku about rivers."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

Editors, chat apps and agent frameworks that take an "OpenAI-compatible" base URL work the same way. The server API lists every endpoint and field.

See what a file holds

kinetic info model.gguf

prints the architecture, size, context length, weight types and tokenizer, and says if the architecture is not supported yet.

Measure your machine

kinetic bench model.gguf

runs llama-bench's two tests (a 512-token prompt and 128 generated tokens) and an editing test, and prints a table you can compare with llama-bench on the same file. See Benchmarks.

Next

• Every option of every command: Command line.
• Images, tools, JSON output, embeddings: Server API.
• GPU or CPU, memory, large MoE models: Devices and offload.

Switch from llama.cpp

Your GGUF files work unchanged when their model family is supported (see Models and formats). Each case below is one step.

You run llama-server

Write kinetic serve where you wrote llama-server, and keep every flag:

llama-server  -m model.gguf -c 8192 -ngl 99 -fa on --jinja --port 8080
kinetic serve -m model.gguf -c 8192 -ngl 99 -fa on --jinja --port 8080

The same address answers llama-server's own API (/completion, /tokenize, /detokenize, /props, /slots, /health, /embedding), the OpenAI API under /v1 and Ollama's under /api:

$ curl -s http://127.0.0.1:8080/completion -d '{"prompt": "The capital of France is", "n_predict": 4, "temperature": 0}'
{"index":0,"content":" Paris, and the","tokens":[],"id_slot":0,"stop":true,"model":"LFM3-1.7B-Q4_K_M","tokens_predicted":4,"tokens_evaluated":5,...,"stop_type":"limit",...}

If an app starts a binary named llama-server itself, copy kinetic to that name instead; it then behaves as kinetic serve. llama-cli becomes kinetic chat, or kinetic run when given a prompt, and llama-completion becomes kinetic run:

cp kinetic llama-server            # Windows: copy kinetic.exe llama-server.exe

Flags for what Kinetic does by itself, such as -fa, --jinja or -b, are accepted and ignored. The few with no equivalent stop with a line saying what to use; -hf points at kinetic pull. The full list: llama.cpp flags. What still differs in the HTTP API is listed under llama-server API.

You use llama-cpp-python

pip install llama-cpp-kinetic

import llama_cpp then loads Kinetic's version of the Llama class, so your code runs unchanged:

$ cat app.py
from llama_cpp import Llama

llm = Llama(model_path="LFM3-1.7B-Q4_K_M.gguf", n_ctx=4096, n_gpu_layers=0, verbose=False)
out = llm("The capital of France is", max_tokens=4, temperature=0)
print(out["choices"][0]["text"])
$ python app.py
 Paris, and the

An installed llama-cpp-python is shadowed, not touched: pip uninstall llama-cpp-kinetic brings it back. The covered API is in Python, C and Rust.

You use Ollama

Stop Ollama, then serve the model on Ollama's port:

kinetic serve qwen3:4b --ollama

Open WebUI, Continue and other Ollama clients keep their settings, since the server answers at http://127.0.0.1:11434. kinetic models lists the models Ollama already downloaded, and a name like qwen3:4b runs Ollama's own file.

$ curl -s http://127.0.0.1:11434/api/version
{"version":"0.13.0"}
$ curl -s http://127.0.0.1:11434/api/chat -d '{"model": "LFM3-1.7B-Q4_K_M", "messages": [{"role": "user", "content": "Say hi in two words."}], "stream": false}'
{"model":"LFM3-1.7B-Q4_K_M","created_at":"2026-11-08T12:01:01.544798100Z","message":{"role":"assistant","content":"Hi there! ..."},"done":true,"done_reason":"stop",...}

You link libllama

Not yet: there is no drop-in libllama library. Kinetic has its own C API, kinetic.h, which covers loading a model, generating and embedding, but code written against llama.h needs porting to it.

Command line

kinetic <COMMAND> [OPTIONS] [MODEL] ...

kinetic alone in a terminal checks the computer, lists two or three models that run well on it with their size and expected speed, models already on disk first, and waits for one key: 1-3 or Enter picks, q quits. A model not on disk yet is downloaded as pull does, then chat opens. Nothing is downloaded before you pick. With input or output redirected, kinetic prints the help as before.

Every command except verify and verify-receipt takes the model as its first argument or with -m/--model, as a .gguf or .ktc file. kinetic --version prints the version and kinetic <command> --help the options of a command.

Kinetic exits with 0 on success, 1 when the command fails (a missing file, a model that cannot load) and 2 for an invalid command line, including a llama.cpp flag it does not take; see llama.cpp flags.

run

kinetic run model.gguf "Explain recursion in two sentences."
kinetic run -m model.gguf -p "Explain recursion."
kinetic run model.gguf < prompt.txt

The prompt goes through the model's chat template as one user message, after the --system prompt if there is one, and the answer streams to stdout. Without a prompt, run reads it from stdin. When the answer is done, a status line on stderr gives the number of tokens, the speed of the answer and of the prompt, and the device:

[45 tokens at 25.5 tok/s; prompt 37 tokens at 62.8 tok/s on the CPU]

When stdout goes to a file or a pipe, the speed is also shown live on stderr. Only the answer goes to stdout, so kinetic run ... > answer.txt captures just the text.

--receipt FILE also writes a receipt for the answer: the model's hash, the build, the device class, the prompt tokens, every sampling setting with the seed, and the output tokens. Without --seed, a fresh seed is picked and recorded.

kinetic run model.gguf "Name three rivers." --temp 0.8 --receipt answer.json
kinetic verify-receipt answer.json

Generation options

run and chat take these:

The sampler flags carry llama.cpp's names and meanings, and samplers run in its default order: logit bias, penalties, DRY, top-n-sigma, top-k, typical, top-p, min-p, XTC, temperature; with Mirostat, temperature then Mirostat. Sampling defaults come from the model file (general.sampling.*) where it has them. Whatever the settings, speculative decoding never changes the answer: greedy output is token for token what plain decoding gives, and a seeded answer is the same for the same seed.

Runtime options

run, chat and serve take these:

Without -c and --gpu-layers, a context whose cache would not fit next to the model on the CUDA GPU shrinks to the largest that does, a multiple of 1024 and at least 4096, with one line such as context 8k to fit the GPU; -c to change, rather than moving layers to the CPU. An explicit -c always wins.

Every speed technology is exact, so --off changes only speed: use it to measure what each one gives, or to rule one out when reporting a bug. kinetic serve prints which run on the device it picked, for example Speed technologies on: RAD-1, Warm Start, Zero Wait, Torque, Convoy; off: Shortlist.

LoRA adapters

--lora takes adapters as llama.cpp's convert_lora_to_gguf.py writes them (general.type = adapter, adapter.type = lora, lora_a and lora_b per adapted weight). Each adapted matrix becomes W + scale * alpha / rank * B A at load, held in F32, so the model runs as fast as without the adapter on every device but takes 4 bytes per weight of each adapted matrix. Adapters on the token embeddings or on expert tensors are refused. Warm Start is off while an adapter is in use.

chat

kinetic chat model.gguf --system "You are a concise assistant."

Reads one message per line; end a line with \ to go on writing on the next one. Each turn reuses the work of the previous ones, and the message runs on the model while it is being typed or pasted (Zero Wait), so after Enter only its last words are processed. Ctrl+C stops an answer, and the conversation goes on.

A model with a vision projector, stored with it or given with --mmproj FILE, sees images; the header says so. Drop an image file on the window (the terminal pastes its path) or use /image: the image goes out with the next message, and the prompt shows how many are waiting. PNG, JPEG, WebP, BMP and GIF files work. On Linux, /image alone reads the clipboard with wl-paste or xclip. Images are encoded on the GPU when the model runs there; turns with an image run whole, without Zero Wait.

When stdin is not a terminal, chat reads one message per line and prints each answer, which is handy for scripted tests. Takes the generation and runtime options.

serve

kinetic serve model.gguf
kinetic serve model.gguf --port 9000 -c 8192
kinetic serve Qwen3.5-4B-Q4_K_M.gguf --mmproj Qwen3.5-4B-mmproj-F16.gguf

and the runtime options. The start-up line names the model, the device and the address:

Serving Qwen3-0.6B-Q8_0 on the CPU at http://127.0.0.1:8080/v1

The model's name in the API is its file name without the extension, or --alias. A text model gets the chat and completion endpoints, an embedding model the embeddings endpoint and a reranker the rerank endpoint; see the server API. Embedding and reranking models run on the CPU.

Warm Start keeps the states of earlier prompts in memory (up to 1 GiB) and under --cache-dir (up to 2 GiB, least recently used out first), so a prompt that starts like an earlier one, even from before a restart, only runs its new tokens. It works on the CPU and on the GPU, resumes only when that is faster than recomputing, and gives exactly the answer a cold run gives.

Any --host other than a loopback address makes the server reachable by other computers, which Kinetic warns about; pair it with --api-key.

bench

kinetic bench model.gguf
kinetic bench model.gguf --device cpu -t 6 -r 5 --json

Three tests, each the best of -r runs:

• pp512: a 512-token prompt, as in llama-bench;
• tg128: 128 generated tokens, as in llama-bench, with speculative decoding off so every token costs one pass over the weights;
• edit256: 256 tokens of an answer that rewrites a short Rust file with one name changed, with speculative decoding on, the way an editing request runs. Its output is compared with plain decoding.

| model | size | params | backend | threads | test | t/s |
| --- | ---: | ---: | --- | ---: | ---: | ---: |
| SmolLM2-135M-Instruct-f16 | 0.25 GiB | 0.13 B | CPU | 4 | pp512 | ... |
| SmolLM2-135M-Instruct-f16 | 0.25 GiB | 0.13 B | CPU | 4 | tg128 | ... |
| SmolLM2-135M-Instruct-f16 | 0.25 GiB | 0.13 B | CPU | 4 | edit256 | ... |

on the CPU, best of 3, load 0.53 s
speed technologies on: RAD-1, Shortlist; Warm Start, Zero Wait and Convoy take no part
edit: drafted output identical to plain decoding: yes
peak memory 0.31 GiB

To see what one technology gives, run bench twice, with and without --off naming it, and compare the rows: --off rad1 runs edit256 without drafting, --off shortlist and --off torque change tg128 and edit256. The line under the table says which ran; Warm Start, Zero Wait and Convoy need repeated, typed or concurrent requests, which bench never sends, so it says they take no part.

If the drafted output ever differs from plain decoding, bench says NO and exits with 1; please report it. --json prints:

{
  "model": "SmolLM2-135M-Instruct-f16.gguf",
  "backend": "cpu",
  "device": "the CPU",
  "threads": 4,
  "repetitions": 3,
  "load_seconds": 0.0135,
  "size_bytes": 270885952,
  "parameters": 134515008,
  "prefill_tokens": 512,
  "prefill_tokens_per_second": 354.05,
  "decode_tokens": 128,
  "decode_tokens_per_second": 49.01,
  "edit_tokens": 256,
  "edit_tokens_per_second": 118.9,
  "edit_identical": true,
  "technologies": "on: RAD-1, Shortlist",
  "peak_rss_bytes": 333955072
}

backend is cpu or cuda, and device names the GPU.

info

kinetic info model.gguf

Name, architecture and whether Kinetic supports it, size and parameter count, context length, layers and width, whether the file has a prediction block that drafts for speculative decoding, the weight types with their tensor counts and sizes, the tokenizer and the size of the chat template. It reads only the file's header, so it is instant even for large files. See Models and formats for an example.

verify

kinetic verify model.ktc

Recomputes the BLAKE3 hash of every section of a .ktc file and reports whether it is signed. Exits with 1 when any hash does not match.

verify-receipt

kinetic verify-receipt answer.json
kinetic verify-receipt answer.json --model other/path/model.gguf --device cpu

Checks a receipt: the token hashes and, if it is signed, the signature; then the model file's BLAKE3 (from a cache, so a large file is hashed once); then reruns the recorded prompt tokens with the recorded settings, alone, with drafting, the prompt cache, Zero Wait and Convoy off. It prints identical or the first output token that differs, with the reason when the build, the device class or a load setting differs. The rerun goes to the receipt's device unless --device says otherwise; -t sets the CPU threads. Exits with 1 when anything does not match.

receipt:   LFM3-1.7B-Q4_K_M.gguf, 15 prompt and 18 output tokens, seed 42, made on cpu-x86_64-avx2 by Kinetic 0.1.0 (build 378534cb64f30a15)
signature: valid, public key 8e79e51c11472a6cffd70635753a7cce14c4335e1f18435459c3ebb35b3f720b
model:     LFM3-1.7B-Q4_K_M.gguf matches (BLAKE3 f4240740be2392a2)
rerun:     identical, 18 of 18 output tokens on cpu-x86_64-avx2 (BLAKE3 7bd0632303b6fcbd)

quantize

kinetic quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

Reads a GGUF model with F32, F16, BF16 or Q8_0 weights and writes it as Q8_0, Q6_K, Q5_K_M, Q4_K_M or Q4_0. Each tensor gets the type llama-quantize gives it for that target (the _M mixtures keep some attn_v and ffn_down layers and the output in Q6_K; rows that are not a whole number of 256-value blocks fall back to Q5_0, Q5_1 or Q8_0), and the blocks are encoded as llama.cpp's reference quantizers encode them: without an importance matrix the output file is byte for byte the one llama-quantize writes, and llama.cpp loads it. -t sets the threads. Importance matrices (--imatrix) are not read yet.

perplexity

kinetic perplexity model.gguf wiki.test.raw -c 512
kinetic perplexity model-f16.gguf wiki.test.raw --chunks 20 --kl-divergence-base base.kld
kinetic perplexity model-Q4_K_M.gguf --kl-divergence base.kld

Scores the text as llama-perplexity does: the tokens are cut into chunks of -c tokens (default 512), each chunk starts with BOS where the model uses one, and only the second half of each chunk is scored. It prints the running estimate after every chunk and Final estimate: PPL = ... +/- .... --kl-divergence-base FILE saves the log-probabilities in llama-perplexity's format, and --kl-divergence FILE compares another model against such a file, from either tool, on the tokens stored in it: mean PPL of both, the mean KL divergence, RMS Δp and how often the top token agrees. Runs on the CPU; -t sets the threads.

models

kinetic models
kinetic run qwen3:4b "Hi"
kinetic serve lmstudio-community/Qwen3-4B-GGUF:Q4_K_M

Lists the GGUF models already on this computer: Kinetic's own folder (KINETIC_MODELS, or %LOCALAPPDATA%\kinetic\models and ~/.local/share/kinetic/models), Ollama's store (OLLAMA_MODELS or ~/.ollama/models), LM Studio's (~/.lmstudio/models), the Hugging Face cache (HF_HUB_CACHE, HF_HOME), llama.cpp's (LLAMA_CACHE) and, on Windows, C:\models. Only file headers are read.

NAME                                     SIZE  QUANT     ARCH            RUNS ON KINETIC         FROM
LFM3-1.7B-Q4_K_M                     1.02 GiB  Q4_K_M    lfm3            yes                     folder
gemma-4-E2B_q4_0-it                  3.12 GiB  Q4_0      gemma4          no                      folder

Every command takes such a name in place of a file. A bare Ollama name takes its latest tag, or its only tag; a repository with several files takes :<quant>. A name that matches nothing lists the closest ones. serve also picks up the vision projector stored with the model.

pull

kinetic pull ggml-org/gemma-3-1b-it-GGUF
kinetic pull unsloth/Qwen3-4B-GGUF:Q4_K_M
kinetic pull org/repo --file model-Q8_0.gguf --no-mmproj

Downloads a GGUF model from Hugging Face into Kinetic's models folder (or --dir) and prints the name to run it by. It takes the file of the named quantization, or when none is named the largest of Q4_K_M, Q5_K_M, Q6_K and Q8_0 that fits the GPU's free memory (or half the free RAM without one) with room for context, Q4_K_M at least, and says which and why; every part of a split model, and the repository's vision projector unless --no-mmproj. Before downloading it prints the source, the files and their size, and it refuses when the disk would keep less than 2 GiB free. Downloads resume where they stopped, are checked against the SHA256 that Hugging Face publishes and are renamed into place only when they match. HF_TOKEN is sent for gated repositories and HF_ENDPOINT names a mirror. The transfer runs through the system's curl.

Kinetic never connects to the network, except when you run kinetic pull; run, chat, serve and the others never download anything.

When a command fails, the error ends with a line saying what to do: a path to check, a smaller model or -c, a driver to install, a free --port, or a download to repeat.

doctor

kinetic doctor

One screen with the operating system, the CPU and its AVX2, FMA and F16C support, memory and its measured read speed, every GPU that CUDA and Vulkan report with its driver and free memory, the models found and the free disk, followed by what to fix.

Kinetic 0.1.0
  system   Windows 11 (build 26300)
  cpu      AMD Ryzen 5 6600H with Radeon Graphics, 12 threads, AVX2 yes, FMA yes, F16C yes
  memory   15.2 GiB, 6.4 GiB available, reads at 40 GB/s (measured)
  gpu      NVIDIA GeForce RTX 3060 Laptop GPU via CUDA: 5.0 of 6.0 GiB free, 336 GB/s nominal; CUDA 13.3, compute 8.6
  gpu      NVIDIA GeForce RTX 3060 Laptop GPU via Vulkan: 5.9 GiB; NVIDIA 610.62, Vulkan 1.4.341
  models   4 found, 1 that Kinetic runs (`kinetic models` lists them)
  disk     7.1 GiB free for C:\Users\me\AppData\Local\kinetic\models

To fix:
  - under 10 GiB of disk is free where models are kept

fit

kinetic fit
kinetic fit qwen3:4b
kinetic fit D:\models

For every model found, or the one or the folder given: whether it fits wholly on the GPU, needs its experts kept on the CPU (mixture-of-experts models), or runs on the CPU, with a decode speed range estimated from the bytes it reads per token and the memory bandwidth. kinetic bench measures the real speed.

MODEL                                    SIZE  RUNS ON               EXPECTED DECODE
LFM3-1.7B-Q4_K_M                     1.02 GiB  GPU                   169-262 t/s

bench --compare

kinetic bench model.gguf --compare
kinetic bench model.gguf --compare C:\llama.cpp\llama-bench.exe --json

Runs Kinetic's bench and then llama.cpp's own llama-bench on the same file with the same threads, device (-ngl 99, or -ngl 0 -dev none on the CPU) and repetitions, flash attention on, taking llama.cpp's best run as Kinetic takes its own. llama.cpp's edit test runs through llama-server with its speculative mode for the model (draft-mtp where the model has prediction layers, otherwise ngram-simple). llama-bench is looked for on PATH and in the usual install folders; Kinetic never downloads it. The card is a markdown table with a one-line summary; --json prints the same as JSON.

## Kinetic vs llama.cpp: LFM3-1.7B-Q4_K_M

| test | Kinetic 0.1.0 | llama.cpp b12001 (eb4e244be) | ratio |
| --- | ---: | ---: | ---: |
| prompt pp512 | 6780.5 | 6667.7 | 1.02x |
| generate tg128 | 191.0 | 166.8 | 1.15x |
| edit256 | 702.7 | 207.7 | 3.38x |

NVIDIA GeForce RTX 3060 Laptop GPU, AMD Ryzen 5 6600H with Radeon Graphics, Windows 11 (build 26300)
Same file (1.02 GiB, Q4_K_M), all layers on the GPU, 12 threads, flash attention on, best of 2, t/s.
edit256: greedy copy of a file with one rename; llama.cpp speculative mode ngram-simple; Kinetic's output identical to plain decoding: yes.

llama.cpp flags

Common llama.cpp flags work unchanged, and Kinetic explains the ones it does not need instead of rejecting them blindly; see llama.cpp flags.

llama.cpp flags

If you use llama.cpp, most of your commands work as they are. These flags mean the same in Kinetic:

-m, -p, -n, -c, -t, -s, --temp, --top-k, --top-p, --min-p, --repeat-penalty, --host, --port and --mmproj.

llama-cli    -m model.gguf -p "Hello" -n 64 --temp 0 -ngl 99
kinetic run  -m model.gguf -p "Hello" -n 64 --temp 0 -ngl 99

llama-server -m model.gguf -c 8192 --port 8080 -ngl 99
kinetic serve -m model.gguf -c 8192 --port 8080 -ngl 99

The commands differ: llama-cli is kinetic run (one prompt) or kinetic chat (a conversation), llama-server is kinetic serve and llama-bench is kinetic bench.

A copy of kinetic named llama-server, llama-cli or llama-completion runs as kinetic serve, kinetic chat (or kinetic run with a prompt) and kinetic run, for apps that start those binaries by name.

For every other llama.cpp flag below, Kinetic takes it under its own name, accepts it when it asks for something Kinetic does by itself, or stops with a line that says what to use instead, rather than a bare "unexpected argument":

$ kinetic run model.gguf -ts 1,1 "Hello"
kinetic: -ts is a llama.cpp flag: Kinetic runs the model wholly on one device; choose it with --device.

The flags for prompt files, constrained output and adapters are on the roadmap.

Server API

kinetic serve <model> speaks the OpenAI API at http://127.0.0.1:8080/v1. Any OpenAI client works: point its base URL there and give any API key.

The paths also answer without the /v1 prefix, except /completions and /embeddings, which are llama-server's there (see llama-server API). Bodies are JSON, sent with Content-Type: application/json. A request to an endpoint the loaded model does not serve, such as /v1/embeddings on a text model, is answered with a 400 error that says which endpoint to use.

/v1/health answers {"status": "ok"} once the model is loaded.

Models

curl http://127.0.0.1:8080/v1/models

{"object": "list", "data": [{"id": "model", "object": "model", "created": 1762171200,
  "owned_by": "kinetic", "context_length": 32768}]}

The id is the model's file name without its extension. One model is served; the model field of requests is accepted and not checked.

Chat completions

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "system", "content": "Answer briefly."},
    {"role": "user", "content": "What is the capital of France?"}
  ],
  "max_tokens": 64,
  "temperature": 0
}'

The messages go through the model's own chat template. Content is a string or a list of parts of type text and image_url.

Defaults come from the model file's recommended sampling where it has them.

Samplers run in llama.cpp's default order: logit bias, repeat, presence and frequency penalties, DRY, top-n-sigma, top-k, typical, top-p, min-p, XTC, temperature. With Mirostat it is logit bias, temperature, Mirostat. The names and meanings follow llama-server's. Log-probabilities are those of the model's raw logits, as llama-server reports them, before any sampler. With speculative decoding every sampler stays exact: a seeded answer and its log-probabilities are the same with drafting as without.

/v1/completions takes logprobs as a count of alternatives, 0 to 20, and answers in the legacy shape: tokens, token_logprobs, top_logprobs and text_offset. Speculative decoding never changes an answer: greedy requests give exactly the plain greedy tokens, and seeded ones the same answer for the same seed.

The response:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1762171200,
  "model": "model",
  "choices": [{"index": 0,
    "message": {"role": "assistant", "content": "Paris."},
    "logprobs": null, "finish_reason": "stop"}],
  "usage": {"prompt_tokens": 25, "completion_tokens": 3, "total_tokens": 28,
    "prompt_tokens_details": {"cached_tokens": 0}}
}

finish_reason is stop when the model ended its answer or a stop string was reached, tool_calls when it called tools, and length when max_tokens or the context ran out. For models that think before they answer, with enable_thinking on, the thinking comes separately as reasoning_content. cached_tokens counts the prompt tokens that were not recomputed because an earlier request already ran them.

Streaming

With "stream": true the answer comes as chat.completion.chunk events, each with a delta, and ends with data: [DONE]:

data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Par"},"finish_reason":null}],...}
data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"is."},"finish_reason":null}],...}
data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],...}
data: [DONE]

Tool calls

With tools, the model may answer with calls instead of, or after, text:

"message": {"role": "assistant", "content": null, "tool_calls": [
  {"id": "call-...", "type": "function",
   "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}}]}

Send the call back as the assistant message and each result as a message with role tool, and the model answers from them:

"messages": [
  {"role": "user", "content": "What is the weather in Paris?"},
  {"role": "assistant", "content": "", "tool_calls": [
    {"id": "call-1", "type": "function",
     "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}}]},
  {"role": "tool", "tool_call_id": "call-1", "content": "{\"temp_c\": 18, \"sky\": \"clear\"}"}
]

Give the assistant message's content as a string, "" when it has only calls; some chat templates fail on a null content. Streamed, text before the calls comes as it is written and the calls arrive whole in one delta.tool_calls chunk, numbered by index. Kinetic reads the call formats of Qwen3.5, Qwen3, LFM3, LFM2, Llama 3 and Mistral models.

"tool_choice": "required" and {"type": "function", "function": {"name": "get_weather"}} hold the reply to calls with a grammar in the chat template's own call format: XML calls for Qwen3.5 and LFM3, JSON in <tool_call> for Qwen3 and Hermes-style templates, a JSON list in LFM2's markers or after Mistral's [TOOL_CALLS], and a bare JSON object for Llama 3 and other templates. Names and arguments follow each tool's JSON Schema (in the XML format, the parameter names; values are free text). An open <think> block is closed first, and templates that know <think> may reason in a whole block before the call. The reply holds one call unless the request sets "parallel_tool_calls": true, as in llama-server. Drafted tokens are checked against the same grammar, so speculative decoding gives the reply plain decoding gives. "none" hides the tools; a forced call cannot be combined with grammar or response_format.

Structured output

response_format and grammar hold the answer to a shape while it is generated, so it always parses:

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Give me a person."}],
  "temperature": 0,
  "response_format": {"type": "json_schema", "json_schema": {"name": "person", "schema": {
    "type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
    "required": ["name", "age"]}}}
}'

"message": {"role": "assistant", "content": "{
  \"name\": \"Ethan\",
  \"age\": 25
}"}

{"type": "json_object"} allows any JSON value, and grammar takes a GBNF grammar in llama.cpp's syntax, such as root ::= "yes" | "no". A request may use one of grammar and response_format, not both.

Receipts

"receipt": true on /v1/chat/completions or /v1/completions adds a receipt object to the response (to the last chunk of a stream) that kinetic verify-receipt can rerun; see Receipts. A request without a seed gets a fresh one, recorded in the receipt. Receipts are signed when the server was started with --sign-key. Requests with images cannot have a receipt yet.

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Name three rivers."}],
  "temperature": 0.8, "seed": 42, "receipt": true
}' | python -c "import json,sys; json.dump(json.load(sys.stdin)['receipt'], open('answer.json','w'))"
kinetic verify-receipt answer.json

Reasoning

Models that think before they answer, such as Qwen3 and Qwen3.5, do so when their template's enable_thinking is on, which is the template's own default unless you set it:

"chat_template_kwargs": {"enable_thinking": true}

The thinking then comes apart from the answer, as reasoning_content in the message (or in delta.reasoning_content when streamed), and content holds only the answer.

Images

Started with --mmproj, a vision model reads images given as parts of a message, in PNG or JPEG. Images must be inline base64 data URLs; the server fetches nothing.

{"role": "user", "content": [
  {"type": "text", "text": "What is in this picture?"},
  {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
]}

Prefill

curl http://127.0.0.1:8080/v1/prefill -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Here is the report: ... What does it"}]}'

Zero Wait for apps: send the conversation while the user is still writing the last message, as often as it changes, and the chat completion that follows only runs what is new. The body is that of a chat completion; only messages, tools and chat_template_kwargs matter, and they must match the request that follows. The prompt is run up to just before the last word of the last message, which may still grow, so the reply waits for that part, in whole batches, on a free slot. It answers once that is done, or as soon as a newer prefill or a real request comes in:

{"object": "prefill", "cached_tokens": 19968}

It never changes what a request generates, and a request never waits for more than one batch of it.

Completions

curl http://127.0.0.1:8080/v1/completions -H "Content-Type: application/json" \
  -d '{"prompt": "The capital of France is", "max_tokens": 8, "temperature": 0}'

Continues prompt, a string or a list of one string, exactly as written, without the chat template. It takes the same fields as chat completions apart from messages and chat_template_kwargs, and answers with text_completion objects whose choices carry text.

Embeddings

kinetic serve bge-small-en-v1.5-f16.gguf
curl http://127.0.0.1:8080/v1/embeddings -H "Content-Type: application/json" \
  -d '{"input": ["a cat", "a dog"]}'

input is a string, a list of up to 256 strings, a list of token ids, or a list of such lists. encoding_format is float (the default) or base64 for little-endian 32-bit floats.

{"object": "list", "model": "bge-small-en-v1.5-f16",
 "data": [{"object": "embedding", "index": 0, "embedding": [0.0123, ...]}, ...],
 "usage": {"prompt_tokens": 8, "total_tokens": 8}}

Rerank

kinetic serve qwen3-reranker-0.6b-q8_0.gguf
curl http://127.0.0.1:8080/v1/rerank -H "Content-Type: application/json" -d '{
  "query": "What is the capital of China?",
  "documents": ["The capital of China is Beijing.", "Gravity pulls bodies together."],
  "top_n": 2, "return_documents": true
}'

Each document, a string or an object with text, is scored against the query with the model's rerank template; up to 256 at once. The answer lists them most relevant first, top_n of them [default: all]:

{"object": "list", "model": "qwen3-reranker-0.6b-q8_0",
 "results": [{"index": 0, "relevance_score": 0.9996, "document": {"text": "The capital of China is Beijing."}},
             {"index": 1, "relevance_score": 0.00004, "document": {"text": "Gravity pulls bodies together."}}],
 "usage": {"prompt_tokens": 152, "total_tokens": 152}}

The scores are probabilities, as llama-server's are for the same model.

Errors

Errors come in the OpenAI shape with a matching HTTP status:

{"error": {"message": "max_tokens must be at least 1", "type": "invalid_request_error", "param": null, "code": null}}

Not supported yet: n other than 1.

Requests at once

The server accepts up to 64 connections. With --parallel N (llama.cpp's -np N) up to N requests run at once and share each pass over the weights, on the CPU and on CUDA when the whole model is on the GPU; every answer is exactly what that request would get on its own. The CPU default fits as many full-context sequences as half the free memory holds, up to 4. With a layer split, expert offload or on Vulkan, requests take turns. /health and /slots report idle and busy slots.

A request that continues the previous conversation reuses its work, so only the new tokens run, and any prompt that starts like an earlier one resumes from where that one's state was kept, when that is faster than recomputing it (see Warm Start). cached_tokens in the usage says how many prompt tokens were reused.

Ollama API

The same server answers Ollama's API, so apps written for Ollama work unchanged; kinetic serve <model> --ollama listens on Ollama's port 11434.

options takes temperature, top_k, top_p, min_p, seed, num_predict, stop, repeat_penalty, repeat_last_n, presence_penalty and frequency_penalty; format takes "json" or a JSON schema; tools, think and images work as in Ollama, and tool call arguments come back as objects. suffix, template and context are refused with an error. Requests whose browser Origin is not this computer are refused, since Ollama clients send bodies of any content type.

llama-server API

The same server answers llama.cpp's own server API, so scripts and apps written for llama-server work unchanged.

/completion takes prompt (text or token ids), n_predict, temperature, top_k, top_p, min_p, repeat_penalty, presence_penalty, frequency_penalty, seed, stop, stream, grammar and json_schema; cache_prompt is always on. n_probs is accepted but token probabilities are not returned yet, a stop string reads as "stop_type": "eos", and streamed pieces carry no token ids. scripts/llama_server_compat.py runs both servers on one GGUF and lists every field that still differs. Like the Ollama routes, these take bodies of any content type and refuse requests whose browser Origin is not this computer.

Metrics

GET /metrics (also /v1/metrics) answers in Prometheus' text format with llama-server's metric names, so its dashboards work unchanged. Counters since start: llamacpp:requests_total, prompt_tokens_total (prompt tokens run), prompt_tokens_cached_total (prompt tokens reused from an earlier request), prompt_seconds_total, tokens_predicted_total, tokens_predicted_seconds_total and n_tokens_max. Gauges: prompt_tokens_seconds and predicted_tokens_seconds (average prompt and generation speed since start), requests_processing and requests_deferred (waiting for a free slot). It is always on; llama.cpp's --metrics and --slots are accepted and change nothing.

curl -s localhost:8080/metrics | grep -v '^#'

Security

The server listens on 127.0.0.1 by default and has no authentication unless started with --api-key KEY; then every route but /health needs Authorization: Bearer KEY (or X-Api-Key: KEY, as llama-server takes it) and answers 401 with an authentication_error otherwise. Listening on another address with --host makes the model usable by anyone who can reach that address; use --api-key there, and a TLS proxy if the key crosses an untrusted network. Browsers cannot send JSON to the server from another site, so web pages you visit cannot use it. A client has 30 seconds to send its request. Kinetic never connects to the network, except when you run kinetic pull. The server never downloads anything.

Client compatibility

What was checked against kinetic serve on Windows 11 (CPU, LFM3 1.7B Q4_K_M with -c 512, and bge-small for embeddings). The client libraries themselves were not installed: each row replays the HTTP requests that client sends, with the same paths, fields and headers, from a Python script using requests (User-Agent: OpenAI/Python, Authorization: Bearer ...).

Refused with a 400 and a message, as intended: n other than 1, an empty messages list, a prompt longer than the context, max_tokens below 1, temperature or top_p out of range, unknown response_format types and invalid schemas or grammars. logprobs: true with top_logprobs: 2 returns each token's log-probability; top_logprobs above 20 is refused.

Robustness

scripts/server_abuse.py sends malformed JSON, wrong types, oversized bodies and headers, chunked bodies, HTTP/2 prefaces, bad input to the llama-server and Ollama routes, a client that sends one byte at a time, clients that disconnect before the body and in the middle of a stream, 100 parallel connections and 8 parallel chats, then checks that the server still answers:

python scripts/server_abuse.py -- kinetic serve model.gguf -c 512 --port 8091

A client that sends its headers and then stalls is disconnected after 30 seconds (--long checks this). Up to 64 connections are served at once; more get a 503.

Models and formats

Kinetic reads GGUF files as llama.cpp and the Hugging Face GGUF tools write them, and its own .ktc files. It does not convert anything: you point it at a file on your disk.

The quickest check for a file is kinetic info:

kinetic info Qwen3.5-4B-MTP-Q4_K_M.gguf

Qwen3.5-4B-MTP-Q4_K_M.gguf
  name           Qwen3.5-4B
  architecture   qwen35
  size           2.64 GiB, 4.33 B parameters
  context        262144 tokens
  layers         33, width 2560
  prediction     yes (drafts for speculative decoding)
  weights        Q4_K    149 tensors    1478.0 MiB
                 Q6_K     35 tensors    1018.2 MiB
                 Q5_K     24 tensors     165.0 MiB
                 F32     232 tensors      18.7 MiB
                 Q8_0      1 tensors      13.3 MiB
  tokenizer      gpt2 (qwen35), 248320 tokens, BOS none, EOS <|im_end|>
  chat template  7992 bytes

An architecture Kinetic does not know is marked (not supported yet). A known architecture can still use a feature that is not supported yet, such as a kind of rotary scaling; kinetic run then names it when it loads the file.

Architectures

The names are the GGUF general.architecture values.

With --device auto, a model that cannot run on the GPU runs on the CPU and Kinetic prints why. Through Vulkan (--device vulkan), llama, qwen2, qwen3, lfm3 and qwen35 models run today; see Devices.

Embedding and ranking models

kinetic serve answers /v1/embeddings for these and /v1/rerank for rerankers. They run on the CPU.

A file is served as an embedding model when it is an encoder or its metadata sets a pooling type (<arch>.pooling_type).

Vision

With kinetic serve <model> --mmproj <projector>, chat requests may carry images. The projector file names its vision tower in clip.projector_type (or clip.vision.projector_type when it also holds an audio tower):

When the model runs on a CUDA GPU, images are encoded there too. A prompt with images runs on the GPU when every layer of the model is there, and on the CPU otherwise. More projectors are coming.

Adding a projector type takes one file in crates/kinetic-vision/src/:

1. A struct holding the weights, borrowed from the file through Loader.
2. A load function that reads the hyperparameters and tensors and returns the struct boxed as a Projector.
3. impl Projector: n_embd_out, prepare (resize and normalize the way the model's processor does) and encode (pixels to one row per image token, row-major over the token grid).
4. One line in PROJECTORS in lib.rs mapping the clip.projector_type string to load.

Shared pieces live in ops.rs (linear, layer and RMS norms, GELU, bidirectional attention, patch gathering) and preprocess.rs (decoding, the aspect-keeping resize). Check a new type against llama.cpp's llama-mtmd-debug --image rainbow -n <size> with cargo run --release -p kinetic-vision --example encode -- <mmproj> rainbow:<size>: the first and last values and the sum of the final embeddings should match to a few digits. On the text side, an image's rows enter the prompt as Input::Image; models with (time, row, column) rotary positions place them on the image grid, every other model gives them consecutive positions.

Tokenizers

The tokenizer is read from the file. Kinetic has byte-level BPE (tokenizer.ggml.model = gpt2), SentencePiece (llama) and WordPiece (bert). BPE models also name a pre-tokenizer (tokenizer.ggml.pre); these are supported: qwen35, qwen2, llama3, llama-v3, llama-bpe, lfm2, falcon-h1, dbrx, smollm, refact, gpt-2, exaone4 and hunyuan-dense. kinetic info names a missing one on its tokenizer line.

Chat templates and tool calls

The chat template is the file's own Jinja template, rendered as the model's authors wrote it, or the one --chat-template-file names. Tool calls are read in the formats of Qwen3.5, Qwen3, LFM3, LFM2, Llama 3 and Mistral.

Weight formats

These are the GGUF tensor types, by the names kinetic info prints. A file mixes several (Q4_K_M, for one, is mostly Q4_K with some Q6_K).

"Experts only" means the type runs on the GPU as the routed experts of a mixture-of-experts model; in any other tensor it keeps the model on the CPU. The format definitions follow ggml's, so a GGUF file means the same in Kinetic as in llama.cpp.

On the CPU, AVX2 kernels cover the common formats; any other type still runs, through a slower path that dequantizes each row.

Files that do not load

Split files (model-00001-of-00003.gguf) are not read yet; join them first with llama.cpp's llama-gguf-split --merge.

If a model you need is missing, open a "model not supported" issue with the output of kinetic info, or add it: see Adding an architecture.

.ktc files

A .ktc file holds everything needed to run a model, weights, tokenizer and chat template, with a manifest that records the BLAKE3 hash of every section, an optional Ed25519 signature over the manifest, and an execution profile tuned for a kind of hardware. Every byte of the file is either the header, the manifest, zero padding or a hashed section, so nothing can hide in it.

kinetic verify model.ktc

recomputes every hash and reports whether the file is signed; it exits with 1 when anything does not match. Every command that takes a .gguf file takes a .ktc file too. The format is specified in FORMAT.md and versioned on its own; see Versioning.

Devices and offload

Kinetic runs on x86-64 CPUs, on NVIDIA GPUs through CUDA and on AMD, Intel and NVIDIA GPUs through Vulkan, on Windows and Linux. A model larger than the GPU can still use it: a dense model splits its layers between the GPU and the CPU, and a mixture-of-experts model keeps its experts in system memory.

Choosing the device

run, chat, serve and bench take --device:

none is accepted for cpu, cuda0 for cuda and vulkan0 for vulkan, as llama.cpp spells them. -ngl 0 means --device cpu, and any other -ngl N means --gpu-layers N (see Dense models larger than the GPU).

With auto, Kinetic skips a GPU it cannot use with one line that says why:

kinetic: running on the CPU; CUDA is not usable: <reason>; Vulkan is not usable: <reason>

The reasons are: no NVIDIA driver, a build without the CUDA kernels, a model whose weights and caches do not fit in the GPU's free memory, or a model that uses something the GPU code does not have yet (see the CUDA column in Models and formats). The start-up line of kinetic serve and the status line of kinetic run name the device that is used:

Serving Qwen3.5-4B-MTP-Q4_K_M on NVIDIA GeForce RTX 3060 Laptop GPU at http://127.0.0.1:8080/v1
[4 tokens at 273.9 tok/s; prompt 13 tokens at 1727.7 tok/s on NVIDIA GeForce RTX 3060 Laptop GPU]

Some work always runs on the CPU, whatever --device says: prompts that carry images, and embedding and reranking models.

CPU

Any x86-64 CPU runs Kinetic. The fast kernels need AVX2, FMA and F16C, which Intel CPUs have since Haswell (2013) and AMD CPUs since Excavator and every Ryzen; Kinetic detects them at runtime. Without them it still runs, on a much slower path.

-t sets the number of threads; by default Kinetic uses every logical CPU. Generation is limited by memory bandwidth, so more threads than physical cores rarely help, and on some machines fewer threads are faster. Measure with kinetic bench model.gguf --device cpu -t N.

NVIDIA GPUs

GPUs from the Turing generation (RTX 20 series, GTX 16 series) on are supported, with kernels for compute capability 7.5, 8.0, 8.6, 8.9, 9.0 and 12.0, and PTX that newer drivers compile for later GPUs.

Kinetic opens the driver library at runtime (nvcuda.dll on Windows, libcuda.so.1 on Linux) and links nothing from the CUDA toolkit, so:

• you need the NVIDIA driver, not the CUDA toolkit;
• the cuda build starts on machines without a GPU and runs on the CPU.

Use a recent driver; the kernels are built with CUDA 12.8, and nvidia-smi shows the highest CUDA version your driver supports in its header. Only the first GPU is used; two GPUs are coming.

Vulkan

--device vulkan runs models on any GPU with a Vulkan driver: AMD, Intel and NVIDIA. Today it runs llama, qwen2, qwen3, lfm3 and qwen35 models, with drafting and Warm Start; MoE models and the other architectures are coming. Under auto, Vulkan is tried after CUDA, so an NVIDIA GPU uses CUDA and an AMD or Intel GPU uses Vulkan.

Dense models larger than the GPU

When a dense model does not fit in GPU memory, Kinetic puts as many of its last layers on the GPU as fit, with the cache, and runs the first layers on the CPU, exactly as the CPU path runs them. Only one hidden state per token crosses between the two. The status line says how the model was split:

[... on NVIDIA GeForce RTX 3060 Laptop GPU, 24 of 33 layers, the rest on the CPU]

--gpu-layers N puts at most N layers on the GPU, as llama.cpp's -ngl N does; use it to leave GPU memory for other programs. This works through CUDA; on Vulkan a model runs wholly on the GPU or on the CPU.

Memory

Weights are memory-mapped. Kinetic does not copy the model into its own memory; the operating system reads the file's pages as they are used and keeps them in its file cache. On the CPU, a model needs about its file size in RAM to run at full speed. On the GPU, the weights are copied to video memory at load time.

The context sets the cache size. Each conversation gets a cache for up to -c tokens, 32768 by default (less if the model was trained on less; -c 0 takes everything the model supports). On the GPU that cache is reserved at load time, so a model that does not fit with the default context may fit with -c 8192. Models with recurrent layers (Qwen3.5, LFM2, LFM3, Granite 4.0 H, Falcon-H1) need much less cache per token than pure transformers.

A quantized cache holds about twice the context. On CUDA, --cache-type-k q8_0 and --cache-type-v q8_0 (llama.cpp's -ctk and -ctv) store keys and values in blocks of 32 int8 values with one f16 scale, 34 bytes where f16 takes 64. Every token stays in the cache, but its keys and values are rounded, so answers differ from the f16 cache's: close in quality, not identical, and never what the exactness checks compare against. It is off by default. The CPU and Vulkan paths ignore it.

Warm Start keeps prompt states in up to 1 GiB of RAM and 2 GiB on disk, under --cache-dir (by default kinetic/warm in %LOCALAPPDATA% on Windows and $XDG_CACHE_HOME or ~/.cache on Linux). --no-prompt-cache turns it off.

kinetic bench reports the peak memory of a run.

Mixture-of-experts offload

A mixture-of-experts model that does not fit in GPU memory still runs on the GPU with --device auto or cuda. Kinetic puts the attention layers, the shared weights and the cache on the GPU, gives the remaining free memory to a cache of experts, and leaves the other experts in the memory-mapped file. Each layer then runs the experts it picks where that is cheapest at the moment: from the GPU cache, on the CPU, or copied over. See Flywheel for how it decides.

For this to be fast, the whole file should fit in system RAM next to your other programs. Kinetic reads the experts into memory at load time when they fit with 3 GiB to spare, and otherwise lets the first tokens read them as needed.

When every expert fits on the GPU, there is no offload: routing runs on the GPU too.

Coming

• More on Vulkan: more architectures, speculative decoding, Warm Start and layer splits.
• Two GPUs.

See the roadmap.

Python, C and Rust

Besides the command line and the server, Kinetic is a library. All three interfaces load the same .gguf and .ktc files, use the model file's recommended sampling by default and give the same exactness guarantees. A text model runs on the GPU when the build has CUDA and the model fits, otherwise on the CPU, as --device auto does.

Python

pip install kinetic-engine

import kinetic

model = kinetic.Model("SmolLM2-135M-Instruct-f16.gguf")

# Chat through the model's own template.
print(model.chat([{"role": "user", "content": "Name three rivers."}], max_tokens=64))

# Stream the answer as it is written.
for piece in model.chat([{"role": "user", "content": "Tell me a story."}], stream=True):
    print(piece, end="", flush=True)

# Continue a raw prompt; temperature 0 always takes the likeliest token.
print(model.generate("The capital of France is", max_tokens=8, temperature=0))

# Sampled output repeats for the same seed.
print(model.generate("Once upon a time", max_tokens=32, temperature=0.8, seed=42))

# Embedding models turn texts into vectors.
embedder = kinetic.Model("bge-small-en-v1.5-f16.gguf")
vectors = embedder.embed(["a cat", "a dog"])

Without max_tokens an answer runs until it ends or fills the context. Calls release the GIL, so other Python threads keep running while the model works; requests from several threads take turns on one model. Dropping a stream stops its generation.

The published wheels run on the CPU. For the GPU, build the package from source with the CUDA toolkit installed (see Building from source), or run kinetic serve and use the openai package against it, which also gives you tools, JSON output and images.

kinetic.llama_cpp offers the commonly used part of llama-cpp-python's Llama class. pip install llama-cpp-kinetic makes import llama_cpp load it, so code written for llama-cpp-python runs unchanged; an installed llama-cpp-python is shadowed, not touched, and comes back when you uninstall llama-cpp-kinetic. Or change one import:

from kinetic.llama_cpp import Llama   # was: from llama_cpp import Llama

llm = Llama(model_path="model.gguf", n_ctx=4096, n_gpu_layers=-1, seed=42)
out = llm("Q: Name the planets. A:", max_tokens=64, stop=["Q:"])
for chunk in llm.create_chat_completion(messages=[{"role": "user", "content": "Hi"}], stream=True):
    print(chunk["choices"][0]["delta"].get("content", ""), end="")

It covers __call__/create_completion, create_chat_completion (with stream), embed, create_embedding, tokenize, detokenize, n_ctx, token_bos and token_eos, with the same response shapes and defaults. n_gpu_layers=0 runs on the CPU and any other value on the GPU when the model fits. Grammars, tools, response_format and logprobs raise NotImplementedError rather than being ignored.

C

The release archives carry include/kinetic.h and the library: lib/kinetic_capi.dll with its import library kinetic_capi.dll.lib on Windows, lib/libkinetic_capi.so on Linux. To build it yourself: cargo build --release -p kinetic-capi.

This program, examples/chat.c, asks one question and streams the answer:

#include <stdio.h>

#include "kinetic.h"

static bool print_text(const char *text, size_t len, void *user_data) {
    (void)user_data;
    fwrite(text, 1, len, stdout);
    fflush(stdout);
    return true;
}

int main(int argc, char **argv) {
    if (argc != 3) {
        fprintf(stderr, "usage: %s MODEL PROMPT\n", argv[0]);
        return 2;
    }
    kinetic_model *model = kinetic_model_load(argv[1]);
    if (!model) {
        fprintf(stderr, "kinetic: %s\n", kinetic_last_error());
        return 1;
    }
    kinetic_session *session = kinetic_session_new(model, NULL);
    if (!session) {
        fprintf(stderr, "kinetic: %s\n", kinetic_last_error());
        kinetic_model_free(model);
        return 1;
    }
    kinetic_params params = kinetic_params_default();
    params.max_tokens = 128;
    int64_t tokens = kinetic_session_generate(session, argv[2], &params, print_text, NULL);
    if (tokens < 0) {
        fprintf(stderr, "kinetic: %s\n", kinetic_last_error());
    } else {
        fprintf(stderr, "\n[%lld tokens, kinetic %s]\n", (long long)tokens, kinetic_version());
    }
    kinetic_session_free(session);
    kinetic_model_free(model);
    return tokens < 0;
}

Build and run it:

# Linux
gcc -std=c99 -I include examples/chat.c -L lib -lkinetic_capi -o chat
LD_LIBRARY_PATH=lib ./chat model.gguf "Name three rivers."

:: Windows, in a Visual Studio developer prompt
cl /I include examples\chat.c lib\kinetic_capi.dll.lib
copy lib\kinetic_capi.dll .
chat.exe model.gguf "Name three rivers."

The API in brief; kinetic.h documents every function:

Strings are UTF-8. A model may be shared between threads, whose requests take turns on it; a session belongs to one thread at a time. Each turn of a session reuses the work of the previous ones.

Rust

Add the kinetic crate from the repository:

[dependencies]
kinetic = { git = "https://github.com/Veloresearch/kinetic" }

fn main() -> Result<(), kinetic::Error> {
    let model = kinetic::Model::open("model.gguf")?;
    let request = model.request();
    let messages = [kinetic::Message::new("user", "Name three rivers.")];
    let answer = model.chat(&messages, &request, &mut |text| {
        print!("{text}");
        true
    })?;
    println!("\n{} tokens", answer.completion_tokens);
    Ok(())
}

model.request() starts from the file's recommended sampling; change its sampling (temperature, top-k, top-p, seed and the rest) and max_tokens fields before passing it. generate continues a raw prompt and embed returns vectors from an embedding model. The cuda feature is on by default; default-features = false builds a CPU-only library.

The crates below kinetic (kinetic-models, kinetic-server and the others) are the engine itself. Their APIs are public so that the tools in this repository can use them, but they change between releases.

How Kinetic is fast

Kinetic gets its speed from doing less work per token, never from changing the answer. RAD-1, Warm Start and Shortlist give exactly the output plain decoding gives on the same device, and the test suite checks that on real models. Flywheel changes where the work of a mixture-of-experts model runs.

Generating one token on a personal computer is limited by memory bandwidth, not arithmetic: a dense model reads all of its weights once per token. A 4-billion-parameter model in Q4_K_M is about 2.8 GB, so a GPU with 300 GB/s of bandwidth tops out near 110 tokens per second however fast its cores are. There are four ways around that wall without touching the answer:

None of the ideas is new on its own; the references are below. What Kinetic adds is that they are exact by construction, on by default and need nothing from you: no draft model to pick, no cache to size, no flags.

RAD-1: exact speculative decoding

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) guesses the next few tokens cheaply, then runs the model once over all of them. Reading the weights once costs about the same for one token as for eight, so every correct guess is a token for free. The model keeps the guesses it agrees with, adds one token of its own, and the rest are rolled back.

context   ...  let total = count + 1;  ...  rename count to n:  let total =
draft     n      +      1      ;      }          <- guessed, almost free
verify    one pass over the weights scores all five positions
model     n      +      1      ;      \n         <- what plain decoding picks
kept      n      +      1      ;      \n         4 accepted + 1 = 5 tokens, 1 pass

Where the guesses come from

RAD-1 needs no separate draft model. It runs several cheap sources side by side and lets whichever has lately been right most propose the next draft:

• The context. The tokens that followed the last occurrence of the current few tokens, as in prompt lookup decoding. Right whenever the answer repeats the input: edits of code, quotes, structured output.
• Earlier requests. An index of every token the server has seen, across requests. Agents resend the same files and tool output over and over; this source drafts from all of it.
• The model's own prediction block. Some models ship one or more multi-token prediction (MTP) layers next to the main model (Gloeckle et al., 2024; DeepSeek-V3), as Qwen3.5 does. Kinetic runs that block to draft from the model's hidden state, in the spirit of Medusa (Cai et al., 2024) and EAGLE (Li et al., 2024). kinetic info shows prediction yes for such files.

A draft can branch into a tree of alternatives, all checked in the same pass, with each row seeing only its own path (as in SpecInfer, Miao et al., 2023). Draft length follows how well recent drafts did, and after drafts that miss outright, drafting pauses for longer and longer, so text that cannot be guessed costs almost nothing extra.

Why the output does not change

Verification chooses each token exactly as plain decoding would: from the same logits, with the same history, by the same rule.

• Greedy decoding (temperature 0) keeps a guess only when it is the model's top token. The output is plain greedy decoding, token for token.
• Sampling uses rejection sampling against a drafter that proposes one token with certainty: a guess x survives with probability p(x), and on rejection the token is drawn from p without x, renormalized. Every kept token is drawn from exactly the distribution plain sampling uses, with one random number per token in the same order, so the same seed gives the same answer with drafting and without.

Both hold only if running eight tokens at once gives each of them the same numbers as running it alone. Kinetic's kernels are written to be batch-invariant, bit for bit, and every architecture is tested for it: a batch must give the logits its tokens give one by one, a tree row the logits of its path alone, and drafted decoding the tokens of plain decoding. kinetic bench repeats the last check on your machine and prints drafted output identical to plain decoding: yes.

Warm Start

Chat clients and agents send the whole conversation with every request: the same system prompt, the same tool definitions, the same history, and a few new tokens at the end. Without a cache, every request pays for all of it again.

request 1   [system prompt][tools][user: fix the bug        ]
request 2   [system prompt][tools][user: fix the bug][answer][user: now add a test]
            |<---------- resumed from request 1's state ---------->|<-- runs -->|

Warm Start keeps the states that earlier prompts left behind and starts each new prompt from the longest prefix it can return to exactly. This is prefix caching, as in vLLM and SGLang, and llama.cpp's prompt cache.

Attention layers can be cut back to any length. Recurrent layers, such as the Gated DeltaNet layers of Qwen3.5, the short convolutions of LFM2 and LFM3 and the Mamba-2 layers of Granite 4.0 H and Falcon-H1, cannot be rewound; for those Kinetic records a snapshot every 256 tokens (or a larger multiple when a snapshot is large next to what it saves), and resumes from the last snapshot before the prompts part.

• States are kept in memory (up to 1 GiB) and in your cache directory (up to 2 GiB, least recently used out first), so they survive a restart.
• A state is restored only when the prompt processing it saves outweighs restoring it, at speeds Kinetic measures as it goes; files are written in the background.
• A restored state is bit for bit the state its run left behind, and prompts are split into the same batches cold or warm, so a warm start gives exactly the answer of a cold one.

The usage.prompt_tokens_details.cached_tokens field of a server response says how many prompt tokens were not recomputed. --no-prompt-cache turns Warm Start off and --cache-dir moves it; see the command line.

Zero Wait

Warm Start makes the history free; the message itself still has to run after Enter, and a pasted 20,000-token document takes seconds. Zero Wait runs it while it is being written:

typing      [history][user: <pasted document> What does the second sec|
running     [history][user: <pasted document> What does the]           ahead, in batches
after Enter                                          [ second section say?][assistant]

kinetic chat sends every edit to a thread that runs the conversation up to just before the last word of the message (its tokens may still merge with the next keystroke) and never the end of the turn. Apps do the same through /v1/prefill.

• It runs only the whole batches the request would run first, from where the request would start, so the request continues with exactly the batches it would have run itself and gives exactly the same answer.
• When earlier text changes, it cuts back to the last batch both still share; on models with recurrent layers it records a snapshot where it started and on the Warm Start grid to return to.
• It gives way after every batch: a request never waits for more than one. Input never waits at all, since the model runs on its own thread.

Shortlist

Choosing the next token normally reads the whole output layer: one row of weights per entry of a vocabulary that can be a quarter of a million tokens long. In a small model that layer is a large share of all the bytes read per token.

Shortlist keeps a smaller Q4_K copy of the output layer with a proven bound on the error of every row. Each token then:

1. scores every row with the copy, which gives an upper bound for each row's true score;
2. computes the exact scores of the best-looking rows with the original weights, which gives a floor the winners cannot be below;
3. computes exact scores for the few other rows whose bound reaches that floor, and skips the rest.

copy score + bound    ████████▌   ███████▌   ██████▏   ███▊   ██▍   ...   (250k rows)
exact floor           ─────────────────────────┤
exact scores for      row 1, row 2, row 3                 skipped: everything below the floor

The chosen tokens, their order and their scores are those of the full layer, bit for bit, because every exact score comes from the layer's own kernel. Shortlist serves greedy decoding, top-k sampling and draft verification. It works on the CPU, for output layers stored as Q6_K or Q8_0.

Flywheel

A mixture-of-experts (MoE) model has many expert feed-forward blocks per layer and uses only a few of them for each token: Qwen3.5-35B-A3B uses about 3 billion of its 35 billion parameters per token. The model is large, but the work per token is small, if the right experts are at hand.

GPU memory    [attention, shared weights, KV cache][ expert cache: the experts in use ]
system RAM    [ every expert, memory-mapped from the model file                     ]

per layer     the router picks a few experts
              in the cache   -> run on the GPU
              not in cache   -> run on the CPU, or copy to the GPU, whichever is cheaper now
              next layer     -> likely experts prefetched while this one runs

Flywheel gives the experts all the GPU memory left after the rest of the model, one sequence's state and a reserve. It measures how fast the CPU runs an expert and how fast one copies to the GPU, keeps adjusting for other programs that slow the CPU down, and splits the work accordingly. When every expert fits, routing happens on the GPU and each one-token step runs as one recorded CUDA graph. Expert offloading itself is known (for example Eliseev and Mazur, 2023, and llama.cpp's --override-tensor); Flywheel makes the placement automatic.

See Devices and offload for what runs where.

References

• Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023. arXiv:2211.17192
• Chen, Borgeaud, Irving, Lespiau, Sifre, Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. 2023. arXiv:2302.01318
• Miao et al. SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification. ASPLOS 2024. arXiv:2305.09781
• Cai et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML 2024. arXiv:2401.10774
• Li, Wei, Zhang, Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. arXiv:2401.15077
• Gloeckle et al. Better & Faster Large Language Models via Multi-token Prediction. ICML 2024. arXiv:2404.19737
• DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. arXiv:2412.19437
• Saxena. Prompt Lookup Decoding. 2023. github.com/apoorvumang/prompt-lookup-decoding
• Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). SOSP 2023. arXiv:2309.06180
• Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024. arXiv:2312.07104
• Eliseev, Mazur. Fast Inference of Mixture-of-Experts Language Models with Offloading. 2023. arXiv:2312.17238
• llama.cpp and ggml, which define the GGUF format and its quantization types.

Benchmarks

Speed claims are only useful if you can check them on your own machine. This page describes how Kinetic is measured against llama.cpp, how to repeat it, and the results on our reference machines.

Method

• Same file, same threads. Both engines run the same GGUF file. On the CPU both get the same thread count; on the GPU both run the whole model on the device.
• The llama-bench tests. pp512 processes a 512-token prompt and tg128 generates 128 tokens. kinetic bench runs the same two tests as llama-bench, with speculative decoding off, so every generated token costs one pass over the weights on both sides. Each number is the best of the repetitions.
• An editing request. Real use often repeats text: an assistant rewriting code, quoting a document, filling a template. The edit test sends one greedy chat request that renames a variable in a short Python file and asks for the whole file back, to kinetic serve and to llama-server with llama.cpp's own speculative decoding on (--spec-type ngram-mod, or draft-mtp for models with a prediction block). Speed is generated tokens over the time from the first to the last token, measured on the client the same way for both. Both answers are greedy, so they should be identical; the script reports whether they are.
• Exactness is checked, not assumed. kinetic bench also runs its own edit test and fails if drafted output differs from plain decoding.
• A quiet machine. No other programs using the CPU or the GPU, laptops on mains power with the performance power plan, best of three runs.

Reproduce it

You need a llama.cpp build with llama-bench and llama-server and a model file. On the CPU, one script runs everything and prints a markdown table:

python scripts/bench_vs_llamacpp.py model.gguf --llama-dir path/to/llama.cpp/bin -t 6

The script hides the GPUs from llama.cpp and passes --device cpu to Kinetic, starts one server at a time on 127.0.0.1 and stops it afterwards.

On the GPU, run the two tools by hand:

kinetic bench model.gguf --device cuda -r 3
llama-bench -m model.gguf -p 512 -n 128 -r 3 -ngl 99 -fa on

Results

Kinetic 0.1.0 against llama.cpp build b12001, tokens per second, higher is better. Measured on a laptop with a Ryzen 5 6600H (6 cores, DDR5) and an RTX 3060 Laptop GPU (6 GB).

Prompt processing on the GPU (pp512) is an area where llama.cpp is ahead today; it is on the roadmap.

Share your numbers

Results from other hardware are welcome. Open a performance issue with the script's output, or the two commands above and their output, your CPU, GPU, RAM and driver version, and the llama.cpp build you compared with.

Receipts

Kinetic's output does not depend on anything outside the request: not on speculative drafting, not on other users sharing the batch, not on the prompt cache, not on a prompt run ahead while it was typed. A request gives the same tokens alone on an idle machine as on a server answering four users at once with every speed technology on. A receipt turns that into something you can keep and check.

Same input, same output

scripts/same_answer_demo.py sends one chat request ten times at temperature 0.8 with a fixed seed while three other clients keep the server busy with different long requests, and hashes every answer. LFM3-1.7B Q4_K_M on a Ryzen 5 6600H, CPU only, both servers with --parallel 4:

Kinetic, kinetic serve --parallel 4
run  answer (first 40 characters)                sha256 of the full answer
  1  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  2  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  3  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  4  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  5  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  6  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  7  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  8  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
  9  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
 10  **Lunar Lattice Café** — its name evokes    070ad8ff1e6fc95f
     1 distinct answer in 10 runs

llama.cpp, llama-server --parallel 4
run  answer (first 40 characters)                sha256 of the full answer
  1  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
  2  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
  3  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
  4  **Lunar Bean & Crater Café**  The name e    9b21683015c20844
  5  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
  6  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
  7  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
  8  **Lunar Bean & Crater Café**  The name e    30430d6b74effe9b
  9  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
 10  **Lunar Bean & Crater Café**  The name e    cac5bf18cb341aad
     3 distinct answers in 10 runs

Kinetic gives the same answer every time. llama-server gives the same first words but three different answers in ten runs (which runs differ, and how, changes from one try to the next), most likely because how a request is batched with the others changes how its sums are rounded, and sampling turns a tiny difference into another token. Run it yourself:

python scripts/same_answer_demo.py --model LFM3-1.7B-Q4_K_M.gguf \n    --kinetic kinetic --kinetic-args "--device cpu" --receipt last.json \n    --llama-server llama-server --llama-args "-dev none -ngl 0 -c 16384"

The receipt of Kinetic's last run, made on the busy server, then verifies as identical when rerun alone with drafting off. On an RTX 3060 Laptop GPU (--kinetic-args "--device cuda", class cuda-sm86) Kinetic also gives one distinct answer in ten runs, and that receipt verifies the same way.

Making inference batch-invariant is not a new idea: it has been described before (Thinking Machines, 2025) and vLLM has a batch-invariant mode. What Kinetic adds is that it is the default, together with drafting, the prompt cache, several users and run-ahead, at full speed.

Making a receipt

kinetic run model.gguf "Name three rivers." --temp 0.8 --seed 42 --receipt answer.json

In kinetic chat, /receipt answer.json saves the receipt of the last answer. On the server, "receipt": true on /v1/chat/completions or /v1/completions adds a receipt object to the response; see Server API. A request without a seed gets a fresh one, which the receipt records.

A real receipt, signed:

{
  "receipt": 1,
  "kinetic": {
    "version": "0.1.0",
    "build": "378534cb64f30a15"
  },
  "model": {
    "file": "LFM3-1.7B-Q4_K_M.gguf",
    "path": "LFM3-1.7B-Q4_K_M.gguf",
    "size": 1099627808,
    "blake3": "f4240740be2392a21da1668fab289a441684b47db70bd96d8f863b576f44bfb4"
  },
  "device": {
    "class": "cpu-x86_64-avx2",
    "name": "the CPU",
    "settings": {
      "ctx-size": 32768
    }
  },
  "input": [
    {
      "role": "user",
      "content": "Name three rivers."
    }
  ],
  "prompt": {
    "count": 15,
    "blake3": "591484bfe2293f5c7603c970515f9d044a607c9e5ef92bbdc51c05e474672f73",
    "ids": [124894, 124899, 5922, 207, 2617, 1703, 15311, 22, 124900, 207, 124899, 63514, 207, 124901, 124902]
  },
  "sampling": {
    "temperature": 0.8,
    "top_k": 50,
    "top_p": 0.95,
    "min_p": 0.05,
    "repeat_penalty": 1.1,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "repeat_last_n": 64,
    "seed": 42,
    "max_tokens": null,
    "stop": []
  },
  "output": {
    "finish": "stop",
    "text": "1. The Nile River  \n2. The Amazon River  \n3. The Mississippi River",
    "tokens": {
      "count": 18,
      "blake3": "7bd0632303b6fcbdb7f440d8c19415fa73e0afacd6e941137104bf438ebd8662",
      "ids": [25, 22, 440, 35844, 5094, 3103, 26, 22, 440, 12225, 5094, 3103, 27, 22, 440, 17939, 5094, 124900]
    }
  },
  "signature": {
    "public_key": "8e79e51c11472a6cffd70635753a7cce14c4335e1f18435459c3ebb35b3f720b",
    "ed25519": "2dcfee41a820c6294aa1dbf60c916f1dc5ebab6b9575cb87b3f07f3f29f079dcd40020e0423be7df33f53471840ccd1bdcb604763b03ceeb89dfa2a741f2af00"
  }
}

Token hashes are BLAKE3 over the ids as little-endian 32-bit integers.

Verifying

kinetic verify-receipt answer.json

receipt:   LFM3-1.7B-Q4_K_M.gguf, 15 prompt and 18 output tokens, seed 42, made on cpu-x86_64-avx2 by Kinetic 0.1.0 (build 378534cb64f30a15)
signature: valid, public key 8e79e51c11472a6cffd70635753a7cce14c4335e1f18435459c3ebb35b3f720b
model:     LFM3-1.7B-Q4_K_M.gguf matches (BLAKE3 f4240740be2392a2)
rerun:     identical, 18 of 18 output tokens on cpu-x86_64-avx2 (BLAKE3 7bd0632303b6fcbd)

verify-receipt checks the token hashes and the signature, hashes the model file (or takes the hash from the cache) and refuses a file that is not the receipt's, then reruns the recorded prompt tokens with the recorded settings on one slot with RAD-1 drafting, Warm Start, Zero Wait and Convoy off: the plainest way Kinetic can decode. It prints identical, or the first output token that differs together with what differs between the two runs. --model points at the model when it moved, and --device reruns elsewhere.

Signing

openssl rand -hex 32 > receipt.key
kinetic run model.gguf "Name three rivers." --receipt answer.json --sign-key receipt.key
kinetic serve model.gguf --sign-key receipt.key

The key file holds an Ed25519 secret key as 32 bytes or 64 hex digits; keep it private. The signature covers every other field of the receipt as compact JSON, so changing the seed, a setting, a token or the text breaks it. verify-receipt prints the public key it checked against: a signature only says who made the receipt if you know that key is theirs.

What a receipt proves, and what it does not

A receipt that verifies as identical proves that this model file, run by this build on this device class with these settings, turns these prompt tokens into these output tokens. Anyone with the same file and build can check it, on any machine of the same class, without trusting the server that made it.

It does not prove:

• Anything across device classes. The CPU, CUDA and Vulkan paths sum in different orders, so they round differently; bits are promised only within one class and one build. A receipt from a GPU reruns on the same compute capability (sm_86 to sm_86). verify-receipt says so when the class or build differs, and a difference then is expected rather than a bug. Different CPU core counts are fine: threads split rows, not sums. On Vulkan the driver compiles the shaders, so a different driver version may round differently too.
• That the answer is correct. A receipt pins how an answer was made, not whether it is true.
• That the server ran the receipt's model, unless you rerun it. A receipt is a claim until verify-receipt reproduces it; a signature only shows who made the claim.
• Images. Requests with images cannot have a receipt yet.

The text in output.text is decoded from the output tokens after stop strings and reasoning are split off; the tokens are what is compared.

Building from source

Kinetic is a Cargo workspace of small crates. Building it needs only Rust; the CUDA kernels also need the CUDA toolkit.

Requirements

• Rust 1.91 or newer. Install it with rustup.
• Windows: the MSVC build tools (Visual Studio 2022 or its Build Tools, with "Desktop development with C++").
• Linux: a C toolchain (build-essential on Debian and Ubuntu).
• For NVIDIA GPUs: the CUDA toolkit 12.8 or newer, for nvcc. It is needed only at build time: the binary opens the driver at runtime and links no CUDA library.

Build

git clone https://github.com/Veloresearch/kinetic
cd kinetic
cargo build --release -p kinetic-cli
target/release/kinetic --version

The cuda feature is on by default. The build looks for nvcc in $NVCC, then in $CUDA_PATH/bin, then on PATH. Without it the build still succeeds and prints warning: nvcc not found: CUDA kernels not built; that binary runs on the CPU and says so when you ask for --device cuda. To build without CUDA at all:

cargo build --release -p kinetic-cli --no-default-features

On Windows, nvcc needs cl.exe. From a plain shell, the build finds the newest MSVC toolset through vswhere and passes it to nvcc, so a developer prompt is not required.

The kernels are compiled for every NVIDIA architecture from Turing (sm_75) to Blackwell (sm_120), plus PTX that newer drivers can compile for later GPUs.

Install the binary into ~/.cargo/bin:

cargo install --path crates/kinetic-cli

The libraries

cargo build --release -p kinetic-capi        # kinetic_capi.dll / libkinetic_capi.so

The Python package lives outside the workspace, so that building the engine never needs Python, and builds with maturin:

cd bindings/python
pip install "maturin>=1.5,<2"
maturin build --release
pip install target/wheels/kinetic_engine-*.whl

Docker

docker build -f docker/Dockerfile -t kinetic .              # CPU
docker build -f docker/Dockerfile.cuda -t kinetic:cuda .    # NVIDIA GPUs

Both images run kinetic serve on port 8080 as an unprivileged user; see Install.

Tests

cargo test --release --workspace

Tests that need a real model are marked #[ignore]. Point them at a file with KINETIC_TEST_GGUF and include them:

KINETIC_TEST_GGUF=SmolLM2-135M-Instruct-f16.gguf \
  cargo test --release -p kinetic-models --test architectures -- --ignored

The CUDA kernel tests run when a GPU and the kernels are present and print skipped: ... otherwise; the model tests on the GPU take --features cuda on kinetic-models. Test in release mode: real models in a debug build are very slow.

Checks before a pull request

cargo fmt --all -- --check
cargo clippy --release --workspace --all-targets --all-features -- -D warnings
cargo test --release --workspace

CI runs the same on Windows and Linux, builds the C example with gcc and cl, and builds and tests the Python package.

The documentation

This book is built with mdBook:

cargo install mdbook --locked
python scripts/llama_flags_doc.py    # refresh the llama.cpp flag table
mdbook serve docs

Contributing

Bug reports, benchmarks on your hardware, fixes and new model support are all welcome. The short version of the rules is in CONTRIBUTING.md; this chapter explains how the code fits together and walks through the two most common contributions: a new architecture and a new weight format.

For anything larger than a small fix, open an issue first so we can agree on the approach. The good first issues are a good place to start.

How the code is laid out

What every change must keep

• Exactness. Anything that speeds up decoding must give the same tokens as plain decoding. In practice this means a batch of tokens must produce, bit for bit, the logits those tokens produce one at a time. The identity tests below check it; a change that breaks them is not mergeable, however fast.
• Own code. No code copied from other inference engines. Reading llama.cpp's sources to understand a format or an architecture is fine and expected; tables and grids of a format are data, and their source goes into NOTICE.
• No network. Kinetic never connects to the network, except when you run kinetic pull. Nothing else may open a connection.
• Small files with one job, doc comments on public items, comments only where the reason is not obvious, unsafe only at FFI, SIMD and kernel boundaries with a // SAFETY: comment.
• Numbers for speed claims: model, hardware, command, before and after, best of several runs.

Adding an architecture

Adding a model family walks through it: a scaffold script, the blocks and what each covers, mapping GGUF tensors, the family descriptor, the conformance check against llama.cpp and the identity gates.

Adding a weight format or a kernel

Every format is defined once, by its reference dequantization, and every fast kernel is tested against that definition.

1. The type. crates/kinetic-quant/src/types.rs: the GGUF type id, elements and bytes per block, and the name.
2. The definition. A block decoder in crates/kinetic-quant/src/dequant/, wired into dequantize. Straightforward scalar code; this is what "correct" means for the format. Lookup grids go in grids/ with their source noted in NOTICE.
3. The CPU kernel. A dot product against 8-bit activations in crates/kinetic-cpu/src/kdot/, with an AVX2 version where it pays. kdot/tests.rs compares every kernel with the reference on random blocks. Until a kernel exists, matvec falls back to dequantizing rows, so a new format works on the CPU from step 2 on.
4. The CUDA kernel. A GEMV in crates/kinetic-cuda/cuda/, its entry in WEIGHT_TYPES (src/gemv.rs) and its field layout (src/layout.rs), tested in tests/gemv_quants.rs against the reference and an f64 dot product, alone and batched. Expert kernels for mixture-of-experts models are separate (src/moe.rs, tests/moe.rs).
5. The docs. The format table in Models and formats.

Batched kernels must give each row exactly what the single-row kernel gives. That is what keeps speculative decoding and Warm Start exact; the GEMV tests check it.

Sign your work

Kinetic uses the Developer Certificate of Origin (DCO) 1.1. By adding a Signed-off-by line to a commit you certify that you wrote the change or otherwise have the right to submit it under the project's license:

git commit -s -m "models: add lfm2"

Pull requests with unsigned commits cannot be merged. To sign commits you already made, run git rebase --signoff main and force-push your branch.

Kinetic is licensed under the Apache License 2.0, and contributions are accepted under the same license.

Adding a model family

A family is one file in crates/kinetic-models/src/arch/ that maps a GGUF file's tensors onto shared blocks, plus one line in the family list. The file declares a Family: the GGUF architecture names it answers to, the function that builds its layers, and the few ways its embedding and head differ. Loading, kinetic info, kinetic models and the encoder checks all read the list, so nothing else needs registering.

The short path

python scripts/new_family.py model.gguf          # prints the layout, writes arch/<name>.rs, registers it
# map the tensors of one layer onto blocks in arch/<name>.rs
cargo build --release -p kinetic-cli
python scripts/conformance.py model.gguf --kinetic target/release/kinetic --llama path/to/llama.cpp
KINETIC_TEST_GGUF=model.gguf cargo test --release -p kinetic-models --test architectures -- --ignored

new_family.py reads only the header. It prints the architecture name, the <arch>.* metadata and the tensors of layer 0 with their shapes, then writes a starter that builds: a Llama-style stack of attention and a gated feed-forward. Change what differs and delete what the model lacks.

Worked example: LFM2

LFM2 is gated short convolutions and attention with query and key norms. Its pull request touches exactly these files:

crates/kinetic-models/src/arch/lfm2.rs     new, about 80 lines
crates/kinetic-models/src/arch/mod.rs      two lines
docs/src/models.md                         one table row
README.md                                  the model in the supported list
CHANGELOG.md                               one line

The registration is the module and one entry in FAMILIES:

 pub(crate) mod lfm2;
 ...
 pub(crate) const FAMILIES: &[Family] = &[
     qwen35::FAMILY,
     lfm3::FAMILY,
+    lfm2::FAMILY,

The family file starts with its descriptor. LFM2 stores the final norm under another name, so it overrides one field of the default decoder:

#![allow(unused)]
fn main() {
pub(crate) const FAMILY: Family = Family {
    output_norm: "token_embd_norm.weight",
    ..Family::decoder(&["lfm2"], |l, b| Ok(layers(l, b.d, b.eps)?.into()))
};
}

and then builds one Layer per block; a layer without key-value heads is a convolution layer:

#![allow(unused)]
fn main() {
pub(crate) fn layers<'a>(l: &Loader<'a>, d: usize, eps: f32) -> Result<Vec<Layer<'a>>, LoadError> {
    let n_layer = l.usize("block_count")?;
    let n_head_kv = l.array("attention.head_count_kv", n_layer, |v| {
        v.as_u64().map(|n| n as usize)
    })?;
    // ...
    (0..n_layer)
        .map(|i| {
            let t = |s: &str| format!("blk.{i}.{s}");
            let mixer = if n_head_kv[i] == 0 {
                Mixer::ShortConv(ShortConv {
                    in_proj: l.matrix(&t("shortconv.in_proj.weight"), d, 3 * d)?,
                    // ...
                })
            } else {
                Mixer::Attention(Attention { /* ... */ })
            };
            Ok(Layer {
                mixer_norm: Some(l.vector(&t("attn_norm.weight"), d)?),
                mixer,
                ffn_norm: Some(l.vector(&t("ffn_norm.weight"), d)?),
                ffn: Some(l.swiglu(i, d, n_ff)?),
                // ...
            })
        })
        .collect()
}
}

The descriptor

#![allow(unused)]
fn main() {
pub(crate) const FAMILY: Family = Family {
    names: &["myarch"],                    // GGUF general.architecture values it answers to
    body: Body::Decoder(build),            // Decoder, Bidirectional (embeddings only) or Encoder
    output_norm: "output_norm.weight",     // tensor of the norm before the output head
    sqrt_embd_scale: false,                // embeddings times sqrt(width), as in Gemma
    logit_scale: false,                    // logits divided by <arch>.logit_scale, as in Granite
};
}

Family::decoder(names, build) fills in the usual values; override a field with ..Family::decoder(...) as LFM2 does. A builder gets the Loader and the Basics every family reads (arch, the width d, the norm epsilon eps and the weight folds) and returns a Stack: the layers and, for a model with a multi-token prediction block, that block (see qwen35.rs). Most builders return layers(...)?.into().

One file may declare several descriptors when variants share a builder: transformer.rs declares the Llama-like family, Gemma 3 and EmbeddingGemma, and its builder reads b.arch for the details that differ. A plain transformer that differs from Llama only in rotary layout, biases or which norms exist belongs there: add its name to transformer::FAMILY and express the difference with the flags that file already has, the way olmo2 and exaone4 are.

Blocks

crates/kinetic-models/src/blocks/ holds the building blocks. A Layer is an optional norm, a mixer, an optional output norm, then the same around a feed-forward:

Layer also has residual_scale for models whose residual branches carry a fixed multiplier (Granite).

Mapping tensors

The Loader reads everything and checks every shape, so a wrong mapping fails at load time with the tensor's name rather than giving bad output:

GGUF stores a matrix with the input width first: blk.0.attn_q.weight [576, 576] in the scaffold's listing is l.matrix(.., 576, 576). For what each tensor does, the model's reference implementation (modeling_*.py on Hugging Face) and llama.cpp's src/models/ are the specifications. Read them; do not copy them.

A tensor the blocks have no place for must make loading fail, not be skipped: list it the way transformer.rs lists UNSUPPORTED.

When a new block is needed

Open an issue first. A new mixer or feed-forward lands in this order:

1. The CPU reference in blocks/: the plain, obviously correct version, with a batch path that gives each token bit for bit what it gets alone.
2. Its state, checkpoint and rollback in blocks/snapshot.rs, so drafting and Warm Start work.
3. CUDA in cuda/ and Vulkan in vulkan/, each checked against the CPU reference; or, until then, a clear refusal at upload (unsupported("...") in vulkan/layer.rs), so the model runs on the CPU instead of giving wrong output on the GPU.

Testing

Compare greedy output with llama.cpp. scripts/conformance.py runs kinetic serve and then llama-server on the same file, one at a time, sends a few fixed raw prompts with greedy sampling, and reports how many tokens agree. Where the two part it prints llama.cpp's two likeliest tokens there; a gap under 0.1 nats is a near-tie that differently rounded kernels may break either way, an early divergence without one is a bug, and the script exits with 1. On an RTX 3060 Laptop, LFM3 and Qwen3.5 part only at near-ties, and Qwen3.5's fourth answer ends after 32 tokens in both:

$ python scripts/conformance.py LFM3-1.7B-Q4_K_M.gguf --kinetic target/release/kinetic --llama llama-b12001 --device cuda
LFM3-1.7B-Q4_K_M.gguf on cuda, 64 greedy tokens per prompt
prompt 1: 64/64 tokens agree
prompt 2: 64/64 tokens agree
prompt 3: 1/64 tokens agree; near-tie at token 1: llama.cpp top two 'Set' -1.167, 'Create' -1.253 (gap 0.086), kinetic 'Create a dedicated works'
prompt 4: 64/64 tokens agree
total: 193/256 tokens agree

$ python scripts/conformance.py Qwen3.5-4B-MTP-Q4_K_M.gguf --kinetic target/release/kinetic --llama llama-b12001 --device cuda
Qwen3.5-4B-MTP-Q4_K_M.gguf on cuda, 64 greedy tokens per prompt
prompt 1: 3/64 tokens agree; near-tie at token 3: llama.cpp top two 'A' -1.484, 'The' -1.499 (gap 0.014), kinetic 'The capital of France is'
prompt 2: 64/64 tokens agree
prompt 3: 64/64 tokens agree
prompt 4: 32/32 tokens agree
total: 163/224 tokens agree

Then run the identity gates, the contract every family meets:

KINETIC_TEST_GGUF=model.gguf cargo test --release -p kinetic-models --test architectures -- --ignored
KINETIC_TEST_GGUF=model.gguf cargo test --release -p kinetic-models --features cuda --test cuda -- --ignored

• a batch gives the logits its tokens give one by one, bit for bit;
• every row of a tree batch gives what its path gives alone;
• speculative decoding reproduces plain greedy decoding token for token.

Last, kinetic run, kinetic bench (it must print drafted output identical to plain decoding: yes) and kinetic serve with a real chat. The tokenizer and chat template usually need nothing; a new BPE pre-tokenizer (tokenizer.ggml.pre) goes in crates/kinetic-tokenizer/src/lib.rs, tested against llama-tokenize on the same file.

Document it

Add the family to the table in Models and formats, to the README's supported models and to the changelog.

FAQ and troubleshooting

Questions

Do I need a GPU? No. Kinetic is built to be fast on an ordinary CPU with AVX2. An NVIDIA GPU makes it faster.

Does it run on macOS, or on ARM? No. Kinetic supports Windows and Linux on x86-64.

Is the output really the same with all the speed-ups? Yes. Greedy decoding gives the same tokens as plain decoding, and sampling with a seed gives the same answer for the same seed. The test suite checks this on real models, and kinetic bench checks it on your machine. The CPU and the GPU round differently, so a CPU answer and a GPU answer to the same prompt can differ; each is exact for its device.

Will my llama.cpp models work? If kinetic info lists the architecture as supported and the file loads, yes, unchanged. See Models and formats.

Does Kinetic collect data or need an account? No. Kinetic never connects to the network, except when you run kinetic pull. run, chat and serve never download anything, and there is no telemetry and no update check.

What does Kinetic store on my disk? Only Warm Start's prompt states, in kinetic/warm under your cache directory (%LOCALAPPDATA% on Windows, ~/.cache on Linux), up to 2 GiB. They are derived from your prompts, so treat them like your chat history. Delete the folder to clear them, or run with --no-prompt-cache to keep nothing.

Can I use Kinetic in a commercial product? Yes, under the Apache License 2.0. The name and logo have their own rules; see TRADEMARKS.md.

How do I report a bug or a vulnerability? Bugs in an issue; security problems privately to contact@veloresearch.com, see SECURITY.md.

Troubleshooting

--device auto uses the CPU although I have an NVIDIA GPU

Kinetic prints the reason on one line, starting with kinetic: running on the CPU, the GPU is not usable:. The common ones:

• The driver is missing or old. Install the current driver from NVIDIA. nvidia-smi should list your GPU and, in its header, a CUDA version of 12.8 or newer.
• You have the cpu archive or a build made without nvcc. Use a cuda archive, or see Building from source.
• The model does not fit. Through CUDA, a dense model that does not fit runs its first layers on the CPU (the status line says how many are on the GPU); through Vulkan it needs to fit whole. If not even one layer fits beside the cache, try a smaller context (-c 8192), close other programs that use the GPU (games, browsers with hardware acceleration, another model), or use a smaller quantization.
• The model uses something the GPU code does not have yet, such as an architecture Vulkan does not run yet. These run on the CPU for now.
• The GPU is older than Turing (GTX 10 series and earlier). These are not supported.

--device cuda turns any of these into an error instead of a fallback.

Out of memory

• On the GPU: lower -c; the default context of 32768 tokens reserves its cache at load time. Use --device cpu to run from system memory.
• On the CPU: a model needs about its file size in RAM, plus the cache. Close other programs or pick a smaller quantization; a Q4_K_M file is about a third of the size of the F16 one.
• Mixture-of-experts models larger than GPU memory run with their experts in system RAM; they are fast only when the whole file fits in RAM.

It is slow on the CPU

• Check that your CPU has AVX2 (Intel Core since 2013, every AMD Ryzen). Without it Kinetic runs on a much slower path.
• Try fewer threads: kinetic bench model.gguf --device cpu -t 6 with the number of physical cores often beats the default of all logical CPUs.
• On a laptop, plug in and choose the best-performance power mode.
• Other programs using the CPU slow generation down a lot.

"architecture X is not supported yet" or another load error

See Files that do not load for what each message means. Then open a model not supported issue with the output of kinetic info.

The model answers in a strange way

• Check the sampling: some GGUF files carry their own defaults, such as a repeat penalty. Compare with --temp 0 --repeat-penalty 1.
• Make sure the file has a chat template (kinetic info shows its size). A base model without one is meant for /v1/completions, not chat.
• Compare with llama.cpp on the same file and prompt, greedy on both sides. If they differ early in the answer, please file a bug.

Windows says the file is not commonly downloaded

New releases have no download history yet, so SmartScreen may show "Windows protected your PC" the first time you run kinetic.exe. Check the archive against SHA256SUMS from the release page, then choose More info and Run anyway. You can also unblock the archive before you unpack it: right-click it, Properties, tick Unblock.

My antivirus flags or slows down kinetic.exe

Some antivirus programs flag new, unsigned binaries, or scan model files every time they are read, which makes loading slow. Verify the download against SHA256SUMS, and if you trust it, add an exception for kinetic.exe and the folder that holds your models. Builds from source behave the same.

The server cannot be reached from another computer or a container

The server listens on 127.0.0.1 by default. Start it with --host 0.0.0.0 to listen on all interfaces, and keep in mind that it has no authentication: anyone who can reach the port can use the model. Put a reverse proxy with authentication in front for anything beyond your own network.

A web page cannot call the server

That is deliberate: the server refuses requests from web pages, so a website you visit cannot use your local model. Call it from a backend, an app or a script instead.

Split GGUF files

Files split into parts (model-00001-of-00003.gguf) are not read yet. Join them with llama.cpp's llama-gguf-split --merge model-00001-of-00003.gguf model.gguf first.

Versioning

Kinetic's version is the one in the workspace Cargo.toml; the command line, the libraries and the Python package all carry it, and kinetic --version prints it.

• 0.1.x releases fix bugs. They add support where it was missing (a pre-tokenizer, a weight type on a device) but do not change how existing commands, flags or API fields behave.
• 0.2 brings new features, such as the Vulkan backend, partial offload of dense models and --parallel. While Kinetic is below 1.0, a minor version may also change behavior; the changelog lists every such change.
• File formats are versioned on their own. A .ktc file starts with the magic KTC1 and a format version, which changes only when the layout does, independent of Kinetic's version. See FORMAT.md.
• Releases are git tags named vX.Y.Z, such as v0.1.0. Each tag builds the release archives, the Python wheels and their SHA256SUMS.

Roadmap

What we are working on next, roughly in order. Nothing here has a date; items move when they are ready, and only after they pass the same exactness and speed checks as everything else. Ideas and pull requests for any of them are welcome; open an issue first so we can agree on the approach.

Landing next

• winget and scoop packages; the manifests are in packaging/.

0.1.x: fixes

Bug fixes, new pre-tokenizers, better error messages and documentation. No changes to the command line or the server API that break existing use.

0.2: next features

• More on Vulkan. Llama, Qwen2.5, Qwen3, LFM3 and Qwen3.5 run on AMD, Intel and NVIDIA GPUs through --device vulkan; MoE models and the other architectures are next.
• Layer splits on Vulkan. Dense models larger than the GPU split their layers between GPU and CPU through CUDA today (--gpu-layers); Vulkan is next.
• Gemma 4 26B A4B, LFM2.5 MoE and more quantization formats.
• Block drafters for RAD-1.
• Shared passes everywhere. Convoy runs on the CPU and on CUDA when the whole model is on the GPU; layer splits, expert offload and Vulkan are next.
• More vision models. Today images work with Qwen3.5 and Gemma 4 and their mmproj projectors; other projectors are next.
• Two GPUs. Splitting one model across two devices.

Later

• More architectures, driven by the "model not supported" issues.
• Constrained output (--grammar, --json-schema) in kinetic run; it is in the server already.
• Split GGUF files (-00001-of-0000N.gguf) read directly.
• n greater than 1 in the server API.
• --imatrix for kinetic quantize.
• Tools to convert and sign .ktc files from the command line.

Kinetic supports Windows and Linux on x86-64. macOS is not on the roadmap.

Changelog

All notable changes to Kinetic are listed here. The format follows Keep a Changelog, and versions follow the rules in Versioning.

Unreleased

Models

• Gemma 4 (gemma4) on the CPU and CUDA, with images through its gemma4v projector.
• LFM2.5 mixture of experts (lfm2moe), experimental, on the CPU and CUDA, with experts kept in system memory when they do not fit the GPU.

0.1.0 - 2026-11-05

The first public release.

Command line

• kinetic run, chat, serve, bench, info and verify.
• Common llama.cpp flags work unchanged (-m, -p, -n, -c, -t, -s, --temp, -ngl, -dev and others); the ones Kinetic does not need are explained instead of rejected.
• kinetic bench runs llama-bench's pp512 and tg128 tests, plus an edit test that checks drafted output against plain decoding.

Server

• OpenAI-compatible /v1/chat/completions, /v1/completions, /v1/models and /v1/health, with streaming, stop strings, seeds, tool calls, response_format (JSON and JSON Schema), GBNF grammars and reasoning output.
• /v1/embeddings for embedding models and /v1/rerank for rerankers.
• Images for Qwen3.5 with its mmproj projector.
• Listens on 127.0.0.1 by default and refuses requests from web pages.

Models

• Architectures: qwen35, qwen35moe, lfm3, lfm2, llama, qwen2, qwen3, gemma3, ernie4_5, hunyuan-dense, exaone4, olmo2, granite, granitehybrid, granitemoe, falcon-h1, bert, nomic-bert and gemma-embedding, and decoder embedding and reranking models built on them.
• Every GGUF weight type in common use: F32, F16, BF16, the legacy and K-quants, the IQ formats, TQ1_0, TQ2_0 and MXFP4.
• The .ktc format, version 1: one file with weights, tokenizer, chat template and a hashed manifest, optionally signed with Ed25519.

Devices

• x86-64 CPUs, with AVX2 kernels.
• NVIDIA GPUs from Turing (RTX 20 series) on, through CUDA. The driver is loaded at runtime, so the CUDA build also starts on machines without one.
• Mixture-of-experts models larger than GPU memory (Flywheel).

Speed

• RAD-1: speculative decoding that is exact and automatic, drafting from the context, from earlier requests and from a model's own multi-token prediction block.
• Warm Start: prompts that begin like an earlier one resume from its exact state, in memory and on disk.
• Shortlist: exact top tokens without reading the whole output layer.

Libraries

• Python package kinetic, C API (kinetic.h, kinetic_capi) and the Rust crate kinetic.
• Release archives for Windows and Linux, CPU and CUDA, and Dockerfiles.