llama.cpp flags
If you use llama.cpp, most of your commands work as they are. These flags mean the same in Kinetic:
-m, -p, -n, -c, -t, -s, --temp, --top-k, --top-p,
--min-p, --repeat-penalty, --host, --port and --mmproj.
llama-cli -m model.gguf -p "Hello" -n 64 --temp 0 -ngl 99
kinetic run -m model.gguf -p "Hello" -n 64 --temp 0 -ngl 99
llama-server -m model.gguf -c 8192 --port 8080 -ngl 99
kinetic serve -m model.gguf -c 8192 --port 8080 -ngl 99
The commands differ: llama-cli is kinetic run (one prompt) or kinetic chat (a conversation), llama-server is kinetic serve and llama-bench
is kinetic bench.
A copy of kinetic named llama-server, llama-cli or llama-completion
runs as kinetic serve, kinetic chat (or kinetic run with a prompt) and
kinetic run, for apps that start those binaries by name.
For every other llama.cpp flag below, Kinetic takes it under its own name, accepts it when it asks for something Kinetic does by itself, or stops with a line that says what to use instead, rather than a bare "unexpected argument":
$ kinetic run model.gguf -ts 1,1 "Hello"
kinetic: -ts is a llama.cpp flag: Kinetic runs the model wholly on one device; choose it with --device.
| llama.cpp flag | In Kinetic |
|---|---|
-sys | works, as --system |
--system-prompt | works, as --system |
--temperature | works, as --temp |
-dev | works, as --device |
-np | works, as --parallel |
-ctk | works, as --cache-type-k |
-ctv | works, as --cache-type-v |
-a | works, as --alias |
-rea | works, as --reasoning |
-ngl, --n-gpu-layers | works: 0 means --device cpu, any other number --device cuda, all as many as fit |
-fa, --flash-attn, --jinja, --mmap, -cb, --cont-batching, -nocb, --no-cont-batching, --no-warmup, --metrics, --slots, --no-webui, --embedding, --embeddings, -no-cnv, --no-conversation, -st, --single-turn, -b, --batch-size, -ub, --ubatch-size, -tb, --threads-batch, --cache-reuse, --numa, --prio, --poll, -to, --timeout, --threads-http | accepted and not needed: Kinetic does this by itself |
-sm, --split-mode, -ts, --tensor-split, -mg, --main-gpu, -ot, --override-tensor | Kinetic runs the model wholly on one device; choose it with --device |
-cnv, --conversation, -i, --interactive, --interactive-first | use kinetic chat <model> |
--chat-template | built-in template names are not supported; pass a Jinja file with --chat-template-file |
-md, --model-draft, --draft, --draft-max, --draft-min, -ngld | speculative decoding is automatic and needs no draft model |
-hf, --hf-repo, -mu, --model-url | download with kinetic pull <org>/<repo>[:<quant>], then run it by name |
-f, --file | pipe the prompt in: kinetic run <model> < prompt.txt |
--image | send images to the server: kinetic serve <model> --mmproj <projector> |
--control-vector, --control-vector-scaled | control vectors are not supported yet; LoRA adapters are, with --lora |
--grammar, --grammar-file, -j, --json-schema | constrained output is in the server, through grammar and response_format |
--api-key-file | pass the key itself with --api-key |
--ssl-key-file, --ssl-cert-file | the server speaks plain HTTP; put a TLS proxy in front |
The flags for prompt files, constrained output and adapters are on the roadmap.