Soup
Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.
Last verified:
What is Soup?
Soup CLI is a zero-config command-line tool for fine-tuning large language models (LLMs). It automates hyperparameter selection through its Autopilot feature, supports 11 training methods including SFT, DPO, KTO, and RLVR (GRPO), and provides 43 ready-made recipes. The tool handles the entire post-training workflow: it writes configuration files, derives evaluations from user data, gates every model save to prevent regressions, and ships trained models.
Key features include a local model registry at ~/.soup/registry.db that tracks lineage with DAG visualization, eval gates that auto-stop training on quality regression, trace-to-preference pipeline for building DPO/KTO pairs from logs, quantization regression checking, and Soup Cans for reproducible recipe bundles. It supports native MLX training on Apple Silicon (M1-M4) without CUDA, and includes migration commands to convert configs from LLaMA-Factory, Axolotl, and Unsloth.
Soup CLI is built for machine learning engineers, data scientists, and developers who want to fine-tune LLMs without managing complex configurations. It integrates with HuggingFace, Ollama, vLLM, DeepSpeed, Unsloth, ONNX, NVIDIA TensorRT, W&B, SGLang, and FlashAttention. The tool supports models like Llama 3.1/3.2, Qwen 2.5/3, Gemma 3, Phi-4, DeepSeek, Mistral, and any model on Hugging Face Hub.
The CLI requires Python 3.9+, works with CUDA or Apple Silicon, and can be installed via pip install soup-cli. Users go from installation to trained model in three commands (install, init, train) with setup time under 60 seconds. It exports models in GGUF, ONNX, TensorRT, and AWQ/GPTQ formats for deployment.
Soup pricing
Pricing model: Freemium
Free and Open Source under MIT License. No credit card required, no sign-up needed, no vendor lock-in, works offline. All features are included in the free tier: Autopilot, MLX backend, eval gates, local registry, Soup Cans, quant-check, trace-to-preference, migration tools, and all 11 training methods with 43 recipes. No paid tiers or premium features exist.
Soup pros
- Zero-config hyperparameter selection with Autopilot
- 11 training methods from SFT to full RLHF
- 43 ready-made recipes for common use cases
- Local model registry with lineage DAG tracking
- Eval gates auto-stop training on quality regression
- Native MLX backend for Apple Silicon M1-M4
- RLVR support with deterministic rewards (math_verify, code_exec, json_schema)
- Trace-to-preference pipeline from LangChain/OpenAI logs
- Quantization regression checking before shipping
- Soup Cans for reproducible, shareable recipe bundles
- One-command migration from LLaMA-Factory, Axolotl, Unsloth
- 30-second setup time (<60s to first model)
- MIT licensed, free forever with no vendor lock-in
- Works offline without sign-up or credit card
- First-class integrations with HuggingFace, Ollama, vLLM, DeepSpeed
- GGUF, ONNX, TensorRT, AWQ/GPTQ export formats
- soup diagnose report card on 6 failure modes
- 2,467 tests across 91 test files for reliability
- SQL-based registry with search, diff, fork capabilities
- Safe extraction with symlink and path-traversal blocking
Soup cons
- Requires Python 3.9 or higher
- CUDA or Apple Silicon required (no CPU-only training)
- Relatively new tool with smaller community
- CLI-only interface (no GUI available)
- 100 MB cap on Soup Can bundles
- MLX backend limited to SFT, DPO, GRPO only
- No cloud experiment tracking built-in (requires W&B integration)
- Learning curve for YAML configuration format
- Limited documentation compared to established tools
- No pre-trained base models included (must use HuggingFace)
Frequently asked questions about Soup
How do I install Soup CLI?
Install via pip with the command: pip install soup-cli. The tool requires Python 3.9+ and either CUDA (for NVIDIA GPUs) or Apple Silicon (M1-M4). After installation, run soup init to create your soup.yaml configuration file, then use soup train to start fine-tuning.
What is Autopilot and how does it work?
Autopilot is Soup's zero-config hyperparameter selection feature. When you pass a model, data, and goal, Autopilot profiles the dataset, model, and GPU, then automatically picks the task, quantization level, PEFT rank, batch size, learning rate, epochs, and max_length. Every choice is justified and documented in the generated soup.yaml.
What training methods does Soup support?
Soup supports 11 training methods via the task config key: Supervised Fine-Tuning (SFT) for instruction-response pairs, DPO (Direct Preference Optimization), KTO (Kahneman-Tversky Optimization), and GRPO with RLVR for verifiable rewards on math, code, and JSON schema tasks. It also covers the full RLHF pipeline.
How does the model registry work?
Every finished run is pushed to ~/.soup/registry.db with its config, eval baseline, and parent relationship. Use soup history to render the lineage DAG, soup registry diff to compare config and eval deltas between versions, tag runs as prod, fork entries into .can bundles, and search by name, base model, task, or notes.
What are Eval Gates?
Eval gates run a declarative evaluation suite at epoch boundaries and automatically stop training when quality drops below a baseline. Baselines can point to registry://<id> entries, files, or fresh runs. Use soup eval gate for post-hoc verdicts on whether a model passed quality thresholds.
Does Soup work on Apple Silicon?
Yes, Soup is the first CLI to ship a native MLX backend for Apple Silicon. It supports SFT, DPO, and GRPO on M1-M4 chips using unified memory, with no CUDA required, no emulation, and no surprises. Training runs natively on macOS without workarounds.
How do I migrate from LLaMA-Factory or Axolotl?
Use the migration command: soup migrate --from llamafactory config.yaml or soup migrate --from axolotl config.yml. Soup automatically converts your existing config file, auto-detects optimizer, scheduler, target modules, and batch size, then generates a clean soup.yaml. No rewriting or guessing required.
What are Soup Cans?
Soup Cans are shareable, reproducible recipe bundles (.can files). A .can is a tar.gz containing manifest, config, and data reference. Use soup can pack/inspect/verify/fork to make every run reproducible. Safe extraction blocks symlink and path-traversal attacks, has a 100 MB cap, and the format version is locked.
What failure modes does soup diagnose check?
The soup diagnose command generates a post-training model report card checking 6 failure modes: forgetting (catastrophic forgetting of base capabilities), refusal (model refusing valid requests), format (output format violations), mode-collapse (limited response diversity), memorization (over-memorizing training data), and contamination (training/test data overlap).
Can I use Soup without internet access?
Yes, Soup works offline without requiring sign-up or a credit card. The local registry at ~/.soup/registry.db, all training operations, and Soup Can bundles work completely offline. You only need internet to initially download the package from PyPI or push models to HuggingFace Hub if desired.