Cog
Containers for machine learning
Last verified:
What is Cog?
Cog is an open-source tool that lets you package machine learning models in a standard, production-ready Docker container. It simplifies the complex process of creating Docker containers for ML models by handling CUDA/cuDNN/PyTorch/Tensorflow/Python compatibility automatically, eliminating what the creators call
Cog pricing
Pricing model: Freemium
Cog is completely free and open-source. There are no paid tiers or subscription plans. The tool itself is free to download and use. You can deploy packaged models to your own infrastructure at no cost from Cog, or deploy to Replicate (which has its own separate pricing for running models on their platform). Installation is free via Homebrew (brew install replicate/tap/cog), install script (sh <(curl -fsSL https://cog.run/install.sh)), or manual download from GitHub releases.
Cog pros
- Eliminates CUDA/cuDNN/PyTorch/Tensorflow/Python compatibility headaches
- Generates Docker containers without writing Dockerfiles manually
- Automatic HTTP prediction server with RESTful API
- High-performance Rust/Axum inference server (coglet)
- OpenAPI schema generation from Python types
- Automatic input/output validation
- Supports GPU acceleration with Nvidia base images automatically
- Efficient dependency caching for faster builds
- Simple cog.yaml configuration file
- Works on macOS, Linux, and Windows 11 with WSL 2
- Installable via Homebrew on macOS
- Open-source and free to use
- Deploy anywhere Docker images run
- Seamless Replicate integration for deployment
- Support for concurrent predictions via cog.yaml configuration
- Health check endpoint for orchestration systems
- Separate model weights layering for optimized pushes
Cog cons
- Requires Docker to be installed as a prerequisite
- Does not currently support Windows natively (requires WSL 2)
- Learning curve for the cog.yaml configuration format
- Must define Python interface for model predictions
- Docker images can be large due to CUDA dependencies
- Experimental features like cog doctor may change
- Limited to machine learning model packaging use case
- Requires understanding of container concepts for advanced usage
Frequently asked questions about Cog
What is Cog used for?
Cog is an open-source tool that packages machine learning models in standard, production-ready Docker containers. It lets researchers and engineers ship ML models to production without dealing with complex Dockerfiles, CUDA version compatibility, Flask servers, and other deployment complexity. You define your environment in cog.yaml and your model interface in Python, then Cog builds a Docker image that can run predictions anywhere Docker works.
How do I install Cog?
On macOS, use Homebrew: brew install replicate/tap/cog. For Linux or macOS, use the install script: sh <(curl -fsSL https://cog.run/install.sh). You can also manually download from GitHub: sudo curl -o /usr/local/bin/cog -L https://github.com/replicate/cog/releases/latest/download/cog_$(uname -s)_$(uname -m) && sudo chmod +x /usr/local/bin/cog. Docker must be installed before running Cog.
What are the prerequisites for Cog?
Cog requires macOS, Linux, or Windows 11 with WSL 2. Docker must be installed first (Docker Engine or Docker Desktop). If using Docker Engine instead of Docker Desktop, you also need to install Buildx. For GPU support, you need NVIDIA GPUs with appropriate drivers and the nvidia-docker runtime.
How does Cog handle CUDA compatibility?
Cog knows which CUDA/cuDNN/PyTorch/Tensorflow/Python combinations are compatible and automatically sets them up correctly. When you add gpu: true to your cog.yaml, Cog uses the appropriate NVIDIA CUDA base image and installs the correct versions based on your Python and deep learning framework versions, eliminating manual CUDA version hunting.
What is the cog.yaml file?
cog.yaml defines the Docker environment your model runs in. It specifies build configuration including GPU usage, system packages, Python version, Python requirements file, and the predict or run interface. Example: build section with gpu: true, python_version: '3.13', python_requirements: requirements.txt, and predict: 'predict.py:Predictor' or run: 'run.py:Runner'.
How do I run predictions with Cog?
Use cog predict -i input=value to run predictions. For file inputs, use @ prefix: cog predict -i [email protected]. You can pass multiple inputs: cog predict -i prompt='sunset' -i width=1024 -i height=768. Outputs can be saved with -o output.png. Cog builds the Docker image if needed, runs the prediction, and returns the result.
How do I deploy a Cog model?
Build a Docker image with cog build -t my-model, then run it with docker run -d -p 5000:5000 --gpus all my-model. Or use cog serve -p 8080 to start an HTTP server directly. The server exposes a /predictions endpoint for POST requests with JSON input. Deploy to any Docker-compatible infrastructure or push to Replicate with cog push r8.im/username/model.
What CLI commands does Cog support?
Main commands include: cog init (create cog.yaml and run.py), cog build (build Docker image), cog run (run command in environment), cog predict (run prediction), cog serve (start HTTP server), cog push (push to registry), cog login (authenticate to registry), cog exec (run arbitrary command in environment), cog doctor (diagnose issues). Each command has options for customization like --tag, --gpu, --port, --input.
Can Cog run multiple predictions concurrently?
Yes, by default Cog processes one prediction at a time. To enable concurrency, add a concurrency section to cog.yaml with max: number (e.g., concurrency: max: 4). You can also set COG_MAX_CONCURRENCY environment variable. This allows the server to process multiple predictions simultaneously, improving throughput for models that can handle it.
What is the Cog HTTP API like?
Cog automatically generates a RESTful HTTP API using a high-performance Rust/Axum server called coglet. The API exposes /predictions endpoint for running predictions, /openapi.json for the OpenAPI schema, and /health-check for health status. Input is passed as JSON with an 'input' field containing the model inputs. Output follows the return type defined in your Python code.