Zml
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild
Last verified:
What is Zml?
ZML is a high-performance AI inference stack built for production that enables running Machine Learning models, particularly Neural Networks, across any hardware without compromise. Written entirely in Zig without any Python in the runtime stack, ZML lowers models directly onto NVIDIA CUDA, AMD RoCM, Google TPU, and AWS Trainium/Inferentia 2 targets from a single codebase. The framework simplifies model serving by ensuring peak performance and maintainability in production environments.
Key features include a complete model lifecycle implementation with compilation caching, asynchronous primitives for parallel compilation and weight loading, tagged tensors for strong type checking, and cross-compilation support for Linux X86_64, Linux ARM64, and MacOS ARM64 platforms. ZML provides Kubernetes-ready deployment, Sagemaker containers, bare metal packages, OpenAI-compatible API, built-in Prometheus monitoring, and predictable flat latency. The framework supports HuggingFace model authentication and includes reference implementations for models like MNIST and Llama 3.1/3.2.
ZML is designed for ML engineers, AI researchers, and production teams who need to deploy models across multiple hardware platforms without vendor lock-in. It targets organizations running inference at scale who want unmatched latency and throughput while avoiding Python-heavy runtime layers. The open-source project has 1.9K GitHub stars and an active community contributing to its development.
Zml pricing
Pricing model: Freemium
ZML is fully open-source with no paid tiers or subscription plans. The framework is free to use, modify, and distribute under its open-source license. Users self-host the infrastructure and only pay for their own hardware (GPUs, TPUs, cloud instances) and HuggingFace access tokens for gated models. There are no per-request fees, token charges, or usage-based pricing. The open-source model means reduced TCO compared to proprietary solutions with multiplicative spend reductions.
Zml pros
- Written entirely in Zig with zero Python in the runtime stack
- Supports NVIDIA CUDA, AMD RoCM, Google TPU, and AWS Trainium from single codebase
- Open-source with 1.9K GitHub stars and active community
- Compilation caching across runs eliminates recompilation for same model architecture
- Asynchronous primitives enable parallel compilation and weight loading
- Cross-compilation to Linux X86_64, Linux ARM64, and MacOS ARM64
- Kubernetes-ready deployment with Sagemaker container support
- OpenAI-compatible API for seamless tool integration
- Built-in Prometheus stack for easy monitoring
- Predictable flat latency for production workloads
- Smallest image sizes enable zero-to-production scaling in seconds
- No vendor lock-in - run on any cloud, multicloud, any hardware
- Strong type checking with Tensor, Buffer, HostBuffer, and Shape types
- HuggingFace token authentication for gated models like Llama
- Compressed TAR archive deployment for easy server distribution
Zml cons
- Requires Bazel and bazelisk for building (additional build tool dependency)
- Needs HuggingFace approval for gated models like Meta Llama 3.1/3.2
- CPU runtime should be avoided - designed primarily for accelerators
- Cross-compilation requires manual TAR archive creation and distribution
- Learning curve for Zig programming language if writing custom models
- No explicit free tier or paid pricing - purely open-source with self-hosting
- Model compilation can be a startup bottleneck without caching
- Requires GPU/TPU hardware for optimal performance
- Documentation focused on tutorials rather than comprehensive API reference
- Inference requires manual weight loading and executable binding
Frequently asked questions about Zml
What is ZML and what does it do?
ZML is a high-performance AI inference stack built for production that helps run Machine Learning models, particularly Neural Networks. It compiles models into accelerator-specific executables for NVIDIA CUDA, AMD RoCM, Google TPU, and AWS Trainium/Inferentia 2 from a single Zig codebase, eliminating Python-heavy runtime layers and vendor lock-in.
Which hardware platforms does ZML support?
ZML supports NVIDIA CUDA, AMD RoCM, Google TPU, and AWS Trainium/Inferentia 2. It also supports cross-compilation to Linux X86_64, Linux ARM64, and MacOS ARM64 target architectures. CPU execution is available but discouraged as it cuts down the framework's performance advantages.
How do I install ZML?
Clone the ZML codebase with 'git clone https://github.com/zml/zml.git', then navigate to the directory. Install bazelisk (Bazel version manager) via 'brew install bazelisk' on macOS or using the curl command on Linux. Bazel will download dependencies automatically when building.
Can I run Llama models with ZML?
Yes, ZML supports Llama 3.1 8B, 70B, and Llama 3.2 1B, 3B Instruct models. These are gated models requiring Meta approval on HuggingFace. You need a HuggingFace token stored at $HOME/.cache/huggingface/token or in the HUGGINGFACE_TOKEN environment variable to download them.
How do I deploy ZML models on a remote server?
Cross-compile on your development machine using 'bazel build --config=release //llama --@zml//runtimes:cuda=true --platforms=@zml//platforms:linux_amd64', create a compressed TAR archive with zstd compression, copy it to the server via scp, extract with 'tar xvf archive.tar.zst', and run the executable with the model weights path.
What is the model lifecycle in ZML?
The lifecycle has these steps: 1) Open model file and read weight shapes (keeping weights on disk), 2) Instantiate model struct with Tensors, 3) Compile into accelerator-specific executable, 4) Load weights onto accelerator memory, 5) Bind weights to executable, 6) Load user inputs and copy to accelerator, 7) Call executable on inputs, 8) Fetch output to host memory, 9) Free resources when done.
Is ZML open-source?
Yes, ZML is fully open-source. The codebase is available on GitHub at https://github.com/zml/zml with 1.9K stars. The project welcomes community contributions and is building the future of AI inference collaboratively.
How does ZML handle compilation caching?
ZML provides compilation caching across runs. If you always use the same model architecture with the same shapes, you can bypass compilation entirely. This significantly reduces startup time since compilation is one of the main bottlenecks alongside weight loading.
What TypeScript/programming language is ZML written in?
ZML is written entirely in Zig, with zero Python code in the runtime stack. This makes it clean, robust, and built for production. The Zig static type system enables strong type checking with differentiated types like Tensor, Buffer, HostBuffer, and Shape.
How do I authenticate with HuggingFace for gated models?
Generate a 'read-only public repositories' token at https://huggingface.co/settings/tokens by clicking 'Create new token', naming it, granting 'Read access to contents of all public gated repos you can access', and copying the token. Store it at $HOME/.cache/huggingface/token or in the HUGGINGFACE_TOKEN environment variable.