Surogate
Training/Fine-tuning at the speed of light
Last verified:
What is Surogate?
Surogate is an open-source, enterprise-grade AgentOps and LLMOps platform designed for fast LLM pre-training, fine-tuning (LoRA/QLoRA), and reinforcement learning on NVIDIA GPUs. Built by Invergent, it features a native C++/CUDA execution engine that pushes GPUs to near speed-of-light throughput, with support for BF16, FP8, and NVFP4 precision formats. The platform bridges the gap between AI experimentation and production deployment, offering both an open-source core and enhanced enterprise features.
Key features include native multi-GPU and multi-node training with Ray-based DDP, smart CPU offloading, pre-built training recipes, adaptive training with automatic hyperparameter adjustment, MoE support with expert parallelism, stacked LoRA training, and a comprehensive evaluation suite with red-teaming capabilities. Surogate Studio provides a GUI for no-code pretraining and fine-tuning, while the Data Hub offers Git-like versioning for datasets and models. The platform supports cloud deployment (AWS, GCP, Azure, Oracle Cloud) and on-premises deployment on DenseMAX Appliances, including optional air-gapped configurations.
Surogate is designed for AI researchers, ML engineers, and enterprises that need to train and deploy their own LLMs on hardware they control. It suits teams working with models like Qwen2.5, Qwen3, Llama 3+, and Nemotron Nano who require predictable scaling, low-latency execution, and production-grade reliability from single-GPU rigs to multi-node datacenter clusters.
Surogate pricing
Pricing model: Freemium
Surogate offers three tiers: Surogate OSS (open-source) is FREE with core features including pretraining, full fine-tuning, LoRA/QLoRA, BF16/FP8/NVFP4 precision, multi-GPU/multi-node training, and native C++/CUDA engine. Surogate Studio includes enterprise features like GUI workflows, no-code pretraining, reinforcement fine-tuning, Data Hub, team collaboration, live monitoring, and advanced serving. Surogate Enterprise is FREE but requires contacting Sales for deployment on DenseMAX Appliances and public clouds; it includes all OSS + Studio features plus SSO, LDAP, RBAC, audit logs, SOC2 compliance, dedicated CSM, SLAs, high availability, backup/restore, security hardening, and air-gapped deployment options.
Surogate pros
- Native C++/CUDA engine for near speed-of-light GPU utilization
- Supports BF16, FP8, and NVFP4 precision for maximum performance
- Works on all NVIDIA GPUs from sm80 to sm121 including RTX 5090 and B200
- Multi-GPU and multi-node training with Ray-based DDP
- Smart CPU offloading for weights, gradients, activations, and quants
- Pre-built training recipes for BF16, FP8, and NVFP4
- LoRA and QLoRA support with BnB/FP8/NVFP4 dynamic quantization
- Adaptive Training with automatic hyperparameter adjustment and early stopping
- Stacked LoRA training to skip offline merging into base model
- Comprehensive evaluation suite with red-teaming for bias/toxicity/leakage
- Data Hub with Git-like versioning for datasets and models
- GUI workflows for no-code pretraining and fine-tuning in Enterprise
- Reinforcement learning support with GRPO, DPO, PPO algorithms
- Model distillation and synthetic data generation capabilities
- Deploy on DenseMAX Appliance or major clouds with optional air-gapped mode
- SSO via SAML/OIDC, LDAP integration, and RBAC for enterprise security
- SOC2 compliance commitment with audit logs and encryption at rest
- KV-aware cache routing and GPU sharding for advanced model serving
Surogate cons
- Requires Ubuntu 24.04 x64 with CUDA 12.8/12.9/13.0
- Only runs on Linux x86_64 architecture, no Windows or macOS support
- NVIDIA GPU required, no AMD or Intel GPU support
- Only x86-64 architecture supported, no ARM support
- DenseMAX Appliance required for some enterprise on-prem features
- Enterprise features require contacting sales for pricing
- Open-source version lacks GUI workflows and no-code interface
- Air-gapped deployment only available in Enterprise tier
- No pre-built binaries for non-Ubuntu Linux distributions
- Model addition requires creating a PR for custom models
Frequently asked questions about Surogate
What is Surogate?
Surogate is an extremely fast production-grade LLM training framework engineered to operate at practical hardware limits, delivering near-speed-of-light throughput, low-latency execution, and predictable multi-GPU/multi-Node scaling. It combines a native C++/CUDA execution engine, a low-overhead Python DSL, an AOT-based Auto Differentiation engine, and a highly optimized multi-threaded scheduler to achieve industry-leading Speed-Of-Light (SOL) utilization on NVIDIA GPUs, outperforming existing training toolkits by a wide margin.
What GPUs does Surogate support?
Surogate runs on all NVIDIA GPUs from SM80 to SM121, including A100, A30 (SM80); A10, A16, RTX 3060, RTX 3090 (SM86); L40, RTX 4090 (SM89); H100, H200 (SM90); B200, GB200 (SM100); B300 (SM103); and RTX 5070, RTX 5080, RTX 5090, RTX PRO 6000 Blackwell (SM120); plus DGX Spark (SM121).
What precision formats does Surogate support?
Surogate supports BF16 (bfloat16 for maximum numerical accuracy with no quantization), FP8 (native E4M3 for weights/activations and E5M2 for gradients with delayed scaling), NVFP4 (CUTLASS FP4 E2M1 with block scaling for Blackwell GPUs), and BnB/FP8/NVFP4 dynamic quantization for QLoRA.
How do I install Surogate?
You can install Surogate via Docker (recommended) using images like ghcr.io/invergent-ai/surogate:latest-cu128 for CUDA 12.8.1, or via the install script: curl -LsSf https://surogate.ai/install.sh | sh. Docker requires Ubuntu 24.04 x64 with CUDA 12.8/12.9/13, NCCL, cuDNN, and recent NVIDIA drivers.
What training modes does Surogate support?
Surogate supports pre-training, full fine-tuning, LoRA, QLoRA, and reinforcement learning with GRPO, DPO, PPO, and GDP algorithms. It also supports stacked LoRA training (training LoRA on top of another LoRA), mixed-precision training, and MoE training with expert parallelism.
What models are supported?
Surogate supports Qwen2.5, Qwen3, Qwen3 MoE, Llama 3+, and Nemotron Nano models. Additional models can be added by creating a PR to the project.
What is the difference between Surogate OSS and Enterprise?
Surogate OSS includes core training features like pretraining, fine-tuning, LoRA/QLoRA, multi-GPU training, and native C++/CUDA engine. Enterprise adds GUI workflows, no-code pretraining, reinforcement fine-tuning, Data Hub with versioning, team collaboration, live monitoring, advanced serving, evaluation suite, red-teaming, SSO/LDAP/RBAC, audit logs, SOC2 compliance, dedicated CSM, SLAs, high availability, backup/restore, security hardening, and air-gapped deployment.
Can I deploy Surogate on-premises?
Yes, Surogate can be deployed on-premises on DenseMAX Appliances or on your own local GPU infrastructure. Enterprise tier supports optional air-gapped deployment, hardened deployment patterns, and works with major clouds (AWS, GCP, Azure, Oracle Cloud) for hybrid setups.
What is Adaptive Training in Surogate?
Adaptive Training is a built-in automated training monitoring feature with automatic phase detection, multi-criteria early stopping (convergence, compute-efficiency, divergence, plateau), automatic learning rate management, MoE imbalance detection, Chinchilla token budgeting, and dynamic epoch adjustment to prevent drift or collapse during training.
Does Surogate support multi-node training?
Yes, Surogate supports native multi-node DDP training using Ray for efficient parallelism across multiple GPU nodes. This enables predictable scaling from single-GPU rigs to multi-GPU nodes and multi-node clusters for large-scale LLM training.