Trainer
Distributed AI Model Training and LLM Fine-Tuning on Kubernetes
Last verified:
What is Trainer?
Kubeflow Trainer is a Kubernetes-native distributed AI platform for scalable large language model (LLM) fine-tuning and training of AI models across a wide range of frameworks, including PyTorch, MLX, HuggingFace, DeepSpeed, JAX, XGBoost, and more. It brings MPI to Kubernetes, orchestrating multi-node, multi-GPU distributed jobs efficiently across high-performance computing clusters, enabling high-throughput communication between processes for large-scale AI training requiring ultra-fast synchronization between GPU nodes.
The platform seamlessly integrates with the Cloud Native AI ecosystem, including Kueue for topology-aware scheduling and multi-cluster job dispatching, as well as JobSet and LeaderWorkerSet for AI workload orchestration. Kubeflow Trainer provides a distributed data cache designed to stream large-scale data with zero-copy transfer directly to GPU nodes, ensuring memory-efficient training jobs while maximizing GPU utilization. With the Kubeflow Python SDK, AI practitioners can effortlessly develop and fine-tune LLMs while leveraging the Kubeflow Trainer APIs: TrainJob and Runtimes.
Kubeflow Trainer is designed for three main user personas: AI Practitioners (ML engineers and data scientists who develop AI models using the Kubeflow Python SDK and TrainJob), Platform Administrators (administrators and DevOps engineers responsible for managing Kubernetes clusters and Kubeflow Training Runtimes), and Contributors (open source contributors working on the Kubeflow Trainer project). The project supports key phases of the AI lifecycle including model training and LLMs fine-tuning.
Key benefits include simple and scalable LLM fine-tuning, extensibility and portability across any cloud or on-premises Kubernetes cluster, distributed AI data caching powered by Apache Arrow and Apache DataFusion, ready-to-use LLM fine-tuning blueprints, optimized GPU efficiency through intelligent dataset streaming, and native Kubernetes integrations with projects like Kueue, Coscheduling, Volcano, or YuniKorn.
Trainer pricing
Pricing model: Freemium
Kubeflow Trainer is free and open-source under Apache License V2.0. There are no paid plans or subscription tiers. However, running Kubeflow Trainer incurs costs associated with Kubernetes infrastructure (depending on cloud provider or on-premises setup), storage and compute expenses for storage, compute nodes, GPUs and other resources used for ML tasks, and operational overheads for time and resources needed to set up, maintain, and manage the platform.
Trainer pros
- Kubernetes-native distributed AI platform for LLM fine-tuning
- Supports multiple frameworks: PyTorch, MLX, HuggingFace, DeepSpeed, JAX, XGBoost
- Brings MPI to Kubernetes for multi-node multi-GPU orchestration
- Integrates with Kueue for topology-aware scheduling
- Distributed data cache with zero-copy transfer to GPU nodes
- Powered by Apache Arrow and Apache DataFusion for data streaming
- Kubeflow Python SDK for effortless LLM development
- TrainJob and Runtimes APIs for simplified training
- Runs on any cloud or on-premises Kubernetes cluster
- Ready-to-use LLM fine-tuning blueprints for generative AI
- Offloads data preprocessing and I/O to CPU workloads
- Intelligent dataset streaming reduces GPU costs
- Native integrations with Volcano, Coscheduling, YuniKorn
- Optimized for high GPU utilization in large-scale training
- Extensible API layer for custom ML frameworks
- Apache License V2.0 open source under CNCF governance
- Supports Distributed Data Parallel (DDP) automatically
Trainer cons
- Project is in alpha status with APIs that may change
- Requires Kubernetes cluster with Kubeflow Trainer control plane installed
- Steep learning curve for platform administrators
- Complex setup compared to single-machine training
- Requires GPU resources for optimal performance
- Migration needed from legacy Training Operator V1
- Distributed data cache requires additional infrastructure
- Best suited for large-scale training not small experiments
Frequently asked questions about Trainer
What is Kubeflow Trainer?
Kubeflow Trainer is a Kubernetes-native distributed AI platform for scalable large language model (LLM) fine-tuning and training of AI models across a wide range of frameworks including PyTorch, MLX, HuggingFace, DeepSpeed, JAX, XGBoost, and more. It brings MPI to Kubernetes for orchestrating multi-node, multi-GPU distributed jobs efficiently.
Who is Kubeflow Trainer for?
Kubeflow Trainer serves three main user personas: AI Practitioners (ML engineers and data scientists who develop AI models using the Kubeflow Python SDK and TrainJob), Platform Administrators (administrators and DevOps engineers managing Kubernetes clusters and Training Runtimes), and Contributors (open source contributors working on the project).
What ML frameworks does Kubeflow Trainer support?
Kubeflow Trainer supports PyTorch, MLX, HuggingFace, DeepSpeed, JAX, XGBoost, TensorFlow, and other ML frameworks. You can integrate other ML libraries such as HuggingFace, DeepSpeed, or Megatron-LM with Kubeflow Trainer to orchestrate their ML training on Kubernetes.
How do I get started with Kubeflow Trainer?
First ensure you have access to a Kubernetes cluster with Kubeflow Trainer control plane installed. Install the Kubeflow Python SDK using 'pip install -U kubeflow'. Define your training function that handles end-to-end model training, check available Training Runtimes, and create a TrainJob using the TrainerClient().train() method.
What is the Kubeflow Python SDK?
The Kubeflow Python SDK allows AI practitioners to effortlessly develop and fine-tune LLMs while leveraging Kubeflow Trainer APIs including TrainJob and Runtimes. It provides TrainerClient for managing jobs, CustomTrainer for defining training functions, and methods like list_runtimes(), train(), get_job(), and get_job_logs().
What are Training Runtimes?
Training Runtimes are Kubernetes-native configurations that define how training jobs execute. Examples include 'torch-distributed' for PyTorch distributed training. Runtimes configure the distributed environment automatically, enabling features like Distributed Data Parallel (DDP) for frameworks like PyTorch.
What is the distributed data cache feature?
The distributed data cache is designed to stream large-scale data with zero-copy transfer directly to GPU nodes. Powered by Apache Arrow and Apache DataFusion, it enables seamless access to large datasets, minimizes I/O overhead, cuts GPU costs, and ensures memory-efficient training jobs while maximizing GPU utilization.
Is Kubeflow Trainer free to use?
Yes, Kubeflow Trainer itself is free and open-source under Apache License V2.0. However, running it incurs costs for Kubernetes infrastructure (cloud provider or on-premises), storage and compute resources (nodes, GPUs), and operational overhead for setup, maintenance, and management.
What is the difference between Kubeflow Trainer v2 and Training Operator v1?
Kubeflow Trainer v2 is the next evolution building on over seven years of Training Operator v1 experience. Trainer v2 is currently in alpha status with new APIs (TrainJob, Runtimes). Users of Training Operator v1 should refer to the migration document. v1 source code is maintained at the release-1.9 branch.
How does Kubeflow Trainer optimize GPU efficiency?
Kubeflow Trainer reduces GPU costs through intelligent dataset streaming and model initialization. It offloads data preprocessing and I/O to CPU workloads, ensuring GPUs stay focused on training. The distributed data cache streams tensors directly to GPU nodes, minimizing I/O overhead and maximizing GPU utilization.