Datadesigner
Datadesigner: 🎨 NeMo Data Designer: Generate high-quality synthetic data from scratch or from seed data.
Last verified:
What is Datadesigner?
Datadesigner: NeMo Data Designer is an open-source orchestration framework for generating high-quality synthetic data from scratch or using seed datasets. It is purpose-built for AI developers who need to create domain-specific synthetic data at scale, offering a declarative Python-first approach that avoids writing imperative LLM call logic. The framework handles batching, parallelism, validation, token metric collection, and reproducible workflows automatically.
Key features include statistical samplers for diverse data distributions, LLM-generated columns with prompt templates, structured outputs with user-defined schemas, dependency-aware field relationships, Python/SQL/remote validators for quality assurance, optional LLM-as-judge scoring, quick preview mode for fast iteration, and integration with NeMo Curator for scalable synthetic data pipelines. It supports multiple LLM endpoints including NVIDIA Build, OpenAI, vLLM, and OpenRouter.
The tool is designed for AI developers, machine learning engineers, and data scientists working on generative AI models, particularly those training LLMs like Nemotron. It's ideal for teams needing pre-training and post-training data, those requiring synthetic person entities with demographics and personality traits, and anyone who needs consistent, reliable synthetic data that goes beyond what one-size-ffits-all LLMs can deliver.
Datadesigner pricing
Pricing model: Freemium
NeMo Data Designer is open-source and free under the Apache 2.0 license. The library itself has no cost, but users must provide their own LLM endpoint API keys (NVIDIA Build, OpenAI, OpenRouter, vLLM, etc.), which may have associated costs depending on the provider. NVIDIA Build API at build.nvidia.com is recommended and appears to offer free tier access, but third-party endpoints like OpenAI charge per token. The tool is currently in early access/beta with Docker Compose deployment for local testing.
Datadesigner pros
- Python-first declarative configuration instead of imperative LLM calls
- Supports multiple LLM providers (NVIDIA Build, OpenAI, vLLM, OpenRouter)
- Statistical samplers for categorical, numerical, and distribution-based data
- Structured outputs with user-defined schemas and nested relationships
- Dependency-aware field relationships between columns
- Built-in Python, SQL, and remote validators for quality assurance
- Optional LLM-as-judge scoring for quality assessment
- Quick preview mode for fast iteration before full-scale generation
- Seeding generation with external datasets to steer content diversity
- Automated batching and parallelism for scalable data generation
- Token metric collection and analysis included
- Reproducible workflows with save/load configuration support
- Generates synthetic person entities with demographics and personality traits
- Multi-modal context support with vision-capable models
- Integrates with NeMo Curator via DataDesignerStage for pipeline scaling
- Async engine overlaps independent columns for faster execution
- Open-source under Apache 2.0 license
- CLI tools for configuring providers, models, and viewing settings
Datadesigner cons
- Early access release with limited support and potential API changes
- Currently only available via Docker Compose, not yet in Helm Chart
- Requires API key from third-party model provider (not free LLM access)
- Requires deployment on laptop or compute instance (not fully managed SaaS)
- Async engine may have unexpected behavior with slow self-hosted endpoints
- Telemetry collection enabled by default (must opt out manually)
- NVIDIA Build endpoint intended for evaluation/testing only, not production
- Documentation transitioning between MkDocs and Fern (some fragmentation)
- Requires Python installation and SDK setup (not no-code tool)
- Third-party endpoint terms apply independently (NVIDIA Build TOS restrictions)
Frequently asked questions about Datadesigner
What is NeMo Data Designer?
NeMo Data Designer is a declarative data generation framework and orchestration tool for generating high-quality synthetic data from scratch or using seed datasets. Instead of writing imperative LLM call logic, you define a configuration describing what columns to generate, how to sample structured fields, and which LLM to use. It handles execution, batching, parallelism, validation, and token metric collection automatically.
How do I install Data Designer?
Install via pip with 'pip install data-designer' or from source by cloning the GitHub repository and running 'make install'. For NeMo Microservices deployment, use 'pip install nemo-microservices[data-designer]' or 'pip install nemo-platform[data-designer]' depending on your deployment target.
What LLM providers does Data Designer support?
Data Designer supports multiple LLM endpoints including NVIDIA Build (build.nvidia.com, recommended), OpenAI (platform.openai.com), OpenRouter (openrouter.ai), and vLLM. You set API keys as environment variables like NVIDIA_API_KEY, OPENAI_API_KEY, or OPENROUTER_API_KEY.
What is preview mode?
Preview mode allows fast iteration on your dataset configuration before running full-scale generation. You call the preview method to generate a small sample, view sample records with display_sample_record(), access the dataset as a Pandas DataFrame, and view analysis reports. Nothing is persisted to disk by default in preview mode.
What column types are available?
Data Designer offers sampler columns (category, numerical distributions), LLM text generation columns, LLM structured columns for schema-conformant outputs, expression columns, validation columns, and seed-based columns. You can mix statistical samplers, LLM columns, and seed datasets in a single pipeline.
How do I validate generated data quality?
Data Designer includes built-in validators: Python validators for custom code checks, SQL validators for database constraints, and remote validators for external validation services. You can also implement optional LLM-as-judge scoring for quality assessment. Automated checks and assessment metrics are part of the data quality workflow.
Can I use my own seed dataset?
Yes, Data Designer supports seeding generation with external datasets to steer the content and diversity of generated data. You can start from your own seed datasets rather than generating entirely from scratch, which helps maintain domain specificity and desired data characteristics.
What is the async engine?
The async engine is the default execution engine that overlaps independent columns and adapts concurrency per (provider, model). It is typically faster with no configuration changes on most pipelines. For slow self-hosted endpoints, set inference_parameters.timeout to your real per-request latency. You can fall back to legacy sync engine with DATA_DESIGNER_ASYNC_ENGINE=0 if needed.
How do I deploy Data Designer?
Deploy using Docker Compose (easiest for local testing) or the NeMo Microservices Helm chart for Kubernetes. The Early Access release is currently only available via Docker Compose and is not yet part of the Helm Chart. You need Access to an NMP deployment or deploy on your laptop/compute instance.
Is telemetry collected?
Yes, Data Designer collects optional anonymous telemetry including model names used and token counts (input and output) to help improve the library. No user or device information is collected. You can opt out by setting NEMO_TELEMETRY_ENABLED=false. The telemetry data will be shared in aggregate with the community.