Hamilton
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Last verified:
What is Hamilton?
Apache Hamilton is a general-purpose, lightweight Python framework for writing dataflows using regular Python functions. At its core, each function defines a transformation, and its parameters indicate dependencies. Hamilton automatically connects individual functions into a directed acyclic graph (DAG) that can be executed, visualized, optimized, and reported on. It supports dataframe transformations (pandas, polars, PySpark), machine learning pipelines, and even Generative AI/LLM-based workflows like async RAG pipelines and document processing.
Key features include built-in observability hooks, automatic DAG visualization without executing code, data and schema validation via decorators, function modifiers to keep code DRY, separation of transformation logic from execution for remote scaling (AWS, Modal, Spark, Ray, duckdb), and a companion open-source UI for visualizing, cataloging, and monitoring dataflows. The UI provides telemetry for executions, a feature/artifact catalog, a DAG visualizer for exploring lineage, and a project explorer for viewing versions.
Apache Hamilton is designed for data scientists, engineers, and data teams who want to write testable, modular, self-documenting dataflows. It facilitates collaboration by generating flat dataflows from well-scoped functions, reduces development time through reusable dataflows, enables platform ownership by integrating with existing stack tools, and scales dataflows via remote execution. It is used by companies like IBM, Stitch Fix, Adobe, Wealth.com, British Cycling, and the UK Government Digital Service for time series forecasting, LLM pipelines, feature engineering, and ML workflows.
Hamilton pricing
Pricing model: Freemium
Apache Hamilton is fully open-source under the Apache 2.0 License and is free to use. There is no paid tier or commercial pricing. The Hamilton UI is also fully open-source and self-hostable, running locally via sqlite3 or on Docker with Postgres for production deployment. Installation is via pip: pip install
Hamilton pros
- Executes a graph of data transformations
- Visualizes lineage easily from code
- Models Generative AI and LLM-based workflows
- Helps structure codebase with flat dataflows
- Is just a library, not a heavy orchestrator
- Runs anywhere Python runs
- Documentation friendly with auto-generated visuals
- Code is always unit testable
- Separates transformation logic from execution
- Supports remote execution on AWS, Modal, Spark, Ray
- Built-in data and schema validation with decorators
- Function modifiers reduce code redundancy
- Companion open-source UI for observability
- Reusable dataflows across projects and contexts
- Easy to extend and customize to needs
- Type-annotated functions hold no Hamilton dependency
- Integrates with pandas, polars, PySpark, duckdb
- Supports async Python LLM document processing pipelines
Hamilton cons
- Still in Apache Incubator (not yet a top-level project)
- No built-in macro orchestration like Airflow
- Not a feature store (unlike Feast)
- UI requires self-hosting (local sqlite or Docker/Postgres)
- Graphviz must be installed separately for visualizations
- UI Docker builds require 10GB+ Docker memory
- Limited self-hosting documentation for cloud platforms
- No native looping or conditional logic (use Burr for that)
Frequently asked questions about Hamilton
What is Apache Hamilton?
Apache Hamilton is a general-purpose framework to write dataflows using regular Python functions. Each function defines a transformation and its parameters indicate dependencies. Hamilton automatically connects functions into a DAG that can be executed, visualized, optimized, and reported on. It is a lightweight library, not an orchestrator, and runs anywhere Python runs.
How is Hamilton different from Airflow?
Hamilton is just a library that executes graphs of data transformations, while Airflow is a macro orchestration system. Hamilton can visualize lineage easily, models Generative AI/LLM workflows (which Airflow cannot), helps structure codebase, runs anywhere Python runs, is documentation-friendly, and code is always unit testable. Hamilton separates definition from execution, whereas Airflow is heavier and more infrastructure-bound.
What types of workflows can Hamilton handle?
Hamilton handles dataframe transformations (pandas, polars, PySpark), machine learning pipelines, regular software engineering API request workflows, and Generative AI/LLM-based workflows including async RAG pipelines, async Python LLM document processing, and prompt engineering research. It cannot handle loops or conditional logic—that requires the sister library Burr.
Can I use Hamilton with existing Python code?
Yes. Transformations are regular Python functions organized into modules that must be type-annotated but hold no dependency with Hamilton and can be reused outside of it. Add the HamiltonTracker adapter to your existing code to track executions in the UI. The notebook extension also prevents pain when migrating code from notebooks to Python modules.
Is Hamilton production-ready?
Yes. Hamilton was battle-tested under intensive enterprise data workloads since 2019 at Stitch Fix before the creators founded DAGWorks Inc. It is used in production by companies like IBM, Adobe, Wealth.com (Ester AI legal assistant), UK Government Digital Service, British Cycling, and Federal Reserve Board. It is currently in Apache Incubator.
How does Hamilton handle data validation?
Hamilton provides built-in data and schema validation. Decorate functions with @check_output to validate output properties and raise warnings or exceptions. Add the SchemaValidator() adapter to automatically inspect dataframe-like objects (pandas, polars, Ibis) to track and validate their schema.
Can Hamilton scale for enterprise workloads?
Yes. Hamilton separates transformation logic from execution, allowing seamless scaling via remote execution (AWS, Modal, etc.) and specialized computation engines (Spark, Ray, duckdb). It was battle-tested under intensive enterprise data workloads and supports self-serve designs with customizable features.
How do I get help with Hamilton?
Reach out via the Apache Hamilton Slack community for help and troubleshooting. You can also use GitHub Issues for bug tracking, participate in mailing lists ([email protected], [email protected]), consult official documentation, view examples on GitHub, read the DAGWorks blog guides, or watch video tutorials on the DAGWorks YouTube channel.