Chainforge
An open-source visual programming environment for battle-testing prompts to LLMs.
Last verified:
What is Chainforge?
ChainForge is an open-source visual programming environment for prompt engineering, LLM evaluation, and experimentation. It is designed to help people test prompt ideas and model behavior with little to no coding, while still supporting more advanced workflows when needed.
The core idea is to let you query multiple LLMs at once, compare outputs across prompt permutations, models, and model settings, and then evaluate those outputs systematically. It also supports holding multiple conversations in parallel, templating chat messages, and inspecting results at each turn.
A major focus of the product is experimentation and analysis. ChainForge can use code-based or LLM-based scorers to define evaluation metrics, then automatically plot results so you can compare prompts, parameters, models, or settings visually.
ChainForge is also built for sharing and collaboration. The online version includes a Share Button that generates a unique link for an experiment, and the local version is intended for users who want a more private or configurable setup with their own API keys.
The tool is aimed at prompt engineers, researchers, developers, and teams building or testing LLM apps, including people evaluating RAG, agents, or custom model chains.
Chainforge pricing
Pricing model: Freemium
ChainForge offers a free online web version at chainforge.ai/play, but it is limited. The main local version is open-source and can be installed for free with pip install chainforge, then run locally with chainforge serve. The website does not list paid plans or subscription tiers; instead, access depends on the model providers you connect, which may require their own API keys and pricing. For local use, you can set API keys as environment variables to avoid entering them repeatedly.
Chainforge pros
- Open-source and transparent
- Visual programming interface
- Little to no coding required
- Supports prompt engineering workflows
- Queries multiple LLMs at once
- Compares prompt permutations
- Compares across models
- Compares across model settings
- Supports evaluation metrics
- Works with code-based scorers
- Works with LLM-based scorers
- Automatically plots results
- Supports parallel conversations
- Templates chat messages
- Inspect outputs at each turn
- Shareable experiment links
- Supports local installation
- Supports custom providers
- Compatible with common browsers
- Useful for research and hypothesis testing
Chainforge cons
- Active beta, not fully mature
- Web version is limited
- Requires API keys for models
- API keys must be re-entered in web usage
- Best experience may require local install
- Can be awkward for dev-mode users
- Red errors may appear in npm start dev mode
- Custom providers require Python coding
- Some workflows need evaluation pipelines
- Manual comparison across many models is cumbersome
- Template-variable system has learning curve
- Not all provider setups are plug-and-play
- May require environment-variable setup
- Some features are oriented toward developers
- Built-in sharing depends on the online version
Frequently asked questions about Chainforge
What is ChainForge used for?
ChainForge is used for prompt engineering, LLM evaluation, and experimentation. It helps users test prompt variations, compare model outputs, define evaluation metrics, and inspect results visually without writing much code.
Can I compare multiple prompts or templates in ChainForge?
Yes. The documentation recommends template chaining for comparing prompt templates, and the Response Inspector can show templates in table columns. For manual inspection, the docs suggest using a single LLM; for comparisons across multiple LLMs, they recommend setting up an evaluation pipeline.
Does ChainForge support multiple models and settings?
Yes. You can query multiple LLMs at once or compare the same model under different settings. The docs specifically mention comparing across prompts, across models, and across model settings, including values such as temperature.
Can I use my own model or custom chain?
Yes. The FAQ says you can use Custom Providers to bring your own model, chain, or RAG or agent output. That requires writing a Python completion function for the provider.
How do I inspect responses in ChainForge?
You can click the footer of a node to open the full-screen inspector, or use the pop-out inspector drawer. The docs also mention an Inspect Node for examining responses without a pop-up, and the inspector can export data to an Excel file.
What can be exported from ChainForge?
The Response Inspector can export an xlsx file. The exported file includes prompt variables, LLM nicknames, prompts, responses, and any evaluation scores that were collected.
How do I install ChainForge locally?
The docs say to install it with pip install chainforge, then run chainforge serve. After that, you open localhost:8000 in a supported browser such as Chrome, Firefox, Edge, or Brave.
Does ChainForge work online?
Yes. The website offers an online version at chainforge.ai/play. The web version is described as limited, but it includes a Share Button that creates a unique link for an experiment.
What browsers does ChainForge support?
The docs say it is compatible with Google Chrome and Mozilla Firefox, and the install instructions also mention Chrome, Firefox, Edge, and Brave as supported browsers for local use.
Do I need API keys to use ChainForge?
Yes, you need API keys for the LLM providers you want to use. The docs say ChainForge does not store API keys in a cookie, localStorage, or server, so you must set them each time you load the web version unless you install locally and set environment variables.