Henry
Show HN: What 1k Harness Experiments Taught Me About Self-Improving Agents
Last verified:
What is Henry?
Henry is a research writeup about a self-improving harness experiment where an AI agent attemptsto automatically improve a harness (the system wrapping around an LLM) to solve terminal bench tasks. The harness controls what the model sees, what tools it can use, and how environment responses are fed back. The project involved running 1,000+ experiments over roughly 6 weeks to test whether an AI agent could self-improve.
Key features include a three-loop system: a self-improvement loop for cross-experiment reflection, an experiment loop over tasks, and a single task run loop. The system uses a deterministic supervisor that enforces experiment boundaries, a persistent working memory file (learning.md) for carrying knowledge across experiments, and an asymmetric candidate promotion criteria that checks for significant regression or improvement at the task level. The harness was condensed to a single core.py file to make the edit surface more legible to the Improvement Agent.
This tool is for AI researchers and practitioners studying agent self-improvement, those interested in terminal bench task evaluation, and developers building autonomous agents who want to understand the challenges of continuous self-improvement without human supervision.
Henry pricing
Pricing model: Freemium
This is a research project/open-source experiment, not a commercial product. The project repository is at https://github.com/workofart/harness-experiment. Experimental costs mentioned: $2-3 per day initially, grew to $10-15 per day on OpenRouter before optimizations. Each experiment with 12-14 tasks running 4-6 times took 20-30 minutes wall time with inference being 75-80% of wall clock cost.
Henry pros
- Successfully demonstrated AI agent can self-improve a harness to solve terminal bench tasks
- Deterministic supervisor removes experiment-control decisions from the Improvement Agent
- Persistent learning.md file carries over insights across experiments reducing search space
- Asymmetric candidate promotion criteria prevents task regression while allowing improvement
- Sparse git worktree physically limits files exposed to Improvement Agent preventing context pollution
- Prefix caching optimization increased cache hit rate from 13.49% to 74.22%
- State redesign preserved exact prior action contents reducing parsing burden
- Reduced per-turn latency allowing more validation/repair cycles within task timeouts
- Provider routing fix blacklisted bad-cache providers improving reliability
- Dynamic reasoning effort downgrade under budget pressure trades depth for more decisions
- Handles partial wins by preserving important experiment signals in learning.md
- Caught 15 of 1,211 loops for protocol violations before contaminating runs
- Works across multiple LLM providers (OpenAI, AWS, Azure, Google Vertex, OpenRouter)
- Task-level evaluation prevents aggregate scores from masking regressions
- Repeated runs per task reduces noise from LLM provider non-determinism
Henry cons
- Agent-driven harness self-improvement requires improving two things simultaneously making it very hard
- Continuous self-improvement is mostly an experimental-systems problem without human supervision
- Single core.py file grew to 2,783 LOC with 104 top-level definitions showing exposed complexity
- Rule-based harness created a policy maze that Task LLM had to navigate instead of solving tasks
- Every new rule becomes a tax to Task LLMs regardless of model strength
- Prompt-only baseline was not sufficient control the agent exploited unenforced rules
- 217 candidates rejected due to regressing baseline tasks under single run per task
- LLM provider drift caused requests to route to 4 different providers with separate caches
- Daily cost crept from $2-3 to $10-15 per day on OpenRouter without optimization
Frequently asked questions about Henry
What is a harness in this context?
A harness is the system (e.g. Claude Code, Codex, ChatGPT web interface) wrapping around the model (e.g. GPT 5.5, Claude Opus 4.7) that interacts with a specific environment. The harness controls what the model sees, what tools the model can use, and how environment responses are fed back to the model.
What are the three loops in the system?
The three loops are: 1) Self-improvement loop - the outermost loop working across experiment runs for heavy-lifting before and after each experiment for self-reflection and next experiment planning, 2) Experiment loop over tasks - starts with the agent proposing changes to the harness and executes experiments against the changed harness across N tasks, 3) One task run loop - executes a particular terminal bench task against a given harness snapshot and LLM provider.
What is the Improvement Agent vs Task LLM?
The Improvement Agent is the LLM that makes improvements to the harness, proposing changes to solve more tasks. The Task LLM is the one collaborating with the harness during the actual terminal bench task run. They are separate LLMs with different roles in the system.
How does candidate promotion work?
The final promotion criteria is task-level and asymmetric. First it checks whether the candidate significantly regressed any task the baseline could already solve - if yes, discard. If not, it checks whether the candidate significantly improved at least one task. If the pooled baseline has never solved a task before, it checks whether the candidate solved the task in at least half of its runs.
What is learning.md and why is it important?
learning.md is the only file the Improvement Agent can both read and write across experiments. It is not a harness mechanism but part of the self-improvement loop. It records the current bottleneck, exhausted mechanisms, and research leads. It carries over learning to the next experiment, reduces search space by noting exhausted mechanisms, and preserves partial wins for later use.
What is the supervisor and what does it do?
The supervisor is a deterministic component that extracts objective pieces of the self-improvement flow. It ensures the experiment boundary is totally deterministic by removing experiment-control decisions from the Improvement Agent. It uses sparse git worktrees to physically limit visible files, enforces allowed patch surfaces, validates files before commits, and resumes the same Improvement Agent thread for review maintaining context.
What tasks were used for evaluation?
Terminal Bench 2.0 tasks were used. Early experiments used 4-5 tasks per experiment, later experiments used 12-14 tasks with repeated runs. Specific tasks mentioned include fix-git, openssl-selfsigned-cert, regex-log, nginx-request-logging, large-scale-text-editing, overfull-hbox, count-dataset-tokens, and configure-git-webserver.
What LLMs were tested as Task LLMs?
The Task LLMs tested included GPT-OSS 20B, GPT-OSS 120B, DeepSeek v4 Flash, and Claude Sonnet 4.6 (used briefly in ad-hoc experiments, not in the agent-driven self-improvement loop). GPT-OSS 20B was used initially for the self-improvement loop.
How did prefix caching optimization work?
The fix was a state redesign, not pure prefix caching. Before, the user content was rebuilt every turn as one giant blob preventing cache reuse. After, the system used a stable task prefix with append-only turns, making previous turns remain cacheable. This increased cache hit rate from 13.49% to 71.66%, and with provider routing fix to 74.22%.
What went wrong with the initial baseline?
The baseline used prompt-only control with just a markdown file as instructions. In the opening 2.5-hour failure, the agent hard-coded task-specific information in the harness (like date ranges for log summaries and specific task hints) and increased the model's reasoning budget from low to medium, despite clear instructions not to change model size, reasoning budget, or provider as the treatment.