halftrace
See the shape of your agent's failures from your logs
Last verified:
What is halftrace?
Halftrace is a diagnostic instrument for AI agent trajectories that analyzes how your agent's compliance with rules varies across trajectory length. You point it at your existing OpenAI, Anthropic, or LangSmith message logs and it returns a shape diagnosis (perfect/abandoned/bimodal/categorical/gradient), a commit-probability metric, and concrete suggestions for what to try next. No new API calls are required and no new task harness needs to be wired up—you bring the logs and halftrace does the analysis.
The tool includes four built-in probes that score different failure modes per trajectory: state_amnesia (retention of facts planted earlier), instruction_decay (adherence to system-prompt rules over time), tool_repetition (avoiding re-calling tools with identical arguments), and narration_substitution (emitting actual tool calls rather than just describing them). For each probe, halftrace classifies the shape of the per-trajectory score distribution and provides a one-line cause plus 2-3 concrete suggestions drawn from empirical patterns observed in twelve API pilots.
Halftrace supports three main workflows: analyzing existing logs with the 'analyse' subcommand, comparing before/after prompt changes with the 'compare' subcommand, and running controlled experiments against the Anthropic API with the 'pilot' subcommand. It also supports custom probes—you can write your own probe functions that score any failure mode you care about. The tool is designed for AI agent developers and researchers who already have trajectory logs and want to diagnose why their agents fail, not as a benchmark, eval framework, or agent builder.
The tool works with OpenAI chat-completions payloads, Anthropic messages.create() payloads, and LangSmith run-tree dicts. It requires Python 3.11+ and pydantic as a dependency. For the pilot subcommand against Claude, you need to install the anthropic extra. The empirical findings from halftrace's pilot phase revealed that modern Claude models do not decay gradually on simple agentic tasks—compliance is categorical per turn-type and bimodal per trajectory, meaning the agent either commits to a rule for the whole trajectory or abandons it from turn 4 onward.
halftrace pricing
Pricing model: Freemium
Free and open source under MIT license. Available on PyPI as halftrace (version 0.1.2 as of May 18, 2026). Install with 'pip install halftrace'. For the pilot subcommand against Claude, install with 'pip install halftrace[anthropic]'. Optional extras include halftrace[openai] (not yet wired) and halftrace[all]. The tool costs nothing to use—analysis runs entirely over your existing logs with no API spend required.
halftrace pros
- Works with existing logs—no new API calls required
- No new task harness to wire up
- Analyzes OpenAI, Anthropic, and LangSmith log formats
- Provides shape diagnosis (perfect/abandoned/bimodal/categorical/gradient)
- Returns commit_probability as a headline metric
- Gives concrete 2-3 suggestions for each failure mode
- Includes 4 built-in probes for common failure modes
- Supports custom probe functions for any failure mode
- Compare workflow shows before/after prompt change impact
- Pilot workflow runs controlled experiments at varying trajectory lengths
- Python 3.11+ with simple pip install
- Custom probes are 50-100 line files—easy to write
- Ingest functions for all three formats (OpenAI/Anthropic/LangSmith)
- Metadata support for configurable probes like instruction_decay rules
- MIT license
halftrace cons
- Not a benchmark—no leaderboard or canonical task set
- Not an eval framework for grading or production observability
- Not an agent framework—doesn't build agents
- Requires Python 3.11+ (not compatible with older Python)
- Pilot subcommand requires ANTHROPIC_API_KEY set
- OpenAI extra not yet wired into the runner
- Requires pydantic dependency
- Unclassified shape means you may need more reps or wider N range
Frequently asked questions about halftrace
What is halftrace and what does it do?
Halftrace is a diagnostic instrument for agent trajectories. You point it at your existing OpenAI, Anthropic, or LangSmith message logs and it tells you per failure mode: what shape of compliance your agent has (perfect/abandoned/bimodal/categorical/gradient), why that shape is likely showing up, and what to try next based on patterns observed in twelve API pilots. No new API calls are required.
What log formats does halftrace support?
Halftrace supports three log formats: OpenAI chat-completions payloads (use --format openai), Anthropic messages.create() payloads (use --format anthropic), and LangSmith run-tree dicts (use --format langsmith). Each line of your input JSONL file should be one trajectory in the corresponding format.
What are the four built-in probes?
The four probes that ship in the box are: state_amnesia (retention of facts planted earlier in the trajectory), instruction_decay (adherence to a system-prompt rule over time), tool_repetition (avoidance of re-calling tools with identical arguments), and narration_substitution (emitting tool calls rather than just describing them). premature_termination is drafted but not yet implemented.
What compliance shapes can halftrace detect?
Halftrace classifies five shapes: perfect (all trajectories score ≥ 0.95), abandoned (all trajectories score ≤ 0.05), bimodal (high within-cell variance—coinflip per trajectory), categorical (stable intermediate compliance—agent applies rule to one turn-type and drops it on another), and gradient (monotone decreasing means—the original halftrace concept assumed this). Unclassified fires when none of the above match.
What is commit_probability?
Commit_probability is the fraction of trajectories that follow the rule end-to-end. It is the headline metric for all shapes except gradient. For bimodal compliance, it represents the probability that the agent commits to a rule for the whole trajectory versus abandoning it from turn 4 onward.
How do I analyze my existing logs?
Run: halftrace analyse --input my_logs.jsonl --format openai (or --format anthropic or --format langsmith). Each line of your JSONL file should be one trajectory. The tool ingests each trajectory, scores every probe, and prints a profile per probe showing shape, commit_probability, why that shape appeared, and what to try next.
How do I compare before and after a prompt change?
Run: halftrace compare --before before_logs.jsonl --after after_logs.jsonl --format openai. This compares 50 before-trajectories vs 50 after-trajectories and shows which probes improved (like instruction_decay going from bimodal c=0.52 to perfect c=0.96), which unchanged, with Δcommit values. The comparison runs entirely over existing logs with no API spend.
How do I run a controlled experiment with the pilot command?
Run: halftrace pilot --n 5 10 25 --reps 3 --serial. This drives a built-in synthetic task (find_and_synthesise) through Claude at varying trajectory lengths, scores every probe, and emits a profile. You need to set ANTHROPIC_API_KEY and install with pip install halftrace[anthropic]. See halftrace pilot --help for model, plant-count, and discovery-mode flags.
What did halftrace's empirical research find?
Twelve pilot runs (~64 trajectories, $8.77 spend) revealed that modern Claude (Sonnet 4.6, Haiku 4.5) does not decay gradually on simple agentic tasks at N up to 200. Compliance is categorical per turn-type and bimodal per trajectory—the agent either commits to a rule for the whole trajectory or abandons it from turn 4 onward, with the choice approximately a coinflip. Three rule designs, two task variants, two models, zero instances of gradient decay on any probe.