StressingLLMs
Complexity Benchmark
Last checked:
What is StressingLLMs?
StressingLLMs is a benchmark framework that evaluates local language models' ability to reverse-engineer obfuscated C binaries and recover encrypted messages. It tests model reasoning, binary analysis through Ghidra, and code generation capabilities across increasing complexity levels, with results tracking pass rates per model and round.
StressingLLMs pricing
Pricing model: Freemium
StressingLLMs pros
- Real binary analysis via Ghidra MCP integration rather than synthetic tasks
- Evaluates complete workflow: binary inspection, reasoning, Python code generation, and sandbox execution
- Tests progressive complexity with configurable transformation rounds and state chaining
- Transparent results with pass/fail outcomes, failure categories, and runtime metrics per model
StressingLLMs cons
- Limited to synthetic fixtures from one family of transformations; does not test diverse binary styles
- Results are hardware and timeout-dependent (runs on single NVIDIA DGX Spark); not universally reproducible
- Uneven sample sizes per model and adaptive search strategy make pass rates not directly comparable
- Requires substantial hardware investment to reproduce; evaluates model-plus-tool workflow, not isolated reasoning
Frequently asked questions about StressingLLMs
What models have been benchmarked?
Various local models including deepseek-v4-flash-dspark, ollama-based variants (cogito-32b, command-r-35b, deepseek-r1-32b, gemma4 models), tested on a single NVIDIA DGX Spark.
What is the overall pass rate?
36.8% pass rate across 76 completed rounds (28 passed, 48 failed), consuming 149.99M cumulative tokens.
Is this open source?
Yes, the project is referenced on GitHub (github.com/alexander-hanel/StressingLLMs); running it requires NVIDIA DGX hardware and Ghidra MCP setup.