Check what else your agent's bug fix changed with a regression hunter
Trail of Bits published two agent plugins: patching-bugs reproduces a reported bug, fixes it and proves the fix with a three-arm test (old code fails, patch passes, revert fails again), and regression-hunt checks whether the change broke previously working behavior, with a live proof for every claim. Add THREAT_MODEL.md and PATCHING_GUIDELINES.md to your repo to guide them.
Source
tob_scott_a on Hacker News · 2026-10-21 · Project link
Written in our own words. Credit to the original author. Link checked 2026-10-23.
What they did
Trail of Bits published two agent plugins: patching-bugs reproduces a reported bug, fixes it and proves the fix with a three-arm test (old code fails, patch passes, revert fails again), and regression-hunt checks whether the change broke previously working behavior, with a live proof for every claim. Add THREAT_MODEL.md and PATCHING_GUIDELINES.md to your repo to guide them.
Result (as claimed)
In their evals regression-hunt produced 25% fewer false positives than Codex simply asked to "review the PR for regressions".
Tools they used
Cost: OpenAI Codex: free plan, paid from $8/mo. Checked 2026-10-14.
Set it up
Pick a contained repo or task
Start on a branch or side project where mistakes are cheap.
Install and configure
Set up OpenAI Codex, Trail of Bits plugins. Read what it can run on your machine and limit permissions where possible.
Run it on a real task
Give it a well-scoped issue and let it work. Review the diff like any pull request.
Add guardrails
Keep tests and CI in the loop, and never let an agent push to main or touch production secrets unattended.
Copy the full setup prompt
Paste into your agent and it will walk you through the setup, asking before anything that sends, posts or spends.
You are helping me set up this AI use case end to end in Claude Code. Goal: Check what else your agent's bug fix changed with a regression hunter What the original author did: Trail of Bits published two agent plugins: patching-bugs reproduces a reported bug, fixes it and proves the fix with a three-arm test (old code fails, patch passes, revert fails again), and regression-hunt checks whether the change broke previously working behavior, with a live proof for every claim. Add THREAT_MODEL.md and PATCHING_GUIDELINES.md to your repo to guide them. Reported result (as claimed by the author): In their evals regression-hunt produced 25% fewer false positives than Codex simply asked to "review the PR for regressions". Source: https://news.ycombinator.com/item?id=50183253 Tools they used: OpenAI Codex, Trail of Bits plugins Category: Dev & ops Please: 1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like). 2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure. 3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data. 4. Finish with a checklist to run it again next time.
You are helping me set up this AI use case end to end in Codex. Goal: Check what else your agent's bug fix changed with a regression hunter What the original author did: Trail of Bits published two agent plugins: patching-bugs reproduces a reported bug, fixes it and proves the fix with a three-arm test (old code fails, patch passes, revert fails again), and regression-hunt checks whether the change broke previously working behavior, with a live proof for every claim. Add THREAT_MODEL.md and PATCHING_GUIDELINES.md to your repo to guide them. Reported result (as claimed by the author): In their evals regression-hunt produced 25% fewer false positives than Codex simply asked to "review the PR for regressions". Source: https://news.ycombinator.com/item?id=50183253 Tools they used: OpenAI Codex, Trail of Bits plugins Category: Dev & ops Please: 1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like). 2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure. 3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data. 4. Finish with a checklist to run it again next time.
You are helping me set up this AI use case end to end in Grok Bot. Goal: Check what else your agent's bug fix changed with a regression hunter What the original author did: Trail of Bits published two agent plugins: patching-bugs reproduces a reported bug, fixes it and proves the fix with a three-arm test (old code fails, patch passes, revert fails again), and regression-hunt checks whether the change broke previously working behavior, with a live proof for every claim. Add THREAT_MODEL.md and PATCHING_GUIDELINES.md to your repo to guide them. Reported result (as claimed by the author): In their evals regression-hunt produced 25% fewer false positives than Codex simply asked to "review the PR for regressions". Source: https://news.ycombinator.com/item?id=50183253 Tools they used: OpenAI Codex, Trail of Bits plugins Category: Dev & ops Please: 1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like). 2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure. 3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data. 4. Finish with a checklist to run it again next time.
The Daily 5
Get five new use cases tomorrow morning
Free. Real sources. Unsubscribe any time.
Guides for this use case
How to test your app with AI agents before you ship · How to use Claude Code skills and plugins · How to run multiple coding agents together
Related use cases
Check that your coding agent's improvements to an AI app are actually reliable
A team built Autoeval, an open-source tool that lets Claude Code, Codex or Cursor keep improving an AI app while measuring reliability across repeated trials and judges. Releases are gated on that evidence.
Turn every bug fix into a rule your coding agent can't break
He built antibody: each fix becomes a rule, proven against the real fix and checked on every edit, so Claude Code, Cursor and Codex stop repeating mistakes you already fixed.
Let an agent review and fix every open PR overnight, and watch your usage
A developer let Amp work overnight on his open-source repo with one prompt: review every PR, wait for CI and Codex bot review comments, resolve them, re-request review, run an end-to-end test and prove everything worked. Claude Code hit its usage limit for 5 hours mid-run.
See which coding agent still has usage left from your Mac menu bar
Rate-limited across tools, he built Quotchi, a free Mac menu bar app that shows which of Claude Code, Codex and Gemini still has room. It needs no account and never reads your prompts.