Real AI agent use cases, with sources and setup prompts

Benchmark your agent harness against Claude Code with hidden judges

The founders of Relay, a local-first agent supervisor, pre-registered a 5-test suite with hidden judges and ran their tool, Claude Code, Prime Agent and Hermes through the same token meter on Sonnet 5.5 and Haiku 5.5, then published the losses along with the wins.

$17/mo1 dayAdvancedSource linked

Source

@ArnavvSharmaa on X · 2026-10-18

Written in our own words. Credit to the original author. Link checked 2026-10-19.

What they did

The founders of Relay, a local-first agent supervisor, pre-registered a 5-test suite with hidden judges and ran their tool, Claude Code, Prime Agent and Hermes through the same token meter on Sonnet 5.5 and Haiku 5.5, then published the losses along with the wins.

Result (as claimed)

Claude Code won most accuracy tests (83.3% vs 71.8% on Terminal-Bench 2.1; 5.9% vs 26.5% false "done" claims), while Relay was cheapest per solved long task at $1.24 vs $2.48.

Tools they used

Claude CodeClaude Sonnet 5.5Claude Haiku 5.5Hermes Agent

Cost: Claude Code: from $17/mo; Claude Sonnet 5.5: from $2 in / $10 out per 1M tokens; Claude Haiku 5.5: from $0.10 in / $0.50 out per 1M tokens; Hermes Agent: free plan, paid from $20/mo.

Set it up

  1. Pick a contained repo or task

    Start on a branch or side project where mistakes are cheap.

  2. Install and configure

    Set up Claude Code, Claude Sonnet 5.5, Claude Haiku 5.5, Hermes Agent. Read what it can run on your machine and limit permissions where possible.

  3. Run it on a real task

    Give it a well-scoped issue and let it work. Review the diff like any pull request.

  4. Add guardrails

    Keep tests and CI in the loop, and never let an agent push to main or touch production secrets unattended.

Copy the full setup prompt

Paste into your agent and it will walk you through the setup, asking before anything that sends, posts or spends.

You are helping me set up this AI use case end to end in Claude Code.

Goal: Benchmark your agent harness against Claude Code with hidden judges
What the original author did: The founders of Relay, a local-first agent supervisor, pre-registered a 5-test suite with hidden judges and ran their tool, Claude Code, Prime Agent and Hermes through the same token meter on Sonnet 5.5 and Haiku 5.5, then published the losses along with the wins.
Reported result (as claimed by the author): Claude Code won most accuracy tests (83.3% vs 71.8% on Terminal-Bench 2.1; 5.9% vs 26.5% false "done" claims), while Relay was cheapest per solved long task at $1.24 vs $2.48.
Source: https://x.com/ArnavvSharmaa/status/2111841042144378964
Tools they used: Claude Code, Claude Sonnet 5.5, Claude Haiku 5.5, Hermes Agent
Category: Dev & ops

Please:
1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like).
2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure.
3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data.
4. Finish with a checklist to run it again next time.

The Daily 5

Get five new use cases tomorrow morning

Free. Real sources. Unsubscribe any time.

Guides for this use case

How to vibe code an app with Claude Code · How to run local LLMs with AI agents · How to set up Hermes Agent workflows · How to cut AI coding agent costs and token use

Related use cases