Crawl the web for agents without an LLM in the crawl loop

Built Jevcrawl, an MCP and CLI crawler that finds answer pages more often without calling an LLM while crawling.

Free tier1-2 hoursIntermediateSource linked

Source

rayenmlayeh on Hacker News · 2026-10-13 · Project link

Written in our own words. Credit to the original author. Link checked 2026-10-15.

What they did

Built Jevcrawl, an MCP and CLI crawler that finds answer pages more often without calling an LLM while crawling.

Result (as claimed)

Open-sourced MCP/CLI for agent web research.

Tools they used

Claude Code Cursor

Cost: Claude Code: free plan, paid from $17/mo; Cursor: free plan, paid from $8/mo. Checked 2026-10-14.

Set it up

Gather your sources
Collect the papers, notes, links or files you want to work from in one place.
Load them into the tool
Open Claude Code, Cursor and add your sources. Keep the original files so you can check claims later.
Ask focused questions
Ask for summaries, comparisons or connections with citations to the source passages.
Verify and save
Spot check claims against the sources, then save the output as notes you can reuse.

Copy the full setup prompt

Paste into your agent and it will walk you through the setup, asking before anything that sends, posts or spends.

You are helping me set up this AI use case end to end in Claude Code.

Goal: Crawl the web for agents without an LLM in the crawl loop
What the original author did: Built Jevcrawl, an MCP and CLI crawler that finds answer pages more often without calling an LLM while crawling.
Reported result (as claimed by the author): Open-sourced MCP/CLI for agent web research.
Source: https://news.ycombinator.com/item?id=50080450
Tools they used: Claude Code, Cursor
Category: Research & learning

Please:
1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like).
2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure.
3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data.
4. Finish with a checklist to run it again next time.

You are helping me set up this AI use case end to end in Codex.

Goal: Crawl the web for agents without an LLM in the crawl loop
What the original author did: Built Jevcrawl, an MCP and CLI crawler that finds answer pages more often without calling an LLM while crawling.
Reported result (as claimed by the author): Open-sourced MCP/CLI for agent web research.
Source: https://news.ycombinator.com/item?id=50080450
Tools they used: Claude Code, Cursor
Category: Research & learning

Please:
1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like).
2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure.
3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data.
4. Finish with a checklist to run it again next time.

You are helping me set up this AI use case end to end in Grok Bot.

Goal: Crawl the web for agents without an LLM in the crawl loop
What the original author did: Built Jevcrawl, an MCP and CLI crawler that finds answer pages more often without calling an LLM while crawling.
Reported result (as claimed by the author): Open-sourced MCP/CLI for agent web research.
Source: https://news.ycombinator.com/item?id=50080450
Tools they used: Claude Code, Cursor
Category: Research & learning

Please:
1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like).
2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure.
3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data.
4. Finish with a checklist to run it again next time.

The Daily 5