Let Claude Code control mouse and keyboard via screen vision
Built Rosply, a desktop agent that controls mouse and keyboard from screen vision and hooks into Claude Code as a global MCP server.
Source
@harkixsha on X · 2026-07-12
Written in our own words. Credit to the original author. Link checked 2026-10-15.
What they did
Built Rosply, a desktop agent that controls mouse and keyboard from screen vision and hooks into Claude Code as a global MCP server.
Result (as claimed)
Shared as a build-in-public MCP integration.
Tools they used
Cost: Claude Code: free plan, paid from $17/mo. Checked 2026-10-14.
Set it up
Map the task
Write the trigger, the inputs, the decisions and the output. Note which steps need a human yes.
Connect the tools
Set up Claude Code with the narrowest permissions that work. Use a test account or sandbox first.
Build the workflow
Create the steps in order. Add an approval step before anything sends, pays or deletes.
Run it on real but low-stakes data
Run a few times, read every output, and fix prompts or rules that misfire.
Schedule and monitor
Turn on the schedule or trigger, and add a daily summary or alert so you notice failures.
Copy the full setup prompt
Paste into your agent and it will walk you through the setup, asking before anything that sends, posts or spends.
You are helping me set up this AI use case end to end in Claude Code. Goal: Let Claude Code control mouse and keyboard via screen vision What the original author did: Built Rosply, a desktop agent that controls mouse and keyboard from screen vision and hooks into Claude Code as a global MCP server. Reported result (as claimed by the author): Shared as a build-in-public MCP integration. Source: https://x.com/harkixsha/status/2076436397896982706 Tools they used: Claude Code Category: Automation & business Please: 1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like). 2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure. 3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data. 4. Finish with a checklist to run it again next time.
You are helping me set up this AI use case end to end in Codex. Goal: Let Claude Code control mouse and keyboard via screen vision What the original author did: Built Rosply, a desktop agent that controls mouse and keyboard from screen vision and hooks into Claude Code as a global MCP server. Reported result (as claimed by the author): Shared as a build-in-public MCP integration. Source: https://x.com/harkixsha/status/2076436397896982706 Tools they used: Claude Code Category: Automation & business Please: 1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like). 2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure. 3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data. 4. Finish with a checklist to run it again next time.
You are helping me set up this AI use case end to end in Grok Bot. Goal: Let Claude Code control mouse and keyboard via screen vision What the original author did: Built Rosply, a desktop agent that controls mouse and keyboard from screen vision and hooks into Claude Code as a global MCP server. Reported result (as claimed by the author): Shared as a build-in-public MCP integration. Source: https://x.com/harkixsha/status/2076436397896982706 Tools they used: Claude Code Category: Automation & business Please: 1. Ask me 3 short questions about my situation (accounts I have, budget, what I want the output to look like). 2. Propose a step-by-step plan using these tools or cheaper alternatives, with the time and cost of each step. Use official pricing pages; say "check pricing" if unsure. 3. Then do the setup with me one step at a time, waiting for my OK before anything that sends messages, posts publicly, spends money or deletes data. 4. Finish with a checklist to run it again next time.
The Daily 5
Get five new use cases tomorrow morning
Free. Real sources. Unsubscribe any time.
Related use cases
Let Claude Code read WeChat messages on your Mac
Built WeChat Portal for read-only local access to WeChat messages, media and Feishu docs from Claude Code or Codex.
Vibe-code an agentic Skool community ops platform
Vibe-coded MySkool so agents can score churn risk, welcome members and answer the feed for Skool community owners.
Generate videos with code using Claude Code and Remotion
Built an automated video generation pipeline with Claude Code and Remotion.
Code from your phone with Claude Code on an old laptop
Set up an old laptop running Claude Code, reachable over Tailscale and driven from the Termius app on a phone.