Which AI tools do people use to make a hand-tracking or gesture-controlled game?

Across the 6 sourced use cases in this guide, the most used tools are GPT-6, OpenAI Codex, Claude, Claude Code, GPT-6 Astra.

How much does it cost to get started?

2 of the 6 use cases can start on a free plan. The most common paid starting price is $30/mo (1 use cases). Prices come from each tool's official pricing page.

How long does setup take?

1 of the 6 use cases lists a setup time of an hour or less. Each card shows the setup time the builder reported.

Are these real examples?

Yes. Every use case links to its public source (an X post, a Hacker News thread or a GitHub repo) with the author and date, so you can check it yourself.

How to make a hand-tracking or gesture-controlled game with AI

Short answer: You can make a hand-tracking or gesture-controlled game with AI by having a coding agent wire a hand-tracking input (a VR headset's hand tracking, or a webcam hand-tracking library in the browser) to simple game actions like pinch, point and swipe. The builders below did this. Vector Tango is an air traffic control game for Meta Quest built entirely with AI, where you pinch a plane and draw its path with your finger. An artist built a Quest 3 mixed-reality app with Grok Bot in Unity, with passthrough, hand tracking and AI-generated 3D assets. A first-time vibe coder made a globe you spin with your bare hand. Start with one gesture, test it in real lighting, and add more once it feels reliable.

Sourced examples of people building games and installations you control with your hands, using Claude, Codex, Grok Bot and other AI tools. 6 matching use cases below, each with a public source.

Looking for something else? Search all 1889 use cases or .

Get started in 3 steps

Start with one or two gestures, like pinch to grab, and make them forgiving before adding more.
Test in real rooms and lighting; hand tracking that works at your desk can fail in a bright or busy space.
Ask the agent for a debug overlay that shows the tracked hand points, so you can see why a gesture misfires.

Vibe coding & appsX · @xbh_artist

Vibe code a mixed-reality Quest 3 app with hand tracking and AI-made 3D assets

An artist built a Quest 3 mixed-reality app with Grok Bot in Unity and Meta's MR Utility Kit, with every camera asset generated in Tripo, passthrough and hand tracking, a cloth-tear simulation and in-app sounds.

$30/moweeksAdvanced

Grok BotTripoUnity

Vibe coding & appsX · @yibaili530

Make a hand-gesture flying game in the browser with Codex and three.js

After an earlier gesture racing game, a builder made a seaside flight version: turn on your camera and steer a flying car with your palm. It is built with Codex and three.js and plays in the browser.

Free tier1 dayIntermediate

OpenAI Codex three.js

Dev & opsX · @KyleBoas_

Swipe between your coding agent sessions in an iPhone app

A developer made an iOS app that flips between individual Claude Code, pi and other coding agent sessions with swipe gestures, and shows mockups right in the app instead of local or Tailscale links.

$17/mo1 hourIntermediate

Claude CodePi

Vibe coding & appsX · @scobelverse

Build a hand-tracked air traffic control game for Meta Quest with AI

Vector Tango is an air traffic control game for Meta Quest played with bare hands: pinch a plane on the table, draw its path with your finger, then look up and watch it land outside the tower window. It was built entirely with AI.

See pricing pageweeksAdvanced

GPT-6ClaudeMeta Quest

Vibe coding & appsX · @Argona0x

Spin a 3D globe with your bare hand in a vibe-coded installation

In his first vibe-coding project, a 19-year-old built a globe you spin with your bare hand in the air using Codex. The post says four hotel lobbies paid him $11,600 for it; that figure is the poster's claim.

Free tierweeksIntermediate

OpenAI CodexGPT-6

Vibe coding & appsHacker News · callbacked

Control your Mac with gestures from a Meta Neural Band

The author had Astra take over a 10-month-old proof-of-concept repo that reads sEMG data off the Meta Neural Band, then turned it into Kinesis: swipe between desktops, open Mission Control and turn a virtual volume knob with finger gestures.

$10 in / $50 out per 1M tokens1 dayAdvanced

GPT-6 Astra

Common questions

How do you make a hand-tracking or gesture-controlled game with AI?: You can make a hand-tracking or gesture-controlled game with AI by having a coding agent wire a hand-tracking input (a VR headset's hand tracking, or a webcam hand-tracking library in the browser) to simple game actions like pinch, point and swipe. The builders below did this. Vector Tango is an air traffic control game for Meta Quest built entirely with AI, where you pinch a plane and draw its path with your finger. An artist built a Quest 3 mixed-reality app with Grok Bot in Unity, with passthrough, hand tracking and AI-generated 3D assets. A first-time vibe coder made a globe you spin with your bare hand. Start with one gesture, test it in real lighting, and add more once it feels reliable.
Which AI tools do people use to make a hand-tracking or gesture-controlled game?: Across the 6 sourced use cases in this guide, the most used tools are GPT-6, OpenAI Codex, Claude, Claude Code, GPT-6 Astra.
How much does it cost to get started?: 2 of the 6 use cases can start on a free plan. The most common paid starting price is $30/mo (1 use cases). Prices come from each tool's official pricing page.
How long does setup take?: 1 of the 6 use cases lists a setup time of an hour or less. Each card shows the setup time the builder reported.
Are these real examples?: Yes. Every use case links to its public source (an X post, a Hacker News thread or a GitHub repo) with the author and date, so you can check it yourself.

More how-to guides