Hand Gesture Recognition

gestures: 7
keypoints: 21 / hand

MediaPipe
WASM
React
Vite
Tailwind CSS

Real-time hand gesture recognition that runs entirely in the browser — no backend, video never leaves your device. MediaPipe tracks 21 keypoints per hand through a GPU-accelerated WASM runtime and classifies seven gestures.

What is mine here, and what is Google's

Worth saying first, because it is the obvious question. The seven gestures are MediaPipe's own bundled classifier, a canned model shipped inside the task file, recognising a fixed set that happens to include the ASL sign for I love you. I didn't train it and this project doesn't claim to. What is engineered is everything between that model and something a stranger can open on their phone and have work: turning a noisy per-frame reading into a gesture, failing usefully when the hardware will not cooperate, and getting the mirroring right in the two places it has opposite requirements. That is a smaller claim than a trained model, and it is the true one.

Six frames before it counts

The recogniser returns a label and a score for every frame, and taken literally that output flickers. A hand halfway between two poses produces whichever one currently wins, several times a second. So a reading has to clear three separate hurdles before the interface calls it a gesture: it must not be the model's none category, it must score at least 0.5, and it must be the same label for six consecutive frames. Six is what makes the score readable rather than strobing, and it is the same idea as the fall detector's tracker at a much smaller scale. A single frame does not know what a person is doing, and the fix is to require persistence rather than a better classifier. Both games sit on top of that stabilised label rather than the raw one, which is what makes a 30-second score meaningful.

Failing in a way the user can act on

The model loads with the GPU delegate first and CPU second. Before that fallback existed, a machine whose GPU delegate was unavailable (software rendering, a locked-down driver, a browser without WebGL) failed setup outright and was shown an error telling it to check its network connection. That message was neither true nor actionable, which is a worse defect than the failure it reported. CPU is slower and entirely fine for one hand. The WASM runtime and the model file are self-hosted instead of pulled from a CDN, so the deployed demo has no third-party runtime dependency at all; they are also requested from the site root rather than relative to the current path, because as relative paths they only resolved when the app happened to be served from the domain root.

The preview flips, the classifier sees the real hand

A front camera shows you a mirror image, and a preview that is not flipped feels immediately wrong — move your hand right and the person on screen moves left. So the preview is mirrored, exactly as every native camera app does it. Classification runs on the unmirrored frames, because the model has opinions about handedness and flipping the pixels would quietly hand it a different hand. Same footage, two opposite requirements, and the only way to satisfy both is to keep the display transform out of the pipeline. Beyond that it is ordinary care: front and back camera switching on phones, real keyboard focus rings, labelled controls, thumb-sized targets, and reduced motion honoured, which matters more than usual here, because the game states animate.