case study
Hand Gesture Recognition
Real-time hand gesture recognition that runs entirely in your browser, with no backend, so your video never leaves your device. MediaPipe tracks 21 keypoints per hand through a GPU-accelerated WASM runtime and classifies seven gestures.
What is mine here, and what is Google's
Worth saying first, because it's the obvious question. The seven gestures come from MediaPipe's own bundled classifier, a canned model shipped inside the task file, recognising a fixed set that happens to include the ASL sign for I love you. I didn't train it and this project doesn't claim to. What I engineered is everything between that model and something a stranger can open on their phone and have work: turning a noisy per-frame reading into a gesture, failing usefully when the hardware won't cooperate, and getting the mirroring right in the two places where it has opposite requirements. That's a smaller claim than a trained model, and it's the true one.
Six frames before it counts
The recogniser returns a label and a score for every frame, and taken literally that output flickers. A hand halfway between two poses produces whichever one is winning at that instant, several times a second. So a reading has to clear three hurdles before the interface calls it a gesture: it can't be the model's none category, it has to score at least 0.5, and it has to be the same label for six frames in a row. Six is what makes the score readable instead of strobing, and it's the same idea as the fall detector's tracker at a much smaller scale. A single frame doesn't know what a person is doing, and the fix is to require persistence instead of a better classifier. Both games sit on top of that stabilised label instead of the raw one, which is what makes a 30-second score meaningful.
Failing in a way the user can act on
The model loads with the GPU delegate first and CPU second. Before that fallback existed, a machine without a usable GPU delegate (software rendering, a locked-down driver, a browser without WebGL) failed setup outright and was shown an error telling it to check its network connection. That message was neither true nor something anyone could act on, which is a worse defect than the failure it reported. CPU is slower and perfectly fine for one hand. The WASM runtime and the model file are self-hosted instead of pulled from a CDN, so the deployed demo has no third-party runtime dependency at all. They're also requested from the site root instead of relative to the current path, because as relative paths they only resolved when the app happened to be served from the domain root.
The preview flips, the classifier sees the real hand
A front camera shows you a mirror image, and a preview that isn't flipped feels wrong straight away: move your hand right and the person on screen moves left. So the preview is mirrored, exactly as every native camera app does it. Classification runs on the unmirrored frames, because the model has opinions about handedness and flipping the pixels would quietly hand it the other hand. Same footage, two opposite requirements, and the only way to satisfy both is to keep the display transform out of the pipeline. Beyond that it's ordinary care: front and back camera switching on phones, real keyboard focus rings, labelled controls, thumb-sized targets, and reduced motion honoured, which matters more than usual here because the game states animate.