case study
Human Anomaly Detection System
- YOLOv8
- ONNX
- OpenCV
- Node.js
- Prisma
- Socket.IO
- React
My final year engineering project, restored and rebuilt. Real-time AI surveillance: it detects falls, sitting, squatting and standing from live video using a pretrained YOLOv8 pose model plus a posture classifier I built from keypoint geometry and calibrated against 3,106 detections drawn from 4,924 images.
The model I trained is the one I threw away
The version submitted for my degree was a yolov8n fine-tuned on 8,340 Roboflow images to predict posture directly, in three classes. Replacing it was not a preference, it was a measurement. The eval harness replays every labelled fixture through six perturbations (horizontal flip, greyscale, darken 40%, 3px blur, downscale to 320px wide, and an 80% centre crop), and both the old detector and its replacement score perfectly on the clean images. Only the sweep separates them: the trained detector fell to 23 of 30, the pose-plus-geometry pipeline held 30 of 30. The per-class breakdown said why. Under perturbation the sit class had precision 0.545 against recall 1.000. It never missed and it was wrong nearly half the time it fired. Blur or shrink a standing person and the answer became sit at 0.63 to 0.88 confidence. That is a fallback attractor, the signature of a model keyed on scene appearance (bench, floor, indoor room) rather than on the body. More data or a bigger backbone would not have fixed it, because the shortcut was available in the task framing itself.
What replaced it is 17 keypoints and ordinary code
The detector that ships is COCO-pretrained YOLOv8n-pose, unmodified. It is 12.9 MB, one class (person) and 17 keypoints, no fine-tuning, exported straight from the official release asset. It answers a far better conditioned question than three-way posture classification from 8,340 images: find the joints, on a problem trained across COCO's roughly 200,000 person instances with heavy augmentation. Posture is then decided in ordinary code from the geometry of those joints: torso angle off vertical, knee drop relative to torso length, knee interior angle, projected thigh-to-shin ratio, a signed hip-to-ankle drop, stance offset, and box aspect. Keypoint geometry has no equivalent shortcut to learn, because a shoulder-to-hip angle is the same blurred, greyscaled or 320px wide. What can still fail is finding the joints, and that is somebody else's well-solved problem. A useful side effect fell out of it. The pose model finds everyone in frame, so the street-fall fixture now returns the faller plus four background pedestrians where the old model saw one object.
A gate, not a vote
The thigh-to-shin rule is the clearest example of why this is written as an ordered chain rather than a scoring function. The femur and tibia are within about 10% of each other in real length, so their projected ratio sits near 1 whenever both lie in the image plane, which is what standing with vertical legs means. A thigh much shorter than its own shin can only mean the thigh points toward or away from the lens. It exists because of a real miss: a woman seated facing the camera with her legs stretched forward came back as stand at 63% confidence, because both other leg features genuinely read as standing, with knees well below hips at 0.64 and an almost straight leg at 172 degrees. Only the foreshortened thigh, 0.58 of its own shin, disagreed. Left as one signal among three it would lose two to one in precisely the case it exists to catch. Overriding the other two is justified because this is a statement about projection geometry rather than a correlation: no standing pose puts a thigh at 0.58 of its shin. Measured separation backs the 0.75 threshold: standing subjects at 1.00 to 1.11, front-on seated at 0.47 and 0.58.
A man flat on his back, returned as sit
A 51-second aikido clip produced a frame of a man flat on his back returning sit at 0.77. Reading back why it escaped is the most useful thing in this project. His torso measured 25 degrees, under the 50-degree fall gate. His box aspect was 1.96, but the wide-box escape hatch then also required 30 degrees of torso angle, so he missed it by 4.6. Falling through to the seated branch, his knee drop was minus 1.11: knees a full torso-length above his hips, which is a body on its back and never a seated posture. A minimum on that feature had already been added for exactly this shape, on the stated reasoning that a fall relabelled as something calm is a missed alarm, but it was applied to the squat gate and nowhere else, so the leak did not close, it moved to sit. The wide-box condition was self-defeating in the same way: it existed for bodies foreshortened along the view axis, which have a low torso angle by construction, and then demanded a high one. Both were fixed against the corpus rather than by argument. Among detections predicted sit, a knee drop below minus 1.00 splits 36 falls to 0 sits; the change recovers 254 falls against 9 sits, about 28 to 1. The floor sits at minus 0.25 rather than 0 because a genuine bench-sit with legs drawn up measures minus 0.16, and a floor at zero would have called that real sit a fall.
Every accuracy figure was measured on people who had already fallen
The fixture set reports 89.5% clean and 89.5% perturbed, macro-F1 0.901, and the repo says in as many words that this isn't a deployment accuracy — the fixtures were selected to be hard. But the deeper problem was what nothing measured at all. The 2023 corpus is entirely accident scenes, so there was no population of confirmed not-fallen people to count false alarms against, which is the one number that decides whether an anomaly detector is usable in a room where nothing is wrong. The POLAR posture dataset supplies it: 5,784 matched people across sit, squat and stand, containing no falls, so every fall the system emits on it is wrong by construction. It calls 666 of them a fall (11.5%), and 470 of those, 8.1%, at a confidence the tracker would act on. Two-thirds of those are crouching, which is the deliberate kneeling trade finally carrying a number instead of an acknowledgement.
The tracker absorbs a crouch and cannot absorb a bed
A single frame cannot tell a fall from a person bending over, so a tracker votes each person's posture over a rolling window and only confirms a fall after it persists. Whether that actually absorbs the 11.5% was the last claim in the project resting on reasoning rather than measurement, so it got measured: the real detector run over the UR Fall dataset's 40 activity-of-daily-living clips (1,502 frames, five minutes, domestic interiors, no falls), feeding the real tracker in order. The per-frame rate came back at 11.5%, reproducing POLAR's figure exactly on a different dataset, in a different country, in a different decade, labelled by different people. Confirmed alarms: zero at 1 fps, zero at 2 fps, five at 5 fps. Frame rate changes the answer in the counter-intuitive direction, because the confirmation threshold is in milliseconds while the vote window is a fixed number of results, so the window spans 1.4 seconds at 5 fps and 7 seconds at 1 fps, and a brief crouch dominates a short window while being diluted in a long one. The slower, worse-sounding rates are the robust ones.
The finding that spoils the good number
Three of those five alarms were transient, two men bending to pick something up and one kneeling. Two were a man lying on a bed. Reading zero-per-hour at 1 and 2 fps as evidence the system does not alarm on someone in bed would have been exactly wrong, and the only reason those two did not confirm at the slower rates is that the clips are about seven and a half seconds long: no posture in them lasts long enough for a slow sample rate to accumulate. That is a property of the footage, not of the system. Looping the settled-in-bed tail into 40 continuous seconds settles it, with a confirmed fall at every rate, within 1.2 to 2.0 seconds, held for the full duration and never cleared. The tracker's only mechanism is whether a posture persisted, which separates a fall from a transient crouch and cannot separate one from any sustained horizontal posture, because persistence is what those two share. On the floor is a fall, on a bed is not, and nothing in this system distinguishes them. Fixing it needs a signal geometry does not carry: a floor and furniture zone mask, a ground-plane homography from a fixed camera, or the transition into the pose rather than the pose itself. It is written down as the most deployment-relevant gap rather than left for a demo to discover.
One learned component ships, at the narrowest scope that worked
A trained keypoint classifier was the obvious next step, so it was built and then rejected. Testing it by holding out whole domains rather than shuffled images showed it did not transfer as a unit: stand precision fell from 0.876 to 0.744, which was the criterion set in advance. There is also a structural limit worth stating. The two corpora are nearly class-disjoint, with 1,970 falls in one and none at all in the other, so fall cannot be cross-domain tested with local data however the experiment is arranged. But the measurement split the model cleanly: the squat half transferred where the fall half collapsed. So the squat half ships, as a binary probe consulted in exactly one situation, where the geometric gates have already decided sit, at full-visibility tier, on a body whose feet are under it. Three invariants make that safe structurally rather than behaviourally: the fall gates run earlier and return before it, so it cannot suppress an alarm at any threshold or in any unseen domain; it never runs on a stand; and its worst case is a relabel between two non-alarming classes. Tests assert all three and exit non-zero if either of the first two breaks. It moved exactly one fixture, from 84.2% to 89.5%, and that's the honest size of it.