Two Labeling Jobs on the Same Footage
The team's models reconstruct how a person moves from ordinary video, and they learn from short clips of the team's own footage. Two kinds of labeling are needed. In the first, an automatic tracker has already overlaid a 3D hand model on every clip, and a person decides whether the overlay really follows the hands: accept, reject, or keep only the seconds where tracking holds. In the second, every frame is labeled by hand: which of 13 body parts touch the scene, whether each contact holds still or slides, and which frames cannot be labeled or open a new shot.
The rules are strict. Swapped left and right hands reject a clip, minor jitter does not. A wrong label is worse than a missing one, because the model learns it.
Why the Tracker Cannot Grade Itself
- Letting the tracker's confidence decide. A swapped hand or a drifting mesh is reported as confidently as a clean track. Reviewing a tracker with its own signal repeats its mistakes.
- Labeling contact from the pose. A foot near the floor is not a foot on the floor, and in one frame a sliding foot looks planted. Contact is judged across neighboring frames.
- Handing out single frames to a crowd. Frames judged out of order lose the motion. A heel lifting is visible only in sequence.
- Checking quality at the end. One annotator reading a rule differently spreads the error across a whole batch before anyone sees it.
- Treating the sheet as free text. The sheets are read by software downstream. A stray space or a moved row quietly shifts labels onto the wrong frame.
Inside the Pipeline
The pipeline takes the clips straight from the client's cloud storage and returns completed sheets in the client's template.
- Prepare. Clips stream in with a manifest per batch. Duration, frame rate, resolution, codec and file naming are checked before any work starts. Frames are extracted in order and mapped one-to-one onto the rows of a prefilled sheet.
- Pre-screen. For the overlay review, a model analyzes the tracking over time and proposes a verdict with a reason code (hands swapped, overlay drifting, impossible finger geometry, heavy jitter, more than one person in frame) and a confidence. Its thresholds are first calibrated against human verdicts on the client's own clips.
- Annotate. The reviewer watches the clip and confirms or overrules the proposal. For contact, people label all 13 columns with the clip playing in sequence. As an optional assist, the model can prefill the flags for unlabelable frames and shot cuts, and the annotator checks each one.
- Double-check. A random 10% of the work is labeled again by a different annotator, and disagreement is reported per clip and per body part while the batch runs.
- Deliver. Every sheet is validated against the template, then returned to the agreed location with a report per batch.
Technology Stack
Labeling Rules
The model sorts the clips. People make the labels.
- Every model verdict goes through a person. The confidence shown beside a proposal tells the reviewer where to look hardest.
- A missing label is better than a wrong one. Where a contact cannot be determined with confidence, it stays blank rather than being filled in to look complete.
- Keep usable footage. A clip with a clean stretch is trimmed to that range instead of being thrown away, and the range is recorded precisely.
- The client's template is a contract. Headers, row order and allowed values come back untouched.
Quality Targets
The quality targets below are part of the pipeline's specification.
- Thirteen body parts judged on every frame, by people.
- A random 10% of the work labeled twice, with agreement measured during the batch.
- Every sheet validated before delivery: allowed values only, no rows moved, time ranges in the exact machine-readable format.
- A clip that will not decode or breaks the naming is reported with the reason.
- An interrupted batch resumes where it stopped, with no duplicate rows.
Teams Whose Models Learn From Their Own Video
The same pipeline suits any company whose AI learns from, or makes decisions about, its own video of people:
- an automatic model or tracker exists, but its output needs a person's check before use;
- labeling is done in-house by hand and holds up the next release;
- the result has to be defended to an auditor, a clinician or an insurer;
- the footage arrives in large batches.
Robotics, driver monitoring, rehabilitation and gait analysis, workplace safety, sports analytics and hand tracking all work with footage like this. The pipeline can take other footage, label sets and output formats, with the double-checked share agreed per project. Our earlier project on data annotation for AI-driven sports analytics was labeling of a related kind, and annotation, human-in-the-loop and our data annotation services give more detail.
body parts judged on every frame
of the work labeled twice, independently
of sheets validated against the client's template