InputMiddleweight
Hand tracking (MediaPipe)
The camera reads 21 points per hand.
The camera finds 21 points on each hand. Pinch, point, spread and wave become controls: grab an object, dim a light, scale a shape. It lags on slow devices and loses the hand at the edge of the frame.
- Breaks when
- lag on low-end devicesloses hands at frame edges
- Also called
- HandLandmarker, hand pose, pinch gesture
Prompts
Pinch to scale, drag to move
MediaPipe HandLandmarker (2 hands, GPU) at video rate: pinch strength = 1 − clamp(dist(thumbTip, indexTip) / (0.35 × palm width)), smoothed; a pinch above 0.7 grabs the nearest Three.js object to the index tip’s projected position, moving it while pinched and scaling it with the distance between two pinching hands. Draw the 21 landmarks and bones as a subtle overlay. When the hand is lost for 250 ms, release with a spring.
Hand as a light source
Use the palm center and normal (from wrist, index MCP and pinky MCP) to position and aim a SpotLight in the scene; finger spread controls the cone angle; a fist turns it off with a 200 ms fade. Explain the coordinate conversion from normalized landmarks to world space with the camera’s aspect.
Fallback and calibration
Add a 3-second calibration that measures the person’s palm width and distance from the camera, then normalizes all gesture thresholds by it; show a countdown and a silhouette. Provide the no-camera fallback: the pointer emulates the index tip and a key emulates the pinch.
Related
Resources · 3
- MediaPipe Hand LandmarkerWeb task API; 21 landmarks, GPU delegate.googlesite · code · demoJul 10, 2026
- MediaPipe Gesture Recognizergooglesite · code · demoJul 10, 2026
- ml5.jsFriendly wrappers; slower than raw MediaPipe.ml5jssite · code · demoJul 1, 2026