P—04 Case study · AI & ML
Gesture-Based Mouse Control
Role
Computer Vision Engineer
Status
completed
Started
Jul 2025
Last activity
Jul 2025
Stack
6 technologies
Gesture-Based Mouse Control
Python · OpenCV · MediaPipe
Overview
A real-time touchless human-computer interaction system that uses a webcam to capture hand gestures and control mouse functions. The system recognizes a 5-gesture vocabulary — Cursor Control (open hand with index finger extended), Left Click (index-middle finger tap), Right Click (middle-ring finger tap), Drag & Drop (closed fist with movement), and Mission Control (all five fingers spread). A calibration system uses adaptive skin detection and background subtraction to work across different lighting conditions and skin tones. Achieves 92% recognition accuracy with 95-100ms end-to-end latency using a custom 5D nearest-neighbor classifier on MediaPipe hand landmarks.
Outcomes
Accuracy
Avg Latency
Gestures
Required
Impact
- 01Achieved 92% gesture recognition accuracy in bright lighting with 95-100ms end-to-end latency
- 02Built custom 5D nearest-neighbor classifier using MediaPipe hand landmarks with no GPU required
- 03Implemented JSON-based calibration persistence enabling cross-session zero-setup usage
- 04Designed 5-gesture vocabulary with state-machine-driven UI navigation (cursor, click, drag, mission control, exit)
Features
01
21-Point Hand Tracking
MediaPipe landmark detection maps 21 3D hand keypoints at 30fps — no GPU required, runs entirely on CPU
02
5-Gesture Vocabulary
Cursor (index extended), Click (finger tap), Drag (fist + move), Mission Control (5-finger spread), and Exit (peace sign)
03
Touchless HCI
Complete mouse control via webcam — movement, clicking, dragging, and system navigation without physical contact
04
Calibration System
Adaptive skin detection with background subtraction and JSON-based persistence for zero-setup across sessions
05
Performance Metrics
92% accuracy in bright lighting, 95-100ms latency, robust to varying backgrounds and hand orientations
06
5D NN Classifier
Custom nearest-neighbor classifier on MediaPipe landmarks — no deep learning needed on client side
Architecture
Pipeline: Webcam capture → OpenCV preprocessing (color space conversion, histogram equalization) → MediaPipe Hand landmark detection (21 keypoints) → Feature vector (5D: inter-finger distances + wrist proximity) → Custom k-NN classifier → Gesture state machine → PyAutoGUI system command. The state machine tracks gesture transitions with debouncing to prevent accidental triggers.
Stack
Development log
· Started
Gesture-controlled mouse interface
Accessibility-focused gesture mouse interface with JSON calibration and low-latency recognition. The camera detects hand gestures and decides for navigation controls such as selecting, closing, opening, and drag and drop. Can be improved with potential use cases in VR products.