Gesture-Based Mouse Control
Computer Vision EngineerImpact
- Achieved 94.7% gesture recognition accuracy in bright lighting with 95-100ms end-to-end latency
- Built custom 5D nearest-neighbor classifier using MediaPipe hand landmarks with no GPU required
- Implemented JSON-based calibration persistence enabling cross-session zero-setup usage
- Designed 5-gesture vocabulary with state-machine-driven UI navigation (cursor, click, drag, mission control, exit)
Skills
Project Abstract & Details
A real-time touchless human-computer interaction system that uses a webcam to capture hand gestures and control mouse functions. The system recognizes a 5-gesture vocabulary — Cursor Control (open hand with index finger extended), Left Click (index-middle finger tap), Right Click (middle-ring finger tap), Drag & Drop (closed fist with movement), and Mission Control (all five fingers spread). A calibration system uses adaptive skin detection and background subtraction to work across different lighting conditions and skin tones. Achieves 94.7% recognition accuracy with 95-100ms end-to-end latency using a custom 5D nearest-neighbor classifier on MediaPipe hand landmarks.
Key Features
21-Point Hand Tracking
MediaPipe landmark detection maps 21 3D hand keypoints at 30fps — no GPU required, runs entirely on CPU
5-Gesture Vocabulary
Cursor (index extended), Click (finger tap), Drag (fist + move), Mission Control (5-finger spread), and Exit (peace sign)
Touchless HCI
Complete mouse control via webcam — movement, clicking, dragging, and system navigation without physical contact
Calibration System
Adaptive skin detection with background subtraction and JSON-based persistence for zero-setup across sessions
Performance Metrics
94.7% accuracy in bright lighting, 95-100ms latency, robust to varying backgrounds and hand orientations
5D NN Classifier
Custom nearest-neighbor classifier on MediaPipe landmarks — no deep learning needed on client side
Architecture
Pipeline: Webcam capture → OpenCV preprocessing (color space conversion, histogram equalization) → MediaPipe Hand landmark detection (21 keypoints) → Feature vector (5D: inter-finger distances + wrist proximity) → Custom k-NN classifier → Gesture state machine → PyAutoGUI system command. The state machine tracks gesture transitions with debouncing to prevent accidental triggers.
Architecture diagram placeholder