PkVision — Roadmap

Infrastructure

• Docker + docker-compose — Containerize the full stack (API + worker + Redis + PostgreSQL). Single docker-compose up to run everything. GPU passthrough support for training with NVIDIA Container Toolkit. Separate Dockerfile.api (lightweight, inference only) and Dockerfile.train (full ML deps + CUDA/MPS).
• CI/CD with GitHub Actions — Auto-run pytest on every push/PR. Lint with ruff. Type-check with mypy. Block merge if tests fail. Separate workflows for test.yml (fast, no GPU), train-smoke.yml (weekly, runs 5-epoch training on fixture data to catch regressions).
• PostgreSQL migration — Switch from SQLite to PostgreSQL for production. Alembic for schema migrations. Keep SQLite as dev default.
• Pre-commit hooks — ruff format + ruff check + mypy on staged files.

Real-time Pipeline

• Live webcam analysis — WebSocket endpoint that streams detections in real-time. YOLO processes frames at ~15fps, detection runs on sliding windows, scores update live.
• RTMP/RTSP stream input — Accept live video feeds from competition cameras. Integrate with OBS or professional streaming setups.
• Low-latency mode — Optimized pipeline for sub-second detection: skip frames, reduce YOLO input resolution, batch inference.
• Live overlay — OpenCV overlay on video feed showing skeleton, detected trick name, confidence bar, running score.

Benchmark & Metrics

• Accuracy benchmark suite — Standard test set of labeled clips with ground truth. Report per-trick precision, recall, F1. Compare angle threshold vs ST-GCN accuracy.
• Confusion matrix dashboard — Visual confusion matrix after each training run. Identify which tricks get confused (e.g. gainer vs back flip).
• Latency profiling — Measure end-to-end time: video load → pose extraction → detection → scoring. Track per-component timing. Target: < 2x video duration for offline, < 100ms per frame for real-time.
• Model versioning — Track model versions with metrics (accuracy, loss, training data size). MLflow or W&B integration for experiment tracking.
• Regression tests — Golden test set that must maintain >= X% accuracy. Fail CI if a model change drops below threshold.

Detection Improvements

• 3D pose estimation — Integrate MotionBERT or VideoPose3D for monocular 3D pose lifting. Critical for twist detection (rotations along camera axis).
• Multi-camera fusion — Combine 2+ camera angles for true 3D keypoints. Triangulation pipeline. Required for competition-grade accuracy.
• Execution quality scoring — Rate landing stability, body alignment, height. Separate D-score (difficulty) and E-score (execution) like FIG gymnastics.
• Combo detection — Detect trick sequences (back flip → twist → landing). Score combos with flow/transition bonuses.
• Trick phase visualization — Show approach/takeoff/execution/landing phases overlaid on the video timeline.

Community & Ecosystem

• Web dashboard — Next.js frontend for uploading videos, viewing results, browsing the trick catalog. Dark theme, responsive.
• Clip submission portal — Web form (not just GitHub Issues) for athletes to submit clips. Upload to S3/Blob, auto-notify maintainers.
• Leaderboard — Public leaderboard of highest-scoring runs submitted by the community.
• Mobile app — React Native app for filming + instant analysis. Film a trick, get immediate feedback on what was detected.
• Multilingual catalog — Expand beyond EN/FR: ES, DE, PT, JP, AR. Community-contributed translations.
• Plugin system — Allow third-party detection strategies. Community can train specialized models for niche tricks and share them.

Competition Integration

• FIG notation export — Export analysis results in FIG-compatible notation format. Align difficulty ratings with official Code of Points.
• Judge tablet interface — iPad-optimized UI for competition judges. View AI suggestions, apply overrides, submit final scores.
• Multi-athlete tracking — Detect and track multiple athletes in the same frame. Assign tricks to specific athletes.
• Competition mode — Locked-down mode with audit logging, no model updates during competition, tamper-evident results.
• Replay system — Slow-motion replay with skeleton overlay for judges to review contested detections.