Table-Python

Work in progress. This is an actively developed personal project, not a finished product — expect incomplete features, rough edges, and breaking changes.

Computer-vision and control application for a real-time, gesture-controlled smart tabletop for tabletop wargaming (Warhammer 40k). This app captures video from multiple overhead cameras, tracks players' hands in 3D over the physical table, and streams the result over UDP to the Table-UE Unreal Engine project, which projects an interactive overlay (rulers, markers, HUDs) directly onto the tabletop.

Overview

• PySide6 desktop app: camera setup/preview, calibration, and game control in one window, with a live log console.
• Multi-camera capture via OpenCV, with per-camera calibration (intrinsics + extrinsics) so cameras can be freely positioned above the table.
• ChArUco-board calibration to recover camera intrinsics and pose relative to the table, producing the projection matrices used for triangulation.
• MediaPipe Hands for per-camera 2D hand landmark detection (21 points/hand + handedness + confidence).
• Cross-camera hand matching + stereo triangulation: 2D detections from different cameras are paired up (via reprojection error) and triangulated into 3D hand landmarks in table space.
• Custom UDP protocol (HandUdpSender) streams the resulting 3D hand data to the Unreal Engine side every tick.

Architecture

MainWindow (PySide6)
 ├─ SetupPage           — pick/preview cameras, run ChArUco calibration
 ├─ HomeWindow          — camera list / calibration overview
 └─ GameWindow          — active session view

TrackingController (per-frame loop, driven by a QTimer)
 ├─ _detectHands_all_cameras()   — MediaPipe hand detection per camera → Hand2D
 ├─ _allocateDetectedHands()     — match 2D hands across cameras via reprojection error
 ├─ triangulate_hands_with_metadata() — 2D → 3D triangulation using calibrated P matrices
 └─ HandUdpSender.send_hands3d() — pack + send 3D hands over UDP → Table-UE

Key modules

Module	Responsibility
source/main.py	App entry point, top-level MainWindow (sidebar + page stack + log console)
source/UI/SetupPageWidget.py	Camera discovery, live preview, add/cycle cameras
source/UI/HomePageWidget.py	Calibration workflow and camera/projection overview
source/UI/GameWindow.py	Active tracking session view
source/Tbd/TrackingController.py	Core per-tick pipeline: detect → match → triangulate → send
source/Tbd/helper.py	Shared dataclasses (CameraObject, Hand2D, CharucoDetection, CalibrationSample)
source/Helpers/ChArUco.py	ChArUco board detection for calibration
source/Helpers/HandUdpSender.py	Binary UDP packet encoding for 3D hand data
source/Helpers/saveManager.py	Persisting/loading calibration and session data

UDP protocol

HandUdpSender sends one datagram per tick to 127.0.0.1:9000 (configurable):

Header:  magic "HAND" (4s) | version (B) | seq (I) | timestamp_ms (Q) | hand_count (B)
Per hand: hand_id (B) | point_count (B) | handedness (B: 0=Unknown,1=Left,2=Right) | confidence (f)
          21 × (x, y, z) float32 landmarks

The Unreal project decodes this same format in ADetectionManager.

Requirements

• Python 3.10+
• One or more USB/webcam-style cameras positioned above the table
• Printed ChArUco calibration board (charuco_A4.png in the repo root)

Dependencies (no pinned requirements.txt yet — install as needed):

pip install PySide6 opencv-python mediapipe numpy

Running

cd source
python main.py

1. Setup page: select each connected camera by index and add it to the active camera list.
2. Home page: run ChArUco calibration for each camera to compute intrinsics/extrinsics and the projection matrices used for triangulation (at least 2 calibrated cameras are required for 3D triangulation).
3. Click Start Game to begin the tracking loop and stream 3D hand data over UDP to a running instance of Table-UE.

Status

Personal/hobby project, actively evolving — tracking robustness, calibration UX, and gesture coverage are still being improved. Some files under source/Tbd/ (e.g. import cv2.py, setup.py, camera.py) are early experiments kept around during active development.