Work in progress. This is an actively developed personal project, not a finished product — expect incomplete features, rough edges, and breaking changes.

Unreal Engine side of a real-time, gesture-controlled smart tabletop for tabletop wargaming (Warhammer 40k). A downward-facing multi-camera rig tracks players' hands over the physical table; this project receives the 3D hand data over UDP, recognizes gestures, and drives an interactive projected overlay (rulers, markers, HUDs, etc.) directly on the tabletop surface.

The companion computer-vision/UI application that captures and streams the hand-tracking data lives in Table-Python.

Overview

Built with Unreal Engine 5.8 (C++ gameplay module + Blueprints).
Receives per-frame 3D hand landmark data (21 points/hand, MediaPipe topology) via a raw UDP socket, decoded into FHandPacketHand structs.
Spawns and updates AHandActor instances per tracked hand and exposes them to Blueprints for visualization/debugging.
A JSON-driven gesture binding system matches live hand poses against configurable gesture definitions (e.g. "point with index finger") and triggers corresponding Gesture Actions.
Gesture Actions are self-contained, extendable Actors that own their own on-table visuals ("Table Icons") — e.g. a Ruler gesture action that draws a measuring line with tick marks between two points and displays the distance in real-world units.
A calibration scale factor (UEUnitsPerUnit) converts between real-world measurements (inches/cm on the physical table) and Unreal units, set up per session.

Architecture

Table-Python (multi-camera capture, MediaPipe hand tracking, stereo triangulation)
        │  UDP  (custom binary hand packet protocol)
        ▼
ADetectionManager      — UDP receiver, decodes packets into hand data
        │
        ▼
AHandManager            — spawns/despawns AHandActor per tracked hand id
        │
        ▼
UGestureManagerComponent — loads gesture definitions from JSON, evaluates hand poses each tick
        │
        ▼
AGestureAction (+ subclasses, e.g. AGestureAction_Ruler)
        │
        ▼
ATableIconActor (+ subclasses)  — the actual on-table visuals (lines, labels, procedural meshes, UMG widgets)

Key classes

Class	Responsibility
`ADetectionManager`	Owns the UDP socket, parses incoming hand-tracking packets
`AHandManager`	Tracks the set of currently-active hands, broadcasts added/removed delegates
`AHandActor` / `FingerTip`	Represents a single tracked hand and its fingertip positions in world space
`UGestureManagerComponent`	Loads a JSON gesture-binding config and evaluates gestures against tracked hands
`UGestureDefinition`	Scores how well a hand pose matches a named gesture (overridden per gesture)
`AGestureAction` / `AGestureAction_Ruler`	Lifecycle (init/update/end) for an active gesture, e.g. a ruler measurement
`ATableIconActor` / `ATableIconLineActor`	Visual elements drawn on the table for the duration of a gesture

Gesture-to-action bindings are configured in JSON (see Content/WorldObjects/GestureActions/), so new gestures can be wired up without touching C++.

Requirements

Unreal Engine 5.8
Visual Studio 2022 (Windows) with the "Game development with C++" workload
A running instance of Table-Python sending hand-tracking data over UDP (default 127.0.0.1:9000) if you want live hand input; the level can otherwise be explored/tested without a live feed.

Getting started

Clone the repo and generate project files by right-clicking Table.uproject → Generate Visual Studio project files (or open it directly in the Unreal Editor, which will prompt to build missing modules).
Open Table.sln in Visual Studio and build, or open Table.uproject in Unreal Editor.
Press Play to start the ADetectionManager UDP listener.
Start the Python tracking application to begin streaming live hand data to the table.

Status

Personal/hobby project, actively evolving — gesture set, table icons, and HUD content are still being expanded.