# Table-UE

> **Work in progress.** This is an actively developed personal project, not a finished product — expect incomplete features, rough edges, and breaking changes.

Unreal Engine side of a real-time, gesture-controlled smart tabletop for tabletop wargaming (Warhammer 40k). A downward-facing multi-camera rig tracks players' hands over the physical table; this project receives the 3D hand data over UDP, recognizes gestures, and drives an interactive projected overlay (rulers, markers, HUDs, etc.) directly on the tabletop surface.

The companion computer-vision/UI application that captures and streams the hand-tracking data lives in [Table-Python](https://git.bitrux.de/Ottos_Projects/Table-Python).

## Overview

- Built with **Unreal Engine 5.8** (C++ gameplay module + Blueprints).
- Receives per-frame 3D hand landmark data (21 points/hand, MediaPipe topology) via a **raw UDP socket**, decoded into `FHandPacketHand` structs.
- Spawns and updates `AHandActor` instances per tracked hand and exposes them to Blueprints for visualization/debugging.
- A **JSON-driven gesture binding system** matches live hand poses against configurable gesture definitions (e.g. "point with index finger") and triggers corresponding **Gesture Actions**.
- Gesture Actions are self-contained, extendable Actors that own their own on-table visuals ("Table Icons") — e.g. a `Ruler` gesture action that draws a measuring line with tick marks between two points and displays the distance in real-world units.
- A calibration scale factor (`UEUnitsPerUnit`) converts between real-world measurements (inches/cm on the physical table) and Unreal units, set up per session.

## Architecture

```
Table-Python (multi-camera capture, MediaPipe hand tracking, stereo triangulation)
        │  UDP  (custom binary hand packet protocol)
        ▼
ADetectionManager      — UDP receiver, decodes packets into hand data
        │
        ▼
AHandManager            — spawns/despawns AHandActor per tracked hand id
        │
        ▼
UGestureManagerComponent — loads gesture definitions from JSON, evaluates hand poses each tick
        │
        ▼
AGestureAction (+ subclasses, e.g. AGestureAction_Ruler)
        │
        ▼
ATableIconActor (+ subclasses)  — the actual on-table visuals (lines, labels, procedural meshes, UMG widgets)
```

### Key classes

| Class | Responsibility |
|---|---|
| [`ADetectionManager`](Source/Table/Public/DetectionManager.h) | Owns the UDP socket, parses incoming hand-tracking packets |
| [`AHandManager`](Source/Table/Public/HandManager.h) | Tracks the set of currently-active hands, broadcasts added/removed delegates |
| [`AHandActor`](Source/Table/Public/HandActor.h) / [`FingerTip`](Source/Table/Public/FingerTip.h) | Represents a single tracked hand and its fingertip positions in world space |
| [`UGestureManagerComponent`](Source/Table/Public/GestureManagerComponent.h) | Loads a JSON gesture-binding config and evaluates gestures against tracked hands |
| [`UGestureDefinition`](Source/Table/Public/GestureDefinition.h) | Scores how well a hand pose matches a named gesture (overridden per gesture) |
| [`AGestureAction`](Source/Table/Public/GestureAction.h) / [`AGestureAction_Ruler`](Source/Table/Public/MyGestureActions.h) | Lifecycle (init/update/end) for an active gesture, e.g. a ruler measurement |
| [`ATableIconActor`](Source/Table/Public/TableIconActor.h) / [`ATableIconLineActor`](Source/Table/Public/MyTableIconActors.h) | Visual elements drawn on the table for the duration of a gesture |

Gesture-to-action bindings are configured in JSON (see `Content/WorldObjects/GestureActions/`), so new gestures can be wired up without touching C++.

## Requirements

- Unreal Engine **5.8**
- Visual Studio 2022 (Windows) with the "Game development with C++" workload
- A running instance of [Table-Python](https://git.bitrux.de/Ottos_Projects/Table-Python) sending hand-tracking data over UDP (default `127.0.0.1:9000`) if you want live hand input; the level can otherwise be explored/tested without a live feed.

## Getting started

1. Clone the repo and generate project files by right-clicking `Table.uproject` → **Generate Visual Studio project files** (or open it directly in the Unreal Editor, which will prompt to build missing modules).
2. Open `Table.sln` in Visual Studio and build, or open `Table.uproject` in Unreal Editor.
3. Press **Play** to start the `ADetectionManager` UDP listener.
4. Start the Python tracking application to begin streaming live hand data to the table.

## Status

Personal/hobby project, actively evolving — gesture set, table icons, and HUD content are still being expanded.