DuOtto
69 lines
4.6 KiB
Markdown
69 lines
4.6 KiB
Markdown
# Table-UE
|
|
|
|
> **Work in progress.** This is an actively developed personal project, not a finished product — expect incomplete features, rough edges, and breaking changes.
|
|
|
|
Unreal Engine side of a real-time, gesture-controlled smart tabletop for tabletop wargaming (Warhammer 40k). A downward-facing multi-camera rig tracks players' hands over the physical table; this project receives the 3D hand data over UDP, recognizes gestures, and drives an interactive projected overlay (rulers, markers, HUDs, etc.) directly on the tabletop surface.
|
|
|
|
The companion computer-vision/UI application that captures and streams the hand-tracking data lives in [Table-Python](https://git.bitrux.de/Ottos_Projects/Table-Python).
|
|
|
|
## Overview
|
|
|
|
- Built with **Unreal Engine 5.8** (C++ gameplay module + Blueprints).
|
|
- Receives per-frame 3D hand landmark data (21 points/hand, MediaPipe topology) via a **raw UDP socket**, decoded into `FHandPacketHand` structs.
|
|
- Spawns and updates `AHandActor` instances per tracked hand and exposes them to Blueprints for visualization/debugging.
|
|
- A **JSON-driven gesture binding system** matches live hand poses against configurable gesture definitions (e.g. "point with index finger") and triggers corresponding **Gesture Actions**.
|
|
- Gesture Actions are self-contained, extendable Actors that own their own on-table visuals ("Table Icons") — e.g. a `Ruler` gesture action that draws a measuring line with tick marks between two points and displays the distance in real-world units.
|
|
- A calibration scale factor (`UEUnitsPerUnit`) converts between real-world measurements (inches/cm on the physical table) and Unreal units, set up per session.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Table-Python (multi-camera capture, MediaPipe hand tracking, stereo triangulation)
|
|
│ UDP (custom binary hand packet protocol)
|
|
▼
|
|
ADetectionManager — UDP receiver, decodes packets into hand data
|
|
│
|
|
▼
|
|
AHandManager — spawns/despawns AHandActor per tracked hand id
|
|
│
|
|
▼
|
|
UGestureManagerComponent — loads gesture definitions from JSON, evaluates hand poses each tick
|
|
│
|
|
▼
|
|
AGestureAction (+ subclasses, e.g. AGestureAction_Ruler)
|
|
│
|
|
▼
|
|
ATableIconActor (+ subclasses) — the actual on-table visuals (lines, labels, procedural meshes, UMG widgets)
|
|
```
|
|
|
|
### Key classes
|
|
|
|
| Class | Responsibility |
|
|
|---|---|
|
|
| [`ADetectionManager`](Source/Table/Public/DetectionManager.h) | Owns the UDP socket, parses incoming hand-tracking packets |
|
|
| [`AHandManager`](Source/Table/Public/HandManager.h) | Tracks the set of currently-active hands, broadcasts added/removed delegates |
|
|
| [`AHandActor`](Source/Table/Public/HandActor.h) / [`FingerTip`](Source/Table/Public/FingerTip.h) | Represents a single tracked hand and its fingertip positions in world space |
|
|
| [`UGestureManagerComponent`](Source/Table/Public/GestureManagerComponent.h) | Loads a JSON gesture-binding config and evaluates gestures against tracked hands |
|
|
| [`UGestureDefinition`](Source/Table/Public/GestureDefinition.h) | Scores how well a hand pose matches a named gesture (overridden per gesture) |
|
|
| [`AGestureAction`](Source/Table/Public/GestureAction.h) / [`AGestureAction_Ruler`](Source/Table/Public/MyGestureActions.h) | Lifecycle (init/update/end) for an active gesture, e.g. a ruler measurement |
|
|
| [`ATableIconActor`](Source/Table/Public/TableIconActor.h) / [`ATableIconLineActor`](Source/Table/Public/MyTableIconActors.h) | Visual elements drawn on the table for the duration of a gesture |
|
|
|
|
Gesture-to-action bindings are configured in JSON (see `Content/WorldObjects/GestureActions/`), so new gestures can be wired up without touching C++.
|
|
|
|
## Requirements
|
|
|
|
- Unreal Engine **5.8**
|
|
- Visual Studio 2022 (Windows) with the "Game development with C++" workload
|
|
- A running instance of [Table-Python](https://git.bitrux.de/Ottos_Projects/Table-Python) sending hand-tracking data over UDP (default `127.0.0.1:9000`) if you want live hand input; the level can otherwise be explored/tested without a live feed.
|
|
|
|
## Getting started
|
|
|
|
1. Clone the repo and generate project files by right-clicking `Table.uproject` → **Generate Visual Studio project files** (or open it directly in the Unreal Editor, which will prompt to build missing modules).
|
|
2. Open `Table.sln` in Visual Studio and build, or open `Table.uproject` in Unreal Editor.
|
|
3. Press **Play** to start the `ADetectionManager` UDP listener.
|
|
4. Start the Python tracking application to begin streaming live hand data to the table.
|
|
|
|
## Status
|
|
|
|
Personal/hobby project, actively evolving — gesture set, table icons, and HUD content are still being expanded.
|