From c9e46eeb41c6e4ecc4755cb2e29f9fb13c6d6de8 Mon Sep 17 00:00:00 2001 From: DuOtto Date: Mon, 27 Jul 2026 09:32:25 +0200 Subject: [PATCH] Added README --- Table/README.md | 68 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 Table/README.md diff --git a/Table/README.md b/Table/README.md new file mode 100644 index 0000000..76d49ed --- /dev/null +++ b/Table/README.md @@ -0,0 +1,68 @@ +# Table-UE + +> **Work in progress.** This is an actively developed personal project, not a finished product — expect incomplete features, rough edges, and breaking changes. + +Unreal Engine side of a real-time, gesture-controlled smart tabletop for tabletop wargaming (Warhammer 40k). A downward-facing multi-camera rig tracks players' hands over the physical table; this project receives the 3D hand data over UDP, recognizes gestures, and drives an interactive projected overlay (rulers, markers, HUDs, etc.) directly on the tabletop surface. + +The companion computer-vision/UI application that captures and streams the hand-tracking data lives in [Table-Python](https://git.bitrux.de/Ottos_Projects/Table-Python). + +## Overview + +- Built with **Unreal Engine 5.8** (C++ gameplay module + Blueprints). +- Receives per-frame 3D hand landmark data (21 points/hand, MediaPipe topology) via a **raw UDP socket**, decoded into `FHandPacketHand` structs. +- Spawns and updates `AHandActor` instances per tracked hand and exposes them to Blueprints for visualization/debugging. +- A **JSON-driven gesture binding system** matches live hand poses against configurable gesture definitions (e.g. "point with index finger") and triggers corresponding **Gesture Actions**. +- Gesture Actions are self-contained, extendable Actors that own their own on-table visuals ("Table Icons") — e.g. a `Ruler` gesture action that draws a measuring line with tick marks between two points and displays the distance in real-world units. +- A calibration scale factor (`UEUnitsPerUnit`) converts between real-world measurements (inches/cm on the physical table) and Unreal units, set up per session. + +## Architecture + +``` +Table-Python (multi-camera capture, MediaPipe hand tracking, stereo triangulation) + │ UDP (custom binary hand packet protocol) + ▼ +ADetectionManager — UDP receiver, decodes packets into hand data + │ + ▼ +AHandManager — spawns/despawns AHandActor per tracked hand id + │ + ▼ +UGestureManagerComponent — loads gesture definitions from JSON, evaluates hand poses each tick + │ + ▼ +AGestureAction (+ subclasses, e.g. AGestureAction_Ruler) + │ + ▼ +ATableIconActor (+ subclasses) — the actual on-table visuals (lines, labels, procedural meshes, UMG widgets) +``` + +### Key classes + +| Class | Responsibility | +|---|---| +| [`ADetectionManager`](Source/Table/Public/DetectionManager.h) | Owns the UDP socket, parses incoming hand-tracking packets | +| [`AHandManager`](Source/Table/Public/HandManager.h) | Tracks the set of currently-active hands, broadcasts added/removed delegates | +| [`AHandActor`](Source/Table/Public/HandActor.h) / [`FingerTip`](Source/Table/Public/FingerTip.h) | Represents a single tracked hand and its fingertip positions in world space | +| [`UGestureManagerComponent`](Source/Table/Public/GestureManagerComponent.h) | Loads a JSON gesture-binding config and evaluates gestures against tracked hands | +| [`UGestureDefinition`](Source/Table/Public/GestureDefinition.h) | Scores how well a hand pose matches a named gesture (overridden per gesture) | +| [`AGestureAction`](Source/Table/Public/GestureAction.h) / [`AGestureAction_Ruler`](Source/Table/Public/MyGestureActions.h) | Lifecycle (init/update/end) for an active gesture, e.g. a ruler measurement | +| [`ATableIconActor`](Source/Table/Public/TableIconActor.h) / [`ATableIconLineActor`](Source/Table/Public/MyTableIconActors.h) | Visual elements drawn on the table for the duration of a gesture | + +Gesture-to-action bindings are configured in JSON (see `Content/WorldObjects/GestureActions/`), so new gestures can be wired up without touching C++. + +## Requirements + +- Unreal Engine **5.8** +- Visual Studio 2022 (Windows) with the "Game development with C++" workload +- A running instance of [Table-Python](https://git.bitrux.de/Ottos_Projects/Table-Python) sending hand-tracking data over UDP (default `127.0.0.1:9000`) if you want live hand input; the level can otherwise be explored/tested without a live feed. + +## Getting started + +1. Clone the repo and generate project files by right-clicking `Table.uproject` → **Generate Visual Studio project files** (or open it directly in the Unreal Editor, which will prompt to build missing modules). +2. Open `Table.sln` in Visual Studio and build, or open `Table.uproject` in Unreal Editor. +3. Press **Play** to start the `ADetectionManager` UDP listener. +4. Start the Python tracking application to begin streaming live hand data to the table. + +## Status + +Personal/hobby project, actively evolving — gesture set, table icons, and HUD content are still being expanded.