4E Data Flow System

This system lets you control anything in a Unity scene using your body.

Stand in front of your webcam. Bend your arm. A character's mouth opens. Raise your hand. A light gets brighter. Dance around. Particles explode. The system tracks 33 points on your body (shoulders, elbows, wrists, hips, knees, etc.) and turns those movements into numbers that can drive any property on any Unity object.

It also works with audio (music drives visual effects), LFOs (automatic oscillating patterns), and keyboard/mouse input.

One small open-source library included (websocket-sharp, MIT license). No paid assets. No package manager setup needed.

How it works (simplified)

Think of it like a postal system. Your body movements create "letters" (numbers like "right elbow bend = 90 degrees"). These are stored in named "mailboxes" (the Data Bus). Unity objects check their mailbox each frame and update themselves accordingly.

• Unity 6 (version 6000.x) installed via Unity Hub. If you don't have it, download Unity Hub first, then install Unity 6 from inside the Hub.
• A webcam — built-in laptop camera or USB webcam. Any resolution works.
• Google Chrome or Microsoft Edge — the body tracking runs in the browser. Safari and Firefox may not work reliably.
• This project open in Unity — you should see the Unity editor with "EKO_02" in the title bar.

GameObject

Any object in a Unity scene — a cube, a light, an invisible container. Everything in your scene is a GameObject.

Component

A behaviour or feature attached to a GameObject. A light has a "Light" component. A 3D model has a "Mesh Renderer" component. You add components to make objects do things.

Inspector

The panel on the right side of the Unity editor. When you click on a GameObject, the Inspector shows all its components and their settings.

Hierarchy

The panel on the left showing every object in your scene, listed by name. Click an object here to select it.

Project Window

The panel at the bottom showing your files and folders — scripts, models, textures, etc. Like a file explorer inside Unity.

Play Mode

When you press the ▶ Play button at the top of Unity. Your scene runs in real-time. Press it again to stop.

MediaPipe

A free AI library by Google that can detect body poses, faces, and hands from a webcam video. It runs in your web browser.

WebSocket

A way for two programs to send messages to each other in real-time. Here, the browser sends body data to Unity over a WebSocket connection.

DataBus

The central "post office" of this system. All data flows through named channels on the DataBus. Sources write to it, targets read from it.

Channel

A named value in the DataBus. For example, pose/joint/rightElbow/bend is a channel that holds the current bend angle of your right elbow (0–180 degrees).

Landmark

One of the 33 body points that MediaPipe tracks — nose, shoulders, elbows, wrists, hips, knees, ankles, etc.

Binding

A connection between a DataBus channel and a property on a Unity object. "When this number changes, update that property."

Processor

A component that reads a channel, transforms the value (scale it, smooth it, convert it to true/false), and writes the result to a new channel.

BlendShape

A way to deform a 3D model smoothly — commonly used for facial expressions. A value of 0 means "default shape", 100 means "fully deformed".

ScriptableObject

A special Unity file that stores data as a reusable asset. The Driver Mapping is a ScriptableObject — you create it once and can swap it between scenes.

LFO

Low Frequency Oscillator — a number that automatically goes up and down over time, like a wave. Useful for creating pulsing, breathing, or rhythmic effects without any body input.

The Signal Chain

From the DataBus, data can optionally pass through Processors (to smooth, scale, or threshold values), then finally reaches:

The Key Idea: Everything Is Decoupled

No component knows about any other component. They all just read and write named channels on the DataBus.

This means you can:

• Swap what controls what — without changing any code
• Add audio-reactive effects alongside body tracking
• Mix LFO patterns with pose data
• Create multiple "performance presets" and swap them at runtime

• 1
  In Unity, go to the top menu bar and click Edit → Project Settings. A new window opens.
• 2
  In the left sidebar of the Project Settings window, click Player.
• 3
  Scroll down to the section called Other Settings. You may need to expand it by clicking on the header.
• 4
  Find the row labelled Api Compatibility Level. It's under the "Configuration" sub-heading.
• 5
  Make sure it's set to .NET Standard 2.1. If it says ".NET Standard" or ".NET Framework", click the dropdown and change it to .NET Standard 2.1.
• 6
  Close the Project Settings window. Unity may recompile scripts — wait for the progress bar at the bottom to finish.

A. Create the System Object

This is the "brain" of the system — it receives body data and converts it into channels.

• 1
  In the Hierarchy panel (left side of Unity), right-click on an empty area.
• 2
  From the menu that appears, click Create Empty. A new object called "GameObject" appears in the Hierarchy.
• 3
  The object should already be selected (highlighted blue). If not, click on it. Look at the Inspector panel on the right side.
• 4
  At the very top of the Inspector, you'll see the name "GameObject". Click on that name and type [4E System] then press Enter. The brackets are just for visibility — they make it sort to the top of the list.
• 5
  Now you need to add 6 components. In the Inspector, click the Add Component button (at the bottom of the Inspector).
• 6
  A search box appears. Type DataBus and click on Data Bus when it appears in the list. The component is now added.
• 7
  Click Add Component again. Search for and add each of these, one at a time:
  
  MainThreadDispatcher
DataFlowMonitor
WebSocketServer
LandmarkConverter
PoseChannelSource
ChannelSliderPanel (F11 debug sliders)
• 8
  Connect the reference: On the PoseChannelSource component, you'll see a field called Converter that says "None (Landmark Converter)". You need to tell it where to find the LandmarkConverter.
  
  How to drag a reference: Click and hold on the [4E System] object in the Hierarchy (left panel), then drag it into the Converter field on the PoseChannelSource component (right panel). Unity will automatically find the LandmarkConverter component on that object.

B. Create the Sources Object (Optional)

These add audio, timing, and keyboard data to the system. You don't need them for basic body tracking, but they're useful for more complex effects.

• 1
  Right-click in the Hierarchy → Create Empty. Name it [4E Sources].
• 2
  Add components (same process — click Add Component and search):
  
  TimeSource — gives you automatic oscillating values (LFOs). Ships with 3 pre-configured LFOs: slow (0.25 Hz), medium (1 Hz), fast (4 Hz).
  
  InputSource — tracks keyboard keys (Space, Enter, Shift) and mouse position. You can add more keys in the Inspector.
  
  AudioAnalysisSource — analyses audio for bass, mid, high frequencies and beat detection. Requires an AudioSource (see note below).

C. Create the Driver Object

This is what connects DataBus channels to your scene objects. It reads a "mapping" file that describes which channel drives which property.

• 1
  Right-click in the Hierarchy → Create Empty. Name it [4E Driver].
• 2
  Add Component → search DriverMappingRunner → add it.
• 3
  Leave the Mapping field empty for now — we'll create and assign it in Step 5.

What is MediaPipeSender.html?

It's a web page that runs Google's MediaPipe AI directly in your browser. It accesses your webcam, detects 33 body landmarks in real-time, and sends the data to Unity over a WebSocket connection.

The file is located at: Assets/4E_System/Browser/MediaPipeSender.html

Step-by-step

• 1
  Find the file: Open File Explorer (Windows) or Finder (Mac). Navigate to your Unity project folder → Assets → 4E_System → Browser. You'll see MediaPipeSender.html.
• 2
  Open in Chrome: Double-click MediaPipeSender.html, or right-click → "Open with" → Google Chrome. Do not open it from inside Unity — it needs to run in a real browser.
• 3
  Allow camera: The browser will ask "Allow this page to access your camera?" — click Allow. You should see your webcam feed appear in the main area of the page.
• 4
  Wait for tracking: After a moment, you should see a cyan skeleton overlay drawn on top of your body in the video. The purple dots are individual landmarks. If you don't see a skeleton, make sure your full upper body is visible to the camera.

The Browser UI

The page has these areas:

• 🔴
  Status Badge (top-right corner): Shows the current state.
  ● Disconnected — not connected to Unity
  ● Connected — WebSocket connected, waiting for pose
  ● Tracking — everything working, pose data flowing
  ● Error — something went wrong
• 📷
  Video Area (centre): Your webcam feed with skeleton overlay. The skeleton appears in cyan with purple landmark dots.
• ⚙
  Side Panel (right side):
  CONNECTION — WebSocket URL field and Connect/Disconnect buttons
  SETTINGS — Target FPS (default 30) and Model Complexity (0=fast, 1=balanced, 2=accurate)
  STATS — Frames sent, actual FPS, pose detected, latency
  CAMERA — Start/Stop camera buttons
• 📋
  Console (bottom strip): Log messages showing connection events and errors.

Connect to Unity

• 1
  Start Unity: Go back to the Unity editor. Press the ▶ Play button at the top centre of the editor. The Game view appears and the Console should show: [4E WebSocket] Server started — ws://localhost:8080
• 2
  Connect browser: Go back to Chrome. In the side panel under CONNECTION, the URL should already show ws://localhost:8080. Click the Connect button.
• 3
  Verify: The status badge should turn cyan (Connected) then green (Tracking). The Stats panel should show frames counting up and "Pose detected: YES".
• 4
  Check Unity: In the Unity Console, you should see: [4E WebSocket] Client connected.

• 1
  With Unity still in Play mode and the browser connected, click on the Game view in Unity (so it has focus/is the active window).
• 2
  Press F12 on your keyboard. A dark overlay appears in the top-left corner of the Game view.
• 3
  You should see a list of channels with names like pose/landmark/nose/x, pose/joint/rightElbow/bend, etc. The numbers change in real-time as you move in front of the camera.
• 4
  Each row shows: the channel key (name), current value, a bar graph (visual representation), and the owner (which source wrote it).
• 5
  Try bending your right arm — watch pose/joint/rightElbow/bend change from ~180 (straight arm) to ~50 (bent arm).
• 6
  Press F12 again to close the overlay.

A Driver Mapping is a reusable asset file — like a recipe that says "take this channel and feed it to that object". You can create multiple mappings for different performances or scenes and swap between them.

• 1
  Stop Play mode first — press ▶ again to stop. You should never create assets while in Play mode.
• 2
  In the Project window (bottom panel), navigate to a folder where you want to save the mapping — for example, Assets/ or create a new folder called Mappings.
• 3
  Right-click in the Project window → Create → 4E → Driver Mapping.
• 4
  A new file appears with the name highlighted. Type a name like MyFirstMapping and press Enter.
• 5
  Now connect it to the Driver: In the Hierarchy, click on [4E Driver]. In the Inspector, find the DriverMappingRunner component. Drag your new MyFirstMapping asset from the Project window into the Mapping field.

What's a Binding?

A binding is one self-contained connection: Source channel + input range → output range + target property.

Each binding has a built-in remap. A single mapping can have many bindings — one for each thing you want to control.

When your arm is at 0° (down), the cube is at Y=0. When your arm is at 90° (raised), the cube is at Y=5. The remap converts between the two ranges automatically.

How Do I Find the Right Channel?

There are 100+ channels. Here's how to find the one you need:

• 1
  Press F12 in Play mode to see all live channels. Move your body and watch which values change.
• 2
  Press F11 to open the slider panel. Drag sliders to see what each channel does.
• 3
  Use the filter on DataFlowMonitor — type pose/joint to show only joint angles, or audio/ for audio channels.

Common Tasks → Channels

Example 1: Rotate a cube with your elbow

Let's start with the simplest possible binding.

• 1
  First, create something to control. Right-click in Hierarchy → 3D Object → Cube. A cube appears in the scene.
• 2
  Now select your MyFirstMapping asset in the Project window. The Inspector shows its details, including a Bindings list (currently empty).
• 3
  Click the + button at the bottom of the Bindings list. A new binding entry appears with default values.
• 4
  Set Channel Key to: pose/joint/rightUpperArm/raise
  (Type it exactly — this is the channel name from the DataBus)
• 5
  Set the Value Mapping section:
  Use Remap: checked
  Input Min: 0 · Input Max: 90 ← arm raise range in degrees
  Output Min: 0 · Output Max: 5 ← cube Y position range in metres
  Clamp: checked
• 6
  Set Target Type to Transform.
• 7
  Set Transform Property to LocalPositionY.
• 8
  Drag the Cube from the Hierarchy into the Target Transform field.
• 9
  Press ▶ Play. Connect the browser. Raise your right arm — the cube rises! Lower it — cube drops.

Example 2: Control light intensity with body velocity

Make a light respond to how much you're moving.

• 1
  Create a Point Light: Hierarchy → right-click → Light → Point Light.
• 2
  Select your mapping asset. Click + to add another binding.
• 3
  Set Channel Key to: pose/body/velocity
• 4
  Set Value Mapping:
  Input Min: 0 · Input Max: 1 ← velocity is already 0–1
  Output Min: 0.2 · Output Max: 8 ← dim when still, bright when moving
• 5
  Set Target Type to LightIntensity.
• 6
  Drag the Point Light from the Hierarchy into the Target Light field.
• 7
  Play and move around — the light brightens as you dance, dims when you stand still.

Example 3: Drive a BlendShape with elbow bend

Open a character's mouth (or any morph target) by bending your arm. This requires a 3D model with blend shapes — many free characters on the Asset Store have them.

• 1
  Import a character with blend shapes. Select it in the scene.
• 2
  Find its SkinnedMeshRenderer component in the Inspector. Expand BlendShapes — you'll see a list of shapes with sliders (e.g. "MouthOpen", "Smile", "BrowUp"). Note the index number (0, 1, 2…) of the one you want.
• 3
  Add a new binding to your mapping. Set Channel Key: pose/joint/rightElbow/bend
• 4
  Set Value Mapping:
  Input Min: 60 · Input Max: 180 ← elbow range (60=tight bend, 180=straight)
  Output Min: 0 · Output Max: 100 ← BlendShapes use 0–100
• 5
  Set Target Type to BlendShape.
• 6
  Drag the character's SkinnedMeshRenderer into Target Mesh.
• 7
  Set Blend Shape Index to the number you noted in step 2 (e.g. 0 for the first shape).
• 8
  Play — bend your arm and watch the blend shape deform!

Example 4: Rainbow colour + particles from LFO

Make an object cycle through rainbow colours and emit particles in sync with a slow wave — no body input needed.

• 1
  Select a 3D object in the scene (or create a Sphere).
• 2
  Add binding. Channel Key: time/lfo/slow
• 3
  Value Mapping: Input 0–1, Output 0–1, Clamp checked.
• 4
  Target Type: MaterialColor. Drag the object's Renderer into Target Renderer. Set Material Property to _BaseColor (the URP main colour property).
• 5
  Add another binding. Same channel time/lfo/slow.
• 6
  Target Type: ParticleEmission. Add a Particle System to the object first (Add Component → Particle System). Drag it into Target Particles. Set Max Emission Rate to 50.
• 7
  Play — the object pulses through the rainbow and particles breathe in and out on a 4-second cycle.

All Binding Target Types

Why Processors?

Raw data doesn't always match what you need. Your elbow gives 0–180°, but a BlendShape wants 0–100. A sensor is jittery, but you want smooth motion. A value changes gradually, but you need a clear on/off switch.

Processors read one channel, transform the value, and write the result to a new channel. You chain them together.

How to Add a Processor

• 1
  Select the [4E System] object (or create a new empty called [4E Processors] — either works).
• 2
  Click Add Component → search for the processor type (see below) → add it.
• 3
  Set the Input Key (which channel to read), Output Key (what to name the result), and configure the operation.
• 4
  In your binding, use the Output Key as the Channel Key instead of the raw source.

MathProcessor — Remap, Scale, Clamp

Changes the number range or applies mathematical operations.

Example: Remap elbow to BlendShape range

Now use mapped/elbowBlend as your binding's Channel Key instead of the raw elbow channel.

SmoothProcessor — Remove Jitter

Smooths out rapid changes so values feel natural instead of twitchy.

Factor controls how much smoothing: 0.1 = very responsive (almost no smoothing), 0.9 = very smooth (big lag).

ThresholdProcessor — Convert to On/Off

Converts a continuous value into a true/false trigger. Has hysteresis to prevent flickering at the boundary.

Hysteresis adds a deadband around the threshold. If threshold=90 and hysteresis=10, it turns ON at 85 and OFF at 95 — preventing rapid on/off switching when hovering near 90.

Writes both outputKey (bool) and outputKey/float (0 or 1) plus outputKey/rising and outputKey/falling for edge detection.

CombineProcessor — Mix Two Channels

Takes two input channels and combines them into one output.

MediaPipe detects 33 named points on your body every frame. Each point has an X, Y, Z position and a visibility score (0–1, how confident the AI is that it can see that point).

Pose Channels

Joint names available: leftElbow, rightElbow, leftWrist, rightWrist, leftKnee, rightKnee, leftHip, rightHip, headTilt, leftUpperArm, rightUpperArm, leftForeArm, rightForeArm, leftThigh, rightThigh, shoulderWidth, hipWidth

Audio Channels

Only available if you added AudioAnalysisSource with an AudioSource playing music.

Time Channels

Only available if you added TimeSource. Ships with 3 pre-made LFOs: slow, medium, fast.

You can add custom LFOs in the Inspector: change the shape (Sine, Triangle, Saw, Square, Noise), frequency, and phase.

Input Channels

Only available if you added InputSource. Tracks keyboard and mouse by default.

Default tracked keys: space, enter, shift. Add more in the Inspector by expanding the Keys list and clicking +. Uses Unity's new Input System package.

AudioAnalysisSource

TimeSource

InputSource

Recipe 1: BlendShape driven by elbow bend + audio beat

A character's mouth opens when you bend your arm, with extra punch on bass beats.

Step A — Add a MathProcessor

• 1
  Select [4E System] → Add Component → MathProcessor
• 2
  Set Input Key: pose/joint/rightElbow/bend
• 3
  Set Output Key: mapped/elbowBlend
• 4
  Set Operation: Remap, In Min: 60, In Max: 180, Out Min: 0, Out Max: 100

Step B — Add a CombineProcessor

• 1
  Add Component → CombineProcessor
• 2
  Input Key A: mapped/elbowBlend
• 3
  Input Key B: audio/beat/float
• 4
  Output Key: final/elbowPunch
• 5
  Mode: Blend, Blend T: 0.8

Step C — Add Binding

• 1
  Select your mapping asset. Click + to add a binding.
• 2
  Channel Key: final/elbowPunch
• 3
  Target Type: BlendShape
• 4
  Target Mesh: drag your character's SkinnedMeshRenderer
• 5
  Blend Shape Index: 0 (or whichever blend shape you want to control)

Recipe 2: Object scale pulsing with body velocity × LFO

An object grows and shrinks rhythmically, stronger when you move faster.

Step A — CombineProcessor

• 1
  Add Component → CombineProcessor
• 2
  Input Key A: pose/body/velocity · Input Key B: time/lfo/medium
• 3
  Output Key: combined/velocityLFO · Mode: Multiply

Step B — MathProcessor (remap to scale range)

• 1
  Add Component → MathProcessor
• 2
  Input Key: combined/velocityLFO · Output Key: final/scale
• 3
  Operation: Remap · In Min: 0 · In Max: 1 · Out Min: 0.8 · Out Max: 2.5

Step C — Three bindings (one per axis)

• 1
  Add 3 bindings to your mapping, all with Channel Key: final/scale
• 2
  All set to Target Type: Transform. Drag the same object into Target Transform for all three.
• 3
  Set Transform Property to: LocalScaleX, LocalScaleY, LocalScaleZ (one each)

Recipe 3: Animator trigger when arm is raised

Play a "Wave" animation when you raise your right arm above your shoulder.

Step A — ThresholdProcessor

• 1
  Add Component → ThresholdProcessor
• 2
  Input Key: pose/joint/rightUpperArm/raise
• 3
  Output Key: trigger/armRaised
• 4
  Mode: LessThan · Threshold: 50 · Hysteresis: 15

Step B — Binding

• 1
  Channel Key: trigger/armRaised/float
• 2
  Target Type: AnimatorTrigger
• 3
  Drag your character's Animator into Target Animator
• 4
  Animator Param: Wave (must match the trigger name in your Animator Controller)
• 5
  Bool Threshold: 0.5

• F12
  DataFlow Monitor — your first debugging tool. Press F12 in Play mode to see all channel values live. If channels are there, the data pipeline is working. If they're not, the problem is upstream (WebSocket, browser, etc.).
• ?
  "No channels appearing in F12 overlay"
  1. Is Unity in Play mode? The WebSocket server only runs during Play.
  2. Check Console for [4E WebSocket] Server started. If you don't see it, make sure WebSocketServer has Auto Start checked.
  3. Is the browser connected? Status badge should be green "Tracking".
  4. Is the PoseChannelSource's Converter field filled in? It must reference the LandmarkConverter.
• ?
  "WebSocket connection refused"
  1. Unity must be in Play mode before you click Connect in the browser.
  2. Check Windows Firewall isn't blocking port 8080. Open Command Prompt and run netstat -an | findstr 8080 — you should see a LISTENING entry.
  3. Try a different port: change the Port field on WebSocketServer to 9090, and update the browser URL to ws://localhost:9090/mediapipe.
• ?
  "Compile errors — namespace not found"
  1. Check API Compatibility Level is .NET Standard 2.1 (Step 1).
  2. Make sure all scripts are inside Assets/4E_System/ — don't move them outside Assets.
  3. If referencing 4E classes from your own scripts, add using FourE.DataFlow; at the top.
• ?
  "Binding not working — object doesn't move"
  1. Press F12 and check the channel key exists and has values. The key in your binding must exactly match the key shown in the monitor (case doesn't matter).
  2. Check the binding's Active checkbox is ticked.
  3. For Transform bindings: make sure you dragged the correct object into Target Transform.
  4. For BlendShape: make sure the index is valid (try 0 first).
  5. Right-click the DataBus component → Log All Channels to dump all keys to the Console.
• ?
  "Values are wrong range"
  The most common issue. Raw elbow angles are 0–180° but BlendShapes need 0–100. Use a MathProcessor with Remap operation to convert ranges. See Step 7.
• ?
  "Movement is jittery"
  Add a SmoothProcessor between the source channel and your binding. Start with Lerp mode, factor 0.3. Increase factor for more smoothing.
• ?
  "Camera not working in browser"
  1. Make sure you allowed camera permission when the browser asked.
  2. Try a different browser (Chrome recommended).
  3. Check no other app is using the camera (Zoom, Teams, etc.).
  4. On some laptops, there's a physical camera privacy switch — make sure it's open.
• ?
  "Skeleton not appearing on my body"
  1. Make sure your full upper body is visible to the camera (head, shoulders, arms).
  2. Good lighting helps — avoid backlighting (window behind you).
  3. Try increasing Model Complexity to 2 in the browser settings panel.
  4. Move back from the camera so more of your body is visible.
• ✓
  Test without a webcam
  Create a simple test script on any GameObject:
  

  using FourE.DataFlow; using UnityEngine; public class TestWriter : MonoBehaviour { void Update() { // Simulate an elbow bend that oscillates 60°–180° float v = Mathf.Lerp(60f, 180f, Mathf.Sin(Time.time) * 0.5f + 0.5f); DataBus.Write("pose/joint/rightElbow/bend", v); } }

  This writes a simulated elbow value so you can test bindings without any webcam or browser setup.

These all work with just a Cube and the [4E System] running. Add one binding per experiment.

Start with experiment 1 or 2 — they give the most obvious visual feedback. Once those work, try combining multiple bindings on the same object for richer effects.