Modern Spatial Development with Leap Motion Hyperion

The New Era of Hand Tracking

The Leap Motion controller has long been a staple for experimental interaction design. However, with the release of Ultraleap Hyperion (v6.2), the development landscape has shifted. For developers used to the “plug and play” WebSocket nature of older V4 drivers, the new Gemini and Hyperion engines present a unique challenge: the native browser WebSocket server is gone.

In this post, I’ll walk through how I bypassed these limitations to build a suite of modern spatial tools, including a 3D Air Canvas and a Native MIDI Bridge.

The Problem: The Missing WebSocket

Modern Ultraleap drivers focus on the LeapC API—a high-performance C-style interface. While great for VR headsets and native apps, it leaves web developers in the dark. To bring hand tracking back to the browser, I had to implement a dedicated data bridge.

The Solution: A C-Based WebSocket Bridge

Using the official Ultraleap Tracking WebSocket project, I built a lightweight C application that:

1. Connects to the Hyperion service via LeapC.
2. Serializes hand coordinates, pinch strength, and finger states into JSON.
3. Broadcasts that data over a standard WebSocket on port 6437.

This bridge restores compatibility with modern frontend frameworks like Three.js and Vite.

Building the 3D Air Canvas

With data flowing, the next step was creation. Using Three.js, I built a 3D painting application where the user’s hand becomes the brush.

• Pinch-to-Draw: By monitoring pinchStrength, the app initializes a new BufferGeometry the moment the thumb and index finger meet.
• Fist-to-Clear: To keep the interaction natural, I mapped a high grabStrength (a clenched fist) to a canvas reset command.

Beyond the Browser: The MIDI Bridge

Spatial control shouldn’t be limited to the web. I also developed a Leap-to-MIDI Bridge in Python. By mapping the Y-axis to Pitch (CC 10) and X-axis to Panning (CC 11), the Leap Motion transforms into a professional-grade musical instrument.

This setup allows for controlling high-end DAWs like FL Studio or Ableton Live with zero physical contact—true “Air Music.”

Conclusion: The Future is Spatial

Developing for Leap Motion in 2026 requires a bit more infrastructure than it used to, but the rewards are worth it. The precision of the Hyperion tracking engine combined with modern web tools like Vite and Tone.js opens up possibilities for interaction that were previously impossible.

You can check out the source code for these prototypes on my GitHub:

• AirTouchCanvas
• LeapMidiBridge
• AirSynth

Happy hacking in 3D space!