Using VR headset as an input to Simulink

Hello,

I am looking for methods to integrate a VR headset as a coordinate input device in Simulink. My goal is to allow the user to look in a chosen direction, with the gaze angles (relative to a calibrated origin, where both Θ and Φ are 0°) represented in 3D polar coordinates. The distance is not a factor in this setup, and the headset includes its own built-in coordinate calibration.

I have already developed a Simulink system that processes external inputs and computes real-time responses (please see the attached image). I would like to incorporate the VR headset as an input to this system, replacing the pitch and yaw axis of a joystick.

Any suggestions on how to achieve this or references to relevant documentation would be greatly appreciated.

Answer 1

[colordepth]

Hi Oscar,

To integrate a VR headset as a coordinate input device in your Simulink model, you can utilize the capabilities of Simulink 3D Animation with “Unreal Engine”.

In “Unreal Engine”, you can create VR actors, which are interactive entities within your simulation. To learn more, I suggest referring to additional “Unreal Engine” and VR development tutorials of your liking available on the internet.

Here are some resources to help you get started with integrating Simulink with “Unreal Engine”:

1. Using “Unreal Engine” with Simulink- https://www.mathworks.com/videos/series/using-unreal-engine-with-simulink.html
2. “Unreal Engine” Simulation Environment Requirements and Limitations - https://www.mathworks.com/help/sl3d/unreal-engine-simulation-environment-requirements-and-limitations.html

I hope this helps!

Answer 2

[Velma]

Cool project! You can stream VR headset data into Simulink using tools like Unity or Python with UDP or ROS. If your headset supports OpenVR/OpenXR, extracting gaze angles should be doable. Also, using wireless earbuds for gaming really helps during long testing sessions keeps things immersive and tangle-free. Let us know what headset you’re using!

[Oscar Su]

Thank You!

I ended up going with the Unreal Engine set up and to answer your question, I'm using Oculus Rift S

There was another problem I ran into, as detailed in this post: Editing an Aircraft Actor in Simulink-Unreal Simulation - MATLAB Answers - MATLAB Central

But I'm also pass that point now, and I've wrapped up my project.