How to process live stream from "3D Simulation Camera" UAV toolbox?

Dear Community,

I'm working on "Unreal Engine Simulation" on UAV.

As I need to perform image processing operations, How do i input live stream data into code?

for Example, we read a ".avi" video file with "vision.VideoFileReader('xxxx.avi')" and use "vision.DeployableVideoPlayer" to further carry out BlobAnalysis/Thresholding/Kalman filter for detection/motion estimation operations.

Similarily, How to input live stream from "3D Simulation Camera".

Your help would be very much appreciated.

Thank you,

Maheedhar

Answer 1

[Remo Pillat]

Hi Maheedhar,

It's not entirely clear to me if you are working in MATLAB or Simulink, so let me answer your question for both:

Simulink

The "Image" output of the "Simulation 3D Camera" block returns the image from the Unreal simulation as m-by-n-by-3 matrix. This "Image" signal can be used by all image processing blocks, e.g. from Computer Vision Toolbox, so it should be straightforward for you to further process the image. For example, if you want to convert the color image to a grayscale image, you can use the Color Space Conversion block.

MATLAB

If you have image processing code in MATLAB, you can call it from your Simulink model containing the "Simulation 3D Camera" block. To call MATLAB code from Simulink, it's usually easiest to use the MATLAB Function block. Here is a good tutorial that walks you from the basic steps: Create Custom Functionality using MATLAB Function block. You can pass the "Image" signal from the camera block directly to the MATLAB Function block, process the image in MATLAB and then pass the result back into your Simulink model.

Let me know if this resolves your question or not.

Thanks,

Remo

[Maheedhar Korimi]

Hi Remo,

Thank you for your reply! Glad that I'm getting such quick response.

I apologise for not explaining my question properly.

Objective

My work is regarding vision guidance, where I'm trying to follow another UAV in its trajectory. So, this involves thresholding (for object detection, identification based on colour), blob analysis (To get centroid of the detected object) and further taking the centroid value to maneuver my UAV.

I'm using simulink for this entire model. But, I understand that "Autothreshold" block doesn't take colour parameters to detect a specific colour range, like it is done in this example to detect the green ball(https://www.mathworks.com/videos/object-detection-using-blob-analysis-108372.html; (time 5:47)).

Issue

So, I need to add a function block to perform color thresholding for a specific range of color values.

As I have to process live data from "3D Simulation Camera" i.e., to visualize and record data to perform thresholding and also to use Kalman filter to estimate motion.

Request

I just need to read the live data, and I'm not sure on how to do it!

Hope I'm clear with my question now. Please correct If I'm anywhere wrong about the tools.

Will be awaited for your reply.

Thank you

Maheedhar