Help regarding image recognition inside a live video game

Hello,

I want to do some image recognition and object detection for a video game in which objects move constantly. The first step to do this is to provide MATLAB with the input images by taking a screen shot and saving it inside a folder. This however may be a slow approach for a fast paced game.

What I really want to do is get MATLAB to continuously do object recognition on a video game that will be running maximized (The end goal is to use machine learning to study these images). I came across an article https://uk.mathworks.com/help/vision/examples/face-detection-and-tracking-using-live-video-acquisition.html where they do what I want but instead using a webcam video input. Is it possible to do the same thing but with the computer display itself?

Please also note that I do not want to record a video, save it and then do recognition on that as the game is real time so object detection should be real time as well.

Thank you.

Answer 1

[Image Analyst]

Essentially you put getframe() in a loop and then analyze the image it returns.

[Navtej Dhot]

Ok i will look into the getframe() function. In the meantime more workarounds and solutions are welcome from others. Thank you.

[Image Analyst]

If it's your game, you should know what's on the screen at any given time. If it's not, and you're using a video camera for example to look at a different screen, then you can use getsnapshot() to get images from your video camera and then do your analysis on them.

[Navtej Dhot]

It is not my game. The idea is that the input data should be given to the machine in the same way as a human receives it which is just looking at the pixels on the screen. I will try to implement the getframe() and getsnapshot() functions in a for loop and see if the system responds fast enough in analysis of each frame and the action it takes inside the game for this to be feasible. I have seen people do face tracking on moving people in real time. If they are using the getframe() function to do this from their webcam then i should have no problems either.

[Image Analyst]

• If you're using a video camera to view anohter screen, you can use getsnapshot().
• If you're using MATLAB and playing the game from MATLAB source code and have access to the figure window, then you can use getframe (sounds like this is not the case).
• If it's just some other completely different app running on your computer then I don't know what you'd do. Somehow you'll have to get screenshots of your computer screen.