Identify Objects Within Live Video Using ResNet-50 on Raspberry Pi Hardware

Required Hardware

Prerequisite

Configure the Raspberry Pi network, using the Hardware Setup screen. During this process, ensure that you download the MathWorks® Raspbian image for deep learning. Instead of the MathWorks Raspbian image, if you choose to customize an existing image on your hardware, ensure that you select the option to install the ARM Compute Library.

Step 1: Connect the Raspberry Pi Hardware

Tip: Before you start this example, we recommend you complete the Getting Started with MATLAB Support Package for Raspberry Pi Hardware example.

Connect the micro end of the USB cable to the Raspberry Pi and the regular end of the USB cable to the computer. Wait until the PWR LED on the hardware starts blinking.

To connect a video device to the hardware, do any one of the following:

In the MATLAB Command Window, create a connection to the Raspberry Pi hardware.

r = raspi;

Step 2: Open the Object Classification Algorithm

This example uses the ResNet-50 network to show image classification with the ARM Compute Library and returns the prediction score in the output.

For more information on the function, enter this command in the MATLAB Command Window.

type raspi_webcam_resnet

function raspi_webcam_resnet()
%#codegen
% Copyright 2020 The MathWorks, Inc.

%Create raspi & webcam obj
raspiObj = raspi();
cam      = webcam(raspiObj,1);

%Initialize DNN and the input size
net        = coder.loadDeepLearningNetwork('resnet50');
inputSize  = [224, 224,3]; %net.Layers(1).InputSize;

%Initialize text to display
textToDisplay = '......';

% Main loop
start = tic;
fprintf('Entering into while loop.\n');
while true
    %Capture image from webcam
    img = snapshot(cam);
    
    elapsedTime = toc(start);
    %Process frames at 1 per second
    if elapsedTime > 1
        %Resize the image
        imgSizeAdjusted = imresize(img,inputSize(1:2));
        
        %Classify the input image
        [label,score] = net.classify(imgSizeAdjusted);
        maxScore = max(score);
        
        labelStr = cellstr(label);
        textToDisplay = sprintf('Label : %s \nScore : %f',labelStr{:},maxScore);
        start = tic;
    end
    
    %Display the predicted label
    img_label = insertText(img,[0,0],textToDisplay);
    displayImage(raspiObj,img_label);
end
end

You can modify the function by using the edit command.

edit raspi_webcam_resnet

Step 3: Create a Hardware Configuration Object

Create a hardware configuration object by using the targetHardware function in the MATLAB Command Window.

board = targetHardware('Raspberry Pi');

Verify the DeviceAddress, Username, and Password properties listed in the output. If required, change the value of the properties by using the dot notation syntax. For example, to change the device address to 172.18.185.123, enter,

board.DeviceAddress = '172.18.185.123'

Set the language of the generated code to C++ using the TargetLang property.

board.CoderConfig.TargetLang = 'C++'

board = 
targetHardware with properties:

               Name: 'Raspberry Pi'
      DeviceAddress: '172.18.182.234'
           Username: 'pi'
           Password: '*********'
           BuildDir: '/home/pi'
    EnableRunOnBoot: 0
        BuildAction: 'Build, load, and run'
        CoderConfig: [1×1 coder.CodeConfig]

Create an arm-compute deep learning configuration object. Specify the architecture of Raspberry Pi.

dlcfg = coder.DeepLearningConfig('arm-compute');
dlcfg.ArmArchitecture = 'armv7';

In the MATLAB Command Window, execute this command. This command displays the version of ARM Compute Library on Raspberry Pi.

r.system('strings $ARM_COMPUTELIB/lib/libarm_compute.so | grep arm_compute_versio | cut -d\  -f 1')

ans = 
    'arm_compute_version=v19.05
     '

Copy the version number and assign it to the ArmComputeVersion property of the deep learning configuration object.

dlcfg.ArmComputeVersion = '19.05';

Set the DeepLearningConfig property of the code generation configuration object to the deep learning configuration object.

board.CoderConfig.DeepLearningConfig = dlcfg

board = 
targetHardware with properties:

               Name: 'Raspberry Pi'
      DeviceAddress: '172.18.182.234'
           Username: 'pi'
           Password: '*********'
           BuildDir: '/home/pi'
    EnableRunOnBoot: 0
        BuildAction: 'Build, load, and run'
        CoderConfig: [1×1 coder.CodeConfig]

Step 4: Deploy the Function on the Hardware

Deploy the function as a standalone executable on the hardware by using the deploy function.

deploy(board,'raspi_webcam_resnet')

 Deploying code. This may take a few minutes. 
Location of the generated elf : /home/pi/MATLAB_ws/R2020a/L/Work/raspi_example/Examples/raspberrypiio-ex56701774
Warning: Function 'raspi_webcam_resnet' does not terminate due to an infinite loop.

Warning in ==> raspi_webcam_resnet Line: 42 Column: 1
Code generation successful (with warnings): View report

The deploy function initiates code generation of the raspi_webcam_resnet function. Once the code generation is complete, MATLAB generates a code generation report. Use this report to debug the raspi_webcam_resnet function for any build errors and warnings in the generated code.

After successfully generating the code, the support package loads and runs the object classification algorithm as a standalone executable on the hardware. The executable starts classifying the objects in the acquired video, and displays the predicted labels and their associated probability. To view the Raspberry Pi screen, use a VNC viewer and perform a remote session on the hardware to get the display. You can alternatively connect a HDMI cable from the monitor to the hardware.

See Also

Classify Static Image Using Deep Learning on Raspberry Pi