Frame-Based Modeling of Video Processing for Hardware

This example shows how to develop a frame-based video processing algorithm in MATLAB®, accelerate its simulation using MATLAB Coder™, and generate sample-based HDL code from the design using the HDL Coder™ Frame to Sample optimization. The algorithm enhances edge detection from noisy color video.

Test Bench

In the FrameBasedEnhancedEdgeDetectionHDLTestBench.m file, the videoIn object reads each frame from a color video source, and the imnoise function adds salt and pepper noise. The test bench calls the Design Under Test (DUT), FrameBasedEnhancedEdgeDetectionHDLDesign.m, to process each frame in turn. The reference design, FrameBasedEnhancedEdgeDetectionHDLReference.m, performs the same operations using Image Processing Toolbox™ functions. The test bench compares the output of the reference function and the output of the DUT. The function FrameBasedEnhancedEdgeDetectionHDLViewer.m displays both video outputs.

The workflow above is implemented in the following lines of FrameBasedEnhancedEdgeDetectionHDLTestBench.m.

...
frameFull = zeros(frameHeight, frameWidth, 3, 'uint8'); 
frameIn   = zeros(frameHeight, frameWidth, 3, 'uint8');      
for ii = 1 : numFrames
    frameFull(:) = readFrame(videoIn);                  % Read a new frame
    frameIn(:)   = imnoise(frameFull, 'salt & pepper'); % Add noise

    % Call the design under test
    frameOut = FrameBasedEnhancedEdgeDetectionHDLDesign(frameIn);

    % Call the reference design
    [frameGray, frameDenoise, frameReference] = ...
        FrameBasedEnhancedEdgeDetectionHDLReference(frameIn);

    % Compare the results
    if nnz(imabsdiff(frameReference, frameOut)) > 50
        fprintf('frame %d: reference and design output differ in more than 50 pixels.\n', ii);
        return;
    end

    % Display the results
    FrameBasedEnhancedEdgeDetectionHDLViewer(frameWidth, frameHeight, ...
        [frameGray, frameDenoise, uint8(255*frameOut)], ...
        [frameFull, frameIn]);
end
...

Since frameGray and frameDenoise are uint8 data type while frameOut is logical, uint8(255*frameOut) maps logical false and true to uint8(0) and uint8(255), respectively, so that matrices can be concatenated.

Before the test bench terminates, the code displays the frame rate to illustrate the simulation speed.

For the functions that do not support C code generation, such as tic, toc, imnoise, and fprintf in this example, use coder.extrinsic to declare them as extrinsic functions. Extrinsic functions are excluded from MEX generation. The simulation executes them in the regular interpreted mode. Since imnoise is not included in the C code generation process, the compiler cannot infer the data type and size of frameIn. To fill in this missing piece, add the statement frameIn = zeros(frameHeight, frameWidth, 3, 'uint8') before the outer for-loop.

Frame-Based HDL Design

The function defined in FrameBasedEnhancedEdgeDetectionHDLDesign.m accepts a 3D RGB frame and returns the 2D boolean edges.

In this example, the hdlrgb2gray function converts a color image to grayscale, hdlmedfilt2 removes the salt and pepper noise, and hdledge highlights the edge. The code is shown below.

frameGray    = hdlrgb2gray(R,G,B);      % Convert RGB to grayscale
frameDenoise = hdlmedfilt2(frameGray);  % Remove noise
frameEdge    = hdledge(frameDenoise,'Sobel',7/255); % Detect edges

Reference Design

When designing a complex video processing algorithm, it is a good practice to develop a parallel reference design using functions from the Image Processing Toolbox™ to verify the implementation.

The function FrameBasedEnhancedEdgeDetectionHDLReference.m contains a similar set of three functions as in the FrameBasedEnhancedEdgeDetectionHDLDesign.m.

Due to the implementation difference between edge function and hdledge function, the reference and design output are considered matching if frameOut and frameReference differ in no more than 50 pixels.

Create MEX File and Simulate the Design

Generate and execute the MEX file.

visionhdlFrameToSampleSetup;
codegen('FrameBasedEnhancedEdgeDetectionHDLTestBench');

Code generation successful.

FrameBasedEnhancedEdgeDetectionHDLTestBench_mex;

100 frames have been processed in 3.87 seconds.
Average frame rate is 25.83 frames/second.

The upper video player displays the original color video on the left, and its noisy version after adding salt and pepper noise on the right. The lower video player, from left to right, represents: the grayscale image after color space conversion, the de-noised version after median filter, and the edge output after edge detection.

Note that in the lower video chain, only the edge output (right-most video) is generated from design under test. The other two are the intermediate videos from the reference design as FrameBasedEnhancedEdgeDetectionHDLDesign.m is the target function for HDL code generation. To display all of the three videos from the design under test, you would have written the design file to output three frames.

HDL Code Generation

Generating HDL using HDL Coder's frame to sample optimization generates an Entity with I/O suited for AXI-Stream interfaces and provides a input and output FIFO for memory contention and backpressure. The size of the FIFOs can be configured in the HDL Coder setings.

To create a new project, enter the following command in the temporary folder

coder -hdlcoder -new FrameBasedEnhancedEdgeDetectionProject

Then, add the file 'FrameBasedEnhancedEdgeDetectionHDLDesign.m' to the project as the MATLAB Function and 'FrameBasedEnhancedEdgeDetectionHDLTestBench.m' as the MATLAB Test Bench. Refer to Get Started with MATLAB to HDL Workflow (HDL Coder) for a tutorial on creating and populating MATLAB HDL Coder projects.

Launch the Workflow Advisor. In the Workflow Advisor enable the Frame to Sample optimization:

HDL Code Generation > Optimizations > General > Aggressive Dataflow Conversion

HDL Code Generation > Frame to Sample Conversion > Enable Frame to Sample Conversion (Requires Aggressive Dataflow Conversion)

To generate code right-click the 'HDL Code Generation' step. Choose the option 'Run to selected task' to run all the steps from the beginning through HDL code generation.

Examine the generated HDL code by clicking the links in the log window.

The design was synthesized and put through place and route on a Xilinx™ Zynq™ ZC706 board.

	Resources
LUT	1962
LUTRAM	308
FF	965
BRAM	0.5
DSP	5