MATLAB Coder: How do I build TensorFlow Lite for Deep Learning C++ code generation and deployment?

I see a few deep learning networks supported for code generation using MATLAB Coder:

• Deep Learning Networks and Layers Supported for C++ Code Generation

I'm looking to generate code from my deep learning network (like ResNet, GoogLeNet, SqueezeNet, VGG-16/19, etc) to run on boards supported by TensorFlow Lite using MATLAB Coder. What are the steps to do this?

[Weidong Yang]

Hi Bill, could you also provide the steps for MacOS?

I assume AI models supported by tensorflow lite should be supported by this workflow? e.g., if I have a transformer model which is supported over tensorflow lite, then it should work with C++? My goal is to generate a mex function so I can call it repeatedly in my matlab code.

Thanks!

[Sayan Saha]

Hello @Weidong Yang, generating code for TFLite models using MATLAB Coder is not yet supported on Mac OS. I'll inform the dev team about your request.

Thanks,

Sayan

[Weidong Yang]

Okay, Thanks Sayan!

Answer 1

[Bill Chou]

Note: The following applies to R2023a and newer releases of MATLAB Coder and TensorFlow Lite library version 2.8 as described here. For older releases, see the answer below.

Use on Windows targets

To generate and run C++ code that performs inference with TensorFlow Lite models on Windows targets, you must have theTensorFlow Lite library on the Windows hardware. To build the TensorFlow Lite library version 2.8 for Windows targets on your host Windows platform, execute the following steps.

Requirements:

• To build the TensorFlow Lite dynamic library, you must install bazel (version 4.1.0 to 4.99.0) on the host Windows computer. Note that these build steps were validated using bazel version 4.2.1. See this link for more information: Installing Bazel on Windows
• You may need to add PYTHON_BIN_PATH to the bazel command if Bazel is not able to find the python paths
• Start the process of bazel-build from the Developer Command Prompt terminals provided with Visual Studio installations

Build Instructions:

1. Open Command Prompt

2. Execute the following commands:

% Clone the git repo
git clone https://github.com/tensorflow/tensorflow.git

% This is the TFLITE ROOT FOLDER
cd tensorflow

% Checkout version 2.8
git checkout r2.8

% Install dependencies
pip --no-cache-dir install numpy

% This step is important to correctly set the python location if using virtual environments
python ./configure.py

% Set to full-path to the VC directory of MSVS (see https://docs.bazel.build/versions/4.2.4/windows.html for details if using other compilers)
set BAZEL_VC="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC"

% Build the dynamic library
bazel build -c opt //tensorflow/lite:tensorflowlite.dll

% Copy generated dll and lib files
mkdir -p lib\tensorflow\lite
cp bazel-bin\tensorflow\lite\tensorflowlite.dll lib\tensorflow\lite
cp bazel-bin\tensorflow\lite\tensorflowlite.dll.if.lib lib\tensorflow\lite\tensorflowlite.lib # note the filename is being changed when copying

% Copy flatbuffer
cp -r bazel-tensorflow\external\flatbuffers .

% To configure the MATLAB environment for TensorFlow Lite code generation, set the environment variables TFLITE_PATH
% <Full path to TFLITE ROOT DIR> is the full-path to `tensorflow` folder where you have downloaded the source files
setenv('TFLITE_PATH’, ‘Full path to TFLITE ROOT DIR’);

Use on Linux host computer

To generate and run C++ code that performs inference with TensorFlow Lite models on your Linux host computer, you must have the TensorFlow Lite library. To build the TensorFlow Lite library version 2.8 on your host Linux platform, follow these steps.

Requirements:

• To build the TensorFlow Lite dynamic library, you must install bazel (version 4.1.0 to 4.99.0) on the host Linux computer. Note that these build steps were validated using bazel version 4.2.1. See this link for more information: Installing Bazel on Ubuntu - Bazel main

Build instructions

1. Open Linux terminal

2. Execute the following commands:

% Clone the git repo
git clone https://github.com/tensorflow/tensorflow.git

% This is the TFLITE ROOT FOLDER
cd tensorflow

% Checkout version 2.8
git checkout r2.8

% Install dependencies
pip --no-cache-dir install numpy

% This step is important to correctly set the python location if using virtual environments
python ./configure.py

% Build the dynamic library
bazel build -c opt //tensorflow/lite:libtensorflowlite.so

% Copy generated .so file
mkdir -p lib\tensorflow\lite
cp bazel-bin\tensorflow\lite\libtensorflowlite.so lib\tensorflow\lite

% Copy flatbuffer
cp -r bazel-tensorflow\external\flatbuffers .

% To configure the MATLAB environment for TensorFlow Lite code generation, set the environment variables TFLITE_PATH
% <Full path to TFLITE ROOT DIR> is the full-path to `tensorflow` folder where you have downloaded the source files
setenv('TFLITE_PATH’, ‘Full path to TFLITE ROOT DIR’);

Answer 2

[Bill Chou]

Note: The following applies to R2022b and older releases of MATLAB Coder and TensorFlow Lite library version 2.4.1 as described here. For newer releases, see the answer above.

Use on Windows targets

To generate and run C++ code that performs inference with TensorFlow Lite models on Windows targets, you must have theTensorFlow Lite library on the Windows hardware. To build the TensorFlow Lite library version 2.4.1 for Windows targets on your host Windows platform, execute the following steps.

Requirements:

• To build the TensorFlow Lite dynamic library, you must install bazel (version 3.1.0 to 3.99.0) on the host Windows computer. See this link for more information: Installing Bazel on Windows
• You may need to add PYTHON_BIN_PATH to the bazel command if Bazel is not able to find the python paths
• Start the process of bazel-build from the Developer Command Prompt terminals provided with Visual Studio installations

Build Instructions:

1. Open Command Prompt

2. Execute the following commands:

% clone the git repo
git clone https://github.com/tensorflow/tensorflow.git  

% git clone will automatically download the code into a folder named `tensorflow`
cd tensorflow

% checkout version 2.4.1
git checkout v2.4.1

% install dependencies
pip --no-cache-dir install numpy future

% Download flatbuffers and other dependent files by running the following command from within the command prompt in Windows:
./tensorflow/lite/tools/make/download_dependencies.sh

% configure tensorflowlite installation
python ./configure.py

% build the dynamic library
bazel build -c opt //tensorflow/lite:tensorflowlite.dll

% it might be necessary to provide python path if it's not picked up automatically
bazel build -c opt //tensorflow/lite:tensorflowlite.dll --action_env PYTHON_BIN_PATH=<python-path>

% Create a directory named lib and copy the contents of the bazel-bin directory into it.
cd bazel-bin
cp -r ../lib/

% To configure the MATLAB environment for TensorFlow Lite code generation, set the environment variables TFLITE_PATH
% <TFLite Root folder> is the full-path to `tensorflow` folder where you have downloaded the source files
setenv('TFLITE_PATH’, ‘<TFLite Root folder>’);

Use on Linux host computer

To generate and run C++ code that performs inference with TensorFlow Lite models on your Linux host computer, you must have the TensorFlow Lite library. To build the TensorFlow Lite library version 2.4.1 on your host Linux platform, follow these steps.

Requirements

To build the TensorFlow Lite dynamic library, you must first install bazel (minimum version 3.1.0) on the host Linux computer. Note that these build steps were validated using bazel version 3.7.2:

1. Install bazel on Linux host (see Installing Bazel on Ubuntu - Bazel main for more information):

2. Install bazel dependencies:

sudo apt-get install curl gnupg 

3. Install bazel from source on host Linux:

curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg   
sudo cp bazel.gpg /etc/apt/trusted.gpg.d/   
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list   
sudo apt update && sudo apt install bazel   

4. If the version of bazel is not compatible, install a compatible bazel version. Minimum bazel version needed is 3.1.0.

sudo apt install bazel-3.7.2 
sudo ln -s /usr/bin/bazel-3.7.2 /usr/bin/bazel 
bazel --version  # 3.7.2 

5. If you encounter any issues install:

sudo apt-get install python-pip python3-pip 
pip --no-cache-dir install grpcio h5py keras_applications keras_preprocessing mock numpy requests future 
    (or) 
pip3 --no-cache-dir install grpcio h5py keras_applications keras_preprocessing mock numpy requests future 

Build instructions

1. Download TensorFlowsource code for version 2.4.1 (available atGitHub - tensorflow/tensorflow at v2.4.1).

2. Unzip the source code to a folder, then:

a) Open Linux terminal

b) Change the directory to the TensorFlow Lite source folder by running the following command.<TFLite Root folder> is the location where you downloaded the source code:

cd <TFLite Root folder> 

3. Download flatbuffers and other dependent files by running these commands.

chmod +x ./tensorflow/lite/tools/make/download_dependencies.sh 
./tensorflow/lite/tools/make/download_dependencies.sh 

4. To generate dynamic library, run:

bazel build -c opt //tensorflow/lite:libtensorflowlite.so 

5. Create a directory named lib and copy the contents of the bazel-bin directory into it.

cd bazel-bin 
cp -r ./ <TFLite Root folder>/lib/ 

6. To configure the MATLAB environment for TensorFlow Lite code generation, set the environment variables TFLITE_PATH and LD_LIBRARY_PATH at the MATLAB command window:

setenv('TFLITE_PATH’, ‘<TFLite Root folder>’);
setenv('LD_LIBRARY_PATH',[getenv('LD_LIBRARY_PATH'),':', ...
getenv('TFLITE_PATH'),'/lib/tensorflow/lite']) ; 

Use on ARM targets

To generate and run C++ code that performs inference with TensorFlow Lite models on ARM targets, you must have theTensorFlow Lite library on the ARM hardware. To build the TensorFlow Lite library version 2.4.1 for ARM targets on your host Linux platform (by using a cross-compiler toolchain), execute the following steps.

Requirements:

• The target device must have either armv7 (32-bit) or armv8 (64-bit) ARM architecture. To verify the architecture, run this command at your device terminal:

arch 

• You must have the Linaro AArch32 or AArch64 toolchain installed on the host Linux computer.

For armv7 target, install the GNU/GCCg++-arm-linux-gnueabihftoolchain:

sudo apt-get install g++-arm-linux-gnueabihf 

For armv8 target, install the GNU/GCCg++-aarch64-linux-gnutoolchain:

sudo apt-get install g++-aarch64-linux-gnu

• To build the TensorFlow Lite dynamic library, you must install bazel (minimum version 3.1.0) on the host Linux computer. To install bazel on Linux host (Ref: Installing Bazel on Ubuntu - Bazel main):

1. Install bazel dependencies:

sudo apt-get install curl gnupg 

2. Install bazel from source on host Linux:

curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg   
sudo cp bazel.gpg /etc/apt/trusted.gpg.d/   
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list   
sudo apt update && sudo apt install bazel   

3. If the version of bazel is not compatible, install a compatible bazel version. Minimum bazel version needed is 3.1.0.

sudo apt install bazel-3.7.2 
sudo ln -s /usr/bin/bazel-3.7.2 /usr/bin/bazel 
bazel --version  # 3.7.2 

4. If you encounter any issues install:

sudo apt-get install python-pip python3-pip 
pip --no-cache-dir install grpcio h5py keras_applications keras_preprocessing mock numpy requests future 
(or) 
pip3 --no-cache-dir install grpcio h5py keras_applications keras_preprocessing mock numpy requests future 

We have validated the build steps on host Linux platform using the following software:

• Build tool: bazel version 3.7.2
• Toolchain:
• GNU Arm Embedded Toolchain (g++-arm-linux-gnueabihf) version 8.3.0 for armv7 targets such as Raspberry Pi
• AArch64 toolchain (g++-aarch64-linux-gnu) version 8.3.0 for armv8 targets such as Hikey960

Build instructions

1. Download the TensorFlowsource code for version 2.4.1 (available atGitHub - tensorflow/tensorflow at v2.4.1) to your host Linux computer.

2. Unzip the source code to a folder. Then:

• Open Linux terminal
• Change directory to the TensorFlow Lite source folder by running this command.Here, <TFLite Root folder> points to the location where you downloaded the source code

cd <TFLite Root folder>

3. Download flatbuffers and other dependent files by running these commands:

chmod +x ./tensorflow/lite/tools/make/download_dependencies.sh 
./tensorflow/lite/tools/make/download_dependencies.sh 

4. Generate the TensorFlow Lite dynamic library by running these commands:

• For armv8 architecture

bazel build --config=elinux_aarch64 -c  
opt//tensorflow/lite:libtensorflowlite.so 

• For armv7 architecture

bazel build --config=elinux_armhf -c  
opt//tensorflow/lite:libtensorflowlite.so 

5. Create a directory lib. Copy the contents of bazel-bin into lib.

cd bazel-bin 
cp -r ./ <TFLite Root folder>/lib/ 

6. Copy the <TFLite Root folder> directory to the target hardware.

7. On the target hardware, set the environment variables TFLITE_PATH and LD_LIBRARY_PATH as follows:

export TFLITE_PATH=<TFLite Root folder>
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:<TFLite Root folder>/lib/tensorflow/lite