Deploy YOLOv2 to an NVIDIA Jetson - MATLAB
Video length is 18:58

Deploy YOLOv2 to an NVIDIA Jetson

From the series: Perception

In this video, Connell D’Souza joins Neha Goel to demonstrate how to deploy a deep neural network to an NVIDIA® embedded GPU using GPU Coder™ and the GPU Coder support package for NVIDIA GPUs. The example discussed in this video is deploying a multiclass YOLOv2 neural network to an NVIDIA Jetson® TX1.

Connell first introduces the different MathWorks and third-party prerequisite libraries needed to generate and deploy CUDA code to an NVIDIA GPU and demonstrates how to verify the setup using the coder.checkGpuInstall app.

Next, Connell will discuss preparing MATLAB code for GPU code generation. He demonstrate how to generate and deploy an executable to an NVIDIA Jetson TX1 to sample video frames from a camera connected to the Jetson and detect objects of interest in the video frames.

Resources:

Published: 3 Jan 2020

Hi, guys. We are back with our final video for the deep learning for object detector series. And we have come here-- in our previous three videos, we actually labeled the data, and then trained the data, designed the data, and also imported the network from outside, like TensorFlow. And so we have a network, and we want that network to be on some hardware and to see the usage, how we can actually use that network. So Connell here will explain you how you can actually deploy that network on your NVIDIA Jetson.

All right, so as Neha mentioned, we are assuming that we have a trained deep learning network now in MATLAB. And what we're going to do is we're going to go ahead and deploy it to this little NVIDIA Jetson TX1 that we have here. So let's actually go ahead and see how we can do that from the MATLAB environment. Remember, what we're doing now is we're going to convert MATLAB code into CUDA code so it can run natively on the Jetson.

All right, so let's jump in. The first thing we need to do is we need to actually install a bunch of prerequisite products between the Jetson and the MATLAB host. So the toolboxes that we require to perform this particular application are listed up here. The ones in particular that I want to call out are the GPU Coder Interface for Deep Learning Libraries and the GPU Coder Support Package for NVIDIA GPUs. You need to download this from File Exchange or from the Add-Ons Manager. So you can go ahead go up here, and then, under the Get Add-Ons tab, you can go ahead and grab these two libraries from there.

The next thing that we need to do is we need to actually do some significant setup on the Jetson. Those of you that are familiar with working with the Jetson are probably already done this. These are, again, one-time setups that need to be done to flash the Jetson with its OS.

The way we do this setup is that you've actually got to connect your Jetson to a host computer and then flash the Jetson from a host computer. Now this host computer could be either a Windows or Linux. I've used a Windows machine to do this. I had to set up a virtual Linux machine on Windows to get this done, but if you're doing it from a Linux environment, it's a lot more straightforward.

The first thing they need to do to flash the OS is to install Jetpack. Jetpack is NVIDIA's OS for the Jetson. There is some excellent documentation on how you can get through setting this up. And I've put in a link as to how you can do this. This takes a significant amount of time, so you want you want to leave yourself with some time to do that.

The next thing that you need is the simple DirectMedia layer, or the SDL library. This helps you visualize camera outputs and stuff for that on the Jetson. As you know, the Jetson is a standalone computer. You can hook a keyboard and a mouse and a monitor to it and run it like a computer.

And the last step that you need to do is you actually need to set up some environment variables to point the Jetson to where your CUDA libraries are on the Jetson. Again, all of this stuff gets installed directly from the-- all of the CUDA libraries, at least, gets installed directly from Jetpack. But you need to download that simple DirectMedia layer independently.

The next setup that you need to do is you actually need to do some setup on the MATLAB host as well. So if you're using a Linux computer or a Windows computer, there are some libraries that you need to download onto the actual host. The first thing is-- and the first important thing that you need is the C++ compiler. In MATLAB, you can-- the compiler supported for GPU Coder is Microsoft Visual Studio, so either 2013 through 2017 if you're using the CUDA toolkit version 10.0. And if you actually want to download the more recent 10.1 release, you can also use Visual Studio 2019 with that. And then for the Linux environment, the C++ compiler works.

A good way to run a compiler check is to use the max dash setup function in MATLAB. So let's go ahead and run this section real quick and we should see that our MATLAB on our end has been configured to use Microsoft Visual C++ 2019. And then these are some other options and a hint on how you can set that-- or you can set that the other compilers on.

The next few libraries that you need to install is the CUDA toolkit and the drivers. Again, we've put in links to the NVIDIA website from where you can download these. Once you download this, this goes and gets saved in your Program Files folder on your Windows computer. The next thing that you need is the cuDNN and the Tensor RB libraries again these are two to 2 deep learning libraries that are used on NVIDIA GPUs and links for them as well.

And then the last thing that you need is you need some OpenCV libraries. Now the OpenCV library that get shipped with the Computer Vision Toolbox in MATLAB, it does not have all the required libraries, and so what you need to do is you actually want to download that standalone and add that to your path as well. Once you download all these libraries, you want to tell your computer where these libraries are. So you want to set some environment variables. You can use the setenv command to set environment variables in MATLAB.

So does it register-- does the MATLAB register the environment variables permanently?

That's a good question. MATLAB does not register these environment variables permanently, and we'll get these-- environment variables will get erased every time you close down your MATLAB session or you restart MATLAB. So one way to get around that is to actually have a startup script, and there's enough documentation on how you can do that. You can go ahead and add these lines of code into your startup script that will get launched every time you open a new session of MATLAB.

But as you can see we've got a few environment variables for your CUDA path. This points to the NVIDIA GPU computing toolkit and your CUDA libraries. So you want to set that variable. You want to set the NVIDIA CUDNN variable, the TENSORRT variable, and then the OpenCV directory. And then you also want to add all of these folders to your Windows path, if that makes sense.

At this point, we've set up our MATLAB host to actually to generate CUDA code. Now the next thing we want to do is we want to actually go ahead and connect to the NVIDIA Jetson. So MATLAB provides you with a function, the jetson function, that will help you connect that. You need to give it some parameters that can help them access the board. Again, if you look at our little Jetson board up here, we've got this little ethernet port, which is helping us talk to our host machine.

So again, the IP address username and password-- these are values that are specific for your Jetson board. You may want to use something like an advanced IP scanner to find the board. But what I went and did was I went and actually set a static IP address to the Jetson, and that's why I know it's this value. Again, you have to change this for your device.

What we can do is if I go ahead and select this and evaluate the selection, what I should see in the command window is that I have the hardware object created. And then it's also telling me that it's been connected to the board and what are my different versions on there. And it's saying that I have a Microsoft LifeCam connected and then the Tegra X1 GPU.

So that is good that, at every step, we are seeing what all we have done and what all hardware installations is complete.

Yep, and OK, so this looks good. It looks like our MATLAB is registered with the Jetson now. We can actually talk to the Jetson.

The next step that we want to take is we want to actually check if we can generate code properly onto the Jetson. And what our code team has provided us with is a coder.checkGpuInstallApp. So I'm going to go ahead and run this in the command window. And we can see what I'm talking about.

So what this app actually does?

That's a great question, and we will talk about it right now. So what the GPU environment check does is it's basically going to run a bunch of checks. So under the hood, what it's doing is it's taking a very simple file and generating code and seeing if we've got all the necessary libraries and all the necessary versions that can help us generate the code for the Jetson. So I'll show you how we can do it.

Let's go ahead and choose the hardware. So we say, OK, we want to select the Jetson. And it's going to ask us for the device address. Now, as you can see, it's already grabbed that from our variable in there. We're already connected to the board. Let's actually go and run some workflow checks.

So we go-- let's test the basic code generation, and let's check deep learning code generation. And you can choose both these options. So the Generate Code will just give you CUDA code. Generate and Execute will actually compile the executable and run it on the Jetson. So we're just going to choose the Generate Code option for now. And let's choose the cuDNN library. And let's hit the Run Checks button.

So once the app runs the environment checks, it's going to give us a report, telling us all that it has checked. So as you can see, it's found that it's connected to the GPU. So it's found a compatible GPU. It's checked our CUDA environment, so it knows that we have all these libraries installed. It's checked the cuDNN environment. Again, we're using the cuDNN library to install code. And then, as we scroll down, we see our basic code generation check. So we said that it's passed as well as the deep learning. So what this means is that our code is now set up, and we're good to go in terms of generating and deploying the code.

So let's go back to our script real quick. Now what we've provided here in this little block of code is basically what we did in the app. But we just can do that programmatically as well. So instead of running the checkGpuInstallApp function, we just ran the checkGpuInstall. And this will do the same thing. It's just a programmatic way of doing it.

So now that we've set up our environment, we've set up the Jetson board, we know that we can generate code, let's actually go and see how we can prepare the MATLAB code to generate-- to deploy to the Jetson. So I'm just going to go ahead and run this section real quick and open up our roboSubPredict function.

The first thing that you need to do before you generate C, C++, or even CUDA code from MATLAB, you want to take the algorithmic part of your code and convert it into a function. So if you remember, earlier, what we had was we just had that for loop that was calling-- that was loading the network and calling the detect function. Now we've just gone in and put that detect function into another function.

So is there any other way that I can learn more about this code generation thing?

Yeah, so we actually-- our student competition team actually put out a whole series on code generation. Again, we haven't touched on CUDA code in that-- on CUDA code generation in that series. But we talk extensively about generating code from both MATLAB and Simulink. And I've actually put a link to that series here, and you can go and check that out on your own. But there's a lot of good information there.

So at a very high level, once you've converted the algorithm into a function, you need to use the codegen directive. What the codegen directive does is the codegen directive runs-- it does a very preliminary check on your function to make sure that you aren't using any functions in MATLAB that are not supported for code generation. So there is a small subset of the functions, things like the plotting functions and stuff like that, that are not code general. And the code generator will run a quick pass through your function to make sure that you're not using any of those functions. And it will give you a warning, rather, if you are actually using those kind of functions.

Then, what we're doing is the next thing we need to do is we need to-- I want to talk about the Jetson object. So remember, we set up the Jetson object earlier to actually connect and check the CUDA environment. We've also got some utilities here. So the webcam method of this Jetson object helps us connect to any webcam that's connected to the Jetson. So, for example, if you have a webcam connected there, and you want to sample images, you can create a webcam object for the Jetson and then run a snapshot method to grab images from there. And the same with the image display-- this helps you visualize the output on the Jetson board. And we'll see that in a little bit.

The next thing that we're doing is we want to use the coder.loadDeepLearningNetwork function. Now what we have here is this is where we're actually loading our detector in this function. So we've saved our detector as a Mac file detector-- YOLOv2, if you remember this from the previous video. We need to declare this mynet detector object as persistent.

And the reason you want to do that is if it's not declared as persistent and not declared within this if statement, it's going to reload the network every time this function gets called. And you don't want to do that. So persistent variables in MATLAB are equivalent to static variables. Once you register them, it's going to maintain its memory. So you don't want to keep loading this every time you call this function. That's why you want to declare this as persistent. We have a network loaded in here. And if you scroll down, we'll see that we're actually calling the detect method of this object here.

So are we also resizing our images?

Yeah, actually, that's a very good point. We had a discussion on input sizes to our network. And if you see here, what we're doing is we're grabbing a snapshot of the image, and then just resizing it to 416 by 416, just because that's what our network is best suited to work with. And then, lastly, we use this image function to actually visualize the output.

Let's go back to our deployToNVIDIAJetson script. We've got a MATLAB function that we can generate code for now. Now the next thing is to actually take this and dump it onto the hardware. So what we need to do for that is we need to set up some coder configurations. And we can do it-- so we're telling it that we want to generate an executable for the NVIDIA Jetson hardware.

And then this remoteBuildRoboSubPredict-- remember this directory. This is the directory in which all the files are going to be loaded onto the Jetson. So once you generate the code, you go and look for the remoteBuildRoboSubPredict directory on the Jetson. You will find it there. And that's where all your code will be. And then we're telling it that we want to generate an example main, so it'll generate the code and compile an executable.

And then, lastly, what we do is we choose the codegen command, give it these configuration variables and then the function that we want, and then we ask it to-- we want to generate a report. So this is what's actually going to generate the code and deploy it onto the Jetson. So let's go ahead and run this section real quick.

Once you run the codegen command, we get-- MATLAB tells us the code generation was successful. Let's go ahead and view the report and see what it looks like.

So other than using this code, do we have any other way?

Yeah, you can also use the GPU Coder app to generate this. If you guys go through the student competition code generation tutorial series, you actually see that we actually use the MATLAB Coder app over there. So there is an app workflow as well. Again, it's just a matter of preference.

But what this report tells us is the report shows us all the files that are generated. So if you see this, this talks about-- this is our DeepLearningNetwork CUDA file. And then you can also see the GPU kernels that have been created here. We actually use the Trace Code option to see what that translates to in CUDA. So roboSubPredict is our function name, and then we've got the roboSubPredict.c, which is the CUDA version of the function.

And if you want to actually look at any particular function, you can use this Trace Code option. So if I say I want to see what is the code that corresponds to the mynet.detect function call, I click on this, and it will tell me that it's a whole host of functions that are highlighted over here. And I can scroll through them and see what this looks like. So there's a significant amount of code that gets generated from it. But it speaks to the advantage of using GPU Coder because you can write a few lines of code, and then it'll spit out all that you need to write it on to the Jetson.

So what we've done now, by running this codegen command, is we've actually gone ahead and flashed our little Jetson here with the executable. And we can actually go into the Jetson environment and see what this looks like over there.

So what I understand is that we are doing is directly flashing the Jetson with our code. So what if I have something to take another environment to integrate with or like other code I have?

Oh, that's actually a really good question. If you go back into the report real quick, I'll show you what we can do. What we've done right now is we've created an executable. You can also create libraries or DLLs or other settings that are available. And then, once we generate the code, you can use this Package Code option. So what this will do is it'll package all the files that are generated and all the files that are needed to run that code, and you can just take that package into whatever other environment you're using and integrate it with any other code that you have. So MATLAB can fit into parts of your workflow that way.

So let's hop over to the Jetson environment real quick, and we can show you how this code runs on the Jetson.

What we have here is a monitor connected to our Jetson computer. If we go into our Files tab, we should see our remoteBuildRoboSubPredict folder in here. And if we scroll through this, we'll see our generated code in here. And this is our roboSubPredict.elf file, which is our executable. So if I go ahead and click this guy, what we should see here is our network is able to identify images. Again, it's played off a computer screen right now because we can't obviously put this into the water. But this is the code running on the Jetson now.

So now that you have seen that we have deployed our network on the Jetson and it's running perfectly well, and so you can actually look at all our resources where we talk about every kind of training, the code generation trainings, and these all the videos. Or any other questions you have you can just email at roboticsarena, and feel free to deploy the code on your Jetson and give us the feedback.

Yeah, let us know what you think, and we're happy to help. See you soon.

Up Next:

View full series (11 Videos)

View more related videos