BackProgation Neural network on GPU

Hi everyone,

Can you help me how to write backprogation in neural network on GPU?

I assume that my data is big, and number of input, hidden, output neuron are 784, 1024 and 10. this like MNIST data.

I write code manually, not use function in neural network tools.

I wonder about update weight is not parallel, so How can use GPU here?

Thanks so much.

[Joss Knight]

Nobody is going to be able to answer your question in its current form. It's impossible to tell what help you need. If you don't know how to use the GPU in MATLAB, then read the documentation on that. If you don't know how to write the equations for backpropagation then you should be specific about what operation you want to backpropagate through. To get help, you need to be specific, and you need to provide some example code.

[Tiki Tiki]

Thank. I wrote backprogation code not use function 'train' support by matlab. So my mean is in the backprogation, we have to update weight based on previous weight. so I dont know how to use parallel programming here. is it mean we put 60000 samples of MNIST once time. and then caculate error?. please correct me if i am wrong? Thank.

Answer 1

[Diwakar Ravichandran]

Hi Tiki,

As I try to understand you want to implement back propagation by hard coding the variables. To do the same you have to know the back propagation algorithm and understand how it works. You cant code on a GPU you code and the process runs on a GPU.

This article online explains what back propagation is as mentioned below,

https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

Hope this helps,

Cheers!

[Tiki Tiki]

Thank you. Actually, I have already written code backprogation on CPU. it works well.

But My problem is how to use GPU in backprogation.

I dont understand much about how GPU work when training network use backprogation.

As I asked, I put 60000 samples MNIST at once time to run parallel. it it right?

please, help me more?

[Diwakar Ravichandran]

When you want to use back propagation, and you write your code, and run it, MATLAB will be using a GPU if it is available or it will use your CPU to run the code. If you put all of the 60000 examples and ask it to run every time, your code will run very slowly irrespective of where you are running it.

[Tiki Tiki]

Sorry, something I dont understand. in backprogation, i need to use update phase to update weight from previous weight. So I have to run one sample at one time. and I define function gpuArray for samples. it not means that parallel in GPU. So actually I dont understand how GPU can speed up and solve update phase in Backpogation.

Can you explain for me? Thank

[Diwakar Ravichandran]

I would suggest you to go through all algorithms related to gradient descent to understand how it works internally.

[Tiki Tiki]

actually, I read many times, and my backprogation code on cpu is work well. i put each samples at one time to update weight, so this is serial not parallel. but now i have GPU, so i want to ask you how to use GPU for this case. Can you explain how to use GPU in training network?

Thank.

[Diwakar Ravichandran]

https://www.mathworks.com/discovery/matlab-gpu.html

I think this should help

[Tiki Tiki]

Thank. I already run code use function with GPU supported by matlab like train or feedforward. however, I cannot see inside function, So I had to write step by step to describe backprogation in my network. but i still dont understand how GPU can use.

Anyway, Thank for your support.