Double precision in deep learning

Hello guys,

i want to train my LSTM using double precision but the deep learning toolbox only allows single precision. Can i change that to double precision or how do i train my neural network using double precision ?

Thanks.

[KSSV]

To my knowledge, the present deep learning toolbox uses single precision. Though you give input in double, it will be converted to single and trained.

[bah]

but using custom training loops it is possible to use double precision right ?

[Matt J]

I don't know, but you should be mindful that training could be slower on the GPU, not just because of the increased precision, but because some GPUs are not optimized for double precision.

[bah]

Thanks a lot for the information.

Answer 1

[Joss Knight]

This is possible with a dlnetwork but it is (currently) more of a workaround than anything else. Once your dlnetwork is ready to go, convert it to double precision:

net = dlupdate(@double, net);
net.State = dlupdate(@double, net.State);

Then make sure your input data to the network is always in double precision. Everything should then remain in double as you train and use the network.

Let me know if this doesn't work for you.