How to implement rematerialization when using autodiff?

I am using autodiff instead of finite differences and I am running into a memory problem. I read about rematerialization but I don't know how to or if it is possible to implement on matlab.

Answer 1

[Zuber Khan]

Hi,

I am assuming that by rematerialization, you are referring to Gradient Checkpointing. Based on my understanding, there is no in-built functionality in Matlab so far to minimize memory consumption by discarding certain intermediate outputs and recalculating them during the backward pass.

You might need to write a custom code to save the important intermediate outputs (checkpoints) and recompute the rest during the backward pass with the help of these checkpoints.

Since you have not provided any details, it is difficult to write the exact code for identification and usage of these checkpoints. Further, in order to know how to use them for recomputation, it would require understanding of your objective function as well as the computational graph of derivatives.

I hope this will give you certain clarity.

Regards,

Zuber