Evaluate deep learning model for custom training loops
dlfeval to evaluate custom deep learning models for
custom training loops.
For most deep learning tasks, you can use a pretrained neural network and adapt it to your own
data. For an example showing how to use transfer learning to retrain a convolutional neural
network to classify a new set of images, see Train Deep Learning Network to Classify New Images. Alternatively, you can create and train
neural networks from scratch using the
trainingOptions function does not provide the training options that you need for your task, then you can create a custom training loop using automatic differentiation. To learn more, see Define Deep Learning Network for Custom Training Loops.
Compute Gradient Using Automatic Differentiation
Rosenbrock's function is a standard test function for optimization. The
rosenbrock.m helper function computes the function value and uses automatic differentiation to compute its gradient.
function [y,dydx] = rosenbrock(x) y = 100*(x(2) - x(1).^2).^2 + (1 - x(1)).^2; dydx = dlgradient(y,x); end
To evaluate Rosenbrock's function and its gradient at the point
[–1,2], create a
dlarray of the point and then call
dlfeval on the function handle
x0 = dlarray([-1,2]); [fval,gradval] = dlfeval(@rosenbrock,x0)
fval = 1x1 dlarray 104
gradval = 1x2 dlarray 396 200
Alternatively, define Rosenbrock's function as a function of two inputs,
x1 and x
function [y,dydx1,dydx2] = rosenbrock2(x1,x2) y = 100*(x2 - x1.^2).^2 + (1 - x1).^2; [dydx1,dydx2] = dlgradient(y,x1,x2); end
dlfeval to evaluate
rosenbrock2 on two
dlarray arguments representing the inputs
x1 = dlarray(-1); x2 = dlarray(2); [fval,dydx1,dydx2] = dlfeval(@rosenbrock2,x1,x2)
fval = 1x1 dlarray 104
dydx1 = 1x1 dlarray 396
dydx2 = 1x1 dlarray 200
Plot the gradient of Rosenbrock's function for several points in the unit square. First, initialize the arrays representing the evaluation points and the output of the function.
[X1 X2] = meshgrid(linspace(0,1,10)); X1 = dlarray(X1(:)); X2 = dlarray(X2(:)); Y = dlarray(zeros(size(X1))); DYDX1 = Y; DYDX2 = Y;
Evaluate the function in a loop. Plot the result using
for i = 1:length(X1) [Y(i),DYDX1(i),DYDX2(i)] = dlfeval(@rosenbrock2,X1(i),X2(i)); end quiver(extractdata(X1),extractdata(X2),extractdata(DYDX1),extractdata(DYDX2)) xlabel('x1') ylabel('x2')
Compute Gradients Involving Complex Numbers
dlfeval to compute the value and gradient of a function that involves complex numbers. You can compute complex gradients, or restrict the gradients to real numbers only.
Define the function
complexFun, listed at the end of this example. This function implements the following complex formula:
Define the function
gradFun, listed at the end of this example. This function calls
complexFun and uses
dlgradient to calculate the gradient of the result with respect to the input. For automatic differentiation, the value to differentiate — i.e., the value of the function calculated from the input — must be a real scalar, so the function takes the sum of the real part of the result before calculating the gradient. The function returns the real part of the function value and the gradient, which can be complex.
Define the sample points over the complex plane between -2 and 2 and -2 and 2 and convert to
functionRes = linspace(-2,2,100); x = functionRes + 1i*functionRes.'; x = dlarray(x);
Calculate the function value and gradient at each sample point.
[y, grad] = dlfeval(@gradFun,x); y = extractdata(y);
Define the sample points at which to display the gradient.
gradientRes = linspace(-2,2,11); xGrad = gradientRes + 1i*gradientRes.';
Extract the gradient values at these sample points.
[~,gradPlot] = dlfeval(@gradFun,dlarray(xGrad)); gradPlot = extractdata(gradPlot);
Plot the results. Use
imagesc to show the value of the function over the complex plane. Use
quiver to show the direction and magnitude of the gradient.
imagesc([-2,2],[-2,2],y); axis xy colorbar hold on quiver(real(xGrad),imag(xGrad),real(gradPlot),imag(gradPlot),"k"); xlabel("Real") ylabel("Imaginary") title("Real Value and Gradient","Re$(f(x)) = $ Re$((2+3i)x)$","interpreter","latex")
The gradient of the function is the same across the entire complex plane. Extract the value of the gradient calculated by automatic differentiation.
ans = 1×1 dlarray 2.0000 - 3.0000i
By inspection, the complex derivative of the function has the value
However, the function Re() is not analytic, and therefore no complex derivative is defined. For automatic differentiation in MATLAB, the value to differentiate must always be real, and therefore the function can never be complex analytic. Instead, the derivative is computed such that the returned gradient points in the direction of steepest ascent, as seen in the plot. This is done by interpreting the function Re: C R as a function Re: R R R.
function y = complexFun(x) y = (2+3i)*x; end function [y,grad] = gradFun(x) y = complexFun(x); y = real(y); grad = dlgradient(sum(y,"all"),x); end
fun — Function to evaluate
Function to evaluate, specified as a function handle. If
dlgradient call, then
the gradient by using automatic differentiation. In this gradient evaluation, each
argument of the
dlgradient call must be a
or a cell array, structure, or table containing a
dlarray. The number
of input arguments to
dlfeval must be the same as the number of input
x1,...,xn — Function arguments
any MATLAB® data type |
An input argument
xj that is a variable of differentiation in a
dlgradient call must be a traced
dlarray or a
cell array, structure, or table containing a traced
dlarray. An extra
variable such as a hyperparameter or constant data array does not have to be a
To evaluate gradients for deep learning, you can provide a
dlnetwork object as a function argument and evaluate the forward pass
of the network inside
dlarray([1 2;3 4])
Complex Number Support: Yes
y1,...,yk — Function outputs
any data type |
Function outputs, returned as any data type. If the output results from a
dlgradient call, the output is a
dlgradientcall must be inside a function. To obtain a numeric value of a gradient, you must evaluate the function using
dlfeval, and the argument to the function must be a
dlarray. See Use Automatic Differentiation In Deep Learning Toolbox.
To provide the best performance, deep learning using a GPU in MATLAB is not guaranteed to be deterministic. Depending on your network architecture, under some conditions you might get different results when using a GPU to train two identical networks or make two predictions using the same network and data.
Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.
Usage notes and limitations:
gpuArrayor as a
dlarraythat contains a
For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).
Introduced in R2019b