Extracting mathematical equation from a deep neural network?

Dear all ,

I have built a deep neural network consist of two hidden layers (6 neurons in the first layer and 4 in the second layer), (11 inputs), and (1 output). Below is the code:

net=newff(Norm_inputs,Norm_Outputs,[6 4],{'tansig' 'tansig' 'tansig'}); %% Neural Network with 12 inputs and four Neurons in the Hidden layers
net.divideParam.trainRatio=80;     %% 70% of the data are used for training 
net.divideParam.testRatio=20;      %% 15% of the data are used for testing  
net.divideParam.valRatio=0;       %% 15% of the data are used for the validation  
net.trainParam.lr=0.01;             %% Learning rate  
net.trainParam.min_grad=1e-20;
net.trainParam.goal=1e-30;
net.trainParam.epochs=1000;
net = train(net,Norm_inputs,Norm_Outputs);

where Norm_inputs are the normalized inputs and Norm_Outputs are the normalized outputs:

Below is the output of my network:

To test the accuracy of network I am using the following:

A = net([0.613497 1 1 0.660657 0.675336 0.150442 0.343584 0.162069 0.655799 0.574284 9.83E-05 ]') 

I am comparing the output of A with my raw data to check the performance..

I tried to extract the mathematical form so I can avoid opening matlab everytime to see the results..

I did the following in the command window to exctract the weights and biases:

x =  [0.2 0 0.048095238 0.814509123 1 0.292035398 0.028111384 0 0.749206349 0.35239154 0.0013]' % The input vector as a column
b1 = net.b{1}; b2 = net.b{2}; b3 = net.b{3}; % Biases 
W_ITH = net.IW{1,1}; W_H1TH2 = net.LW{2,1}; W_H2TO = net.LW{3,2}; % Weights 

Then I used the following equation to see the output:

a1 = tansig(b1+W_ITH*x) % Layer 1 
a2 = tansig(b2+W_H1TH2*y1) % Layer 2 
a3 = tansig(b3+W_H2TO*h2) % Layer 3
Output  = tansig(a3)  % Output layer

However, if I compared the output with what I got from A = net([ ]'), the results will be different so I think I am doing something wrong in the last line.. To understand what is going on I used the following function:

genFunction(net)

and this is what I got:

function [Y,Xf,Af] = neural_function(X,~,~)
%NEURAL_FUNCTION neural network simulation function.
%
% Auto-generated by MATLAB, 09-Dec-2022 09:21:01.
% 
% [Y] = neural_function(X,~,~) takes these arguments:
% 
%   X = 1xTS cell, 1 inputs over TS timesteps
%   Each X{1,ts} = 11xQ matrix, input #1 at timestep ts.
% 
% and returns:
%   Y = 1xTS cell of 1 outputs over TS timesteps.
%   Each Y{1,ts} = 1xQ matrix, output #1 at timestep ts.
% 
% where Q is number of samples (or series) and TS is the number of timesteps.

%#ok<*RPMT0>

% ===== NEURAL NETWORK CONSTANTS =====

% Input 1
x1_step1.xoffset = [0;0;0;0;0;0;0;0;0;0;0];
x1_step1.gain = [2;2;2;2;2;2;2;2;2;2;2];
x1_step1.ymin = -1;

% Layer 1
b1 = [-1.6619087105318077757;-0.38036635353628145406;0.070456977515701249559;-1.8091719441920439682;0.84024035176083067267;-2.4082013915970765794];
IW1_1 = [0.17151362799105107637 0.43991566279934013473 -0.94498059911126330856 -0.42006296125946129827 -0.27862563546688645655 -0.83527543485057420547 0.39025369258432240915 -0.59597775914195516567 0.030901804283541799145 -1.5751433471974514156 2.2185153817891616335;1.4170772331681724676 0.35412432286137413007 0.87818875125559181516 -0.17600030163403754258 0.26322893962013838021 -0.84054484142076568709 -0.11892676428554814494 -1.1826941118532154906 1.1218133611751428536 -0.66767731298128440987 3.1106872504963392068;-0.26723479009346129409 0.81357312787374558294 0.72789078646866034461 -0.50921265559455153316 -0.39103833955865030525 -0.62173294964878533797 0.68528117173730473954 -0.34828465558276422431 -0.16312788337813632911 0.21050282043176274382 0.44335360877887930453;0.14487115798130367761 -0.15055640733584960134 0.27543617993750263429 -0.59709575795996938652 -0.75216339591064962367 1.2416027483157188183 1.1298395428727894263 0.84984116490102246111 -0.10259937135856361801 -0.49136865338803564773 -3.4508552900537483588;0.21061050817627047227 -1.0259134414309891703 -0.47307696880542565721 -0.10211219235696178398 -0.97576324395602820339 0.85652978231966880873 0.57414719896924193421 1.0580352911678043704 -0.11113538128158330365 2.0123285342463415049 -1.0707627999428350751;-0.31632362570748778774 1.498261146701386215 1.8694622275477512652 0.54377418209887562472 -0.16850753331158352388 -0.024211365158402883141 0.50363354864679121814 0.053259098864533692408 0.048581620707959571881 -1.3625415675598020471 -4.8017707994150420348];

% Layer 2
b2 = [-2.155229374705729839;-0.46393350865168381247;-0.250561899861700077;-2.030526644413503945];
LW2_1 = [-0.074382195279472429483 3.2640182588225155058 0.17760021246183130272 -3.9560253984244440062 -0.66153193018127776082 0.27963632610539712608;0.44281065487010662363 0.73006863933605237182 -0.59246756174612402734 0.81153976385101223023 -1.876640649290156837 -0.50558075563543158903;-0.95688266706868496669 -1.7303968831455176502 0.27663573250824052829 1.9643907072858886043 -0.46236471788163485686 -0.60703199794076179163;-2.0165635473477441231 -0.63275446155609926802 -0.28943668952945711403 0.3566114961582590448 2.9589004495388779858 4.2633945339043108902];

% Layer 3
b3 = -0.66635787301535986948;
LW3_2 = [3.617152099907068763 1.7855472593354049238 1.8237105913499436216 -3.6510555660572112657];

% Output 1
y1_step1.ymin = -1;
y1_step1.gain = 2;
y1_step1.xoffset = 0;

% ===== SIMULATION ========

% Format Input Arguments
isCellX = iscell(X);
if ~isCellX
    X = {X};
end

% Dimensions
TS = size(X,2); % timesteps
if ~isempty(X)
    Q = size(X{1},2); % samples/series
else
    Q = 0;
end

% Allocate Outputs
Y = cell(1,TS);

% Time loop
for ts=1:TS
    
    % Input 1
    Xp1 = mapminmax_apply(X{1,ts},x1_step1);
    
    % Layer 1
    a1 = tansig_apply(repmat(b1,1,Q) + IW1_1*Xp1);
    
    % Layer 2
    a2 = tansig_apply(repmat(b2,1,Q) + LW2_1*a1);
    
    % Layer 3
    a3 = tansig_apply(repmat(b3,1,Q) + LW3_2*a2);
    
    % Output 1
    Y{1,ts} = mapminmax_reverse(a3,y1_step1);
end

% Final Delay States
Xf = cell(1,0);
Af = cell(3,0);

% Format Output Arguments
if ~isCellX
    Y = cell2mat(Y);
end
end

% ===== MODULE FUNCTIONS ========

% Map Minimum and Maximum Input Processing Function
function y = mapminmax_apply(x,settings)
y = bsxfun(@minus,x,settings.xoffset);
y = bsxfun(@times,y,settings.gain);
y = bsxfun(@plus,y,settings.ymin);
end

% Sigmoid Symmetric Transfer Function
function a = tansig_apply(n,~)
a = 2 ./ (1 + exp(-2*n)) - 1;
end

% Map Minimum and Maximum Output Reverse-Processing Function
function x = mapminmax_reverse(y,settings)
x = bsxfun(@minus,y,settings.ymin);
x = bsxfun(@rdivide,x,settings.gain);
x = bsxfun(@plus,x,settings.xoffset);
end

So everything is similar except the last lines:

    % Output 1
    Y{1,ts} = mapminmax_reverse(a3,y1_step1);

So I am doing something wrong in last step(i.e., the following line is wrong):

    Output  = tansig(a3)  % Output layer 

Any help!

Answer 1

[Udit06]

Hi Mahmoud,

The code generated using “genFunction” indicates that MATLAB internally applies “minmax” processing to the input and output data. For more details you can refer the following MATLAB answer: https://www.mathworks.com/matlabcentral/answers/385206-why-is-predicting-the-output-of-a-trained-neural-network-by-using-net-sim-different-than-manually#answer_307446.

I hope this helps.