Transfer learning CNN with regression at the end?

Hi

I have been trying to test transfer learning with a regression layer at the end, but I end up with a network that always predicts NaNs. What am I doing wrong?

I want to use e.g., VGG19 to estimate timber volume in images but as I cannot release the data I created a modified version of one of the Matlab examples to show the problem. The example that I modified is called 'TrainAConvolutionalNeuralNetworkForRegressionExample', where the rotation of digits should be estimated.

I assume the problem lies in how the layers are put together at the end. Here is the modified code:

clear
close all

% Clear GPU memory
gpuDevice(1)

% Load VGG network
net = vgg19; % vgg16, vgg19 or alexnet

% Load the digit training set as 4-D array data using
[trainImagesSmall,~,trainAngles] = digitTrain4DArrayData;

% Load the digit test set.
[testImagesSmall,~,testAngles] = digitTest4DArrayData;

% Pick out a subset to test
numToTest = 1000;

testImagesSmall = testImagesSmall(:,:,1,round(linspace(1,5000,numToTest)));
testAngles = testAngles(round(linspace(1,5000,numToTest)),1);

% % In case the images have been resized and saved one can load instead
% load('trainImages.mat');
% load('testImages.mat');

% Resize train images to match VGG input
trainImages = zeros(224,224,1,5000);
for i = 1:size(trainImages,4)
    i
    trainImages(:,:,1,i) = imresize(trainImagesSmall(:,:,1,i),[224 224]);
    clc
end
trainImages = repmat(trainImages,1,1,3,1);  
disp('Saving training images...');
save('trainImages.mat', 'trainImages','-v7.3');

% Resize test images to match VGG input
testImages = zeros(224,224,1,numToTest);
for i = 1:numToTest
    i
    testImages(:,:,1,i) = imresize(testImagesSmall(:,:,1,i),[224 224]);
    clc
end
testImages = repmat(testImages,1,1,3,1);    
disp('Saving test images...');
save('testImages.mat', 'testImages','-v7.3');

% Display 20 random sample training digits using |imshow|.
numTrainImages = size(trainImages,4);

figure
idx = randperm(numTrainImages,20);
for i = 1:numel(idx)
    subplot(4,5,i)
      imshow(trainImages(:,:,:,idx(i)))
      drawnow
  end

% Combine all the layers together in a |Layer| array.
layersTransfer = net.Layers(1:end-3);
layers = [layersTransfer
          fullyConnectedLayer(1)
          regressionLayer];

% Train Network
transferLayerOptions = trainingOptions(  'sgdm','MiniBatchSize',25,...
                            'InitialLearnRate',0.001, ...
                            'MaxEpochs',7, 'Verbose', true);

% load('trainedNet_7Epochs.mat');   % Load previously trained network
net = trainNetwork(trainImages,trainAngles,layers,transferLayerOptions);

% Clear GPU memory (don't know if this makes any difference here)
gpuDevice(1)

% Use |predict| to predict the angles of rotation of the test images.
predictedTestAngles = zeros(length(testAngles),1);
for i = 1:length(testAngles)
    i
    predictedTestAngles(i) = predict(net,testImages(:,:,:,i));
    clc
end

% =========================================================================
if any(isnan(predictedTestAngles))
    disp('There were NaNs predicted');
    return;
end

% =========================================================================

% Calculate the prediction error between the predicted and actual angles of
% rotation.
predictionError = testAngles - predictedTestAngles;

figure;
plot(1:length(testAngles),testAngles,'b-', 1:length(predictedTestAngles),predictedTestAngles,'r-');
ylabel('Angle');
legend('True angle', 'Predicted angle');

[Joss Knight]

Are you running R2016b on a GTX 1080?

[Tobias Pahlberg]

I am running R2017a on a GTX 960, why?

[Joss Knight]

Just checking whether you were suffering from the Pascal cuDNN bug, but clearly you weren't.

[Mity sosi]

Hi

how did you make your trainimages ? how did you make digitTrain4DArrayData ?

Answer 1

[Tobias Pahlberg]

Lowering the initial learn rate to 1e-5 did the trick.

Answer 2

[Greg Heath]

In order to get a variety of answers you have to use a different initial random number state for each design.

So, either use a different rng initialization for each run

OR (my technique)

Use a double for loop to loop over number of hidden nodes (outer loop) and random initial weights (inner loop)

A single rng initialization before the first loop will suffice. I have posted zillions of examples in both the NEWSGROUP and ANSWERS.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Tobias Pahlberg]

I think you misunderstood me. The problem is that the trained network gives the same answer, let's say 15 degrees, for every test image I supply to the function predict. (I updated the question to be more clear.)

[Tobias Pahlberg]

Corrected a bug in my code and now it only predicts NaNs for every image I supply.