Error in using trainNetwork with training data and labels

I am trying to classify image datasets using deep learning.

after getting feature vector of each single image I ve got a matrix 18000x24000 which indicates to No. of images x features.

I used:

trainNetwork (X, Y, Layers, Options)

Where X is the train data and Y is the Labels which is 18000x1. But there is an error says Invalid training data. X and Y must have the same number of observations.

I think I should change the train matrix to 4-D but I don't know how, and if it is correct?

[Walter Roberson]

Try

trainNetwork(X.', Y, Layers, Options)

You need to watch out for whether features are to go along rows or along columns.

[Ali Yar Khan]

i have row of feature in data and one distinct value in label for each row of data ... my network is also giving me the same error ... here is my code.

%loading train and test data
trainingData = load('train.mat');
trainData = trainingData.data;
trainLabels = categorical(trainingData.labels);
testingData = load('test.mat');
testData = testingData.data;
testLabels = testingData.labels;

% Define Network Architecture
% Define the convolutional neural network architecture.

layers = [
    imageInputLayer([1 144 1]) % 22X1X1 refers to number of features per sample
    convolution2dLayer(3,16,'Padding','same')
    reluLayer
    fullyConnectedLayer(384) % 384 refers to number of neurons in next FC hidden layer
    fullyConnectedLayer(384) % 384 refers to number of neurons in next FC hidden layer
    fullyConnectedLayer(2) % 2 refers to number of neurons in next output layer (number of output classes)
    softmaxLayer
    classificationLayer];
options = trainingOptions('sgdm',...
    'MaxEpochs',500, ...
    'Verbose',false,...
    'Plots','training-progress');

%train the network
net = trainNetwork(trainData,trainLabels,layers,options);

%test the network
predictedLabels = classify(net,trainD)'
accuracy = sum(predictedLabels == testLabels)/numel(testLabels)

It gives me this error

Invalid training data. X and Y must have the same number of observations.

Error in convnet1d (line 27)
net = trainNetwork(trainData,trainLabels.',layers,options);

Caused by:
    Error using nnet.internal.cnn.util.NetworkDataValidator/assertXAndYHaveSameNumberOfObservations (line 142)
    Invalid training data. X and Y must have the same number of observations.

Any help regarding this . I have this work space ... that may help you about the dimensions of data

[Hamza]

Hi, have you solved the issue?

Answer 1

[Bhartendu]

Try the following:

If your data ( 18000 data points) is in form of images with dimensions say, 120*200 (equals to 24000), then reshape it as follows:

X_train = reshape(X, [120,200,1,size(X,1)]);

This should create 4-D Matrix X_train of size (120,200,1,18000), then train the network using:

net = trainNetwork(X_train,Y,Layers,Options)

[Radians]

I have done exactly that but to no avail.

I

where 'X_1' is my X and 'correspondingvalues' is my Y. basically, x is a 16*9500 items collection of 16*16 images. Y is their labels.

Please help.

Answer 2

[US]

I am also getting the same error (my layers structure is a prediction CNN):

>> whos X Y

Name Size Bytes Class Attributes

X 64x64x52x1800 3067084800 double

Y 1800x1 14400 double

>> MyPrediction = trainNetwork(X, Y, layers, opts);

Error using trainNetwork (line 165)

Number of observations in X and Y disagree.

It gives me the same error when I transpose Y, or when I reshape X to [64*64,52,1,1800].

And I have:

MATLAB Version: 9.6.0.1072779 (R2019a)

I am starting to think that this MATLAB function trainNetwork or a sub-function that it calls is somehow not behaving properly or I am missing something.

Any help will be appreciated.

Answer 3

[michael scheinfeild]

hi i have similar issue and cant solve it i also look at the example Sequence Classification Using Deep Learning

i try to classify between different signals based frequency future

%%generete the signal
fs=500;
f=20;
t=[0:1/fs:5-1/fs];
xsig=10*sin(2*pi*f*t);
sampleLen=length(xsig);
NFFT =  2.^nextpow2(sampleLen);

% signal fft
hwin=hamming(length(xsig))';
Y = fft(xsig.*hwin,NFFT);
magnitudeY = abs(Y); 
xsampleFFT=20*log10(magnitudeY(1:NFFT/2));
figure,plot(xsampleFFT);title('signal fft')

% noise fft
xnoise= randn(size(xsig));
Yn = fft(xnoise.*hwin,NFFT);
magnitudeYn = abs(Yn); 
xnoiseFFT=20*log10(magnitudeYn(1:NFFT/2));
figure,plot(xnoiseFFT);title('noise')

%%make the data
sampleLen = 100;
NFFT   = 128;
hwin=hamming(sampleLen)';

nsamples=length(xsig)/sampleLen;

xsampleFFT={};%zeros(nsamples,NFFT/2);
hwin=hamming(sampleLen)';
kj=1;

for(k=1:sampleLen:length(xsig)-sampleLen+1)

      cursig=xsig(k:k+sampleLen-1);
      Y = fft(cursig.*hwin,NFFT);
      magnitudeY = abs(Y);        % Magnitude of the FFT

      xTrain{kj}=20*log10(magnitudeY(1:NFFT/2));
       %figure,plot(xTrain{kj})
      yTrain(kj)=categorical(1);
      kj=kj+1;

end

disp('data types')
[size(xTrain) size(yTrain)]
[size(xTrain{1})]
class(xTrain) 
class(yTrain)

%append
for(k=1:sampleLen:length(xnoise)-sampleLen+1)

      cursig=xnoise(k:k+sampleLen-1);
      Y = fft(cursig.*hwin,NFFT);
      magnitudeY = abs(Y);        % Magnitude of the FFT

      xTrain{kj}=20*log10(magnitudeY(1:NFFT/2));
       %figure,plot(xTrain{kj})
      yTrain(kj)=categorical(0);
      kj=kj+1;

end

disp('data types')
[size(xTrain) size(yTrain)]
[size(xTrain{1})]
class(xTrain) 
class(yTrain)

figure,plot(yTrain)
figure,subplot(2,1,1),plot(xTrain{10})
subplot(2,1,2);plot(xTrain{30})

function [net] = train_lstm(XTrainLoc,YTrainLoc)

%%lstm

inputSize = 1;
numHiddenUnits = 100;
numClasses = 2;
layers = [ ...
    sequenceInputLayer(inputSize)
    lstmLayer(numHiddenUnits,'OutputMode','sequence')
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

maxEpochs = 1;
miniBatchSize = 100;

options = trainingOptions('adam', ...
    'ExecutionEnvironment','cpu', ...
    'GradientThreshold',1, ...
    'MaxEpochs',maxEpochs, ...
    'MiniBatchSize',miniBatchSize, ...
    'SequenceLength','longest', ...
    'Shuffle','never', ...
    'Verbose',0, ...
    'Plots','training-progress');

% options = trainingOptions('adam', ...
%     'ExecutionEnvironment','cpu', ...
%     'GradientThreshold',1, ...
%     'MaxEpochs',maxEpochs, ...
%     'MiniBatchSize',miniBatchSize, ...
%     'SequenceLength','longest', ...
%     'Shuffle','never', ...
%     'Verbose',0, ...
%     'Plots','training-progress',...
%     'ValidationData',{XValidation,YValidation},...
%     'ValidationPatience',Inf);

%%train
net = trainNetwork(XTrainLoc,YTrainLoc,layers,options);

====== i recive error

[net] = train_lstm(xTrain,yTrain)
*Error using trainNetwork (line 154)
Invalid training data. If all recurrent layers have output
mode 'sequence', then the responses must be a cell array of
categorical sequences, or a categorical sequence.*

_Error in train_lstm (line 42) net = trainNetwork(XTrainLoc,YTrainLoc,layers,options);

Caused by: Error using nnet.internal.cnn.util.NetworkDataValidator/assertOutputModeCorrespondsToDataForClassification (line 380) Invalid training data. If all recurrent layers have output mode 'sequence', then the responses must be a cell array of categorical sequences, or a categorical sequence._

so what is the issue i tried also change y to cell array of category , transpose the internal x, change network in. i think in this fft i have actually one sample each time with nfft feature. this is the same as the Japanese sample but they have 12 features

[Hamza]

Hi, have you solved the issue?

Answer 4

[SARAH LONER]

sir i also have the same issue cant able to train network for an image .

my work was image segmentation based on unet segment cant able to clear the code

it showing error at

net = trainNetwork(imds,layers,options);

where in imds i have taken image

kindly help to solve this error

[Walter Roberson]

Please show the complete error message

[SARAH LONER]

inp=b % my input image is dicom image

DatasetPath=fullfile('C:\Users\Desktop\to');

imds=imageDatastore(DatasetPath, 'IncludeSubfolders', true,...

'LabelSource','foldernames','fileextension',{'.dcm'});

labelDir = fullfile(DatasetPath,'testImages');

I = readimage(imds,1);

I = histeq(I);

imshow(I)

classes = [

"MALIGNANT","BENIGN"

];

labelIDs=[255 0]

inputlayer = imageInputLayer([512 512 1],'Name','inp')

numFilters = 64;

numLayers = 16;

layers = [ ...

imageInputLayer([512 512 1])

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

transposedConv2dLayer(4,numFilters,'Stride',2,'Cropping',1);

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

transposedConv2dLayer(4,numFilters,'Stride',2,'Cropping',1);

convolution2dLayer(5,20)

batchNormalizationLayer

reluLayer

convolution2dLayer(5,20)

fullyConnectedLayer(4)

softmaxLayer

pixelClassificationLayer

]

pxds = pixelLabelDatastore(labelDir,classes,labelIDs);

options = trainingOptions('sgdm', ...

'InitialLearnRate',0.01, ...

'MaxEpochs',1, ...

'Shuffle','every-epoch', ...

'ValidationFrequency',30, ...

'Verbose',false);

ds = pixelImageDatastore(imds,pxds);

net=trainNetwork(ds,layers,options);

I = read(imds);

C = read(pxds)

C = semanticseg(I, net);

% Overlay pixel label data on the image and display.

B = labeloverlay(I, C);

figure(12)

imshow(B)

i got error at

ds = pixelImageDatastore(imds,pxds);

once i cleared that error means again i got error at

net=trainNetwork(ds,layers,options);