Image segmentation using vgg16. Problems in augmenting test images

Hi

I am new to CNN. I was trying to delineate farm boundaries using vgg16. The following is my code:

clear all;

close all;

clc;

vgg16();

imgDir = fullfile(pwd,'TestImages');

imds=imageDatastore(imgDir);

I=readimage(imds,1);

classNames=["lines" "green_farm" "farm_grey"];

pixelLabelID = cell(3,1);

pixelLabelID{1,1} = [2;0];

pixelLabelID{2,1} = 1;

pixelLabelID{3,1} = 3;

labelDir=fullfile(pwd,'image_labelling');

pxds=pixelLabelDatastore(labelDir,classNames,pixelLabelID);

tbl = countEachLabel(pxds);

frequency = tbl.PixelCount/sum(tbl.PixelCount);

figure

bar(1:numel(classNames),frequency)

xticks(1:numel(classNames))

xticklabels(tbl.Name)

xtickangle(45)

ylabel('Frequency')

imageFolder = fullfile(imgDir,'imagesResized',filesep);

imds = resizeBloodSmearImages(imds,imageFolder);

labelFolder = fullfile(imgDir,'labelsResized',filesep);

pxds = resizeBloodSmearPixelLabels(pxds,labelFolder);

[imdsTrain, imdsTest, pxdsTrain, pxdsTest] = partitionCamVidData(imds,pxds);

numTrainingImages = numel(imdsTrain.Files);

numTestingImages = numel(imdsTest.Files);

imageSize = [224 224 3];

numClasses = numel(classNames);

lgraph = segnetLayers(imageSize,numClasses,'vgg16');

imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount;

classWeights = median(imageFreq) ./ imageFreq;

pxLayer = pixelClassificationLayer('Name','labels','ClassNames', tbl.Name, 'ClassWeights', classWeights);

lgraph = removeLayers(lgraph, 'pixelLabels');

lgraph = addLayers(lgraph, pxLayer);

lgraph = connectLayers(lgraph, 'softmax' ,'labels');

options = trainingOptions('sgdm', ...

'Momentum', 0.9, ...

'InitialLearnRate', 1e-3, ...

'L2Regularization', 0.0005, ...

'MaxEpochs', 3000, ...

'MiniBatchSize', 1, ...

'Shuffle', 'every-epoch', ...

'Plots','training-progress', ...

'VerboseFrequency', 1000);

dsTrain = combine(imdsTrain, pxdsTrain);

data = read(dsTrain);

xTrans = [-10 10];

yTrans = [-10 10];

dsTrain = transform(dsTrain, @(data)augmentImageAndLabel(data,xTrans,yTrans));

[net, info] = trainNetwork(dsTrain,lgraph,options);

idx = 2;

I = readimage(imdsTest,idx);

C = semanticseg(I, net);

cmap=colormap('gray');

B = labeloverlay(I, C, 'Colormap', cmap, 'Transparency',0.4);

imshowpair(I, B, 'montage')

pixelLabelColorbar(cmap, classes);

However, I am getting the following error:

Error using trainNetwork (line 184)

Invalid transform function defined on datastore.

Error in trial_2 (line 71)

[net, info] = trainNetwork(dsTrain,lgraph,options);

Caused by:

Error using matlab.io.datastore.TransformedDatastore/read (line 222)

Invalid transform function defined on datastore.

Undefined function 'augmentImageAndLabel' for input arguments of type 'cell'

I am unable to figure out the issue. I followed exactly what is written in https://www.mathworks.com/help/vision/ug/semantic-segmentation-using-deep-learning.html under the Data Augmentation section

xTrans = [-10 10];

yTrans = [-10 10];

dsTrain = transform(dsTrain, @(data)augmentImageAndLabel(data,xTrans,yTrans));

Can someone please help? I will be very graeful for any help on this matter.

Regards

Prateek

Answer 1

[V Sairam Reddy]

Hi Prateek,

I understand that you are following the example - Semantic Segmentation Using Deep Learning but are an issue with the undefined function ‘augmentImageAndLabel’.

Please note that the function 'augmentImageAndLabel' is not a built-in function in MATLAB. You can find the function 'augmentImageAndLabel' at the end of the example mentioned. Hence run the function before using it.

If the error persists, attach a sample image and its corresponding mask to make the debugging easier.

[prateek gantayat]

Thank you for the reply. Right now I am getting the following error:

Error using trainNetwork (line 184)

Invalid training or validation response data. Categorical responses must either be a vector or a single-channel

2-D or 3-D image.

Error in trial_2 (line 71)

[net, info] = trainNetwork(dsTrain,lgraph,options);

[prateek gantayat]

these are the test images

[V Sairam Reddy]

Hi Prateek,

Please provide the Labels and the function code for 'resizeBloodSmearImages' and 'resizeBloodSmearPixelLabels'.

It would be helpful if you could provide your updated code in 'file.m' format or in the code section while asking a query. Please update your code alongside when asking a follow-up query.

[prateek gantayat]

Here are the functions resizeBloodSmearImages.m and resizeBloodSmearPixelLabels.m. All the images that I used for labelling have been attached. The updated code is here:

clear all;

close all;

clc;

vgg16();

imgDir = fullfile(pwd,'TestImages');

imds=imageDatastore(imgDir);

I=readimage(imds,1);

classNames=["lines" "green_farm" "farm_grey"];

pixelLabelID = cell(3,1);

pixelLabelID{1,1} = [2;0];

pixelLabelID{2,1} = 1;

pixelLabelID{3,1} = 3;

labelDir=fullfile(pwd,'image_labelling');

pxds=pixelLabelDatastore(labelDir,classNames,pixelLabelID);

tbl = countEachLabel(pxds);

frequency = tbl.PixelCount/sum(tbl.PixelCount);

figure

bar(1:numel(classNames),frequency)

xticks(1:numel(classNames))

xticklabels(tbl.Name)

xtickangle(45)

ylabel('Frequency')

imageFolder = fullfile(imgDir,'imagesResized',filesep);

imds = resizeBloodSmearImages(imds,imageFolder);

labelFolder = fullfile(imgDir,'labelsResized',filesep);

pxds = resizeBloodSmearPixelLabels(pxds,labelFolder);

[imdsTrain, imdsTest, pxdsTrain, pxdsTest] = partitionCamVidData(imds,pxds);

numTrainingImages = numel(imdsTrain.Files);

numTestingImages = numel(imdsTest.Files);

imageSize = [224 224 3];

numClasses = numel(classNames);

lgraph = segnetLayers(imageSize,numClasses,'vgg16');

imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount;

classWeights = median(imageFreq) ./ imageFreq;

pxLayer = pixelClassificationLayer('Name','labels','ClassNames', tbl.Name, 'ClassWeights', classWeights);

lgraph = removeLayers(lgraph, 'pixelLabels');

lgraph = addLayers(lgraph, pxLayer);

lgraph = connectLayers(lgraph, 'softmax' ,'labels');

options = trainingOptions('sgdm', ...

'Momentum', 0.9, ...

'InitialLearnRate', 1e-3, ...

'L2Regularization', 0.0005, ...

'MaxEpochs', 3000, ...

'MiniBatchSize', 1, ...

'Shuffle', 'every-epoch', ...

'Plots','training-progress', ...

'VerboseFrequency', 1000);

dsTrain = combine(imdsTrain, pxdsTrain);

data = read(dsTrain);

xTrans = [-10 10];

yTrans = [-10 10];

dsTrain = transform(dsTrain, @(data)augmentImageAndLabel(data,xTrans,yTrans));

doTraining = true;

if doTraining

[net, info] = trainNetwork(dsTrain,lgraph,options);

end

idx = 2;

I = readimage(imdsTest,idx);

C = semanticseg(I, net);

cmap=colormap('gray');

B = labeloverlay(I, C, 'Colormap', cmap, 'Transparency',0.4);

imshowpair(I, B, 'montage')

pixelLabelColorbar(cmap, classes);