Partially labelled semantic segmentation

Hi, I'm trying to use partially labelled images as a groud truth for the semantic segmentation training(there are lots of ambiguous regions in my images, so i am hoping to train only with the apparent regions; by inputting the class weight 0(techinically it was 10^-20 as I cannot input zero) to the unlabeled class in pixel classification layer)

And I found that the mini-batch accuracy is fluctuates around 35-40% and never converges.

Are there any ways i could fix this problem?

Is it related to excessive non-labeled regions?

I would appreciate any of your advice concerning this problem.

Answer 1

[Raunak Gupta]

Hi,

Assigning zero class weight to unlabeled pixel will help in removing them from loss calculation during training however when the accuracy metrics are calculated after training, it will include all the pixels and thus giving random categories to unlabeled pixel (I am assuming there is more than 1 class in labeled pixels).

Instead I will recommend assigning a background category to unlabeled pixel and keeping the class weight same as labeled pixel so that network can assign proper categories to all the pixels (background pixel can also be trained). This way network will be more robust for all parts of the image. So, adding one more class named background to the previous classes may help you converge to a higher accuracy.

[byungchan]

Thank you for your answers Gupta,

What I actually want to do is to classify the pixel labels for the ambiguous region(unlabelled region), using a network trained only with the apparent regions(labeled region).

However, if I assign a "background" label to the unlabelled regions, and train with those images, semantic segmentation results from the trained network would segment those ambiguous areas as a "background" rather than specific categories that I wish to verify(my goal is to see those ambiguous regions were classified as one of the categories that I have predefined).

So my question is this.

1. When calculating the mini-batch accuracy during the network training, does it accounts for all the regions in the image? (i.e. if the network assigns certain categories(my case "wet sand", "dry sand") to the non labeled region("NaN" label), would it affect the mini-batch accuracy during training?)
2. If that is the case, is there any possible way that I could see the mini-batch accuracy calculated only by using the labeled regions?

Thank you very much in advance!

[Raunak Gupta]

Hi,

I understand that you are trying to assign labels to unlabeled data in above formulation. However it is not possible to get information whether those labels are correct or not because there is no ground truth available for those pixels.

So, the mini batch accuracy will be calculated for all the pixels and if a category is NaN, it will be treated as miss classification. I think that is the reason for so low mini batch accuracy.

For the second question, while mentioning the Validation data, you can pass only those pixel in the image which are labeled and can ignore the rest of the ambigous pixels. Since you need to give X (input data) and Y (Response variable) into the ValidationData , It is possible to remove the unlabeled pixel using array indexing.

This will not change the mini batch accuracy, but you will get the desired Validation accuracy as only labeled data is passed as Validation data.

Hope this clarifies!

[byungchan]

Thank you very much, this has helped a lot!

[byungchan]

Hello Gupta, I was working on this after I've got your tips, and I've got several more things to ask which I hope you don't mind.

1.I don't know the reason why the training loss have high loss function(always converges over 0.5 after only around 5 epochs, even if I tried numerous trials with different initial rate and different learning rates) when training with partially labeled images.

I can understand that the training accuracy could be small, as it accounts for all the pixels in the image, but I still don't know the reason for this high loss function.

Do you have any clues for this? Or do you think it's enough as it shows convergence?

Also, I don't know how the Matlab calculates its cross-entropy loss for images with unlabeled regions.

Would the sum of loss functions for unlabeled regions calculated as zero in the Matlab?

2. In regards to question 1, and your advice which you've given previously, I tried to make the unlabeled region in the original image as NaN(it automatically becomes 0 and the unlabeled regions become black) and pass it through the trained network, using 'ValidationData'(though it is validation, I used the same image as with the training to know the "true" training accuracy).

But the accuracy is even lower than 30% which I think still accounts for the "unlabeled region."(This is clear that if I use the function "evaluateSemanticSegmentation" manually, the class accuracy for each label is mostly over 80-90%.) Loss function, on the other hand, is around zero which is quite surprising that I thought it would be around 0.5. (Hence, I am asking above question #1.)

So is there anyway I could fix this problem? What I really want to do is to verify the "true" training accuracy and loss function (which only accounts the labeled regions) in "real-time" during the training, so that I can validate the training with some confidence.

Thank you very much in advance and have a wonderful day!

[Raunak Gupta]

Hi,

For both the queries I think I can conclude that the NaN (or simply unlabeled) pixels are used for calculating metric while doing the training in real time. I am not aware of any workflows as of now to exclude the unlabeled data from calculating the metric.

So, I would recommend checking the loss and accuracy through the custom functions you mentioned after any training run is finished. This way you can be sure at the end about the model performance.

For the first query about the convergence I would say that you can stop the training if loss has not been changing for 5-10 epochs since it is good sign that you have arrived to a local minimum in terms of loss.

[byungchan]

Thank you!

[byungchan]

After I've installed 2020b the accuracy problem has been resolved!

Thanks again!