What is the image normalization procedure for trainNetwork?

I'm attempting to train a semantic segmentation network (vgg19) on 1.4x10^6 images and the 'initializing image normalization' phase has been ongoing for 12 hours and counting.

This seems unreasonably long (at 12 hours the processing rate would be 32 images / second).

Prior to this step I ran a script which queried every pixel value in every image and created a new categorical uint8 image at 500 images / second.

1. Why is this part of the process so slow?

2. What is the normalization procedure? (so I can normalize prior to running trainNetwork).

3. Having done #2, how can I skip the image normalization step in trainNetwork?

Thanks!

System: Ubuntu 18.04, Matlab 2018a, i7-6950x, 128GB RAM, 3X Titan V.

[Louis Vaickus]

I've tracked normalization down to (I think?) 4 Functions:

Trainer.m

->initializeNetworkNormalization

-->TrainerGPUStrategy.m

-->Precision.m

Within Trainer.m the function initializeNetworkNormalization contains two lines:

avgI = this.ExecutionStrategy.computeAverageImage(data, augmentations, executionSettings);
net.Layers{1}.AverageImage = precision.cast(avgI);

this.ExecutionStrategy = nnet.internal.cnn.TrainerGPUStrategy:

classdef TrainerGPUStrategy < nnet.internal.cnn.TrainerExecutionStrategy
   % TrainerGPUStrategy   Execution stategy for running the Trainer on the
   % GPU

     %   Copyright 2016 The Mathworks, Inc.

      methods

          function Y = environment(~, X)
              Y = gpuArray(X);
          end        

          function [avgI, numImages] = computeAccumImage(~, data, augmentations)
              data.start();
              avgI = gpuArray(0);
              numImages = 0;
              while ~data.IsDone
                  X = data.next();
                  if ~isempty(X)
                      X = apply(augmentations, X);
                      X = gpuArray(double(X));
                      avgI = avgI + sum(X, 4);
                      numImages = numImages + size(X,4);
                  end
              end
          end

      end

end

I'm no expert but I'm assuming X = data.next() is pulling the individual images and keeping a running tally of average pixel values and the number of images (to divide by?)? In which case I can't see why this should take so long?

[Joss Knight]

Are you using an augmentedImageDatastore with BackgroundDispatch set to true?

To skip the normalization you need to replace your input layer with a new input layer, but with 'Normalization' set to 'none'.

layers(1) = imageInputLayer([227 227 3], 'Normalization', 'none');

Answer 1

[Youshan Zhang]

Did you solve this problem?

[Louis Vaickus]

As I recall, it was zero center normalization.