Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)

I want to train the model in .mat dataset but i am getting the memory error my dataset size is [256,340,2] when i try

 gpuDevice(1)

ans = 

  CUDADevice with properties:

                      Name: 'NVIDIA GeForce GTX 1080 Ti'
                     Index: 1
         ComputeCapability: '6.1'
            SupportsDouble: 1
             DriverVersion: 11.4000
            ToolkitVersion: 11
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 1.1811e+10
           AvailableMemory: 1.0615e+10
       MultiprocessorCount: 28
              ClockRateKHz: 1620000
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
           DeviceAvailable: 1
            DeviceSelected: 1

code is

            location = 'D:\data-11\sir task\dataset\';
imds = imageDatastore(location, 'FileExtensions', '.mat', 'IncludeSubfolders',1, ...
                      'LabelSource','foldernames',...
                      'ReadFcn',@matReader);

[imdsTrain,imdsValidation] = splitEachLabel(imds,0.7, 'randomized');

net = lgraph_1;

inputSize = lgraph_1.Layers(1).InputSize;

[learnableLayer,classLayer] = findLayersToReplace(lgraph_1);
[learnableLayer,classLayer] 

numClasses = numel(categories(imdsTrain.Labels));

if isa(learnableLayer,'nnet.cnn.layer.FullyConnectedLayer')
    newLearnableLayer = fullyConnectedLayer(numClasses, ...
        'Name','new_fc', ...
        'WeightLearnRateFactor',10, ...
        'BiasLearnRateFactor',10);
    
elseif isa(learnableLayer,'nnet.cnn.layer.Convolution2DLayer')
    newLearnableLayer = convolution2dLayer(1,numClasses, ...
        'Name','new_conv', ...
        'WeightLearnRateFactor',10, ...
        'BiasLearnRateFactor',10);
end

lgraph_1 = replaceLayer(lgraph_1,learnableLayer.Name,newLearnableLayer);

newClassLayer = classificationLayer('Name','new_classoutput');
lgraph_1 = replaceLayer(lgraph_1,classLayer.Name,newClassLayer);

miniBatchSize = 128;
valFrequency = floor(numel(imdsTrain.Files)/miniBatchSize);
checkpointPath = pwd;
options = trainingOptions('sgdm', ...
    'MiniBatchSize',miniBatchSize, ...
    'MaxEpochs',100, ...
    'InitialLearnRate',1e-3, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',valFrequency, ...
    'Verbose',false, ...
    'Plots','training-progress', ...
    'CheckpointPath',checkpointPath);

net = trainNetwork(imdsTrain,lgraph_1,options);

I have also tried to change the batch size to 8 but it does't work.

Answer 1

[yanqi liu]

sir，may be use

miniBatchSize = 128;

to

miniBatchSize = 1;

[john karli]

have tried but not working

[yanqi liu]

options = trainingOptions('sgdm', ...
    'MiniBatchSize',1, ...
    'MaxEpochs',100, ...
    'InitialLearnRate',1e-3, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',valFrequency, ...
    'Verbose',false, ...
    'Plots','training-progress', ...
    'CheckpointPath',checkpointPath, ...
	'ExecutionEnvironment','cpu');

[john karli]

why you select executuonEnvironment 'CPU'?

[yanqi liu]

yes，sir，may be the gpu memory not enough，so use cpu to test