It's likely that your custom region proposal method is producing ROIs that are too small to process. The minimum size that we can currently process is limited by the amount of downsampling the network does prior to the last max pooling layer. (Note that we are going to remove this limitation in a future release).
By default, with alexnet the minimum size is 105x105. Knowing which network you're using for training would help determine if this is the cause of the error.
If you find that this is the cause, then you can expand small ROIs returned by your region proposal method so that they are above the minimum. Alternatively, you can resize your training images so that your objects are larger. But you would have to be careful not to make the image too large because this can increase the amount of GPU memory required to process the data. If this happens you can try to resize the image and then crop around the object.
Or you can change the network itself by reducing the output size of the final max pooling layer. For example, with alexnet you can change the last max pooling layer's pool size to 5 so that the output feature map is smaller. This changes the minimum size to 88-by-88:
net = alexnet;
layers = net.Layers;
% Reduce output size of final max pooling layer by increasing pool size to 5.
% This changes the minimum size to 88-by-88.
layers(16) = maxPooling2dLayer(5,'stride',2);
% reset fully connected layers because of the size change.
% Note: This may not be the ideal set of layers and might require some experimentation
% to figure out the best number of layers after making this change to the max pooling
% layer.
layers(17) = fullyConnectedLayer(4096);
layers(20) = fullyConnectedLayer(4096);
layers(23) = fullyConnectedLayer(2)
layers(end) = classificationLayer()
Figuring out which approach to use might require some experimentation with the layers that follow the final max pooling layer.
The other cause of this error could be that the PositiveOverlapRange and NegativeOverlapRange are set to values that make it harder to find training samples that overlap with your custom region proposals. For example, large positive overlap values could make it impossible for any of your region proposals to overlap with the ground truth.
