how to segment a tray image with regular pocket pattern ?

Hello everyone,

I am working on a project which need to find how many pockets (rectangular or square shapes) in a tray image and location of those pockets.

The image is acquired by a webcam with quite uniform illumination. The pockets have same size for the same type of tray but for different tray their size are varied. Please see below link for the image:

https://www.dropbox.com/s/6q1sf294z05ull2/EmptyTrayImage.jpg

Can anyone advise a robust way to segment those pockets out? As the color of the tray may also vary, what is the best approach to handle do such problem?

Thank you very much!

Best regards,

Di

Answer 1

[Image Analyst]

Your illumination may be uniform but your image is not. If your illumination is uniform then you have bad vignetting (shading) due to your lens. Take a blank shot and then fit a 2D polynomial to it, then divide your images by the modeled, perfect background to "flat field" your image.

To find the pockets, simply sum your image vertically and horizontally. Then threshold to find the rows and columns that have dark stuff and bright stuff in them. Knowing the rows and columns locations you can easily find the locations of the rectangles defining the pockets, and of course the count also. Give that a try.

[Image Analyst]

Haven't heard back from you so it looks like you ran into some trouble. I whipped this together in about 10 minutes. I think you can easily adapt it.

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
imtool close all;  % Close all imtool figures if you have the Image Processing Toolbox.
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 11;

% Check that user has the Image Processing Toolbox installed.
hasIPT = license('test', 'image_toolbox');
if ~hasIPT
  % User does not have the toolbox installed.
  message = sprintf('Sorry, but you do not seem to have the Image Processing Toolbox.\nDo you want to try to continue anyway?');
  reply = questdlg(message, 'Toolbox missing', 'Yes', 'No', 'Yes');
  if strcmpi(reply, 'No')
    % User said No, so exit.
    return;
  end
end

% Read in a standard MATLAB gray scale demo image.
folder = 'C:\Users\Di\Documents\Temporary';
baseFileName = 'EmptyTrayImage.jpg';
% Get the full filename, with path prepended.
fullFileName = fullfile(folder, baseFileName);
% Check if file exists.
if ~exist(fullFileName, 'file')
  % File doesn't exist -- didn't find it there.  Check the search path for it.
  fullFileName = baseFileName; % No path this time.
  if ~exist(fullFileName, 'file')
    % Still didn't find it.  Alert user.
    errorMessage = sprintf('Error: %s does not exist in the search path folders.', fullFileName);
    uiwait(warndlg(errorMessage));
    return;
  end
end
grayImage = imread(fullFileName);
% Get the dimensions of the image.  
% numberOfColorBands should be = 1.
[rows, columns, numberOfColorBands] = size(grayImage);
if numberOfColorBands > 1
  % It's not really gray scale like we expected - it's color.
  % Convert it to gray scale by taking only the green channel.
  grayImage = grayImage(:, :, 2); % Take green channel.
end
% Display the original gray scale image.
subplot(3, 3, 1);
imshow(grayImage, []);
axis on;
title('Original Grayscale Image', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);
% Give a name to the title bar.
set(gcf,'name','Demo by ImageAnalyst','numbertitle','off') 

% Let's compute and display the histogram.
[pixelCount, grayLevels] = imhist(grayImage);
subplot(3, 3, 2); 
bar(pixelCount);
grid on;
title('Histogram of original image', 'FontSize', fontSize);
xlim([0 grayLevels(end)]); % Scale x axis manually.

% Determine horizontal shading by looking between columns 50 and 575
% and between rows 25 and 225
% Note: you can also try this method: http://blogs.mathworks.com/steve/2013/06/25/homomorphic-filtering-part-1/
subImage = grayImage(25:225, 50:575);
% Get horizontal profile.
shadingProfile = sum(subImage, 1);
% Fit it to a quadratic
coeffs = polyfit(1:length(shadingProfile), shadingProfile, 2);
% extrapolate to the full size
x = (1 : columns) - 25;
shadingCorrection = polyval(coeffs, x);
% Normalize
shadingCorrection = shadingCorrection / max(shadingCorrection);
subplot(3, 3, 3);
plot(shadingCorrection);
grid on;
title('Horizontal Shading Correction Factor', 'FontSize', fontSize);

% Create a perfect background image estimate.
backgroundCorrectionImage = repmat(shadingCorrection, [rows, 1]);
subplot(3, 3, 4);
imshow(backgroundCorrectionImage, []);
title('Horizontal Shading Correction Image', 'FontSize', fontSize);

% Background correct the image
grayImage2 = double(grayImage) ./ backgroundCorrectionImage; 
subplot(3, 3, 5);
imshow(grayImage2, []);
title('Background Corrected Image', 'FontSize', fontSize);

% Get vertical profile
verticalProfile = sum(grayImage2, 2) / columns;
subplot(3, 3, 6);
plot(verticalProfile);
grid on;
title('Vertical Profile', 'FontSize', fontSize);
% Get horizontal profile.
horizontalProfile = sum(grayImage2, 1) / rows;
subplot(3, 3, 7);
plot(horizontalProfile);
grid on;
title('Horizontal Profile', 'FontSize', fontSize);

% Now threshold at 240
rowsWithPocket = verticalProfile > 240;
columnsWithPocket = horizontalProfile > 240;
% Get image with bands
horizontalBands = repmat(rowsWithPocket, [1, columns]);
verticalBands = repmat(columnsWithPocket, [rows, 1]);
% And them together to find where both rows and columns are true.
pocketImage = horizontalBands & verticalBands;
% Get rid of noise which are regions with an area less 
% than the known minimum pocket size (say, 400 pixels).
pocketImage = bwareaopen(pocketImage, 400);
subplot(3, 3, 8);
imshow(pocketImage, []);
title('Pockets', 'FontSize', fontSize);

% Count them and label the image
[labeledImage numberOfPockets] = bwlabel(pocketImage);
% Get bounding boxes
measurements = regionprops(labeledImage, 'BoundingBox', 'Area');
allAreas = [measurements.Area];
boundingBoxes = [measurements.BoundingBox];

% Display over original image
subplot(3, 3, 9);
imshow(grayImage);
hold on;
title('Pockets over original image', 'FontSize', fontSize);
for b = 1 : numberOfPockets
  thisBoundingBox = measurements(b).BoundingBox;
  rectangle('Position', thisBoundingBox, 'EdgeColor', 'r');
end
message = sprintf('Done!\nThere are %d pockets.', numberOfPockets);
msgbox(message);

[Di]

Thank you so much, Image Analyst!

[Di]

But it seems that in the code you set a fixed threshold of 240 in horizontal and vertical directions to segment the pockets. It may not work for some tray image with different brightness, for example

https://www.dropbox.com/s/hr6pun7u04wa51j/EmptyTrayImage2.jpg

Is there a way to set the threshold adaptive?

Thank you again for your help!

[Image Analyst]

Yes. That's where the art of image processing comes in. You can devise some algorithm to automatically threshold. It might take looking at a lot of histograms and trying to figure out what is it about the shape of the histogram that you can use to arrive at the best threshold.

[Di]

Image Analyst, thank you for your helps! I really appreciate it.

https://www.dropbox.com/s/oz8q0ois79mbxl6/EmptyTray2_result.JPG

The above link is the result of 2nd tray image. There are a few reduntant small regions. To remove them, I am thinking to

1. smooth('rloess') the x and y profile data, find all composed bw areas using a threshold of 0.9* max(profile_value); 2. based on the size of areas, use k-means clustering to segment a set of max areas as the pocket regions.

Do you think will this approach be robust?

[Image Analyst]

It looks like you have serious problems getting a good image. Why can't you get a consistently good image? Anyway, it looks like the tray is in a jig (holder), so why can't you just use fixed, known positions?