How can I detect that which squares are ticked in the image?

How can I detect that which squares are ticked in the image?

I'm think use the normxcorr2 or imshowpair to compare the original image with ticked image, but I not sure how to do this.

Original Image:

Answer 1

[Mark Sherstan]

An almost identical question and solution can be found here

[AlexMT]

Mark thank you

I already know the question and try that solution before. However, my image have the lines between the squares. When i use imfill, some squares will be masked.

I'm try to use bwareaopen or use threshold to remove the line first, but it not work. Do you suggest any method to solve this problem?

[Mark Sherstan]

Here you go :) It is a slightly different approach but basically I took the picture and located all the squares with a red box (file attachted). From there I was able to create a mask and loop through each box and check if there was something inside of it or not. Based off those results I either circled it or left it blank. Let me know if you have any questions.

A couple notes: It would be much easier if you scanned the document instead of taking a picture of it. This would also allow for the results to be repeatable as everything will have the same angles, spacing, perspective, etc... and the mask can always be treated as true. I also had to darken the image slightly as the pen marks where showing up light (may be from posting it online, switching file formats to png, etc...) but again this could easily be fixed by scanning.

clear all
close all

% Read in original and colored mask image with red squares
image = imread('image.png'); original = image;
mask = imread('mask.png');

% Create a mask and apply to image so we can focus on the boxes
BW = createMask(mask);
image(repmat(~BW,[1 1 3])) = 0;
BW2 = imbinarize(rgb2gray(image));

% Get the coordinates for each square
stats = regionprops(BW,'boundingbox','centroid');

% See if there is anything in a square
for ii = 1:numel(stats)
    tempImg = imcomplement(imcrop(BW2,stats(ii).BoundingBox));
    check = round(bwarea(tempImg));
    
    % Only put circle around boxes with something in them 
    if (check > 150)
        original = insertObjectAnnotation(original,'circle',[stats(ii).Centroid 50],'');
    end
end

% Display results
figure
imshow(original)

function [BW,maskedRGBImage] = createMask(RGB)
%createMask  Threshold RGB image using auto-generated code from colorThresholder app.
%  [BW,MASKEDRGBIMAGE] = createMask(RGB) thresholds image RGB using
%  auto-generated code from the colorThresholder app. The colorspace and
%  range for each channel of the colorspace were set within the app. The
%  segmentation mask is returned in BW, and a composite of the mask and
%  original RGB images is returned in maskedRGBImage.

% Auto-generated by colorThresholder app on 14-Dec-2018
%------------------------------------------------------


% Convert RGB image to chosen color space
I = rgb2hsv(RGB);

% Define thresholds for channel 1 based on histogram settings
channel1Min = 0.997;
channel1Max = 0.156;

% Define thresholds for channel 2 based on histogram settings
channel2Min = 0.288;
channel2Max = 1.000;

% Define thresholds for channel 3 based on histogram settings
channel3Min = 0.549;
channel3Max = 1.000;

% Create mask based on chosen histogram thresholds
sliderBW = ( (I(:,:,1) >= channel1Min) | (I(:,:,1) <= channel1Max) ) & ...
    (I(:,:,2) >= channel2Min ) & (I(:,:,2) <= channel2Max) & ...
    (I(:,:,3) >= channel3Min ) & (I(:,:,3) <= channel3Max);
BW = sliderBW;

% Initialize output masked image based on input image.
maskedRGBImage = RGB;

% Set background pixels where BW is false to zero.
maskedRGBImage(repmat(~BW,[1 1 3])) = 0;

end

[AlexMT]

Hey, Mark

Thanks for helping me

I have 2 questions

1. The red squares in mask.png are manually create by yourself ? or you use the code to do it ? ( Because I'm trying to apply your code to other image.)

2. Another is about colorThresholder. I never use this function before. Which color space do you choose to mask HSV or YCbCr ?

[Mark Sherstan]

1. The mak.png was created manually using Photoshop / paint or whatever program you might have.
2. You can be in any color space. The function is autogenerated from a GUI. You can find more information here.

[Image Analyst]

Alex:

The mask approach is a good one if the image is in a fixed location. I was going to do that but then wondered if the image might not be in a fixed location because it almost looks like that image was taken by someone with a hand held camera (like in a phone). In that case you'd have to first find the squares because they might not be in the same location.

We need to know the use case like Mark noted. Does the image wander around because the camera is hand held, or is the image fixed like on a flatbed scanner (as Mark noted) or the camera fixed (like on a tripod or some other mounting jig) and you insert the papers into a template or jig to make sure the papers are also positioned properly. The latter situation (scanner or mounted camera with paper alignment jig) is of course the best situation, rather than hand held. So which do you have?

[AlexMT]

Hey Image Analyst

Now, I only have the image which taken by the hand held camera. (The first image is taken by phone, and cropped the 7*4 table to try to detect.)

May be I need to optimize my method to fixed the squares in the similar location. (like use the tripod or scanner as you and Mark advised)

However, I'm still try to detect which squares are ticked by using the images taken by phone.

Do you have any ideas or suggestions?

[Mark Sherstan]

One thing you could try is use a color marker instead of a pen. Use a color mask to locate a mark and then divide your image into quadrants and check that way instead of looking for specific boxes. I am sure Image Analyst will have some other good suggestions .

[Image Analyst]

It's a little more complicated and you'll have to have labeled training images but you could use a deep learing model to find the squares. Otherwise we'll have to use fancier traditional techniques to find them. Can we guarantee that there will be 28 squares in a 7 row, 3 column format? If so, we could threshold and then find 28 blobs in a certain size range.

Answer 2

[Image Analyst]

You might try the attached script. It automatically finds the boxes and inspects inside for black marks. It still needs work since the boxes are not identified by row and column, and there still needs to be some robustness about inspecting the boxes depending on how it behaves with other images. But at least it's automatically finding your boxes correctly and identifying checked boxes correctly for THIS image.

You might also take a look at a demo I made for finding marked circles in an OMR (Optical Mark Recognition) image.

[AlexMT]

Hey Image Analyst

Thanks for helping and replying!

I'll try this script and attempt to understand all the function inside. :)