How to extract the features of an image frame and obtain the feature vector?

Hi, I am doing my final year B.Tech project on abnormality activity detection in video using MATLAB. I wanted to know how to extract features from all the video frames and make a feature vector. Can anyone please help?

Thank You.

Answer 1

[Walter Roberson]

npix = 20;
nframe = 50;

FF = read first frame  

N = numel(FF);
featidx = randperm(N, npix);   %that is, 20 random indices for the image array

FV(:,1) = FF(featidx);

for framenum = 2 : nframe
  FF = read frame #framenum
  FV(:,framenum) = FF(featidx);
end

You now have a feature vector for each frame; FV will be npix by nframe with the feature vector for each frame going down the columns.

You might be thinking, 'Randomly selected pixels can't be a feature!!", but that would be incorrect. Every deterministic numeric function applied to the data matrix can be a feature. Taking the log of the average hue can be a feature. Taking the maximum intensity of the Green channel can be a feature. The function just has to be consistent about what it does.

The trick with image analysis is not extracting features: it is know what features to extract. And there is no one answer for that: features that are useful in one situation might be meaningless in another situation.

It is very common to extract a bunch of different features for each image, and then to do "feature selection", which is a process of figuring out which of the features is most useful in distinguishing between the various classes of data. It even turns out that random selection can be important ("compressive sensing") so do not dismiss it out of hand.

[Yashaswini MU]

Thank U very much Sir. Your code is working. Is this equivalent to extracting features using radon transform and PCA?

[Walter Roberson]

radon is another deterministic function that can be used for extracting features. PCA is not: it is a technique for deciding which combination of data is important.

[Qazi Arbab Ahmed]

@Walter Roberson, This code doesn't work for me. I have some normal and abnormal thermal images/frames. I want to make data set first and this want to train those. What will be the features that will effect most in detection should I extract. Then Applying different machine learning techniques I want to find out the normal and abnormal frames. Could you please explain how can I do this.

Thanks

[Walter Roberson]

"What will be the features that will effect most in detection should I extract."

So you have some thermal images. Are those:

1. satellite images for remote sensing of vegetation? http://www.gers.uprm.edu/geol6225/pdfs/08_rs_vegetation.pdf
2. vein images of human hands? http://www.sersc.org/journals/JSE/vol6_no1_2009/6.pdf
3. thermal images of circuit boards? http://makezine.com/2016/02/23/troubleshoot-circuitry-diy-magnified-thermal-imaging/
4. thermal images of buildings? http://www.greenbuildingadvisor.com/blogs/dept/musings/introduction-thermal-imaging and http://www.irinfo.org/05-01-2005-grossman/
5. thermal images of breasts for cancer detection? http://www.bmj.com/rapid-response/2011/11/03/digital-infrared-thermal-imaging-breast-cancer-detection
6. thermal images of insects? https://www.youtube.com/watch?v=Ahg14zTF-3o

The features you extract in each of these cases are unlikely to be the same.

[Qazi Arbab Ahmed]

Well, I have thermal Images of a Circuit. I am gonna attach you an image. Actually I have a thermal video from where I have extracted these frames. Thanks

[Walter Roberson]

Shapes are not going to help all that much because components can be any shape (though they would tend to be roughly rectangular.)

Temperature differences relative to the surroundings are not in themselves going to help all that much because it is normal for components to be hot.

This square running at 38C might be abnormally hot, but that square running at 38C might be abnormally cold.

When you look at the images how do you know what is abnormal?

[Qazi Arbab Ahmed]

Authentic Images have known temperature variations while intruded/ Malicious Images have different Temperature variations. So I have to differentiate between Authentic and Intruded Images. Thanks

[Walter Roberson]

So why bother with features? Why not just check the temperature variations?

[Qazi Arbab Ahmed]

Actually, Just getting thermal maps, one can't see temperature and difference is not known. So I want to apply machine learning, let say SVM to classify between normal and abnormal images. For that I need features like pixel Intensity so that I can Differentiate more than 1000 images using SVM. How can I extract these features and will be I able to use them for SVM. Please do let me know if you have a code or so. Thanks

[Walter Roberson]

I am somewhat unclear on how you could have a thermal map but not know the temperature? Unless, that is, that the thermal maps were colored without a provided temperature key but the absent temperature key was known to be consistent between the images??

You can use rgb2gray to get pixel intensity for one sense of the word "intensity". However, if you do not have a temperature key, differences in the numbers you get from that do not have any relative meaning, and they might not have any absolute meaning either since it is possible for two different RGB colors to map to the same brightness.

... So, have you studied how "noise" affects the colors that come out on these images that have no temperature key? That is, if you take the exact same image several times, then you are going to get some variation due to sensor noise. An analysis of how the colors change could give you some clues as to the relationships in the absent temperature map.

Are you taking the images with exactly the same framing and distance and and exactly the same illumination each time? Or can there be small incidental differences in the illumination? If there can be, then again analysis of how the colors change could give you some clues as to the nature of the absent temperature map.

[Qazi Arbab Ahmed]

Thanks, Yes the images are with exactly the same framing, but the illumination is different in different images. How can I analyze the color change? Thanks again

[Walter Roberson]

Please confirm that the thermal maps were colored without a provided temperature key but the absent temperature key was known to be consistent between the images.

Do you have a contact temperature sensor that you can use as part of a calibration process ?

Which manufacturer and model of thermal imaging are you using? Perhaps the temperature to color key is documented.

[Image Analyst]

Illumination has no effect at all on a thermal image. Any term with "lum" or "phot" in the name means it's applicable only to the visual light range (400 - 700 nm), not to the thermal range (several thousand nm) of thermal cameras. Terms what have "rad" in the name, like "irradiance" or "radiance" work for all wavelengths, even invisible/thermal wavelengths.

I would think any decent thermal camera should provide you with the thermal image in units of degrees rather than a pseudocolored RGB image, though I'm not sure about some of the really cheap thermal cameras that are being introduced. Thermal cameras that I've seen all let you look at different temperature ranges and you can apply a colormap (from a choice of several colormaps) to different degree ranges. For example you could apply the jet colormap to temperatures of 1000 to 1500 degrees, or it could be applied to 3000 to 4000 degrees. And the images might look the same - both have red, yellow, blue colors in them despite being in drastically different temperature ranges. So if you can get the original grayscale thermal image, you're all set. If you're using some cheap camera that only gives you a pseudocolored RGB colormapped image, then the only way to get the true temperatures is to know the colormap, but this is a less desirable approach than getting the thermal image initially. If you don't have the colormap, then you're really out of luck. You could be looking at a cryogenically cooled circuit or a super overheated circuit - they could both look the same depending on what the colormap is. You might possibly take a guess at what it would be, but you're guessing. At best you could make a guess and produce relative temperatures, like this is hotter than that, but not calibrated temperatures, like this is 50 degrees while that is 90 degrees.

[Walter Roberson]

"Illumination has no effect at all on a thermal image."

It does have an effect if the illumination is heating the item. For example, sunshine can need to be taken into account in some situations.

... in some of these situations, the illuminating source could be heat from a human hand.

[Qazi Arbab Ahmed]

Thank you both of you for your detailed discussion. I think I could not explained well enough. I am using a Pyrosoft Professional thermal camera and obviously it gives the temperature values. I got thermal videos with the help of camera software and got the frames that I have already attached. Suppose I have two different kind of videos and so I get two different set of Frames having colors Red, Green Blue, as thermal maps have. Consider one of the video has malicious behavior than other. I have to develop a trained model that can Classify the between malicious and benign maps. I have taken simple feature like Mean and S. Deviation that gives me 86 percent accuracy of trained model. I want to make my model more accurate so that I can Validate large data. What are other single value features could be effective for me? I think now you might have understand what I want to do.

Thanks

[Walter Roberson]

If the camera gives temperature values, then you should be working with those rather than with the pseudocolored images. But you wrote "Just getting thermal maps, one can't see temperature and difference is not known." which implies you cannot get to the temperature values ??

If you had a situation where any one kind of component had the same temperature profile, then you could do template matching. However I do not expect that to be the case -- I would expect that the temperature profile of any one kind of component would depend on what it was being used for in the circuit.

[Image Analyst]

I believe you'd have to have a tremendous amount of visible light ("illumination") to affect the temperature of an item. The sun has lots of infrared in it. If you filter out the infrared irradiance with an IR-cutoff filter, and just allow through illuminance (visible light), I doubt it would heat the sample even with 50,000 Lux (100 times office brightness). I know often a layman's inaccurate definition of illuminance includes all irradiance, not just visible light and including infrared, but my optics professor in grad school was a stickler for us using proper terms and units. He even published a paper on it (see attached paper).