How to combine and rgb and a thermal image?

Hello,

I am working with a ZenMuse XT2 camera, and it outputs images in rgb multispectral and in thermal. Both images have be resized into .tiff files with the same size, and I want to create an .rgbt image that has 4 bands on a single image. The images are currently seperate but in the order (visual, thermal). Is there a way to combine each first image to the second image in this way?

edit: RGBA would probably be the better way to combine them

Any help would be appreciated

[DGM]

If they're compatible data ranges and classes, then why couldn't you just concatenate them? The real question is once you have an RGBT image, what are you going to do with it? Is there something that can view such an image without assuming it's RGBA?

[Cal McCormick]

RGBA would probably be the better format

[Cal McCormick]

How would I concatenate images in this format of pairs(visual, themal) repeating?

[Cal McCormick]

Thank you both for submitting awnsers, very helpful to me when working this process in MATLAB

Answer 1

[DGM]

Let's just start with concatenation. We have two test images: an RGB visible image and a single-channel thermal image. How I make them isn't really important, but they can simply be concatenated.

gpict = imread('cameraman.tif'); % raw single-channel source
gradpict = lingrad([256 256 3],[0 1; 1 0],[0 0 1; 1 0 0]*255,'cosine','uint8'); % colored gradient

rgbpict = imblend(gradpict,gpict,1,'moonlight'); % RGB test image
gpict = fliplr(imlnc(gpict,'independent','g',2,'k',3)); % pretend thermal image
rgbtpict = cat(3,rgbpict,gpict); % 4-channel RGBT image

Of course, nothing knows what an RGBT image is. This is just sort of a contrivance of our own design, to fit our own needs. Things like imshow() won't know what to do with it ... of course imshow() doesn't know what an RGBA image is either. Most tools (at least in the MIMT) think a 4-ch image is RGBA. Imwrite() sees a 4-channel image and thinks it's CMYK. All this confusion doesn't mean you can't use these tools to save the data and read it back, but other applications may grossly misinterpret the saved image. The issue of saving a multichannel image is one that I'm not certain I have a good answer for.

Let's start with a simple example of saving the image as RGBA using PNG:

imwrite(rgbtpict(:,:,1:3),'rgbt.png','alpha',rgbtpict(:,:,4))
% which is the same as
imwrite(rgbpict,'rgbt.png','alpha',gpict)

[readbackrgb,~,readbackalph] = imread('rgbt.png'); % read the image back
readbackrgbt = cat(3,readbackrgb,readbackalph); % concatenate if desired

Note that imwrite/imread really don't support 4-ch RGBA images directly. The fourth channel has to be split off when writing and reconcatenated when reading (if you choose to process the image as a 4-channel image). Of course, other applications are going to treat this image as RGBA and display it accordingly as nonsense:

What happens if you just feed the 4-ch image to imwrite with a TIFF output?

imwrite(rgbtpict,'rgbt.tif') % image is attached; can't embed tiffs

readbackrgbt = imread('rgbt.tif'); % read the whole 4-ch image back

Imwrite treats 4-channel images as CMYK. Consequently, other apps will treat it that way. I'm not sure of an appropriate set of property settings for arbitrary multichannel images.

Of course, TIFF is a container format. You can always just put both images in the same file.

imwrite(rgbtpict(:,:,1:3),'rgbt.tif') % write the RGB image
imwrite(rgbtpict(:,:,4),'rgbt.tif','writemode','append') % append the T image
readbackrgbt = cat(3,imread('rgbt.tif',1),imread('rgbt.tif',2)); % read back both as a 4-ch image
% image is attached as rgbt2.tif

So that's a few ways to save such an image. Like I said, there may be better canonical ways of dealing with arbitrary multichannel images. I'm not terribly familiar with the TIFF tools or dealing with hyperspectral imaging. I don't know how helpful any of this is.

As Image Analyst points out, how you handle/store the image might depend on how you want to visualize it. Juxtapositioning is simple, but may make content colocation less intuitive depending on the amount of detail available in either image. Alpha blending is simple, so let's consider that as a baseline. IA already showed how to do it with regular tools. I'm going to use MIMT tools because I'm lazy.

blendpict1 = imblend(gpict,rgbpict,1,'average',0.5); % 25/75 (FG/BG) blend
blendpict2 = imblend(gpict,rgbpict,1,'average',1); % 50/50 blend with default parameter
blendpict3 = imblend(gpict,rgbpict,1,'average',1.5); % 75/25 blend

Of course, it looks as flat as one should expect from averaging. If all that is needed is an informal means of visualization, the requirements may be more flexible. Besides simple alpha compositing, there are many ways to blend image data. It may be desirable to emphasize different parts of one image or the other instead of combining them so linearly. Just as Photoshop can blend images, so can imblend().

blendpict4 = imblend(gpict,rgbpict,1,'softlight',1);
blendpict5 = imblend(gpict,rgbpict,1,'softerflatlight',1);
blendpict6 = imblend(gpict,rgbpict,1,'meanglow',1); % hybrid of average and max()
blendpict7 = imblend(gpict,rgbpict,1,'meanshadow',1); % hybrid of average and min()

... of course finding the right mode and parameter settings might take some time and discretion, and may depend heavily on the content and contrast of the actual images being used. MIMT does have a multilayer composition GUI (imcompose) and thorough blend mode documentation which may help if blending is considered a useful approach.

Of course, if all that's needed is visualization, there are other ways, such as reducing the visible content to some sort of edge/depth map and then combining that with the thermal image -- sort of like what some IR cameras do for the viewfinder display.

I don't know if any of this is going to be directly helpful, but maybe you can use it to decide how you want to deal with the images and ask further questions.

Some of the tools used in these examples (lingrad, imlnc, imblend) are from MIMT on the FEX:

https://www.mathworks.com/matlabcentral/fileexchange/53786-image-manipulation-toolbox

[DGM]

I figured this could use an update. As before, this example relies on MIMT.

Let's assume we have two registered images, an RGB visible-light image, and an RGB pseudocolor IR image. Another way to combine the two images would be to take edge content from the visible-light image and blend it with the pseudocolor IR image.

rgbpict = imread('peppers.png'); % visible-light image
pcpict = imread('pcolorpep2.png'); % pseudocolor IR image

% get the gradient of the visible-light image
Y = mono(rgbpict,'y'); % reduce to luma
Yg = edgemap(Y);
Yg = imlnc(Yg,'rg',0.8); % normalize and stretch gamma at top end

% blend the two images
%outpict = imblend(Yg,pcpict,1,'screen'); % won't blow-out bright regions
%outpict = imblend(Yg,pcpict,1,'malekidodge'); % strong influence over dark regions
outpict = imblend(Yg,pcpict,1,'lineardodge'); % simple addition

imshow(outpict)

If the thermal image is not a pseudocolor RGB image, but is actual temperature data instead, then you can just use gray2pcolor().

% a thermal image scaled in temperature units
S = load('Tpep.mat'); % attached
T = S.T; % a single-channel image
imrange(T) % data range is -50 to 75 degrees

% convert to pseudocolor using any desired colormap
% ccmap() has some default camera maps
% or you could use parula(), etc.
CT = ccmap('flir1',256); 
pcpict = gray2pcolor(T,CT);

imshow2(pcpict)

... at which point, the composition can be performed as described prior.

Answer 2

[Image Analyst]

To do side by side, do

wideImage = [image1, image2];

To do top and bottom do

tallImage = [image1; image2];

I would think that a monochrome thermal image overlaid on an RGB image would be confusing to interpret and that having the thermal image beside the RGB image would be easier to interpret.

For example:

% Get the RGB image
rgbImage = imread('peppers.png');
[rows1, columns1, numberOfColorChannels1] = size(rgbImage);

% Read in gray scale image.
grayImage = imread('cameraman.tif');
[rows2, columns2, numberOfColorChannels2] = size(grayImage);
if rows1 ~= rows2 || columns1 ~= columns2
	grayImage = imresize(grayImage, [rows1, columns1]); % Resize to match RGB image.
end

% Display RGB image
imshow(rgbImage);
% Display gray scale image as alphadata
hold on;
h = imshow(grayImage);
h.AlphaData = 0.5;

g = gcf;
g.WindowState = 'maximized'

Misc Overlay links:

How to overlay a foreground image over a background image? - MATLAB Answers - MATLAB Central

Image overlay using transparency | Steve on Image Processing

https://www.mathworks.com/matlabcentral/answers/100086-how-do-i-superimpose-images-in-matlab?s_tid=ta_ans_results

Also see attached paper.