bar3 colors of zero height graphics issue

I'm trying to use bar3 function to plot various time shifts per measurement, where the height of each bar represents the time shift, but the color represents a normalized cross-correlation value.

After learning how to color by height, I simply utilized the code by transferring the bar3 ZData from cross-correlation value to the CData, like such:

figure(2); b1 = bar3(corrVal1);
figure(1); b = bar3(timeShift1);
for k = 1:length(b)
    zdata = get(b1(k),'ZData');
    set(b(k),'CData',zdata);
    set(b(k),'FaceColor','interp');
end

The problem lies afterwards: the bars that have 0 timeshift (And thus zero height) aren't able to consistently display a color, but the color is sometimes visible if I move the axes around. Example of these semi-visible bar values:

Changing 'FaceColor' to 'flat' doesn't help, and I've tried tinkering with some of the patch properties or using different colormaps, but nothing I could think of could help me solve this. This seems to be a graphics issue, which I have very little knowledge about, so I'm hoping someone here could help me with this issue.