How to project point cloud back to image?

I tried using this code:

 imagePointsRGB = worldToImage(stereoParams.CameraParameters1,eye(3), [0,0,0] ,ptCloud.Location,'ApplyDistortion', true);

and color the points with

    for i = 1:size(imagePointsRGB,1)

        valid = (imagePointsRGB(i,1) >= 1) & ...
        (imagePointsRGB(i,1) < imageSize(2)) & ...
        (imagePointsRGB(i,2) >= 1) & ...
        (imagePointsRGB(i,2) < imageSize(1));

            if(valid)

                    image_proj_RGB2RGB(floor(imagePointsRGB(i,2)), floor(imagePointsRGB(i,1)),:) = ptCloud.Color(i,:);

            end
    end

The resulting image does not overlap with the original (left) image. Is the orientation/location of the camera not correct or is it something else?

Edit: I found out that it corresponds to the rectified frame, translated by some value. But this doesn't make sense to me. Could someone clarify please?