I defined spatial references for the A and B coordinate systems with
RA = imref3d(size(A),[min(xA), max(xA)],[min(yA), max(yA)],[min(zA), max(zA)]);
RB = imref3d(size(B),[min(xB), max(xB)],[min(yB), max(yB)],[min(zB), max(zB)]);
Then i defined an affine transformation AT with my transformation matrix TM
AT=affine3d(TM);
Now i could transform B into the coordinate system of A with
[C,RC]=imwarp(B,RB,AT,'OutputView',RA);
[rp,Bfit,ErrorStats]=absor(PA,PB,'DoScale',1);
TM=rp.M.';