MATLAB being defined as the MATrix LABoratory follows conventional matrix algebra rules --
>> R=[4/3 2/3 0 0;2/3 4/3 0 0;0 0 4/3 -2/3;0 0 -2/3 4/3];
Arrays A, B are only conformable for matrix multiplication C=AB if size(A,2) == size(B,1) (number rows in B is same as number of columns in A). The size of the output array C is then size(A,1) x size(B,2).
If R is 4x4 and A 4x1, then R*A --> 4x4 * 4x1 --> 4x1
Your code above uses the "dot" operator .* which is element-wise multiplication and MATLAB silently expanded the vector to the size of the array to produce the 4x4 result.
As the above shows, all you need is the matrix multiplication operator * applied to the two variables to produce the desired result; the same as Excel implements in a much more convenient format/syntax. :)