As Walter mentioned, the assignment of C(i,:,j) = key_out is not possible as left side is a 1-D vector and right side is 2-D matrix which will give the mentioned error.
From the comments in the code I assume you need to store 4 chunk of 10 x 128 matrices into a xlsx file using writematrix. Since you want the chunk to be placed in a final 2-D matrix row-wise, in this case the size will become 40 x 128. You may find below example code useful for achieving the same.
function key_out=updateKey(key, i)
numOfSteps = 4;
numOfStrings = 10;
numOfBitsperString = 128;
C = zeros(numOfSteps*numOfStrings,numOfBitsperString);
C((j-1)*numOfStrings+1:j*numOfStrings,:) = jj;