Think about what is happening with this loop:
for i = 1:num_iterates
x = A*x;
R(:,i) = x/norm(x);
Suppose x starts out as the unit eigenvector e with associated max eigenvalue lambda. Then
1st iteration: x = lambda*e
2nd iteration: x = lambda^2*e
3rd iteration: x = lambda^3*e
At each iteration x grows in magnitude by factor lambda. Eventually it will overflow to infinity or underflow to 0 (depending on value of lambda) and lead to the NaN issue.
In your other loop, you are storing unit vectors for x at each iteration, which avoids this problem.
In your particular problem, the max eigenvalue is 4. And if we look at the following:
You can see that we would expect an overflow at about iteration 511, and that's what you got.