Performance difference for plotting direction?

I found a very odd behavior that I'd like some help on. When plotting large vectors in matlab, there appears to be a MAJOR performance hit when plotting vectors whose elements contain many decimal places in x vs. y. It appears to be much faster to plot these vectors in x than y.

Here is the minimally functional code.

elems = 1e7;         %Define the number of elements
a = (1:elems);       %Large generic vector with no decimals
b = rand(1,elems);   %Large generic vector of the same length with many decimals

%First with a plotted as x and b plotted as y data
figure; tic; plot(a,b);drawnow;toc %Elapsed time is 1.202868 seconds.
figure; tic; plot(a,b);drawnow;toc %Elapsed time is 1.140523 seconds.
figure; tic; plot(a,b);drawnow;toc %Elapsed time is 1.105982 seconds.
%Now with b plotted as x and a plotted as y data
figure; tic; plot(b,a);drawnow;toc %Elapsed time is 4.654778 seconds.
figure; tic; plot(b,a);drawnow;toc %Elapsed time is 5.054447 seconds.
figure; tic; plot(b,a);drawnow;toc %Elapsed time is 4.788214 seconds.

I am not sure what is going on here, but my program (from which this example code was simplifed) started crashing when I tried to swap x and y data.

Any help would be appreciated. Even an explanation would be nice, as I don't this this is going to be resolvable.