contourfm changing axis location/properties

I'm encountering a weird issue when trying to use contourfm. When I create my map axes using worldmap(), if I then try to use

set(gca, 'mapprojection', 'mercator')

it will shrink the axis size to a fraction of the total figure space, like so:

I haven't encountered this issue with anything but contourfm, other colormap plots like pcolorm will work just fine. Also, a few of the plots don't encounter this issue (I make about 20 of them). The inputs are always the same format though.

Thanks, Aaron