I know this is an old post. But I've searched for an answer numerous times. At the time of posting this,83 others had read this post in the past 30 days at this time. I am maybe 5 of those.
First, Matlab's GUI leaves a lot to be desired. There really is no excuse for not developing a set of 'properties' that allow the font size in the tabs to be changed. But after 7 years, it is obviously not a priority, and doubtful to be changed.
I find that the tabs are the easist way to organize a program with several options. They are integral in our GUI's for simplicity. With that we had to find a solution that worked. I've read solutions that suggest simply enabling and disabling things based on a button but that creates a total mess when trying to layout the GUI. This is a goofy work around but a simple one. It maintains the use of the tab group and does not require any 3rd party control apps etc. The simplicity of this and the fact that Matlab has never suggested this as a solution, at least not that I've found, seems to support the notion that GUI's are an afterthought for Mathworks.
Our solution is to create a button group and simply overlay the actual tab selectors. We then change tabs to display based on the button that is pushed using callback functions on the individual buttons. You can simply delete the label, "button group" that Matlab automatically displays, and then end up with 1 box within which the buttons can be placed.
I attached 1 image with the button group above a tab group, and another where the button group has been placed over the tab selectors. For exageration, I made the buttons 25pt font and the tabs remained their base 12pt font.
I hope this helps some people. I'm interested in better solutions.