I have a late-2013 model macbook Pro that I've tried to install matlab 2008b on. My company only has a license for 2008b, not 2012. The 2008b software wants the X11 app, but os x 10.9 offers only XQuartz.
I think X11 might only be the beginning of your problems. R2008b supported 10.4 and 10.5, and required Java version 1.5. I'm pretty such Apple has not shipped Java 1.5 in a number of years. I know this is not the answer you are looking for, but you face a significant uphill battle, sorry.