" I want the user of d app to be able to type mathematical expressions in textbook format"
I don't believe you're going to achieve that with the equation tool of Excel. For a start it would be very clunky since the user would have to do their inputting in excel. Then, you would be limited in what you can control from matlab. There would be nothing preventing the user for closing excel and you losing your link from matlab.
I've only done limited testing but I don't believe there's an easy way to interact with the equation tool. You could extract the equation after it's been entered by going through the Shapes of the worksheet (assuming you even know which worksheet contains the equation) and extracting the Characters of the TextFrame but you'll have several problems:
- Even a very basic equation results in invalid characters once extracted in matlab. I've not bothered to try to figure out why
- Even assuming you could extract the characters correctly, you still have to figure out what the equation is. Bearing in mind that the equation tool use font formatting to render superscripts/subscripts you would have to use characters and format to decode the equation correctly. Twice as much work!
That last point in particular is why I don't think that the equation tool is any good for what you want. Instead I would advise finding some dedicated program (e.g. pick one from this wikipedia list) that can give you either TeX or probably better MathML equations and let your user use that. Converting a MathML equation into a symbolic equation is probably hard enough, I certainly wouldn't try to convert an equation from a tool whose output is not even documented or standardised.