Generally this error occurs due to improper use or declaration of the symbolic variables.I tried the code on my end. The code runs fine. To my understanding, you dont have symbolic math toolbox installed. Please refer to this community answer to get support on how to install a new toolbox on existing MATLAB installation.
Do you by any chance have a variable named det in your workspace? If so you will not be able to call the det function until that variable is renamed or deleted.
det = 42;
det([x, x.^2; x.^3, 1])
Error using sym/subsindex (line 864) Invalid indexing or function definition. Indexing must follow MATLAB indexing. Function arguments must be symbolic variables, and function body must be sym expression.