The best way to solve this specific problem is to transform it, in the form of a non-dimensional variable. That is, we see the cubic equation:
syms x k m
E = x^3 - 5*x^2*(k/m)+ 6*x*(k/m)^2-(k/m)^3;
In fact though, k and m just confuse things. They are additional variables that introduce units into the probem. Consider instead, this transformation: u = x*m/k. Or, if we solve for x, x = u*k/m.
syms u
Eu = expand(subs(E,x,u*k/m))
Since we want to find the roots, Eu == 0 is our problem. But do you see that now we can factor out k^3/m^3? ANd that leaves us with a very simple problem to solve, since only u appears in the resulting expression.
simplify(Eu*m^3/k^3)
usol = solve(Eu*m^3/k^3 == 0,'maxdegree',3)
Yes, it still looks a bit messy, but the solutions to a cubic polynomial are often a mess of radicals, and there are often some complex roots too. We can resolve all of that into simple numbers using vpa.
usol = vpa(usol)
We are clearly almost there. Now recover x. That just means we need to put back in the ratio of k/m that we took out before. Since we don't care to see those pesky complex solutions almost always, we should consider that the imaginary part of the second and third solutions were infinitessimally tiny. In fact, they are just floating point trash, introduced when we converted into floating point arithmetic using vpa. So we can discard those imaginary parts. And that leaves us with 3 solutions to the problem.
xsol = real(usol)*k/m
Take your pick, as all are equally valid in terms of mathematics, but all I see is the equation. It is often the case that the real world can now intrude, and tell us that two of those solutions are meaningless in terms of physical constraints. But only you know that, since you are the one who proposed the problem.
