An open-top box is to be made by cutting small congruent squares from the comers of a 12-in.-by-12-in. sheet of tin and bending up the sides.

How large the squares should be cut to make the box hold as much as possible.

I dont understand how do i start with this.

[DGM]

Start with a graphical description of the problem, express the volume in terms of the sketch, and then maximize.

If you have symbolic toolbox, you should be able to do this very simply, just as you would on paper. Write the expression for V, find the derivative (use diff), and solve for critical points. You can probably solve for the second derivative to check curvature direction at said CP, but you'll probably only get one non-trivial answer anyway.