There is a description of the algorithm on the documentation page qtdecomp. It says "The qtdecomp function divides a square image into four equal-sized square blocks, and then tests each block to see if it meets some criterion of homogeneity. If a block meets the criterion, it is not divided any further. If it does not meet the criterion, it is subdivided again into four blocks, and the test criterion is applied to those blocks. This process is repeated iteratively until each block meets the criterion. The result can have blocks of several different sizes."
In the comments section you asked about the homogeneity criteria. I asked internally and the default is simply to test if they are all exactly equal. This is also mentioned in the doc. It depends on the syntax used:
S = qtdecomp(I) performs a quadtree decomposition on the grayscale image I and returns the quadtree structure in the sparse matrix S. By default, qtdecomp splits a block unless all elements in the block are equal. Then for threshold
S = qtdecomp(I,threshold) splits a block if the maximum value of the block elements minus the minimum value of the block elements is greater than threshold. You can come up with your own function to determine if a block should be split or not:
S = qtdecomp(I,fun) uses the function fun to determine whether to split a block. Does that help?