Should there be a residual when applying principal component analysis?

I am using Matlab function PCA (principal component analysis) to reduce the dimensionality of a data set with approximately 20 000 observations x 100 dimensions.

After having obtained the principal component coefficients of the data I recreated the input signal in the original coordinate system using the transformation matrix from the PCA function. This yielded a very large residual when comparing with the input signal. I have tried around with different data sets and sizes and it appears to be commonplace to have a large residual. I am not sure yet whether it is due to round-off errors or high SNR in the input data. The dimensionality reduction could of course still be useful, but is this something that one should be cautious about when performing principal component analysis? Or is there another metric that is better to assess the performance of the principal component analysis?

[Adam]

How many components does it leave you with? The obvious interpretation would be that there are too few to accurately represent the data and that you would need to use more components. I haven't use the pca functionality for a few years so I can't remember what it returns by default.

[HaMo]

By default it returns the same number of dimensions as the input data, ordered by maximal variance. But even if I use all 100 dimensions I get a large residual when trying to reconstruct the input signal (which has 100 dimensions). Maybe this is a result of the large number of observations (either due to round-off error or too low SNR), but could the PCA analysis still be valid even if it cannot be used to reconstruct the input signal?

[Adam]

I would have expected there to always be 0 residual if you simply re-arrange the dimensions without throwing any of them away, but maybe I am mis-remembering how PCA works. If it is simply re-orienting the data to the axes of maximal variance it should not be losing data though if you are reconstructing correctly.

[HaMo]

Yeah, you are right Adam. See answer below. Thanks!

Answer 1

[John D'Errico]

If you used all of the components, and you STILl have a large residual, then you are doing the reconstruction incorrectly. The residuals at that point should be on the order of eps. Have you verified that indeed, you did not get tiny numbers,and merely did not see the tiny power of 10 attached to the numbers? With no data and no indication of how you did the analysis or the reconstruction, we cannot know what you did incorrectly.

If I had to guess, you did not properly deal with the mean, or something silly like that.

My recommendation is you attach the data to a comment as a .mat file. Then I can do PCA, and the reconstruction, showing you what to do.

[HaMo]

Problem solved. As you guessed, my input data was not centered. Thanks!

[John D'Errico]

WOW! For once, the crystal ball told me the truth. Usually it does nothing more than tell me my horoscope. :)