Not entirely sure what you are asking.
Do you want to compute the mean for each year for each combination of X and Y? That would result in a 141x41 array.
Do you want to compute the mean over all X and Y, resulting in one element per year, so a vector of length 37? Here, I'd just reshape the rain matrix to be a 141*41 by 37 array, so 2-dimensional. Then take the mean down the columns of that array, so on the first dimension. The result of the computation below would be a 1x37 vector.
MeanRain = mean(reshape(rain,,37),1);
Either could be what you are asking.