I'd call unique(X) to get all the unique x values, just in case they're different from one column to the next.
Then I'd call interp1() on each x,y pairing with the unique values of x. So now you have all the curves sampled at the same x locations. So then I'd just take the mean of all the interpolated y values to get the average at each x
meanY = mean(interpolatedY, 2);
Then do your curve fitting, whether it's via polyfit() or something fancier, on the curve with uniqueX and meanY. Others may have different ideas but it seems reasonable to me. It's worth a shot. Give it a try.