Your matrix has 512*512*3=786432 elements.. to reshape you need to keep the number of elements the same.. That means you cannot reshape if you want just 512*512 elements without losing elements or data..
However if you want to access only the first dimension of your matrix (a) you use some thing like a(:,:,1)
If your data is an image data and it has the three dimension r,g,b you can covert it to grayscale (using im2bw)which should give you a matrix of 512*512
Example for my color image 1.jpg
551.00 1024.00 3.00