Image Processing: Edge Detection on Specific Regions Only..?

I've been stuck on this problem for a while now, and I'm hoping that someone out there might be able to offer a suggestion or two.

Basically, I want to detect only the edges in images where the sky meets a building, or trees, or whatever it happens to be next to.

My initial thought was to use segmentation (this is VERY slow though) to recursively merge areas of the image until 2 classes remain (the background and the objects). I then hoped to perform edge detection on the segmented image, and obtain an estimate of the edge between the building and the sky. This worked for some images, but obviously fails on a large number of images where the background and the objects are fairly close in pixel value. In these cases it merges part of the buildings into the sky. The estimated edge is therefore completely erroneous.

I now think a better solution is to segment into more classes so that I can ensure the structure of the image does not contain any errors. For example, here is the original image:

http://imageshack.us/photo/my-images/269/imagejje.jpg/

I then segment this to obtain the following:

http://imageshack.us/photo/my-images/37/segmented.png/

When I perform edge detection (canny) on this segmented image, a lot of unwanted edges are not present because they have been merged away:

http://imageshack.us/photo/my-images/69/edgek.png/

However, there are still many more edges than those that I am interested in. Does anyone know of a way to reduce the edges to just the edges corresponding to where the sky meets the building?

Or alternatively, can anyone suggest a better way of tackling this problem?

Ideally, the technique would need to work for a wide range of images (all will be of landscapes and include some sky). Whilst I know it is unlikely that there is a "one fits all" solution, I would like to avoid the use of thresholding the blue channel or using the size of the segmented sky regions, since the sky may be cloudy and gray on some images, and there may be less of it.

I hope someone can help me!

Thanks!