Seoninja. Looking at your site I'm not at all surprised that it's been hit. I'm pretty sure that it has nothing to do with the Yahoo directory listing.

This is one of those cases where I think that duplicate content is leading to your site being interpreted as spam. You have many pages which differ very little, except for the page title which refers to a given city / area. I see why you've done it, but I believe that it may have hurt you.