Re: We lost over half our US traffic ... We're a discussion forum with an editorial side, 100% unique content
Mar 15, 2011 3:17 AM
Posted in group:
Crawling, indexing & ranking
>> There are loads of pages still in the index that have been added to my robots.txt disallow list for days.
I haven't followed this thread completely, but I don't think you should use robots.txt to remove any content. It's doesn't remove content, it simply blocks access. If you add Robots META NoIndex, and block access nothing will happen.
I'd strongly suggest you remove bad content ASAP, and this means NOT using robots.txt for this.