Categories: Crawling, indexing & ranking :

DaniWeb lost all its traffic *AGAIN*

Showing 1-145 of 145 messages
DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/11/12 9:31 PM
I have read the FAQs and checked for similar issues: Yes
My site's URL (web address) is: www.daniweb.com
Secret Phrase: I have studied the Help Center, read the FAQs and searched for similar questions.

Soooo ... After being hit hard by Panda and making a complete 110%+ recovery, that eventually climbed up to about 130% of where we were pre-Panda, DaniWeb once again was hit super hard by Google. We rolled out a brand new platform the same time that there was a Panda update mid-March and it's been downhill ever since.

I do not believe that we were hit by Panda. Unlike Panda, which was an instantaneous 50-60% drop in traffic literally overnight, we've instead had a steady decrease in traffic every day ever since our launch. At this point, we're down about 45%. We are using 301 redirects, but our site's URL structure *DID* change. While we're on an entirely new platform, the actual content is entirely the same, and there is a 1-to-1 relationship between each page in the old system and the new system (all being 301-redirected).

First observation that I am not happy with:

If you visit a page such as: http://www.daniweb.com/web-development/php/17 you will see that the article titles have URLs in the format http://www.daniweb.com/web-development/php/threads/420572/php-apotrophe-issue ... However, you can also click on the timestamp of the last post to jump to the last post in the article (a url such as http://www.daniweb.com/posts/jump/1794174)

The /posts/jump/ URLs will 301 redirect you to the full article pages. For example, in this specific example, to http://www.daniweb.com/web-development/php/threads/420572/php-apotrophe-issue/1#post1794174 (the first page of the thread, with an anchor to the specific post).


Why then, does the /posts/jump/ URL show up in the Google search results instead of my preferred URL?? Not only am I doing a 301 redirect away from the /posts/jump/ format, but I am also specifying a rel="canonical" of my preferred URL.

I don't like this at all for a few reasons. Firstly, the breadcrumb trail doesn't show up in the SERPS. Secondly, there is no reason for Google to be sending everyone to shortened URLs, because now nearly every visitor coming in from Google has to go through a 301 redirect before seeing any content, which causes an unnecessary delay in page load time. Thirdly, the /posts/jump/ URLs all tack on a #post123 anchor to the end, meaning that everyone is being instantaneously jumped halfway down the page to a specific post, instead of getting the complete picture, where they can start reading from the beginning. This certainly isn't desirable behavior!

When I do a site:daniweb.com in Google, literally pages 2 through 10 (at least) are *ALL* URLs in the format /posts/jump/.

All of the URLs in my Google sitemaps are the desired, full URLs that match the URLs specified in the rel="canonical" directive. I am not using robots.txt to block either format of URL. However, I am using rel="noindex" on article pages that have zero replies, because I find it very obnoxious when I do a Google search for something, and the SERPs show me a forum thread, only for it to be the exact thing I want to know, but it's unanswered. Therefore, everything is noindexed until it has at least one reply. I have been doing this for the better part of a year, since we were hit by Panda last February (and have since recovered).

I was toying with the idea of adding rel="nofollow" to the link that uses the /posts/jump/ URL. However, I really don't want to do this for a few reasons. Firstly, I'm of the mindset that "nofollow" should be reserved for untrustworthy URLs, and those that are submitted through UGC. It's a borderline case for me to use "nofollow" on internal URLs that are pages that wouldn't make sense for googlebot, such as links to contribute a new post, which would just give permission denied error messages. However, rel="nofollow" is NOT for internal links to our article pages!!! Instead, Google should honor my 301 redirects and my canonical URL directives (either one or the other would work!).

Second observation that I'm not happy with:

After skimming the first 40 or 50 pages of the Google search results for site:daniweb.com, it's essentially entirely a mix of two types of URLs. Those in the /posts/jump/ format, and links to member profiles. Essentially, two types of pages which are both not what I would consider putting our best foot forward.

We currently have nearly one million members, and therefore nearly one million member profiles. However, we choose to use the rel="noindex" meta tag directive on about 850,000 of the member profiles, only allowing those by good contributors to be indexed. I think it's a happy medium between allowing our good contributors to have their profiles found in Google by prospective employers and clients searching for their name, and not having one million member profiles saturate our search results. We allow just under 100,000 of our 950,000+ member profiles to be indexed.

However, as mentioned, it just seems as if member profiles are being ranked too high up and just way too abundant when doing a site:daniweb.com, overshadowing our content. This was no the case before the relaunch, and nothing changed in terms of our noindex approach.

Based on prior experience, the quality of the results when I do a site:daniweb.com has a direct correlation to whether Google has a strong grasp of our navigation structure and is indexing our site the way that I want them to. I noticed when I was going through my Panda ordeal that, at the beginning, doing a site: query gave very random results, listing our non-important pages first and really giving very messy, non-quality results. Towards the end of our recovery, the results were really high quality, with our best content being shown on the first chunk of pages.

Therefore, I'm really confident that there is something quite wrong, but I'm not quite sure what it is or how to fix it. As far as I'm aware, I'm doing everything exactly like I was before the relaunch. I'm noindexing all the same pages. I have all the same pages disallowed in my robots.txt file. I have a similar Sitemap file. I'm specifying canonical URLs on all the same pages. It's all the exact same content.

Why are things so amiss? It just really seems like Google has no grasp of the structure of my site! Nevertheless, if you check out the attached chart.png file from Google Webmaster Tools, you'll see that they've been merrily crawling an average of 600,000 pages a day ever since our relaunch, which is a significant increase from the 300-400K pages a day they used to crawl on average. At 600K pages a day, they should have crawled the entire new version of the site like five times already! While at first I was patiently waiting for them to recrawl everything and get a grasp of the new site, our traffic is slowly dwindling every single day, and I'm starting to grow impatient.

I do want to come clean with something though. This mess is partially my fault, I will have to admit. As mentioned, we changed our URL structure, and I am 301 redirecting the old URLs to the new URLs. However, we also changed our URL structure last February, right after Panda originally hit. I have to admit that when we first went live, I completely forgot about that. While I was 301 redirecting the old version to the new, I was *NOT* redirecting the old old version to the new for about 72 hours, until I remembered! However, by that time, it was too late, and we ended up with over 500,000 404 errors in Google Webmaster Tools. That has been fixed for quite a few weeks already though.

I am also aware that it's one thing to tempt fate by changing URL structures once. But it's another thing to change URL structures a second time just over a year later.

old format (right after Panda hit, February 2011) => www.daniweb.com/web-development/php/123
new format (as of mid-March 2012) => www.daniweb.com/web-development/php/123/my-article-slug

As mentioned, this new version is completely rebuilt from the ground up, so I'd appreciate all suggestions that anyone can throw at me. :) After tweaking the old system little by little for the past eight years, I'm starting my SEO efforts over from ground zero on a completely new system.

Thank you so much in advance to everyone who has even gotten this far. As someone who runs a forum myself, I know the suuuuuper long questions are the ones that tend to go least answered.
Re: DaniWeb lost all its traffic *AGAIN* Steven Lockey 4/12/12 7:27 AM
Sounds like Google is still trying to understand the change-over.

How long ago did you do the system change-over?
Have you tried viewing the inner pages in the canonial via 'view as googlebot' and checked that they could be seen correctly?

Re: DaniWeb lost all its traffic *AGAIN* I know nothing 4/12/12 8:19 AM
I'm struggling with your new site, I bet Google is as well:

404 page not found...

http://www.daniweb.com/web-development/php/123/my-article-slug


Re: DaniWeb lost all its traffic *AGAIN* Steven Lockey 4/12/12 8:31 AM
Hmmm, yep, getting a lot of time-outs and slow loading pages here.
Re: DaniWeb lost all its traffic *AGAIN* KORPG Kevin 4/12/12 8:43 AM
I don't think that was a real URL, just an example of the URL structure.
So that specific 404 is likely not an issue.
Re: DaniWeb lost all its traffic *AGAIN* Steven Lockey 4/12/12 8:49 AM
I was just clicking links on the site and getting the 404s.

Looks at pages were timing out.
Re: DaniWeb lost all its traffic *AGAIN* Suzanneh 4/12/12 9:06 AM
Somebody correct me if I'm wrong, but Google doesn't index links with the "#" tag, right?  Could it be that because you're redirecting to /1#post17whatever -- and despite the canonical reference -- Google is just seeing the posts/jump link and then doesn't want to index that particular redirect (the hash mark) -- and doesn't know what to do, so it indexes the /posts/jump link?  In other words, googlebot doesn't even get a chance to see the canonical reference?


Suzanne
Re: DaniWeb lost all its traffic *AGAIN* Suzanneh 4/12/12 9:26 AM
Call me old school but I'm not quite sure why you're doing a 301 redirect to get to a particular post.  Seems like a roundabout way of doing it?  If I look at vBulletin's forum, they use a query string (?goto=newpost), and that's something they can tell googlebot to ignore.

Just tossing out some ideas.  Maybe you need to rethink how you get people to a post jump?

Suzanne
Re: DaniWeb lost all its traffic *AGAIN* Phil Payne 4/12/12 9:30 AM
http://googlewebmastercentral.blogspot.co.uk/2009/09/using-named-anchors-to-identify.html
Dani of DaniWeb 4/12/12 9:36 AM <This message has been deleted.>
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 9:46 AM
Please forgive me while I try to figure out this new threaded view of Google Groups. I don't know how to reply to everyone at once without writing 50 separate messages.

Steven: We did the changeover mid/late March. I tried viewing the site as Googlebot in webmaster tools and everything looks as intended. Canonical reference in place.

Know Nothing: The /123/my-article-slug was just a sample URL. That specific URL is not meant to exist.

Steven: Are you really getting a lot of time-outs? That's unusual because Google Webmaster Tools is actually showing a huge decrease in page load time with the new version! We actually went from about an 7.5 second load time (according to Webmaster Tools) down to 5 seconds. Can you tell me which pages were timing out? That's a HUGE concern now that I wasn't aware of!

KORP: Yup. :)

Suzanneh: Essentially we're treating the ability to jump to a particular post just like any other short URL (think bit.ly, etc). If Googlebot doesn't like the hash mark, then they should honor the canonical specified on the page, no? We haven't used a URL such as ?goto=newpost in about five years.

Phil: It looks like Google's preferred method is to use rich snippets to create different jump to anchors from within the result. Therefore, they aren't supposed to make the URL with the anchor the primary URL, but instead make it secondary information.

I think what I'm going to do is tack on #post123 to the /posts/jump/ URL and see if Google can figure out that they are NOT meant to be the primary URLs that way!
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 10:15 AM
OK so I just went ahead and added the #post123 anchor information to all links that point to /posts/jump/

Hopefully this will make it more clear to Google that I want them to crawl and index the other URLs that don't have that. ;)
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 10:31 AM
Suzanneh: Just want to also point out that vBulletin's method of a URL ending in ?goto=newpost / ?goto=lastpost do a 301 redirect the exact same way that we do.
Re: DaniWeb lost all its traffic *AGAIN* Grandmaster Flash 4/12/12 10:51 AM
Dani -

Looks like lots of recent Google crawls are on pages with the same <title> as your home page
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 11:12 AM
Thanks for catching that. Why Google is even indexing every page is beyond me. This is yet another example of Google not ranking the appropriate content first. Page 500 of our thread list should not be one of the top ten pages.

I went ahead and added "Page <pagenumber>" to the title to differentiate each page in the list. I also added the noindex meta tag to all pages other than page one.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 2:05 PM
Follow up: I just made sure that all of our important pages have rel=up/prev/next meta tags.
webado 4/12/12 5:45 PM <This message has been deleted.>
Re: DaniWeb lost all its traffic *AGAIN* webado 4/12/12 5:52 PM
http://webcache.googleusercontent.com/search?q=cache:zfBUvHelTOEJ:www.daniweb.com/posts/jump/1671625+&cd=1&hl=en&ct=clnk 

Cached MArch 30. Had you already implemented your 301 redirections at that time? Maybe not.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 6:21 PM
Yes, I did :(

URLs in the format /posts/jump/ have *always* redirected to the same full URLs with a 301 redirect. These URLs have never shown any content in a non-301 way.

What you're showing me is what my issue seems to be ... When I have a short URL that uses a 301 redirect, the short URL is what is apparently being indexed in Google.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 6:27 PM
To give you an example ... Here is a page that was just created 40 minutes ago, and the /posts/jump version got indexed.


This is Google's cache of http://www.daniweb.com/posts/jump/1794641. It is a snapshot of the page as it appeared on Apr 13, 2012 00:59:51 GMT. 

Re: DaniWeb lost all its traffic *AGAIN* webado 4/12/12 6:33 PM
But why are you still creating such urls? 

Why aren't yo creating the FINAL url rather than generate a url that gets 301 redirected? You have probably pinged Google using the short url. Maybe it's even in a feed.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 6:45 PM
We are primarily a forum. Some forum threads are multiple pages. In such a case, they will have multiple URLs in the format /blah/article-title, /blah/article-title/2, /blah/article-title/3, etc.

On the forum listing page, the primary link is to the first page of a multi-page article (or the only page if there is just one page). In other words, the link is /blah/article-title.

However, there is also a secondary link to jump to the current last post in the thread, which is very frequently used. These links are in the format /posts/jump/123, where 123 is the post ID to jump to. These links do 301 redirects to the last page of the thread, and then tack on an #anchor to the last post in the thread. In other words, if it's a ten page forum thread, there needs to be an easy way to jump to the last post on the last page. It's not reasonable to expect someone to manually click through ten pages every time they want to see a new reply.

Therefore, /posts/jump/123 is a 301 redirect to /blah/article-title/10#post123 (in other words, a 301 redirect to page ten and then jump to post 123 on that page.)

Every page has a canonical URL specified. So, /posts/jump/123 is a 301 redirect to /blah/article-title/10#post123, which means 301 redirect to page ten and jump down to post 123, and that page has a canonical URL specified as /blah/article-title/10.

Let's say then someone else posts in the thread and it is post 124. /posts/jump/124 would be a 301 redirect to /blah/article-title/10#post124, and that page would of course also have the canonical URL /blah/article-title/10 because it is the same page with just a different anchor specified.
Re: DaniWeb lost all its traffic *AGAIN* pskullz 4/12/12 6:49 PM
Dani, why dont you just put a rel-nofollow tag on the time stamp? Shouldn't that disallow the post/jump from being indexed?
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 6:53 PM
As mentioned in my original post:
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 7:10 PM
> Why aren't yo creating the FINAL url rather than generate a url that gets 301 redirected?

The final URL is very expensive to create. For example, I know that post 123 is the last post in the thread, but I don't know if post 123 is on page 5 or on page 10. Additionally, for logged in members, we want to jump to the last post since their last visit, which is not necessarily the last post in the thread. Therefore, we can't just take the total number of posts in the thread and divide by the number of posts per page, to figure out how to redirect to the last page of the thread. Sometimes we want to redirect to page 8 of a 10 page thread, for example.

An expensive SQL query needs to be done to figure out which page to redirect to. Therefore, it makes sense to only have to figure it out if it's clicked on, where PHP can figure it out and then send a 301 redirect header. Otherwise, on a listing page with 40 threads, that's an additional 40 SQL queries! It just makes more sense to only do the query if the link is clicked and it needs to be done.

vBulletin does the same thing with their ?goto=latest and ?goto=newest URLs which are 301 redirects to the full version. InvisionBoard does this too with their page_view_getlastpost short URL that does a 301 redirect to the full version. It's a very common technique for forums.
Re: DaniWeb lost all its traffic *AGAIN* webado 4/12/12 7:18 PM
OK then I'm afraid you'll need to wait for some time until the urls are revisited and the redirections are actually found. 
It sounds complicated and maybe your database needs to optimize itself better. Maybe you need some new indices on some fields to facilitate this.

The thing is if your internal navigation links are to urls that always end up getting 301 redirected, crawling will be painfully slow.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/12/12 7:30 PM
As mentioned in my original post, Google is currently crawling 600,000 pages per day according to Google Webmaster Tools. At that rate, they should have circled around my entire site a bunch of times already!!

I totally understand that it's not ideal for internal links to always be 301 redirected, but this seems to be the ideal way of doing it (it's the way all the big forum systems do it and I can't think of a more optimal way, unfortunately). My issue is that I don't feel like Google shouldn't be able to figure it out because it *is* such a common technique.

We've employed this same technique (albeit with different short URLs) for nearly ten years, and it hasn't been a problem up until now.

I agree with just waiting it out since we launched a brand new system and it takes time to adjust, but it's been like three weeks now, and traffic is just dwindling every single day, little by little. By week three, I decided to post this question here on Google Groups because I'm just now starting to get worried.

I also think I'm in a bit of a chicken and egg scenario. Google is penalizing me for a bad user experience, but there is a bad user experience because they're sending users directly to three-quarters down the page almost every time!
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/13/12 11:22 AM
Status update ...

Google Analytics only has a couple of hours of today (Friday) recorded so far. However, when doing an hour-by-hour report, comparing this Friday to last Friday, there is a 15% increase in traffic for each hour that's been accounted for so far.

That would mean that this is the first day since our roll out that any hour of any day has been better than the previous week. I certainly wouldn't call this a recovery, but it's definitely an improvement. It looks like we're on pace to have about 15% more traffic than last Friday, but still 10% less traffic than two Friday's ago. (As mentioned, we've been steadily losing traffic every day for the past 3 weeks.)
Re: DaniWeb lost all its traffic *AGAIN* Suzanneh 4/13/12 12:14 PM
Last Friday was Good Friday -- a holiday.  I don't know about your niche, but in mine, traffic goes down when there's  a holiday. I'm seeing anywhere from a 10 to 19% difference in hourly traffic when comparing today to last Friday.

Suzanne
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/13/12 2:12 PM
Hmm ... I hope that's not the only reason! I was really having high hopes that we stopped the bleeding.

In my niche, as a professional community, we get the most visitors from IT professionals during work hours, or from college students seeking help with their programming courses. Therefore, we get a significant traffic decrease (as much as 40% from the norm) on national holidays when there is no school/work. Holidays that are limited to spending time with family during evenings don't affect us all that much. I'm Jewish, so I wouldn't really know, but at least here in NYC, not being a national holiday, Good Friday was business as usual as far as work/school are concerned.

It certainly was not planned to line our site relaunch up with the Panda update that happened last month. However, it *was* planned to line our site relaunch up with spring break, which was mid/late-March for many universities. I wanted to take advantage of the slow time to work out the first round of kinks in the new system. The spring semester has now officially started for most schools, so this should actually be our busy season right now as students struggle through the first few overwhelming weeks of their courses.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/13/12 2:24 PM
I just want to add that, in browsing the list of pages currently indexed with Google, I'm SOOOO FRUSTRATED that I'm *almost* at the point where I just want to start that nasty technique of cloaking and stripping the #post123 anchor from any URL that comes in from a Google search! It's just such a bad user experience to suddenly start sending every visitor three quarters down the page! C'mon Google! Strip that hash for me!
This message has been hidden because it was flagged for abuse.
Re: DaniWeb lost all its traffic *AGAIN* StevieD_Web 4/13/12 9:26 PM
B2B traffic was down on Good Friday by 31% versus the Friday before.
 
Traffic was down by 60% for a local physican group.
 
might not have been a Jewish holiday, but traffic was definitely tanking for some sites.
Re: DaniWeb lost all its traffic *AGAIN* webado 4/13/12 9:40 PM
Good Friday was the same day as Passover this year or so I was told.


Re: DaniWeb lost all its traffic *AGAIN* StevieD_Web 4/13/12 9:44 PM
Passover.  Yep, forgot about that.  B&H camera was closed.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/14/12 7:00 AM
Dani,

I can't see a reason why Google would be indexing your /posts/jump/<number> style URLs. You're using a 301 redirect and going directly from the short URL to the destination URL, with no intermediate 302 redirects or similar along the way. I did see a thread late last year where a user was having a world of trouble with phantom 301 redirect URLs showing up in the index. Whether or not his scenario is relevant to you or not is another story entirely but it might be worth read through the thread just so you're aware of his problem and what he had to do to solve it eventually.

Regarding your hope that Google will drop off the hash fragment, I don't think that is going to happen. You're linking to a URL including the hash fragment for a reason and Google is honoring that with the link the user is provided in the search results. I don't think this has anything to do with Google providing sitelinks for named anchors in a page, simply that they are following your directive. I'd be inclined to setup a robots.txt rule for your future /posts/jump/<number> URLs so Google doesn't crawl them, as those URLs don't offer anything unique to Google that they aren't going to get via other links on your site. I would make sure that Google can continue to crawl the /posts/jump/<number> post IDs that they have already indexed, with a hope that it'll help them remove those URLs from the index in due course. 

I would offer a couple small pieces of advice to help things along:
  1. Redirect page 1 posts to the base URL with the named anchor

    When redirecting /posts/jump/<number> and the post you're redirecting to is on the first page, you should redirect to /my/friendly/url#123 not /my/friendly/url/1#123. A large percentage of the threads on your forum won't span multiple pages, however because you're redirecting to /my/friendly/url/1#123 - you're needlessly making Google crawl & index a duplicate URL, costing you server bandwidth and resources - which Google they then have to merge using a rel="canonical" tag.

  2. Duplicate trailing slash URLs

    Your URLs don't include a trailing slash (no problem).

    If a thread spans multiple pages, the paginated link to page 1 includes a trailing slash that isn't 301 redirected to the non-trailing slash URL.

    Per point 1, this is another URL that Google would need to needlessly crawl & index costing your bandwidth and server resources.

  3. Always reference your base thread URL when referring to page 1

    Following on from the above again, if you're viewing page 2 of a thread the link to the "start" that uses the < anchor text always links to /my/friendly/url/1.

    I would recommend that you always reference the base URL of /my/friendly/url when referring to page 1, to avoid a duplicate URL that needs to be merged via rel="canonical", bandwidth wastage & server resource wastage.

  4. Subdomains

    I noticed you've got several subdomains indexed that I don't think should be.

    If you haven't set about removing those already, I would add that to your list as a maintenance task as well.
Regards,
Al.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/14/12 8:54 AM
Hi everyone,

Thanks so much for your suggestions :)

Knysna: The text version looks fine to me. When I hover over the links, the title attribute for the links shows up? I have it like that because the descriptions are "secondary" information of sorts ... Essentially, it's a deeper description of the landing page when the link is clicked on, that isn't necessary on the page itself. If all of the descriptions were spelled out on the homepage, I think it would be way too cluttered and confusing. As far as I know, the link title attribute is being used appropriately.

It is terrible though that the homepage looks bad on your Blackberry. Unfortunately, I only have an iPhone and iPad to test with, and it looks fine on those. Is there a way you can take a screenshot?? Otherwise, I'll have to investigate mobile emulators to test what my site looks like on mobile devices.

Al: I will read that phantom redirect discussion. Thanks for linking! :) I'll also definitely remove the trailing /1 during the 301 redirection. Removing the /1/ within the actual page navigation links is a bit trickier because of the page navigation library we use. That's the way that it spits it out, and I will have to modify the library. In the meantime, the canonical should help there (or so we think). To be honest though, I think that removing the trailing /1 during the 301 redirection should make the bigger difference.

We also have our one staging / test subdomain that essentially is a mirror of the live site as of a month ago when we first launched. (It's pulling from a backup of the database.) I inadvertently had Google Analytics on for 48 hours before we went live. Just enough time for Google to find it and spider it, dammit! It's now behind a login wall so the pages are completely inaccessible but I can't figure out how to remove the 15,000 pages (which are completely duplicate content to the live site!) it indexed. Google Webmaster Tools only gives me the option to remove subdirectories or pages, and I'd hate to create an entire new site in Google Webmaster Tools for the subdomain because I don't want to give Google any more encouragement that it exists and is a real site!
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/14/12 4:36 PM
You're at no risk by verifying your subdomain.

I'd verify it and submit a URL removal request for the entire domain, since you have no intention of having that indexed in the future.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/14/12 7:10 PM
You convinced me :) However, I just took a look, and all but 57 pages are already deindexed, out of 15,000 that were there just yesterday. If they're not 100% gone by Monday, then I'll do that, but at this point it seems unnecessary. Lesson learned that Google still indexes and ranks pages that have absolutely no incoming links from anywhere!

Earlier today, I fixed the redirect issue where /posts/jump now redirects to the correct URL when on the first page.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/14/12 7:20 PM
Observation:

As mentioned earlier, vBulletin and InvisionBoard do the same 301 redirect technique that I do in order to jump to the last post in a thread. I just noticed that Xenforo (touted for being SEO friendly) actually does not provide any link to the last post in a thread to guests, and only to logged in members (for which they use the 301 redirection technique that I do, with almost exactly the same URLs, coincidentally).

I wonder if this is their workaround?? If it is, I'm certainly not prepared to not allow any guests to jump to the last post in a thread.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/14/12 10:41 PM
Explain to me the behaviour of the /posts/jump/<number> style URLs for both logged in and logged out users on your site?
This message has been hidden because it was flagged for abuse.
Re: DaniWeb lost all its traffic *AGAIN* Suzanneh 4/15/12 4:20 AM
vBulletin maintains the same base URL throughout the process though.

Jump link on the main forum page:  https://www.vbulletin.com/forum/showthread.php/398763-Lock-Forum-and-Subforums?goto=newpost

User is redirected here: https://www.vbulletin.com/forum/showthread.php/398763-Lock-Forum-and-Subforums?p=2278280#post2278280

What's indexed in Google:  https://www.vbulletin.com/forum/showthread.php/398763-Lock-Forum-and-Subforums

There's nothing for googlebot to get confused about if it's told to ignore "goto".

Anyway, I'm definitely not trying to argue with you! :-)  This just seemed like a simpler solution and I just wanted to point it out in hopes it could possibly  help.

I do hope you get the issue resolved fast!  I was in similar situation (loss of traffic) last year when I made some changes to my site, just before Panda; "stressful" doesn't even begin to cover it...

Suzanne

Re: DaniWeb lost all its traffic *AGAIN* AlbertaSurveyor 4/15/12 8:22 AM
Dani,

In my personal opinion you are loosing ranking because you are obsessed with the SEO.

Build the site for the people. The SEO will follow.

That is the best advice you are going to get.

~ Ben
Re: DaniWeb lost all its traffic *AGAIN* Hardc0d@d 4/15/12 8:45 AM
sorry, I don't agree with your statement @Alberta Surveyor,

I have 12 domains registered in google analytics and the sites WITHOUT ANY SEO has been done show about 50-60% traffic drop.
So I'd determine it has nothing to do with any search engine optimization efforts.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/15/12 8:53 AM
Ben,

Ben,

If you followed my older Panda-related threads around here, you would know this is very much not the case. It just so happens that a lot of SEO issues are related to site usability, and I am more than willing to correct those. However, I always have held firm whenever there was an SEO solution that negatively affected usability in any way, shape, or form.

For example, with this particular issue, as mentioned, XenForo's solution is to not display the /posts/jump/ URLs to guests/Googlebot and only show them for logged in members. That would most likely fix my issue completely as well, but I am not willing to go there because it would remove functionality from guests.

Another example ... We also have a tag cloud at the bottom of the page, which IMO is what made Panda hit us in the first place the first time around, since Google considers it keyword stuffing on every page. I was not willing to part with it, and so I jumped through hoops to figure out a way to get the tag cloud to not hurt me as much.

I have said this time and time again. EVERY THING that I do for my site puts my visitors first. I will definitely go out of my way for usability, because in many cases, I'll discover a problem through SEO and then realize that it is affecting my users as well. But you can read the hundreds upon hundreds upon hundreds of posts in the three threads I have in this forum, and you'll see that there has never been a single thing that I've done for SEO at the expense of my visitors.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/15/12 8:53 AM
> That is the best advice you are going to get.

I agree, because I've given the exact same advice at least a couple thousand times.
Re: DaniWeb lost all its traffic *AGAIN* Shaunguido 4/15/12 10:14 AM
Google won't admit this .. But I've seen it a few times. Major Site Changes kill rankings. I've had a site with a similar pattern to you. Site slowly dropped from the #1 slot to 3 to 5 then 10, 11, 15 then down to over 100.

I think GBot gets scarred when it sees so many changes at once and in a way sandboxes the content until it can access what's going on with the site.

I too, have useless profiles ranking on the first page of site: search results ahead of 100s of pure content pages that are of value and have a number of back links.

You said yourself that Google should know what content is better on your site, but really what they are telling you is that they have no idea what content is important and what is not. They knew on your old site what was and what wasn't .. But they don't believe this is the same stuff - 301 or not.

As to what the answer is ..  I'm not too sure, but I recommend that you be careful making too many more changes. And you need to do your best to re-establish what's good on your site and what's not by using inner links, and outer links pointing to the new urls (vs the old ones 301'd)

Re: DaniWeb lost all its traffic *AGAIN* iBill 4/16/12 11:10 AM
Dani, did you get this message from Google:  https://twitter.com/#!/mattcutts/status/191900489988849664
This might be affecting your site.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/16/12 1:35 PM
Shaunguido, I definitely think that what happened was Googlebot got scared when I redid everything at once and it needs to reanalyze everything to figure out what's what. But it crawled a couple dozen million pages since we launched the new version (according to Google Webmaster Tools) over the course of a month. How much more time does it need? :) I was fully willing to give it a few weeks before I started to get nervous and post this thread.

Wcspaulding, that looks like it's for sites that are redirecting users to weird places because they've been hacked. DaniWeb is behaving as expected.
Re: DaniWeb lost all its traffic *AGAIN* iBill 4/16/12 3:04 PM
Well, I thought maybe that the algorithm used to detect the weird stuff may have mistakenly flagged your site as one of the hacked ones. Mistakes do happen!
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/16/12 4:28 PM
Dani,

Another possibility is that your site has been inadvertently affected by Panda, which is nothing more than a theory.

In January nutsonline.com moved to nuts.com:


They did everything right as far as their site migration is concerned and actually cleaned a lot of things up in the change at the same time. Despite their best efforts and following all of the guidelines, their site got spanked with the change and at the time I theorised that it might have been as a result of the Panda signals not having been computed for their new domain/URLs.

Pre-Panda you could change your URLs or domain, perform the migration correctly and Google wouldn't care a lot. Post-Panda I don't think that is the case at all and that there is now some new signal, possibly part of Panda or elsewhere to do with trust/authority that needs to be re-established. For example between the Jan 6 and the end of March there were 3 different Panda updates from Google but none of them made nuts.com snap back into place to where it was when it was using nutsonline.com

I've been talking to the owner of nuts.com on and off since then to see how things are tracking and their rankings & traffic are coming back slowly but it is definitely a gradual process and not like going up 50 floors in an elevator.

Al.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/16/12 8:06 PM
Check out the following graph which illustrates our traffic patterns:


When we were hit by Panda, the change was completely instantaneous and overnight. We lost half of our traffic in a split second. Our recovery was equally as amazing towards the tail end of the summer. It just came back one morning. There was no slow recovery process at all. In fact, we went up those 50 elevator floors so fast that my ears popped!

Then, the second time we were affected by Panda, it was only for 36 hours. You can see the quick sliver of traffic from the graph during the fall.

Our latest decline seems a lot more gradual from my past experience with Panda. Who knows, it might still be Panda. Third time's a charm, I guess. But it just doesn't fit all the same patterns as the last two times. Of course, it's really hard to tell because Panda rolled out at the exact same time as our relaunch debuted.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/17/12 6:04 AM
Dani,

One thing I didn't think to ask because I just assumed you've checked this, but have you vetted your on site SEO?

For example, are your <title> tags the same format, are you still using <hX> tags in the same way, are you setting or not setting a description like you were before?

If you do a comparison of your traffic by URL type, can you see that you've lost traffic into your new category style URLs compared to your old ones?

What about if you compare traffic by forum (hardware vs software) or subforum if needed?

Are you getting any/many errors reported by your web server and/or Google & Bing webmaster consoles?

I know you mentioned that you think your tags are a good thing but they look a little spammy to me. Have you hooked up event tracking in Google Analytics to measure the real world use of those pages to determine how important they really are?

Al.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/18/12 1:21 PM
Hi,

For the most part, I tried to use H1 tags, meta tags, etc similarly to before. However, the new site structure and design are very different from the old, so it's hard for there to be an exact one-to-one relationship. It should definitely be noted that my reason for posting this thread in the first place was certainly not to complain, but rather to say, "We're on a whole new system now. I'm starting my SEO efforts from ground zero. I'm looking for suggestions and advice." Any helpful tips you could give me about what I might be doing wrong with my on page SEO are hugely appreciated!!!!!

We lost traffic across the board, not in some categories but not in others, or on certain types of pages but not others. In this respect, it does seem like a Panda-like sitewide "penalty" ... or, if not a penalty, just overall less pagerank sitewide.

I also want to mention that you guys were all right and I was wrong about the other Friday being a holiday. Traffic is STILL doing down every day, little by little. About 10% decrease in traffic per day, every day.

Our tags only get about 1,000 clicks per day, but they have one of the lowest bounce rates and lowest exit rates of any other page on the site, as they are primarily used by our active members. Additionally, they are not crawlable or indexable by Googlebot.
Re: DaniWeb lost all its traffic *AGAIN* Cmpsol_Tech 4/18/12 2:56 PM
Dani,
can i see which page you need seo optimization? as i have also lost all my traffic due to panda effect and recover it to 100% now again lost.
Re: DaniWeb lost all its traffic *AGAIN* iBill 4/18/12 3:16 PM
Some others have been reporting steady traffic decreases since the beginning of February:  http://groups.google.com/a/googleproductforums.com/forum/#!searchin/webmasters/Panda$20recovery|sort:date/webmasters/JQYuxNhM4og/YPkdtXXRQeQJ . Could be the nice weather that we are having this year. 
BTW, I do like the new look of your site!
Re: DaniWeb lost all its traffic *AGAIN* iBill 4/18/12 3:18 PM
Sorry, the above link did not post correctly. Just ignore it.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/18/12 5:29 PM
What about your internal link structure?

Do you have the same list of forum categories as before, named identically?

Are you linking from your home page or laterally through your site as you were before?

If your internal link structure has changed, you could be pushing less PageRank into your most important pages which is causing a drop in rankings across the board.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/18/12 10:38 PM
> Do you have the same list of forum categories as before, named identically?

We had about 100 forum categories and sub-categories and sub-sub-categories. I condensed this to about 80 by rolling some of the sub-sub-categories up a level, and 301 redirected them to their parents.

> Are you linking from your home page or laterally through your site as you were before?

No. The forum index that currently exists on the sidebar of the homepage used to be on a different index page that no longer exists. That page was 301 redirected to the homepage.

> If your internal link structure has changed, you could be pushing less PageRank into your most important pages which is causing a drop in rankings across the board.

It *has* changed, but in a way that I think is for the better, as far as usability is concerned. I was expecting there to be a delay while Google recrawled the entire site, but I wasn't expecting *THIS*! We've lost 50% of our traffic at this point and it's still going downhill every single day.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/18/12 11:55 PM
I'm looking at your home page and thinking you've got a lot of links on there, 208 if my CTRL+F was accurate. As I'm sure you're aware, simplistically each link on a page receives the PageRank of the page divided by the number of links. So you're effectively giving each link on your page 1/208th of your home page's PageRank.

In your thread list on your home page, I'd be inclined to remove the links to the author, thread category & tags for each thread. Your users are going to be more interested in reading the thread than clicking into those other type of URLs from your home page. 

I also notice that you include your entire forum hierarchy in your main navigation & repeat primary forum categories in the sidebar of your home page. It might be worth attaching Google Analytics events to each of those two sets of navigation items to see which group gets clicked more. If it turns out that your header navigation forum categories don't get clicked all that often or only your primary forum categories and not the sub-forum categories, it might be worth reducing your main navigation to reduce the total number of links on the page a little further.

Did your old site use tags like you do now?
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/19/12 11:02 AM
Hi,

Thanks for your suggestions :)

Yes, the old site used tags the same way as we do now. Also, all of the links in the top navigation dropdown menu are generated via an external Javascript blocking googlebot via robots.txt, and therefore shouldn't count towards diluting PR, since googlebot has no way of knowing they are even being generated in the first place.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/19/12 4:09 PM
Do you have a Google query to monitor how many of your old URLs are still in the indexed & haven't had their 301 redirects processed?

If you do, are you noticing that the number of indexed old URLs is dropping at a steady pace given that heavy crawling Google is doing at the moment?
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/19/12 10:22 PM
I'm confused by your question

Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/20/12 6:21 AM
In your original post, you said you changed the URL format for your site again in March 2012. I'm wondering if you can see Google reducing the number of indexed URLs using your old format and increasing the number with your new format.

I notice that you've got 37,900 URLs indexed under /forums/ which you said was your old URL scheme. When I follow those URLs, I get redirected to the correct location which is great but I'm wondering why they are still hanging around after such a long time. Checking the HTTP request/response sequence and I see there is an intermediate redirect between your old format and your new format /forums/thread344044.html -> /articles/view/344044 -> /web-development/databases/oracle/threads/344044/transform-data-into-report-format. If you've got a way of going from the first URL to the last in a single step, I'd advise that you implement that in general as it might help speed up getting those old URLs out of the index. 

You have 83,400 URLs indexed under your /members/ section of your site. Do those URLs really serve anyone from Google any good or are they simply useful for people once they get onto your site? If they aren't that useful to people coming from search engines, I'd be inclined to add a meta robots noindex,follow to those URLs to remove them from the index.

Every forum has a zillion threads, which means you end up with a zillion pages listing those threads. At the moment you have all of your paginated URLs indexed but I doubt that you'd get a lot of traffic landing on those URLs. You should check with Google Analytics for all of this but if they aren't useful, I'd yet again remove them with a meta robots noindex,follow or by using a rel="canonical" on pages 2+ back to your base URL that is being paginated.

In a similar vein as the above, I just realised that you've got little tabs above each forum (news, reviews, interviews, ...). If each of those tabs simply lists URLs that are ultimately available via the 'all' option, I would remove all of those from the search results using a meta robots noindex,follow or rel="canonical" tag as well. 

For all of the URL formats you choose to deindex (if any), once all of those URLs have been removed and a site search with inurl/intitle return no results - I'd then block those style URLs with robots.txt so Google isn't wasting your server resources and bandwidth and your're not wasting your crawl allowance with duplicate URLs.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/21/12 8:34 PM
> I'm wondering if you can see Google reducing the number of indexed URLs using your old format and increasing the number with your new format.

No. While the switchover *is* happening, overall we have a LOT fewer pages indexed even though we have just as many pages.

> I notice that you've got 37,900 URLs indexed under /forums/ which you said was your old URL scheme. 

We shouldn't have any URLs indexed under /forums/ anymore. We haven't used this format in over a year.

> If you've got a way of going from the first URL to the last in a single step, I'd advise that you implement that in general as it might help speed up getting those old URLs out of the index. 

Unfortunately, I don't, because as you can see, the old format has no knowledge of the category information, while the new URL contains all of the category information. There needs to be an intermediary page to fetch what that information is from the database since it's not known in the URL itself.

> You have 83,400 URLs indexed under your /members/ section of your site. Do those URLs really serve anyone from Google any good or are they simply useful for people once they get onto your site? If they aren't that useful to people coming from search engines, I'd be inclined to add a meta robots noindex,follow to those URLs to remove them from the index.

As mentioned above in this thread: We currently have nearly one million members, and therefore nearly one million member profiles. However, we choose to use the rel="noindex" meta tag directive on about 850,000 of the member profiles, only allowing those by good contributors to be indexed. I think it's a happy medium between allowing our good contributors to have their profiles found in Google by prospective employers and clients searching for their name, and not having one million member profiles saturate our search results. We allow just under 100,000 of our 950,000+ member profiles to be indexed.

As you can see, we allow just under 100K member profiles to be indexed (the other 850K have noindex), so it's accurate that you're seeing 83,400 of them in Google. We allow this because a lot of our regular members appreciate their profiles being found when people search their name in Google, as their participation on DaniWeb (a professional community) has resulted in job offers, job promotions, career advances, etc.

Every forum has a zillion threads, which means you end up with a zillion pages listing those threads. At the moment you have all of your paginated URLs indexed but I doubt that you'd get a lot of traffic landing on those URLs.

Originally when we relaunched last month, all but the first page had the meta noindex directive, and all of the pages of a multi-page thread had the canonical pointing to the first page. However, as we started to lose traffic, after a few weeks, I decided to set it up the way that it was before our traffic was lost. This seems to be the way recommended by Google: to not use noindex, but instead use the rel=next/prev directives to show Google that it is part of a multi-paginated thread, and the rel=start directive to show Google where to start from, and let Google handle it itself. From what I was able to gather, I actually confused Google by using the same canonical URL for multiple pages that contained different content. Or so it seems.

> For all of the URL formats you choose to deindex (if any), once all of those URLs have been removed and a site search with inurl/intitle return no results - I'd then block those style URLs with robots.txt

According to Google Webmaster Tools, there are about 5 million backlinks currently using the old formats. Using robots.txt on all those links would do more harm than good IMO??
Re: DaniWeb lost all its traffic *AGAIN* TheDonald 4/21/12 10:39 PM

> If you've got a way of going from the first URL to the last in a single step, I'd advise that you implement that in general as it might help speed up getting those old URLs out of the index. 

>>>Unfortunately, I don't, because as you can see, the old format has no knowledge of the category information, while the new URL contains all of the category information. There needs to be an intermediary page to fetch what that information is from the database since it's not known in the URL itself.

Hi Dani,

I don't understand your answer. The unique id for the forum thread in the example above was in the first, second and third URL. Why can't you do a database lookup with the id from the first URL and 301 to the final URL? All category information is in your database. It is crucial that any unnecessary 301's be removed.

I wouldn't recommend this to just anyone, however since you don't have a problem hiding links from Googlebot (you mentioned your tag cloud and drop down menu) why not remove your jumplinks from the HTML and insert them via JavaScript or iframes if possible? Or else temporally only show them to members. If they are confusing Googlebot they are not worth it.

You mentioned that you just started using rel/prev/next on your page 2/3's. Now is not the time to be making additional changes to your site, you are asking Googlebot to index an additional infinite number of pages. If you were using noindex on page 2 previously then you need to use it now. Let Googlebot figure out your basic 301's first. One major change at a time.

I've had Panda problems in the past and I've also done successful large scale redesigning of URL architecture. In my opinion you are likely not dealing with a Panda issue but a large scale URL architecture change gone horribly wrong.

Good luck!
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/22/12 10:41 AM
> Why can't you do a database lookup with the id from the first URL and 301 to the final URL? All category information is in your database.

That's what I'm doing, but I'm using an intermediary page to do it because the PHP framework that I am using now doesn't give me access to the first type's URL structure.

> I wouldn't recommend this to just anyone, however since you don't have a problem hiding links from Googlebot (you mentioned your tag cloud and drop down menu) why not remove your jumplinks from the HTML and insert them via JavaScript or iframes if possible?

I do it with tag clouds because, in addition to the SEO benefits, my method allows the tag clouds on our category pages to be cached on both the server-side and the client-side, therefore causing a dramatic speed improvement. I do it with the navigation menu because, in addition to the SEO benefits, caching Javascript that will load on every page of the site is much faster than having to generate HTML for 100 links on every page load.

Dynamically creating so many individual jump links, placed throughout the entire page, would actually be a very severe decrease in page load time and (therefore) usability as well. Regardless of how much I like to focus on SEO, I do put usability concerns for humans as a top priority.

As mentioned, the Xenforo forum software shows them to members only as their solution to this problem. However, my frustration is they were never a problem for us for the 10 years that I've used them in the past, and I do know that non-logged in members do use them, so I don't want to take such a drastic approach. Obviously Google figured it out before :( It's just frustrating!

> You mentioned that you just started using rel/prev/next on your page 2/3's. Now is not the time to be making additional changes to your site, you are asking Googlebot to index an additional infinite number of pages. If you were using noindex on page 2 previously then you need to use it now. Let Googlebot figure out your basic 301's first. One major change at a time. 

In the old version, we didn't noindex. We used prev/next in the old version. When we switched to the new version, I started to use noindex because I thought that it would improve SEO. When I saw that traffic was decreasing, I removed the noindex and went back to the old method of prev/next because my first instinct was to try to replicate the old system as much as possible.

> I've had Panda problems in the past and I've also done successful large scale redesigning of URL architecture. In my opinion you are likely not dealing with a Panda issue but a large scale URL architecture change gone horribly wrong. 

I agree with you, but I am currently really scared because traffic is still decreasing every single day by about 10% (the bleeding hasn't stopped), and we're now worse off than we were with Panda. I really am just so frustrated at this point that I don't know what to do. I feel like I'm in a catch-22 because on one hand I want to keep trying to improve the situation, but on the other hand the best solution might be to get things as close to the old system as possible, and the third option would be to not touch anything and leave everything as-is to give Googlebot a chance to understand the new architecture. Which option do I choose? I feel so helpless just sitting back and doing nothing and watching traffic dwindle day after day. The problem is that it's been over a month at this point. How much more time can I honestly give??
Re: DaniWeb lost all its traffic *AGAIN* Bigwebmaster 4/22/12 3:23 PM
Dani,

If it makes you feel any better, you are not in the boat alone. The same is happening to our Ozzu site (you advertise there). We are pretty much a forum in the same niche as you and are currently getting the worst traffic levels so far this year. We had a great start to 2012 all the way up to mid February, then the bleeding started. For us it seems like around Thursday or Friday of each week is when we get our biggest declines and then they carry into the next week where we get hit again on Thursday or Friday -- it just keeps repeating every week it seems. We have not changed our structure or anything in the last year, so in my opinion your structure change and the timing of it it just a coincidence.

Matt Cutts did recently mention this:

https://plus.google.com/109412257237874861202/posts/BBDZDq3a5DR

But I think that is not related as he says that is fixed now and our bleeding is still continuing like yours :(
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/22/12 4:07 PM
Hi,

Thanks for your encouragement. :) Our bleeding started a little over a month ago, coinciding exactly with when we switched to the new system, and pretty much follows the same pattern as you. Unfortunately, that parked domains issue seems to be entirely unrelated.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/22/12 4:21 PM
Dani,

I think it might be time to just let it sit for a little while, which I realise isn't what you want to hear.

You've made a huge change to your site, you know the 301 redirects are in place and that Google can crawl your site without any issues.

The only other changes I'd be making now, since we're all striking possible issues from our lists are things to speed up crawling & indexing. For example my comment I made previously about your /posts/jump/ URLs that you fixed or the paginated links is a good example of this. Google is wasting time by crawling duplicate URLs, which then get merged using rel="canonical". Anything you can do to improve the efficiency of your site, so Google is spending time cleaning out your old site and getting on with the business of understanding your new site is a good thing.

What do you think?

Al.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/22/12 5:10 PM
You're right. That's certainly not what I want to hear.

But does that mean I shouldn't continually try to improve the site? Should I further tweak it to be closer to what it used to be, or just let it be as is? I'm at a loss and frustrated as ever.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/22/12 5:28 PM
Dani,

I've read quite a few stories written on several different sites about your problems, starting with your Panda issues, and now your redirect issues.

My main thought is that you have too many redirects and need to kill them all.

While I didn't have as many pages are you did, I still had tens of thousands of pages.
Google will sort things out over time as they did with my pages when I redid my page naming structure.
Yes this will kill some of your traffic, but over time it will pick up again.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/22/12 6:58 PM
I respectfully but completely disagree with OkayNetwork about killing/removing your redirects.


As such, if Dani stopped/removed all of the 301 redirects in her site due to site migrations, all of the inbound links that she has accrued over time would stop being counted & her site would suffer permanently moving forward. Granted it would come back, if & when she amassed the same number/quality of links but when you're talking about a site as large as hers - that could take years - so I definitely wouldn't do that under any circumstance.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/22/12 7:47 PM
I just see what she is going through as a Google index mess and instead of trying to fight it my suggestion was more to just let the pieces fall where they may. If we were talking a few hundred even a few thousand redirects I'd say keep on the fight, but we're talking hundreds of thousands. I'm not saying she shouldn't promote her site and continue to build natural quality links or continue to let her members create quality content. Just that perhaps it's better to stop trying to fight the bot and let it drop the old pages completely. Now I would say that this isn't something that webmasters should do over and over again, as it will have a downside initially that you've pointed out.
Re: DaniWeb lost all its traffic *AGAIN* Steven Lockey 4/23/12 3:52 AM
Its likely the double redirects (301 - > 301 -> 400) are part of the problem. If they get corrected (301 -> 400) I suspect the site might showing improvement relatively quickly.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/23/12 5:53 AM
Dani,

I found another issue with your site, albeit small.

A URL like /{number}, such as /350 that doesn't exist, doesn't return a 404 error.

Have you thought anymore about applying a meta robots noindex,follow to your Solved & Unanswered forum category filters, since they duplicate URLs found within All Articles?

Al.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/23/12 8:52 AM
> A URL like /{number}, such as /350 that doesn't exist, doesn't return a 404 error.

Thanks, that's a bug. I'll fix it right now.

> Have you thought anymore about applying a meta robots noindex,follow to your Solved & Unanswered forum category filters

I went ahead and did this for the Unanswered filter, because, as it turns out, I was doing this in the old version and forgot to carry it over. I was doing this because I noindex all threads with no replies, so essentially the Unanswered listing is a listing of threads that are each noindexed themselves.

I am not doing it for the Solved filter for a few reasons:
- There are not very many solved threads in comparison, so it isn't a hugely duplicated listing
- Our solved threads serve as a knowledge base of sorts (i.e. the best of the forum threads) and so I don't mind it being a starting place for newcomers, and I don't want to noindex the pages on our site that are explicitly navigation for only our best content
- We didn't noindex it before, so I don't want to risk it now

Again, thanks so much for bringing the 404 /posts/jump bug to my attention, and also for pointing out something that I was doing before the relaunch and stopped doing.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/23/12 9:27 AM
Wait ... I'm actually confused :)

Were you trying to point out that /posts/jump/1000000 doesn't give a 404 error even though that post doesn't exist? That's what I thought you meant, and I fixed that.

Were you referring to something else? Currently if you are trying to look at page 100 of an article that only has 5 pages, it redirects to the first page of the article using a 302 (temporary) redirect, since that page could potentially exist in the future.
Re: DaniWeb lost all its traffic *AGAIN* Cmpsol_Tech 4/23/12 1:28 PM
Dani

we have just recovered from google worse effect. after 10 days my website has been recovered 95%.

my recommendation is do not make any changes in website for few days. google will recover all your traffic automatically. 
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/23/12 1:56 PM
Hi,

I'm glad to hear you recovered!

Unfortunately, I've been waiting over a month for Google to figure the new site out. Sometimes you can't just do nothing and wait for the problem to go away ;)
Re: DaniWeb lost all its traffic *AGAIN* Cmpsol_Tech 4/23/12 2:43 PM
well, I've seen your site. it seems you have changed many thing and the page speed seems to increased. 
hoping your site will recover soon.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/23/12 2:50 PM
Thanks.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/24/12 1:26 AM
A URL like /{number}, such as /350 that doesn't exist, doesn't return a 404 error. 

From what I can gather a URL like http://www.daniweb.com/350 is the 350th page of your forum threads. If that page number doesn't exist, it doesn't return a 404 and it should.

Regarding your use of a 302 for a thread page number that doesn't exist, that should also return a 404 error until the thread page number exists. Reason being, if it returns a 200 or 302 - I could spam your forum with millions of low quality URLs by just linking to URLs on your site that don't exist. As a result of the 302 (temporary) redirect, Google will keep the non-existent garbage thread page number in the index when it shouldn't.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/24/12 9:08 AM
> From what I can gather a URL like http://www.daniweb.com/350 is the 350th page of your forum threads. If that page number doesn't exist, it doesn't return a 404 and it should.

I'll fix that.

> Regarding your use of a 302 for a thread page number that doesn't exist, that should also return a 404 error until the thread page number exists.

Here's the reason I don't want to do that ... Thread pages are allowed lets say 15 posts per page. So, suppose there is a two-page thread with 17 posts. Now, suppose two individual posts in the thread end up being deleted, bringing the thread down to just one page. There might still be some links to page two of the thread. From a usability standpoint, I want someone coming in from those links to be redirected to the first page of the thread instead of just getting a 404 error and thinking the entire thread is gone.

And I want to do a 302 redirect because I don't want to tell Google that this page is permanently going to be redirected to the first page of the thread ... it just is for now, in this circumstance.

Once two new posts are contributed to the thread, then the external links will once again correctly point to page 2 of the thread. Page two just temporarily redirected to page one in the interim between the thread post count decreasing and ultimately increasing again.

This is just an incredibly simplistic example, but in the grand scheme of things, many, many posts get deleted for various rule violations every single day, so this would definitely be a fairly common usability concern.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/24/12 10:21 AM
If a page has been removed and does not exist anymore, you shouldn't be doing a 302 redirect to another page, even the 1st page of the thread.

If I have something that's 10 pages long and I remove 7 of them, page 4-10 shouldn't redirect to page 1 simply because I'm afraid of loosing the rankings of pages 4-10, and thinking well eventually they'll be created by my visitors. What if they never are?

This is why I'm thinking you need to bite the bullet and kill all of your redirects.
It just seems to be creating more of a mess for you than actually helping your site.

Really Google does not want to see that you're removing pages and having them redirect until the page exists again, and especially if it takes a long time for the page redirects to stop redirecting and actually be a legitimate page again.

Listen, it's going to harm your site by removing the redirects, but it's also going to clean up a lot of garbage that is also harming your site.
Honestly what if the garbage 302's are the reason Google doesn't like your site as much as it used to?

It's one thing to change naming structure and do 301's, but if you're removing pages for whatever reason and issuing a 302 in hopes that one day those pages might return, I simply think that you're asking for trouble.
Re: DaniWeb lost all its traffic *AGAIN* TimKoster 4/24/12 10:29 AM
I'm sorry to hear that Dani. Same thing has happened to us, though we've worked for over a year on fixing any Panda issues-- without success.  We've provided our searchsystems.net service online for over 16 years and have been ranked #1 for "public records" since Google's inception.  But after a year of punishment they decided to really put the hammer down this week and blocked our home page.  Like you, I'd sure like to know what in the world it is we did wrong and how to fix it.

One thing I do know is that this is incredibly painful.  I have the greatest respect for you for hanging in there.  In our case my wife and I have just been issued a foreclosure notice and will lose our home.  We've lost our retirement and savings trying to keep this alive, and now the business will go to our creditors.  If we'd tried to do something evil I'd understand the punishment-- but we were trying to do something good.
Re: DaniWeb lost all its traffic *AGAIN* ClayburnGriffin 4/24/12 10:42 AM
Wow.  Google is really picking on you.
Re: DaniWeb lost all its traffic *AGAIN* Phil Payne 4/24/12 10:58 AM
> In our case my wife and I have just been issued a foreclosure notice and will lose our home.  We've lost our retirement and savings trying to keep this alive, and now the business will go to our creditors.

Cue 'Hearts and Flowers' on a scratchy violin.

You got some pretty good criticism and advice in http://productforums.google.com/forum/#!profile/webmasters/APn2wQfNs5ZGcyGN2k8S8gNVgCpOo6n71PS8rq2jiBnmVi3iLIgI3MnYifhmfmD8cGjlu4JiPfAJ/webmasters/CpQK1DwyWzA/ncAmJZorcZUJ

Any business model that depends to such a critical extent on the largesse of a free search engine is flawed.


Re: DaniWeb lost all its traffic *AGAIN* iBill 4/24/12 11:42 AM
Your old site pac-info.com is coming up #4 for "public records". When I click on the link, it goes to your present website.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/24/12 3:33 PM
Dani,

Thanks for providing an example of why you're using the 302 redirect on a thread page number that doesn't exist & I agree that is a valid usability concern.

If you're certain that you want to keep a redirect in place for those missing page numbers, I think what I'd ideally want to see is a 302 redirect in place for a period of time after the page count reduces. Subsequently, lets say two weeks for the sake of an example, after the last activity in the thread - if the posts haven't increased to re-instate the lost pages again, you need to either 301 those missing page numbers back to page 1 or issue a 404 for those URLs. 

In general, I think you want to avoid having Google deliberately crawling 302 redirects, for the reasons I outlined in my previous comment about this particular issue.

Al.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/24/12 4:04 PM
> If I have something that's 10 pages long and I remove 7 of them, page 4-10 shouldn't redirect to page 1 simply because I'm afraid of loosing the rankings of pages 4-10, and thinking well eventually they'll be created by my visitors

I'm not doing it because I'm afraid of losing the rankings of pages 4-10. I'm doing it because I don't want valid links from other sites to be 404 pages for users. If they are specifically looking for page five of an article, and I know it doesn't exist, but I know what article they're looking for, I at least want to help them out by pointing them to its first page so that they can ultimately find the information they're after, rather than just saying "Sorry, doesn't exist. Can't help you. Go away now." It's a usability concern.

> This is why I'm thinking you need to bite the bullet and kill all of your redirects.

I really don't see why I should basically make every single external page that's ever linked to me useless. We are not an ecommerce site where most people come in through the homepage and browse around the site. *All* of our traffic is either via our backlinks (aka referral traffic) or Google traffic. We have very little direct traffic in comparison. With all of our google traffic gone, nearly all of our traffic right now are from backlinks. In fact, our referral traffic is our best traffic BY FAR when it comes to time on site, bounce rate, conversion rate, etc. Backlinks don't just exist for SEO!

TimKoster, Best of luck to you and your wife. I know where you're coming from. As you say, hang in there!

Clayburn, fancy seeing you here. And yes. GRRR! Any advice for me? :)

Alistair ... but again, I've been doing it this way for the last X number of years. So do I want to start making changes thinking that *this* is what Google would prefer, or do I want to try to keep things as is / the old way that's worked for 10 years so far?
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/24/12 5:47 PM
As for backlinks,  those should be preserved with a 301 if the page has been moved somewhere else, which I believe you already have done with just about every page you have that was on your old structure, correct? So this preserves your backlink traffic.

Honestly I don't see how you could even think the 302's were not an issue as your rankings and traffic dropped so much from Google.

You did a great job at recovering from Panda, but I still think you're trying too hard to make sure every little bit of traffic is captured and not lost, and you've made a mistake with using 302's. Correctable? Yes. Simply let the 302's die, and keep your 301's to preserve your backlinks. I mean consider that any links that do appear on any site that's of value today is probably using your old naming structure right? Sure you might kill a few newer links, but honestly, things will sort itself out with Google as they always do.

Listen, pages get deleted all the time for one reason or another.
Imagine all the links your members have placed on your site that no longer actually go to the site or page they used to?
How often do you remove those links or do you not even look for them as they're on very old pages?

Also you do not want to keep things the old way that's worked for 10 years simply because search engines have evolved and so should everybody's website. I mean honestly if you don't care about search engine traffic and care more about your referral traffic, then what you are doing is fine, but you'll never recover your search engine traffic keeping on the same path you are.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/24/12 5:59 PM
> As for backlinks,  those should be preserved with a 301 if the page has been moved somewhere else, which I believe you already have done with just about every page you have that was on your old structure, correct? So this preserves your backlink traffic.

In that case, I misunderstood. I thought you were telling me to kill my 301s.

> Honestly I don't see how you could even think the 302's were not an issue as your rankings and traffic dropped so much from Google.

Because we've been using 302s for internal links *forever*. Pre-panda. Post-panda. Now. Etc.

> I mean honestly if you don't care about search engine traffic and care more about your referral traffic, then what you are doing is fine, but you'll never recover your search engine traffic keeping on the same path you are.

Referral traffic is real humans. IMO there has to be a way to not inconvenience humans to fix Google rankings.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/25/12 10:25 AM
Simply because you were doing it for a long time and Google never gave you a penalty for it doesn't mean Google has not changed the rules, as they tend to always be changing the rules, especially when issues are found that can game their index.

Are you sure you're not 301 > 302 > 200 on any of your pages???
Matt Cutts has warned publishers of doing inline redirects.

So while I applaud your efforts to you retain your visitors that stumble upon a link on a site or on a search engine result that goes to your site that may not exist anymore, you're also pissing off the search engines by telling it that the pages do exist but are moved. You didn't actually move the page, so you're doing a bad thing by telling the search engines you did.

So until you remove the 302's, you're not going to recover because you're being punished for telling the search engines a page exists which does not.
You're gaming the system without realizing what you did, and you're being punished algorithmically, so there won't be a notice in Webmaster tools.

I can only imagine how many 302's you must have since you have quite the page count on your site.
Re: DaniWeb lost all its traffic *AGAIN* Suzanneh 4/25/12 10:30 AM
Dani, are you starting to recover a bit?  I was looking at your Quantcast stats and I'm seeing a slight increase the past couple of days.

Suzanne
Re: DaniWeb lost all its traffic *AGAIN* Bigwebmaster 4/25/12 10:36 AM
I am starting to level out as well, this is the first week without decreases.

There is also a major Google Update that started yesterday:

http://googlewebmastercentral.blogspot.com/2012/04/another-step-to-reward-high-quality.html

It is affecting according to Matt Cutts about 3% of searches.
Re: DaniWeb lost all its traffic *AGAIN* ClayburnGriffin 4/25/12 11:40 AM
My guess it's the too many URLs of things that are "moved" or deleted.  You should see what percentage that is compared to the rest of the site.  You probably should be using 301s instead of 302s, unless the content has a chance of ever returning to its URL.  
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/25/12 3:53 PM
ClayburnBriffin,

Let me catch you up on Dani's problem since you don't seem to have read everything that has gone on.

She changed her page naming structure, to which she had redirected all her pages (too many to count) to the new page names. She used 301's which was fine. What she is doing is when a page is removed that is a series of pages, she is redirecting requests for the missing pages with a 302 to the 1st page of the series in hopes that eventually somebody will come along and add those pages back by posting to the series. Basically the 1st page usually is related so you'd think that would be ok, but technically what happens with redirects is the rank juice is transferred to the 1st page.

So she did right by the 301 redirects, but did wrong by then using 302 on pages that have been removed.
She's using the excuse that eventually these pages will be recreated and wants to insure that visitors don't leave her site because they got a 404 error.
To Google this is considered to be gaming the index.

She did some very positive changes to her site when it was hit hard by Panda and she not only recovered, and I seem to remember her being quoted as saying it was 110% recovery, and now the new algorithm changes have hit her site again. I believe her problems are due to her use of 302's which should almost never be used, except in rare cases and she's using them as commonplace.
Re: DaniWeb lost all its traffic *AGAIN* Bigwebmaster 4/25/12 4:54 PM
Good summary for ClayburnBriffin, but I would disagree OkayNetwork. I see no reason why the Google engineers would go... "Oh wait, they are using too many 302s and we need to penalize them for that because they are gaming the system!" -- Do you have any literature or quotes from a Google representative that would indicate that would be considered gaming the system?

Please watch this video:

http://www.youtube.com/watch?v=r1lVPrYoBkA

Matt Cutts goes into extensive detail about 301s and he briefly mentions 302s as well. As you know 301s are for permanent changes. 302s are just that, temporary. I see no reason why Dani can't use the 302s like she is using. They are not permanent and they are a good thing for the visitor as she is helping them find what they were after since the deleted posts were part of that thread which still exists. In my mind if its good for the visitor, then do it -- and I know Google wants what is good for the visitor too. In that video link above Matt Cutts at some point says this:

"If it is going to be temporary, or you might undo it after awhile, that is a good opportunity to use a 302, or temporary redirect."

I really doubt Dani lost half her traffic because of these 302s, that just seems absurd to me. In my opinion there is something going on in the much bigger picture, and that has nothing to do with her structure changes or the 302s. It has to do with Google being busy at changing their algorithm in the recent weeks. Her timing of her structure changes in my opinion were just a coincidence as I had a similar drop and I never changed anything. This week marks a change for me though, traffic is actually up somewhat compared to the previous weeks -- and Google just announced a major change yesterday that affects 3% of search queries.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/25/12 5:24 PM
I am quite confident that I was NOT hit by Panda or a Google algorithm change.

My Google traffic decline coincided exactly with the day of our relaunch. In addition, I finally just noticed that my Bing/Yahoo traffic also took a hugely massive hit the exact same day as well. I think it's just a coincidence that Google released a new algo update at the same time, because otherwise what are the chances that Google and Bing both decided to take a 50% hit the exact day of our massive relaunch.

I really want to say that this update is a case of a relaunch gone massively, massively wrong. As mentioned in my original post in the thread, there was an initial problem with the 301 redirects and Google Webmaster Tools is currently reporting just under 600,000 404 errors.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/25/12 7:00 PM
But this isn't a 302 redirect to tell Google, hey the page has moved.
It's an attempt to prevent a 404 error on a page that was rightfully removed.
I've seen a few spammy sites use this tactic to game Google's index.
So this turning on 302's and turning them off when the actual page is re-created isn't an acceptable use of 302's.

Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/25/12 10:12 PM
Dani,

Just noticed another small thing I'd update on your site, you're not including Google Analytics on your error handlers.

For instance if you view a 404 page on your site - there is no Google Analytics script included.

I'd include Google Analytics on those URLs, because it offers an easy way to see when users are getting errors by viewing Content->Site Content->Pages, using the Page Title tab and filtering for your "Oops! 404 Error" title. Drilling into that item will also show you all of the URLs that users are actually getting the 404 errors on and you're also able to isolate where those errors are being generated from (your site, third party sites, ..).

Using your web server statistics, say awstats or similar - how are you 404 errors trending over time?

Al.


Re: DaniWeb lost all its traffic *AGAIN* Cmpsol_Tech 4/25/12 10:34 PM
Well. All advice O.K.
but i have noticed in my website, i have deleted some pages and google was fetching that page with status 302 instead of 404 and my website lost rankings in google search.

once i have redirected error pages with 301 redirect my website ranking has recovered.
so try 301 redirect instead of 302.
Re: DaniWeb lost all its traffic *AGAIN* Steven Lockey 4/26/12 1:26 AM
Hmmm, would it not be possible to return a 404 status code, then use a on-page meta-redirect to send the users back to page 1?


Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/26/12 5:47 AM
If you're going to do that though Steven, why not simply use a permanent redirect & be done with it?

It upholds Dani's goal of a solid user experience & removes the rubbish URL from the index that shouldn't be there since it no longer has content on it.

I really do appreciate the position that Dani is taking, I just don't believe that long standing temporary redirects for a URL that might exist again in the future is right.

My earlier comment was that if she really wants to use a temporary redirect because she knows (which she could find out with data) that threads that reduce in page count often increase again, that she should use a temporary redirect for 2 weeks past the last thread post. If no one posts in a thread for 2 weeks, then the temporary redirect turns into a permanent redirect to remove the thread page number(s) that no longer exist.

Thoughts?
Re: DaniWeb lost all its traffic *AGAIN* Steven Lockey 4/26/12 6:15 AM
Because Googlebot should see the 404 but ignore the meta-redirect.
Users should get taken to the 404 page like any 404 page, but be instantly redirected to the correct page.
That should give Dani the functionality he wants and at the same time give Google what it expects.

The ideal solution of course would be to have a 'link to' button on each post that uses the post ID instead of the page number, then the system can work out what page to send the user to from the post ID, but that would be more work ofc.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/26/12 12:15 PM
She can't use a 301 because at any time, even after 2 weeks as you suggest, somebody could come along and post more to the thread, basically creating the page that a search engine was told has been permanently moved. Permanent redirects are not meant to be place holders for removed pages until they get created again.

I sort of liked the 404 meta redirect idea that Steven Lockey had, but I wouldn't use a meta redirect, instead I would have the 404 page say that the page was removed, and if there is a related page, such as in a series of pages, display the url of the 1st page in the series. Allow the user to click on the link to be taken to that page if they so desire to still continue eventhough the page they were trying to view has been removed. Careful that if the 1st and only page without a series of pages is removed that you don't also have the 404 error point back to the page that no longer exists. Seems like a simple database query would be able to tell if the 1st page still exists or not for a page that was landed on that is part of a series of pages but was removed for whatever reason.

What I've done on my sites is when visitors get a 404 error they're encouraged to visit my homepage or search around using the search box on my 404 page. I don't just give them a message that encourages them to go away.
Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 4/26/12 4:28 PM
The other option that I see quite often is that a thread gets automatically closed after X days/weeks of inactivity.

If that sort of scenario was implemented, then Dani could use the 301 redirects as soon as the thread is closed.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 4/26/12 9:40 PM
> Hmmm, would it not be possible to return a 404 status code, then use a on-page meta-redirect to send the users back to page 1?

I would consider that a form of cloaking.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/27/12 9:18 AM
So why not use my idea of simply telling the user the page doesn't exist and then if that page that was removed has pages in a series that it belonged to, simply post the url of the 1st page in the series and your visitors simply have to click on the link to be taken to that page? You would of course have to "noindex,nofollow" the 404 page. It's not exactly what you were looking for, but it also doesn't piss off the search engines.

How many 302's are you running right now anyways?
I mean are we talking a few hundred, a few thousand, tens of thousands?


Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 4/28/12 6:38 PM
Ok, now I found what I was looking for......


http://support.google.com/webmasters/bin/answer.py?hl=en&answer=40132


And I quote....

302
The server is currently responding to the request with a page from a different location, but the requestor should continue to use the original location for future requests. This code is similar to a 301 in that for a GET or HEAD request, it automatically forwards the requestor to a different location, but you shouldn't use it to tell the Googlebot that a page or site has moved because Googlebot will continue to crawl and index the original location.

404

The server can't find the requested page. For instance, the server often returns this code if the request is for a page that doesn't exist on the server.

If you don't have a robots.txt file on your site and see this status on the robots.txt page of the Diagnostic tab in Google Webmaster Tools, this is the correct status. However, if you do have a robots.txt file and you see this status, then your robots.txt file may be named incorrectly or in the wrong location. (It should be at the top-level of the domain and named robots.txt.)

If you see this status for URLs that Googlebot tried to crawl (on the HTTP errors page of the Diagnostic tab), then Googlebot likely followed an invalid link from another page (either an old link or a mistyped one).


So I think a solution would be as I suggested to display the url to the end user that goes to the 1st page in the series on the 404 page. If there is a page in the series. This way Google and the visitor both have what they were looking for. Then if you track user behavior of the 404's you can tell if people click through or not. Analytics should be great for that tracking the clicks as events.



Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 5/1/12 3:54 PM
My apologies for the silence the past few days. I have gotten to the bottom of my issue. It turns out that I lost a very prominent backlink at the same time as we launched the new version. Some syndication services also dropped our RSS feeds when we switched to the new version. This resulted in both Google and Bing traffic taking a steady decline at the same time, unfortunately.

At least now I know what I need to do ...
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 5/1/12 4:29 PM
Soooo on that note, I'm now trying to shift my focus to build backlinks. :) Anyone know any good tricks? haha

What do you think of this page: http://www.daniweb.com/stats/get_badge (It gets customized for each individual member)

I'm trying to encourage members to link to their DaniWeb profiles from their own blogs and homepages. Would you do it???
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/1/12 4:50 PM
So after considering all you might have done I still think that your 302's are playing into the already big mess the move already created.

I also think the member badges are great. My suggestion would be to make sure that somehow a link is also included that goes to your homepage, not just the member profile page of that member.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 5/3/12 8:17 PM
Hmm ... It looks like I lost a lot of backlinks because my RSS feeds completely broke. Plus, we're no longer in Google News as of our relaunch.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/3/12 8:56 PM
That really sucks.
What a mess.
I wouldn't know how to fix the problems in Google News, but I'm going to assume the start of the fix is to get your feeds back up.
Are you doing this all by yourself or do you have employees who's job is to make sure stuff like that doesn't happen?
Re: DaniWeb lost all its traffic *AGAIN* iBill 5/6/12 9:10 AM
Dani, has your traffic loss stabilized? Has it started recovering yet? Or are you still losing traffic?
Re: DaniWeb lost all its traffic *AGAIN* brinked 5/6/12 10:01 AM
Why do you have all those tags in your footer? http://www.daniweb.com/web-development/php/17

Not only that, they are almost invisible...as a user they are of 0 use for me.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/6/12 10:28 AM
It's quite obvious that her problems are of her own making.
At least it's become more clear as time goes on and more evidence is revealed.

Yes they are related to the algorithm changes that Google is doing, but it's only because sites are using the same kind of practices to game Google's index.

What you pointed out there brinked is called keyword stuffing. Then you mix on top of it the 302's that redirect to the 1st page in a series when a page is removed and you start to realize that this innocent webmaster is actually using blackhat techniques and has admitted she was doing this kind of behavior for years and has the audacity to ask "Why would these techniques suddenly start to affect me negatively when for years it seemed ok to do?"

If you also factor in the backlink loss she has suffered, you have all the reasons why her site is going down the drain, at least when it comes to organic search engine traffic that is. She's basically using techniques that Matt Cutts has warned many webmasters about using. If the site shuts down eventually, well it'll be an example to publishers of what not to do.

See, even when publishers think they are innocent, they're not necessarily that innocent.

Over time evidence is usually uncovered by somebody that points a different picture than what the publisher is trying to portray to us here, that they either don't know what they did wrong or that they didn't do anything wrong.
This message has been hidden because it was flagged for abuse.
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/6/12 10:34 AM
With the php includes looks like just an html coding issue.  But likely correct that if googlebot picked those up they could penalize some for that.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 5/6/12 8:01 PM
> It's quite obvious that her problems are of her own making.
I am very aware of that. I even said so in my very first post in this thread :) Basically this thread originated with the question, "How can I remedy a relaunch gone bad?"

> What you pointed out there brinked is called keyword stuffing.
Google doesn't see my tag clouds. They are located in a file that is disallowed in robots.txt. We use Javascript to inject them into the page. Even if Google can understand Javascript, they should be following the robots.txt directive and not looking at them. You will notice that Google's cached version of the pages don't have the tag clouds. I don't consider this black hat. Yes, if you want to be particular, the end-user and googlebot are seeing something slightly different, but I don't consider this cloaking because I am not using any tricky techniques to send different content to Googlebot. I am simply blocking content in robots.txt and hope that Google follows the directive. It is not an option for me to get rid of my tag clouds because they are used by my members, and I think this is a happy alternative.

> You cant honestly tell me those titles are useful to the links or the visitors. The link title attribute is meant to describe the links destination, that's no description, its seems more like spam to me. 
We need to HTML-encode quotes in the HTML, because it's not valid HTML to do something like title="This is "text" in quotes", so instead we need to do title="This is &quot;Text&quot;" in quotes", but they don't look like that to the end-user hovering over the links. In fact, we originally didn't have the title attribute when we relaunched, and got a series of complaints from members who missed the "post preview" so we put them back.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/6/12 8:46 PM
Actually it's fine being scripted, sorry I didn't realize it was, but I wouldn't block Google from it.

Just in case Google treats that like you're trying to hide something and gives you a penalty.

It's a catch 22 situation where if you block it, Google doesn't know what or why you're blocking, and it won't go and look to see if what you're blocking is acceptable to be blocked or not. So if it won't go look, it could still treat that as deceptive since it doesn't know what it is. That's at least my thought on that. I mean not that you're doing this, but imagine if you joined a link exchange and did what you were to block Google from seeing the link exchange script.









Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/6/12 8:52 PM
"could still treat that as deceptive"

If that is true, maybe Google should stop trying to be a mind reader.  If true, that opens up a whole other issue many may need to look into.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/6/12 9:39 PM
Yes, Google could still look at the content that is scripted and determine if the use is deceptive or not, however then it breaks the rule of always following the robots.txt file as it was instructed to do so. If you're going to put it on the page for the visitor, you must also allow Google to have the opportunity to look at it. Otherwise I don't see a reason Google shouldn't treat the page as partially deceptive, as technically it is.
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/6/12 10:00 PM
Example of why sometimes using the robots.txt file to block.  Sometimes there can be very old pages that no longer exist thus just block them or something strange in the url that somehow Google got?

But seems to me that people should be able to show content just for their visitors and block it from the bots.  Do not see how that is deceptive.

Think the way to handle those however would be to show them but use the rel=nofollow in the <a href tag for each one in the tag cloud and remove from the robots.txt just in case.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/6/12 10:20 PM
You don't even need to nofollow the links, as they are scripted and they don't even lead anywhere externally.
We're not talking about blocking an entire page, we're talking about blocking part of the page.
So if we were talking about blocking a single page from Googlebot, sure, go ahead, nothing wrong with that.

Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/6/12 10:22 PM
Forget what I just wrote above as can see why Google would not want that as bad sites then could put up bad stuff to surprise visitors.  And with tag clouds bet Google really does not like those as they can look spammy.  So if they have to stay, the best option does seem to be using the robots.txt page to block.

But maybe the real answer is those have to go if not wanting to be penalized for it?  Possibly fix the majority of the issues and see how it goes?
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/6/12 10:33 PM
The reason I put the rel=nofollow is saw one of the top contributors here do that on their pages for their tag cloud and read on other sites that is how they handle those. And sounds like wordpress is doing that for tag clouds.  So maybe that is the best way to handle those and remove some?  Maybe someone who really knows for sure can pipe in. Sorry if I caused more confusion :)
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/6/12 10:41 PM
Remove tag clouds (Matt Cutts suggestion).
 - They could be viewed as keyword stuffing (if they contain over 50 to 75 keywords).
 - Populate tag clouds via javascript so that they are not cached.
 - Blocked tag cloud pages from being spidered in your Robots.txt.
 - This should also improve page load time.

http://blog.site-seeker.com/google-panda-history/ 

Maybe some clarification as to what to do.  And sounds like the OP may have been doing this correctly?  There is a video below that section too.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/6/12 10:59 PM
The best advice for tag clouds is to keep them scripted and allow the bots to walk them.

I suppose yes, you should also mark the links in the tag cloud as nofollow if the links actually lead to tag cloud pages, as well as block the actual tag cloud page. I mean for now I'm not aware that scripted links pass pagerank, but that could change at any time I suppose.

I mean honestly, figure if you have tag cloud pages, you know they're spammy and those page should be blocked, but never should the tag cloud itself be blocked from Googlebot on the page which is simply displaying a few tag cloud words. Google has no way of knowing what you're up to and to be honest if you feel it has to be blocked from a bot to help with SEO, then you shouldn't be putting it on the page in the first place.

They're not going to influence much as a script, at least not currently, and it will show the bots you're not up to something deceptive if they can read scripts, like Googlebot does (well not ajax, but regular javascript).

Also your page load time that Google uses as a quality signal is actually a sampling from your visitor's visits and not the Googlebot's visits. So blocking it or not from the bots won't influence that quality signal.

Anyway, Yes the OP has done it correctly from what I saw, except I didn't check for where the links in the cloud tag go to if anything, but if they go to a page, it wouldn't hurt to nofollow them as well as make sure those pages are not indexable or followed, through the use of robots.txt or even meta tags. Currently I'm unaware that Google is really paying that much attention to scripts except to make sure that nothing bad is showing up, like malware or link exchanges.
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/6/12 11:24 PM
Just looked at one tag cloud and the text color is a light gray.  Think those should be black as light colors can be viewed as hidden text.  And some of those are for words like find or even shorter like re.  Think it is best to use very unique words that are not so common and maybe over 3 characters.  The one I looked at has 100 words so that is way over what that article above says.  But if blocked with the robots.txt and using rel=nofollow maybe ok?  But do not even see them in the view source.  However, if a Google employee was to manually check the page they could still view it as too much spam.  
Re: DaniWeb lost all its traffic *AGAIN* iBill 5/6/12 11:31 PM
Dani, are you recovering at all? Has your decline stopped?
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/7/12 12:18 AM
Something to consider.  Thought I read where the users use those cloud tag internal links.  But if I was a user I would not because they are so light would not want to strain to do that.  So possibly where you have seen users use those links is because they were crawled by the bots and visitors coming in from those.  Or they type that in the browser url address?  So maybe you do not need the tag clouds at all?  But if still wanting them a suggestion might be to limit to a certain character count to get them down to at 50 per page with your script excluding many words that maybe are not unique enough?  Now some might hover over them to sort of see them better but still as a user I would not.  But seeing how you do your urls with the /tags/ would probably do that.  So if I wanted to find more detail about javascript would just type in /tags/javascript after the first part of the url.  Also if keeping the tag clouds might also want to consider making the non highlighted larger ones a larger font size.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 5/7/12 7:11 AM
Recap:

We used to have the tag clouds embedded directly on the page. I personally didn't consider them spammy and didn't think it was a problem. I *think* Panda initially dinged us for keyword stuffing, because switching to this Javascript method is what ultimately got us recovered from Panda in the first place. I did a whole video about it for WebProNews last year.

The landing pages for the tag cloud keywords have the noindex meta tag.

I do not use rel=nofollow on each individual tag keyword because I am of the belief that nofollow should only be used for external sites that one does not trust. (Since the original concept of nofollow was to add it to UGC or content I don't vouch for). I DO vouch for my own internal pages and so I don't think nofollow is appropriate. Instead, the landing pages have the noindex meta tag.

Another huge advantage for the tag clouds being loaded by Javascript is actually a decrease in page load time because the tag clouds get cached client-side. The HTML document for the tag cloud only needs to be downloaded once and can be injected multiple times into multiple documents. This is a huge bonus.

I know that our members use our tag clouds because they tell me so. I have a *VERY* vocal core group of members who complain if any feature is changed/removed.

As mentioned, I explained all of this to DEATH in my Panda thread I had here last year.
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/7/12 9:42 AM
I believe that Google went way too far and wants now for people to design their pages so THEIR algorithms work irregardless if that is the best user experience for visitors.  But on the other hand can see the other side where there are some sites that do popups and bad stuff so ...

But because I and likely others can hardly see those words I still would suggest making them black.

I have had users at a company I worked for too that changing anything can upset them so certainly understand that.  But minor things can be helpful and maybe even necessary.

I have read the two sided debate about the rel=nofollow and tag clouds or any internal links here and other places and some are doing it just to be on the safer side.  None of us really knows for sure and because of too much vagueness by Google people are doing things that may not be exactly the right way but maybe in Google's eyes a "compromise" that gets them out of the "doghouse".  But then again who know if Google will use that against websites later?

And I have read Google has implemented a manual check team.  Since your issue has been widely published I would bet anything they could manually check your site and could still penalize your site to some degree.

But if you believe that issue is resolved and it is not hurting your site in anyway then maybe all these comments could possibly just help others who read here.  But what about if later they decide it will?
Re: DaniWeb lost all its traffic *AGAIN* Panda_Effects 5/7/12 10:21 AM
Just a quick point about the manual team Google has now to review sites.  Much can depend on the quality of training done and how objective they are.

Example.  USPS (and who knows if at FedEx and UPS?) now has what appears to be many more temporary postal people picking up and delivering mail.  A number of times they have not picked up outgoing mail, so had to go find a couple and they were convinced that they did not need to as that is only done in special boxes.  So I would explain that people put bills in their mailboxes to be picked up and if not that could mean they are late with their payments.  They understand eventually after more effort but something is lacking in the training or it was not emphasized enough.

Sad when cannot depend on the mail service like used to.  Sure there were some issues through the years but it has gotten worse.

And just like any service like a manual team there can be issues with training or people being too "aggressive" just like Panda/Penguin etc. I believe did too far too many tiny/small businesses.

So point being I think it is best to not just do things to fix for the automated Googlebot but also for what the manual team may do.


Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/7/12 10:30 AM
Dani,

What you did was what you should have done.
Blocked the bots from your tag cloud pages via robots.txt or meta tags and also script your tag cloud so bots can read the content, but pretty much ignore it. However you are blocking the bots from reading the tag cloud widget and that's something you shouldn't do. Also nofollow isn't about external or internal. There are a few reasons you may want to nofollow internal pages. For example, I nofollow my internal links that appear on all my pages for those links that repeat themselves, like my contact us page or my privacy policy page, or other links that I've spammed on the bottom of my pages on purpose that I want to be there. Basically it prevents the link juice from going to a page that while I find it has a purpose, it's not really something I want to pass internal page rank to, but I didn't block the page in my robots.txt so Google could index it off my xml site map. Like anybody really needs page rank on a privacy policy page or a contact us page? So any page you have blocked Gogolebot from walking in robots.txt should also have nofollow added to the link on any page you have a link to that page. But it doesn't have to be reversed where any link you have nofollow should also be blocked by robots.txt.

If Google was actually following scripted links, it would show up in Webmaster tools as a crawl error since you're blocking the tag cloud pages if the links in your tag cloud widget on your pages actually would cause Google to follow the links. So what a nofollow tag does is tell the bot not to walk the page and the error it would display in GWT would tell you it tried but robots.txt told it not to. It seems silly that if you've told a bot in robots.txt not to walk a page and it sees a link that goes to it that it would not walk that page. Well some of them do, and Googlebot spits up an error saying robots.txt has blocked it from a page you're not nofollowing the link to that appears on your pages.

As for this "core group" you speak of. What would they say if you told them you had to remove or change a feature because the search engines don't like it? Sure let them complain all they want, but when it comes to the sake of your site, who are you going to comply with, the major search engines or your freeloading members? What would they say if you told them it's either do that or watch the site collapse for not making enough money?

I've said it before, you have to find that balance between visitor and search engine. Visitor input is much appreciated, but it can also cause a lot of problems if their want or needs harm your site and you went ahead with them.

So tag clouds are ok to stay, your method of scripting them is acceptable, but you need to unblock them from the bots, else your pages are probably treated as slightly deceptive. It's amazing how you went from recovery to back in the dog house so fast, but that's because you were scared enough to start blocking parts of your pages from the bot, which probably told it you're up to something sneaky, even if you weren't. I believe you're not up to something sneaky, but you have to look at your situation from a bot's point of view. It doesn't know why you've blocked part of the page, and there isn't a way for you to tell it why, so it only knows that you have blocked part of the page. It has to assume that since you're blocking Googlebot specifically and allowing visitors to see the content of the script you have to be up to something it probably wouldn't like to see if it could. It really has no way of knowing and has to treat the page as such because what you did is exactly what the blackhat's do as a method to hide things from Google. I know I'm repeating myself, but I can't stress enough the importance of not blocking content on your pages, especially scripted content.

Listen, I've read enough Google propaganda they've published to know Google simply does not trust publishers and that it has nothing but the page itself to determine if the publisher is whitehat or blackhat. Right now you're slightly on the blackhat side.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 5/12/12 11:52 AM
Dani,

I wanted to follow up and say that at this point since your backlinks are trashed, I would think you would be open to at least giving my suggestions a shot as even if I'm dead wrong, and I have to kind of say that I could be, as there is a lot of secrecy to Google's methods, however the downside is you keep on falling in rankings, but the upside is you start to climb back up.

So hopefully you have fixed your feed and are blocking the actual cloud pages and not any javascript and are no longer using 302's for removed pages.

http://www.youtube.com/watch?v=B9BWbruCiDc

I did find this video about nofollowing javascript links that Matt Cutts posted. Note at that time that video was published it was ok to block javascript, but it's not really anymore as Google discovered that the blackhats were using it to hide stuff from Google. http://www.youtube.com/watch?v=Z5ZTa9yeRgg


Re: DaniWeb lost all its traffic *AGAIN* alistair.lattimore 5/12/12 4:36 PM
I thought I'd add that applying rel="nofollow" to internal links within your site is doing it a disservice.

When Google originally announced rel="nofollow", applying it internally meant that no PageRank flowed through that link and all other links on the page increased proportionally by the amount of PageRank that didn't flow through the rel="nofollow" link. This opened the door for savvy webmasters to engage in what is referred to as PageRank sculpting, which when done naturally is great but people were manipulating it.

More recently Google changed that behaviour and now the PageRank that would normally flow through a rel="nofollow" link evaporates, in that the other links on the page don't get boosted because you applied a rel="nofollow".
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 7/11/12 7:56 AM
Hey everyone ...

My apologies for the silence over the past two months. As you could imagine, there were a lot of snafus related to our  massive site relaunch that were non-SEO related, and we were suffering the after-effects of SEO ... meaning that the community membership and activity was *WAY* down. The business side of things also started to take precedence (advertising sales, etc).

I'd like to report that traffic began slowly popping back up almost three months TO THE DAY of when we lost it all. My gut feeling is that we were in type of 3 month long penalty sandbox for completely redoing our site structure and pages, with nearly everything changing.

We're still not where we used to be, but we're increasing traffic by about 10% week over week, for the past month or so. I can't really pinpoint any SEO thing that we've done. Most of my focus lately hasn't been SEO related because there was just so much that went wrong all at once, and it was overwhelming what to tackle first.
Re: DaniWeb lost all its traffic *AGAIN* OkayNetwork 7/11/12 11:19 AM
Well hopefully you took the time to clean up a lot of the mess you created, especially all those unnecessary redirects for removed pages that may or may never exist again. I'm going to say it again and again. If you remove a page say for spam or malware or even if some person just wants the page removed because they wrote it, then it should be allowed to 404. 404's do not harm your site, even if some other site is still linking to it. There are things beyond our control as publishers that Google is well aware of and doesn't hold against us.

If you didn't take care of those pesky 302's for pages you have removed, then you still might have a penalty placed on your site, and only are recovering from other penalties, or could even perhaps just be the latest refreshes that Google has done that is causing your traffic to increase slightly, or could even just be a summer traffic thing, and you're still being algorithmically penalized, and here you are thinking your penalties have been lifted when they really haven't.
Re: DaniWeb lost all its traffic *AGAIN* Dani of DaniWeb 9/13/12 10:34 PM
I just wanted to give everyone a status update :)

Traffic has been increasing over the past few months, and even more over the past few weeks. We're not quite as high as we were at during our absolute peak post-Panda, but we definitely have fully recovered and have close to the most amount of traffic we've ever had before. Activity is also way up.

I do want to admit that I put the welcome modal back up non-registered members. We had it for many years until it was removed as one of the first things we did as part of our Panda recovery. However, it has always performed so well for us in terms of increasing conversions and lowering the bounce rate, so I decided to put it back.

After thinking about it for awhile, I realized that I've been running around with my head cut off trying to do exactly what Google wants from me, at the sacrifice of community growth, just to please them and get traffic from the SERPs. And what do I have to show for it? ... Unpredictable traffic from the SERPs that I can't rely on. Because I'm sacrificing community growth for increased traffic from the SERPs, that means that if G traffic were to disappear tomorrow, I'd have nothing. At least if I do what it takes to keep the community growing steady, then if we lose G traffic tomorrow, we'd still have a strong community. So that's what I've decided to do. And hopefully Google's algorithms will appreciate that our modal leads to community growth and is there for positive reasons ;)

Regardless, community growth is on the rise. Traffic is strong. Overall, no complaints. Hopefully the upcoming Penguin refresh won't make me changing my tune.
Re: DaniWeb lost all its traffic *AGAIN* Suzanneh 9/14/12 3:50 AM
Glad to hear it, Dani!  And thanks for the update.

Suzanne
More topics »