|COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||1/26/12 9:07 AM|
I have read the FAQs and checked for similar issues: YES
My site's URL (web address) is: nuts.com
Description (including timeline of any changes made):
On January 6th, we submitted a change of address for www.nutsonline.com to nuts.com. We followed everything to the T. 301s, change of address in webmaster tools, and even tried to consolidate old indexed pages with rel=canonical and redirects.
Now nearly 3 weeks have gone by.
Did we miss something?
Does this just take a while?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||1/26/12 4:40 PM|
Google is crawling & indexing your new domain, you can check yourself with a search like [site:nuts.com].
I've checked a few different URLs for your old website & the 301 redirects you've implemented seem to be working.
Regarding the 3 weeks statement, that just isn't enough time for Google to move 130,000+ URLs onto your new domain. You should find over the next 2-3 months that a search for [site:nutsonline.com] returns fewer & fewer results (good) as Google processes the 301 redirects.
Just as a side comment, make sure you're taking care of all the content on your various subdomains:
Moving forward for your nuts.com domain, you want to make sure that you don't have duplicate content issues again.
Verify that your staging sub-domain has a robots.txt blocking it before you even put it online.
For your CDN domain, make sure the only content indexed against that domain are your resources & not your websites content.
Make sure that you're sending the correct 301 redirect when someone accesses www/www1 back to the corresponding URL on http://nuts.com.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||1/27/12 5:16 AM|
Thanks for your help!
Many of the old urls in the index fall under our site search (see: site:www.nutsonline.com/search). On our new domain, we have a robots.txt disallow for these and noindex, nofollow, but any ideas on coaxing these old urls out of the index. For whatever reason, we get this message in webmaster tools for our www.nutsonline.com/search urls: URL restricted by robots.txt Jan 24, 2012. I fear that this will prevent them from coming out of the index.
Or does this not even matter?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||1/27/12 5:48 AM|
That is happening because your nutsonline.com/search URLs are being redirected into nuts.com/search, which is restricted via robots.txt.
To resolve that, you should apply a meta robots noindex,follow to nuts.com/search & remove the robots.txt restriction.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||1/27/12 10:04 AM|
why follow versus nofollow?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||1/27/12 1:46 PM|
When google crawl your search URLs, you want them to follow the URLsthey discover to other pages on your site and you want PageRank to floe through all the URLs. The follow isn't really needed I guess, it is the default behaviour but I say it in general people don't use noindex,nofollow as an example.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||1/27/12 8:26 PM|
thanks. removed the robots.txt restriction.
do we just have to sit back and wait now?
how long have you seen this process take for rankings to come back?
remember, the majority of the old links in the index aren't the pages we were ranking on. our core pages are indexed for our new domain...
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||1/27/12 9:20 PM|
You've got over 100,000 pages indexed still, so it could take 4-8 weeks.
Whenever I've done this in the past, I make sure it is setup correctly and check back once a week or fortnight to check on the progress.
Since your core pages are already indexed, I would focus on improving the quality of your new site, content and links while letting your old domain percolate away in the background as Google dutifully do their job.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||2/17/12 3:24 AM|
Hi Al! We did everything you suggested. Today marks 6 weeks. No improvement in rankings still. Is there something else wrong or that we are missing here?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||2/17/12 6:14 AM|
I don't think so.
Everything seems to be progressing and you've now got ~66000 URLs indexed under your old domain, down from over 130000 URLs.
You might want to review your top landing pages from before/after the change to see if anything has changed. For instance, if during the change your website design change in anyway, it could have affected your internal link structure. Similar, it'd be worth spending the time to compare the <title>, meta description, <hX> headings and copy on each of those URLs to see if and how severely you've been impacted by the change.
The key at this stage for you is to try and discern why your rankings and/or search engine referrals have suffered & see what you can do to correct the issue. Other than the switch in the domain, did anything else change at all on your site during the move or soon thereafter?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||2/18/12 4:01 AM|
The only thing that changed was us eliminating the duplicate content and changing nutsonline to nuts.com in the title tags.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||2/18/12 4:57 AM|
Did the fall in rankings and/or traffic happen exactly when you changed domains or on a slightly different date?
The reason I ask, maybe you've been affected by Google Panda and your just happened to move domains at a similar time & that isn't the cause at all.
According to Google Analytics, what date exactly did your search engine referrals drop?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||2/18/12 5:58 AM|
Yeah, the traffic dropped to the precise day of the change. January 6th!
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||2/18/12 1:28 PM|
The latest Google Panda update to your date was Google Panda 3.2 and was released around January 25, so it isn't that.
What sort of drop in traffic did you see on that date?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||webado||2/18/12 1:35 PM|
The same day as a domain change takes place it's highly likely that the DNS propagation had not yet completed so your site was in fact unreachable for many visitors.
For your robots.txt file you also need an actual directive under the user agent line.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||2/18/12 1:57 PM|
Al, this is our proportional google organic traffic from January 1 of this year. I say proportional because I didn't want to provide our precise traffic, so I divided it by the same factor for each day. You can understand my distress:
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||2/19/12 4:25 AM|
Is Google Webmaster Tools or Bing Webmasters reporting any errors before/after the change of domain?
I mentioned above about comparing two time periods (before/after) using the Top Landing Pages report in Google Analytics with only the non-paid search engine traffic advanced segment selected, did that yield anything of interest? Since you've got a nice URL hierarchy, it might be worth comparing search engine referrals by section of your site as well - for instance before/after for /nuts/, /driedfruit/ and so on. If that turns up anything, drill in further and compare other logical sections of your site such as /nuts/almonds/ and so on until you isolate an area that has suffered.
As a matter of interest, what percentage of your sites traffic was landing into /search via search engines? For instance, as part of the move you've removed an enormous amount of low quality pages from your site by applying the meta robots noindex,follow to your /search URLs (which I think is a great move). However I never asked and you never mentioned if and how much search engine traffic that section of your site contributed. I find in sites I've had involvement with in the past it is normally a very small percentage but thought it worth asking in case your site was different.
It isn't going to be the cause of any problems yet, but you should address the fact that your site is crawlable using HTTPS. Google has currently crawled a small number of URLs on HTTPS, which is going to blow out in due course to your entire site and cause you duplicate content issues. Either add rel="canonical" tags throughout your site directed at the HTTP version of every URL, with the exception of the URLs you actually want to use HTTPS. Alternatively, do the same thing but using a 301 redirect and force users into HTTPS for a handful of secure URLs on your site and force everyone out of HTTPS if they aren't viewing those secured URLs.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||2/20/12 5:43 AM|
Al, thanks again for being some thoughtful on this.
We have a few errors, but all relate to normal things, like 404s for products that were discontinued.
Unfortunately we are seeing a drop across the board, and not related to any one section.
However, with Bing, our traffic has actually improved!
A negligible amount of organic traffic lands on search pages - the majority of which hits /search. Not even sure why that is happening.
Certain types of pages we have already inserted the rel=canonical link elements for the https version. This is just something tricky to do across the board, and doing it incorrectly would send google bad signals. Have you come across any duplicate content issues with https for the new domain?
FYI, we actually got a suggestion passed down to us from someone at google webmaster tools to someone else to then us to remove our change of address in webmaster tools. I tried this late last week.
The saga continues... :)
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||2/21/12 12:50 AM|
You mentioned that your traffic has declined across the board. Have you segmented your keywords to see if you can find anything? For instance you might want to check before/after for brand/non-brand keywords as a starting point.
You can see the HTTPS indexed content using a search like [site:nuts.com inurl:https], it'd be worth checking for yourself.
If doing it with a rel="canonical" tag is going to be hard/funky, I'd do it with mod_rewrite and 301 redirect every URL back to HTTP except the ones you need to be in HTTPS. You can then do the opposite as well and force people into HTTPS just for the URLs you need and back to HTTP for those that don't.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||2/21/12 3:21 AM|
Hey again Al! Yeah, I did segment my keywords. Unfortunately, my top keyword went from ranking 1 to 11 or so, so 95% of that traffic is down.
When I exclude this keyword as well as all brand related keywords, my organic traffic is down 60% from google.
Thanks on the https front. That was helpful.
www.nutsonline.com pages continue to come out of the index, which I guess is a good thing, but our rankings are worse than they were a month ago, so I am not sure it even matters.
I think we have no choice but to focus on link building. Somehow our authority still has not carried over and/or we are being penalized. What do you think?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||2/21/12 10:52 PM|
I have no idea if this is a possibility, however I wonder if you've been temporarily penalised by Google Panda?
As you know, Google periodically compute Google Panda scores. I wonder if it's possible that because you're switching domains that your nutsonline.com scores weren't transferred over to nuts.com or if because of the split content across both domains that you're been penalised.
I'd love for a Googler to explain how that process works now with Google Panda.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||JohnMu||2/22/12 2:00 PM|
I'm sorry to hear that it's been this frustrating for your site. We worked hard to make our systems as fast and as pain-free as possible when it comes to site moves, so it's particularly frustrating for us to see situations where algorithmic fluctuations from such a change last longer than we'd want. Looking into your site, the issues you're seeing are very likely to settle down over time, as our algorithms get used to the new content and are able to migrate all associated signals properly.
Looking into the details, one thing I noticed was that the domain that you moved to was historically more attached to the UK than the US for some reason. Your site, however, appears to be only focused on the US, right? If that's the case, one thing you can do is to set the geotargeting setting in Webmaster Tools to confirm the US-targeting.
Hope it helps & please keep us updated how things progress.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||www.nuts.com||2/22/12 2:12 PM|
Thanks John! Excellent point about geotargeting. Many people from the UK are really looking for a magazine there. I will set it to the US now.
Should we put the change of address back in place in webmaster tools, or does it not matter at this point?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||jonny_london||2/29/12 1:54 AM|
Feel your pain, we went through this process (althought moving from a subdomain to our main www domain) and it was a nightmare. We ended up with 2 or 3 short penalties over the following year or so, but the thing that I worry about most for you is that we monitored our access logs, webmaster tools and noticed that google never stops coming back to check the old subdomain..
In fact it's just started again now, I think roughly 2 years after we started serving 301's, it's recreating the subdomain in webmaster tools and started showing up again using site:subdomain.xxx.com via google search.
Massively frustrating that google doesn't believe a 301 even if you serve the same response for years. We also tried canonicals, domain removal, 404's, 410's etc none of it worked..
Obviously the easy thing would be to use the domain moving part of webmaster tools.. oh that's right, it's doesn't work for subdomains :(
In short, expect to need to maintain 301's from the old domain for the next 10 years or so, monitor you logs and don't ever go near subdomains!
Interesting question how much energy is google wasting by following links that are years out of date over and over again and not trusting 301's? We've noticed the same behaviour redirecting from say xx.html?parameter=useless to xx.html. google comes back over and over and over again for months.. could we save a rainforest by trusting 301's after you get the same response back say 5-10 times? :)
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||2/29/12 4:50 AM|
I guess the thing is that Google needs to check on those URLs because they are still referenced all over the internet. As such, those links produce PageRank for the URL that you've 301 redirected but Google must periodically check nothing untoward has happened with the destination or they might need to devalue/remove it for spam/malicous reasons and so forth.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||webado||2/29/12 6:10 AM|
>>could we save a rainforest by trusting 301's after you get the same response back say 5-10 times? :)
No, it's not normal expected behaviour of a robot to remember details of redirections for everything on the web and apply them before actually following a link to something that gets redirected. Imagine the mess that would cause. Its nto Googlebto that redirects it's the visted site's server that does it.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||jonny_london||3/1/12 2:13 AM|
Al, Webado, thanks for the replies.
Most of the URL's have no links in to them other from within the site itself. The links we did have we asked webmasters to update to the www domain pages.
Maybe less than 1% of the pages had links to them (this was in the day of very long tail pages) and we moved 90% of those links to www, what value does G gain by continuing to try to crawl the entire domain years after it moved? a 301 is moved permanently last time I checked, why not just see it, check it a few times over 2-3 months, then believe it and remove any reference to the old page from it's list of resources that exist on the web?
Another example we saw when we moved onto www is with parameters, there were unwanted some side effects as mentioned, so we 301 x.html?a=1 to x.html and google comes back to those pages almost once a week and has been doing so for about 12 months. What value can possible be gained from seeing a 301 response say 100 times a year for 100,000's of pages that no longer exist? Even when the pages did exist they were exactly the same as the base html page as the parameters didn't even do anything.
In terms of googlebot being a robot, yes it wont personally know that a page is going to 301, but what google has in the background giving the robot a list of URL's to look at should.. Still on the saving a rainforest drive :p
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||3/1/12 4:45 AM|
I don't know what to say other than Google employ the most brilliant minds in the world.
If they think there is a reason to continue to crawl those older URLs, then there is a reason to crawl the older URLs.
While Google have tremendous computing power around the world, it is finite - so I doubt they'l be crawling those URLs more frequently than they feel they need to in general.
If it isn't causing you any harm, leave the 301 redirects in place and forget about them (so to speak) - you're doing your part, rely on Google doing theirs.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||jonny_london||3/1/12 9:17 AM|
Ah I wish I could be that happy person that could forget and doesn't worry.. but it's still causing us problems, a nice penalty every 6 months without fail..
Also being an ex-search engine person (not google) as well, there are great minds in them, but that doesn't mean no bugs, especially with odd domain configs, can tell you that for free :p
Don't want to steal this thread, but the site used to have the home page of the subdomain xxx.yyy.com 301'ing to www.yyy.com/ and this home page had links back into the subdomain.. legacy reasons but you can see the weirdness for google in having a domain point links at a subdomain and the home page of that subdomain 301'ing back to the main domain.
The problem is you just can't raise this directly with anyone at google as you are 1 site out of billions and even if they did look it's extremely hard for them to justify a fix for one or a very small number of sites.. they spend most of their time fighting off the bad people in russia and china, not sorting out our domain :(
I feel this turning into a lamentation..
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||Joooh||4/19/12 4:48 PM|
Traffic is back or not ?
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||4/19/12 9:52 PM|
I've been talking to nutsonline recently & their traffic hasn't fully recovered yet.
It is probably back to 85% of where it was before they changed domains.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||Joooh||4/21/12 9:33 AM|
15% of traffic lost is ok for a 301.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||Phil Payne||4/21/12 9:46 AM|
If you have multiple sitemap files you must connect them with a sitemap index file.
"If you do provide multiple Sitemaps, you should then list each Sitemap file in a Sitemap index file."
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||Spending||4/30/12 3:54 PM|
I've been following this fascinating posting with detailed interest. That's because I've been dealing with issues that may be similar - but for more than a decade - on several dictionary word domains and other descriptives that I booked in the very early web. One such name as an example is spending.com. It used to show thousands of daily unique type ins -- typical of strong direct navigation domains.
But in the late 90s or around 2000 all of its traffic simply vanished. No matter how many times I've switched hosts, dns providers etc it still does not reflect any traffic, whether from search engines or as a result of direct navigation. A few years ago I put it in Google's domain program but again it showed no traffic nor revenue. At one point in 2008 or 2009 a Google employee, whom I was able to reach on the help desk, seemed to suggest their systems indicated I in fact did not own the domain. I told him to verify I did simply by visiting any whois resource. He refused and ended the discussion. So I yanked the name from Google but the problem has persisted.
Indeed, in a decade of on again off again monetization efforts it never generated even one penny in revenue. And yet it is a strong descriptive dictionary word domain that in April 1999 was showing more than 2200 daily raw type in uniques. A few weeks ago I moved it back to Google monetization to see if I couldn't finally solve the problem. It is now at Google's premier monetization partner. Yet since then it has shown virtually no traffic and no revenue.
Some people reported in another Google group posting a few days ago that they could not resolve to the page -- blocked by Chrome in one case. So I tried on my Google phone (I can reach the page no problem on my computer) but my phone's browser could not, getting an error code 400 (bad syntax?). Today I learned a bit about 'uri's' as opposed to 'url's'. Is it possible that Google's systems are attributing traffic related to my domains and my account to an old host or account or something else?
Or could this be an ISP imposed ban or block of some sort?
Any insight you can provide would be greatly appreciated.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||alistair.lattimore||5/3/12 5:29 AM|
Please start your own thread on your issue to keep this thread on topic.
|Re: COA, 301s, rel=canonical, 3 weeks and rankings still down||Spending||5/4/12 10:53 AM|
Understood -- I've got a thread going beside the nuts.com thread, which seems to be a similar kind of thing.