Categories: Crawling, indexing & ranking :

Website Crawl Rate Very Slow Even Take Months

Showing 1-12 of 12 messages
Website Crawl Rate Very Slow Even Take Months Jehon13 12/24/13 12:36 AM
I've read the FAQs and searched the help center.

My URL is: http://bit.ly/10UrjbI and one internal link as a sample http://bit.ly/19n2WyQ


Please note that we update all our website pages twice a month.
Almost 20% pages are updated 4 to 8 time in a month.
Almost 10% pages are updated even more than 10 time in a month.
All the content updated manually.
We change the current month name in title tag and h1, h2 tags etc monthly.

Our website crawl & index rate are too slow. even this take months to re-crawl and re-index a page.

We are facing this problem for more than 6 months, the time when we shift our website hosting (to Godaddy).
We are monitoring our website server status and received a lot alert messages that our site not available due lack of resources issue (503 error). we contact to the hosting company many time and at last this issue resolved and since last month there is not alert for this.
We have tried many things to improve our site load time but there is no change in Google webmasters tools crawl status report.
A crawl status report from Google webmasters tools is attached.

As the Following queries from Google

http://bit.ly/1ckspYo we have just 6 pages indexed for December
http://bit.ly/1bnTftY 239 pages index since September 2013


Our questions are

is this because of Google think this is low quality content?
Is there any internal linking/navigational or structural issue?
Is there any server side issues.?
As per crawl status report is attached, is this rate is normal?

Please guide us to point out the actual issue.

Regards



Re: Website Crawl Rate Very Slow Even Take Months ets 12/24/13 12:59 AM
Hi there,

Let's see if we can help. The first thing I see is that the time spent downloading a page is relatively high (mine are generally about 300ms - yours are up to 5 times greater). Googlebot has only a limited amount of time it can spend on your site so the longer it takes to get each page, the fewer pages it can get from your site each time it visits.

So the first thing I'd want to do would be a speed test. I don't know where in the world your customers/users are, but I'm going to test from Phoenix, AZ, since that's not a million miles from Google HQ in California. Here's the result for the sample URL you listed above:
http://www.webpagetest.org/result/131224_C8_9BR/

That is 50 seconds to download a page - so I'd finger that as a major cause of your issue. You should be looking at less than 5 seconds. Looking in detail at the test:
http://www.webpagetest.org/result/131224_C8_9BR/1/details/

Apparently your server is in Singapore. Is that where your users are as well? Remember that Google is in the USA - so these times are likely the times Googlebot is experiencing as it crawls your site.

Restesting the speed from Singapore:
http://www.webpagetest.org/result/131224_7R_9GN/

That is obviously much faster (6.5s) as we would expect.
Re: Website Crawl Rate Very Slow Even Take Months Jehon13 12/24/13 1:34 AM
Hi ets,

Thanks for your guide.

Our visitors are mostly from USA & Canada.
Google bot comes from Just USA? is there no other data centers to sent crawler for crawling the sites?
Is number of back links can help to increase the crawler visit rate?

What do recommend to optimize this?
Is there any other issue regarding website structure or navigation or internal linking?

Thanks

Re: Website Crawl Rate Very Slow Even Take Months ets 12/24/13 3:45 AM
Our visitors are mostly from USA & Canada.
Google bot comes from Just USA? is there no other data centers to sent crawler for crawling the sites?

If your visitors come mostly from the USA and Canada, having your server in Singapore is a very bad idea, even if you are in Singapore yourself. Googlebot's high crawling times and my speed test show that you are giving a poor experience to users with a very slow website. No-one wants to wait almost a minute for a page to load - and on a dialup or slow connection, it would be much worse. Your best bet would be to move your server hosting to the USA. Even using a CDN is not going to help if your main page is still being served from Singapore.

It is also possible that if Google doesn't consider your site useful or of high quality, it might crawl it less often. I have not looked at the content quality at all - someone else might do that for you.
Re: Website Crawl Rate Very Slow Even Take Months JohnMu 12/24/13 4:07 AM
As ets mentioned, your server is extremely slow in serving content. This definitely makes us crawl your site (and all other sites on the server) a bit slower. So from a technical point of view, if you need to be crawled more often, you should probably either move to a faster server of work on optimizing your site's code. 

That said, just being crawled faster wouldn't change our algorithm's opinion of your site, and at the moment, it looks like they're really not happy with the quality of the content there. 

My personal & direct recommendation here would be to treat this site as a learning experience from a technical point of view, and then to find something that you're absolutely passionate & knowledgeable about and create a website for that instead. Make something that you will want to work on in the long run, make something unique, compelling, and of high quality, something where you're an absolute expert and where you have something to contribute to the web that can't be found anywhere else. This kind of site might be interesting from a technical point of view, but I doubt you dream of "DHGate Coupon Codes Dec 2013," right? :-).

Good luck!

Cheers
John
Re: Website Crawl Rate Very Slow Even Take Months Jehon13 12/24/13 6:47 AM
Hi ets,

Thanks for pointing out the server issue and we will shift our website to USA base server as soon as possible.
Again we are thank full to you for your time.
As per website usefulness or quality, we know there are many things to be improved and any suggestion to improve the usefulness of this website will be highly appreciated. 

Regards

Re: Website Crawl Rate Very Slow Even Take Months Gaieus 12/24/13 6:59 AM
John, I visited your site (and it was not even slow for me - being in Europe) but I must agree with the critiques about the content. Coupon codes may be useful but a site merely providing links to other merchants' sites (even if you correctly marked those links with a nofollow tag) does not say much about itself. All the pages are extremely similar to each other so Google may "think" there's no reason to index all - at least it's not something that it may find "interesting".

One more thing: after clicking on a coupon link, it takes me to a redirect page where it stays a very short time (not even enough to copy that coupon code) then automatically sends to the merchant's site. This is not even useful for visitors this way.
Re: Website Crawl Rate Very Slow Even Take Months Jehon13 12/24/13 9:16 AM
Hi John,

You are right that website download speed and the usefulness are two different things and both contribute their share to perform well in search engine as well as better user experience.
As per your opinion this site has poor content and download speed issue. It means if we work on these issue than this will be perform well in Google as Many Others like this website.?
We started this website to focus a specific niche so that We can deliver quality work to our site visitors this market easily.
Thanks for your points but if you point out the specific issue and suggest to improve than this will be more help full for us.
However thanks for your time.

@Gaieus

People are searching for a specific merchant coupon codes and coupon code websites are offering the coupon codes to the visitors. This is the function of a coupon site.

But here search engine want these sites should add some value along with coupon code, I couldn't understand that why coupon site should add some more things along with coupon codes even the visitors are not interested in more value except the valid coupon code?

Now the quality stander or usefulness for like this websites should be measured by search engines differently than the other type of websites (like content base website).

For example what Search engine doing, just crawl the data/pages, index them and than show the URLs along with description in the response to the user queries. Search engine adding any value to the pages that they are serving to the visitors?

like this coupon code website is doing, these sites are collecting (crawling) the coupons, edit them,make them understandable to their visitors, arrange them, categorize them(indexing) and offers to the visitors who are looking just for them(coupon codes).

So if some one ask me to categorize the different search engine, than should I categorize them on the basis of unique content? or there should be some different parameters as some one use to measure a blog?.

In short, in my opinion, that affiliate websites are a part of retail shopping cycle,  and search engine should treat them differently.
If search engine don't have the ability to evaluate like this (and other content less) websites than this is a search engine fault not the website, so search engine should have to develop the algos for better user experience for their visitors and not should use few website to show on SERPs and say that affiliate sites are not good websites or even start offering the coupon codes it self.

By the thanks for your time to visit our website and point out the issues, However if you suggest any thing that improve the quality of the website than we will be more thank full to you

Cheers
Re: Website Crawl Rate Very Slow Even Take Months Gaieus 12/26/13 9:25 AM
You probably misunderstand me, John. I am not saying your website (and the idea of offering coupons) is bad. The only problem with similar affiliate websites (and there are lots of them) is that since they do not have any unique, quality content (call it as "catch" for Google if you wish), Google cannot really decide which to display to a visitor when someone is searching. There is nothing based on which one or the other could stand out and give "more relevant" search results than the other. Competing, similar websites seem to all offer the same (kind of) content therefore there is nothing that can bump one or the other up.

I know what I am saying may sound cheap but my experience says that Google can indeed index extremely fast (some of my pages were indexed and displayed for relevant queries after a few hours already) ut of course I may be more lucky with my "topics" (true that I do not publish every day). I also guess that with hundreds and thousands of pages (sections) for these coupons it cannot be easy to add some nice description or whatnot to each and also maintain it continuously. But if you do not try to come up with something similar, it won't change I am afraid.

As for the other issue I mentioned: could you not make the redirect a manual action? This (too fast) automatic redirect must be annoying to some of the visitors.
Re: Website Crawl Rate Very Slow Even Take Months Jehon13 12/26/13 9:51 AM
Hi Gaieus,

You help us great to understand the basic issue. We are thankful to you for your precious time.
We are looking on the redirection matter and will adjust it soon.

Best Regards
Re: Website Crawl Rate Very Slow Even Take Months Rana Prathap 12/31/13 9:48 PM
Use a CDN such as http://www.cloudflare.com
Rana Prathap 12/31/13 9:50 PM <This message has been deleted.>