Categories: Crawling, indexing & ranking :

Unable to crawl site. Not appearing in searches. Completely confused. Please help.

Showing 1-37 of 37 messages
Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/16/13 5:08 PM
I've read the FAQs and searched the help center. 

Hi,

I would say 9 times out of 10 a F.A.G. fails with no reason given.

There are no content keywords listed for the site apart from the Loren Ipsum placeholder.

The site is no-where in searches.

I have submitted XML sitemaps and now completely removed robots.txt - Still terrible.

I'm at the end of my tether and don't know what to try next. Would be very grateful if some kind soul could tell me where this is all going so wrong.

Thanks!
Stuart

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. UKIDO 12/16/13 5:22 PM
http://tinyurl.com/q5wnp6v  has the same issue.

I have tried new site maps changing geographic reagion and removing www. prefix totally stumped as the site does seem to rank
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Thomas P. 12/16/13 5:28 PM
OK, I just tried requesting http://www.stephenbryantjoinery.co.uk/ as GoogleBot
The HTTP Respose Header Content-Length, promised me/GoogleBot to deliver 62909 Bytes. But I only got little short of 53KB.
The connection didn't hang or time-out, I was simply cheated out of about 8.6KB - i.e. was served an incomplete document.
 
I think you need to check up on that webserver of yours.
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Thomas P. 12/16/13 5:31 PM
UKIDO, Please start your own thread about your site. - Just click the red button saying "Post A Question" ;-)
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/17/13 3:22 PM
Hello Thomas P.

I've made some amendments and improved a few areas of the site.

Would it be possible to retest on my behalf (or indicate how I could retest it?).

Many thanks,
Stuart
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Thomas P. 12/17/13 3:51 PM
Sure, - I used http://web-sniffer.net/ , setting the UserAgent to Googlebot,
thereby causing use of HTTP Request Header field & value:
  User-Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html)
 
I just retested, and it hasn't gotten better: I was, this time, promised 62715 Bytes, but got 51.8KB.
- i.e. this time around: I was promised more than before, but got even less than before.
 
I'm not an expert on Apache, so why the response mysteriously suffers truncations - is something I dare not make guesswork on. (But ... whatever settings Apache got for traffic control & flow, QOS, and buffers - is where I'd start checking first. If you've already been into all that, then it's probably time for you to go find an Apache forum)
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Benjamin.. 12/18/13 11:39 AM
Your sitemap has less pages on it than Google has in its index - 




You need to redirect one to the other 

Ben 
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/18/13 11:50 AM
Hi Ben,

Thanks for the response - most appreciated. I posted a different thread simply because the webmaster tools was and is showing that URLS have indexed so thought that perhaps that had been resolved somehow and the issue is rankings.

Would I be right in saying that I need to redirect the non www. version of the site to the www. version? 

I'm still a bit of a newbie and appreciate all the help I can get!

Thanks,
Stuart
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/18/13 11:53 AM
UKIDO - Please kindly open your own thread. Click the red "Post a question" button at the top and fill in details. Thanks :)
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/18/13 11:56 AM
Stuart asked:

Hi

I'd really appreciate some guidance as I'm close to the end of my tether with this.

I created a new site for a joiner and think I did everything correctly. Descriptive headers, keywords, good quality content etc.

I submitted the site to Google, submitted the sitemap, performed a F.A.G from wemaster tools, submitted it all to index and ...... nothing. Not a sausage.

No messages to tell me where I might be going wrong, nothing.

The content keywords have not updates. The site does not seem to have been crawled for ages. Webmaster tools say that 30 pages were indexed.....
I don't understand why Google is ignoring my site. If I knew what was wrong I'd be happy to fix it. No crawl errors are listed...nothing like that.

I've read loads of help on SEO, watched all the videos etc - I can't see what I must be missing.

The site has been live for three weeks now and the client is absolutely devastated.
I've launched a couple of sites now following similar guidelines and they show up just fine for relevant searches.

Please, can some kind soul tell me why my site doesn't appear in any search other than a direct search for the name of the domain!!

Would really appreciate any good answers. Surely its not a case of waiting even longer. Is it?

Thanks,
Stuart
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Benjamin.. 12/18/13 12:11 PM
Yes thats right, then i would create a new sitemap, you didnt need to delete your robots txt. Once you've done that in wmt i believe you can add the non www site and then select the www one as your preferred domain - fetch as google and submit (fingers crossed!)

Ben 
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/18/13 12:14 PM
Well first of all, let's keep the crawling/indexing aspect separate from the ranking aspect - since those are quite separate things.

So some questions...

1. Is it definitely being crawled? Forget Fetch as Google for a moment - and let's focus on Crawl stats. Please paste in the three charts that you see under Your site --> Crawl --> Crawl Stats.

Do you see entries for Google crawlers (e.g. "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html") in your server logs? If so, Google is crawling regularly.

It is indexed, as Ben said, so presumably it is being regularly crawled. 

2. Do you see green ticks for DNS, server connectivity, and robots.txt fetch? As you said, I see no robots.txt - so that should not be stopping anything from happening. I see no major DNS issues:

Have you moved the server to another IP address or server or changed DNS records anytime recently? That could account for a failure of FAG (though there are numerous other reasons why it might fail).

3. Moving on to why it's "nowhere in searches", that could be for any number of reasons that have nothing to do with anything above. The fact that it's new is going to be a key factor: sites can't expect to rank immediately in a competitive field. I see a few issues, like a little bit of keyword stuffing on the home page 

Glasgow Joinery
East Lothian Joinery
West Lothian Joinery
Stirling Joinery
Perth Joinery
Borders Joinery
Fife Joinery

There's the www/non-www canonical issue Ben pointed out.

You've got some bad pages indexed, which return 404 but will be causing some problems till they drop out:
Let those fall out of the index naturally; don't remove them.

You've got pages that are totally dependent on javascript and/or images for their content, which means Google will find nothing on them to index. Switch off JS and images on this page and you'll see the problem:


Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/18/13 12:20 PM
I see no backlinks on ahrefs whatsoever, so that won't be helping (or hindering). You must not build backlinks to your own site, but without any links the site will struggle to rank.
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/18/13 12:33 PM
Ben,

THANK YOU. 

I have now applied a .htaccess based 301 redirect from the non www. version to the www. version.

In the process of carrying out the suggestions suggested below too.

Thanks again, Ben.
Stuart

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/18/13 12:34 PM
Great information there.
Working on all of it now. Will be right back.
Thanks!
Stuart
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Benjamin.. 12/18/13 12:38 PM
Dont miss out what ets said especially re keyword stuffing and zero links 

Use these going forward


Good luck

Ben 
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/18/13 1:12 PM
Hi ets,

Thanks for helping. I really appreciate it!

1.


Yes, indeed I do see entries in my site logs for the googlebot:
77092.74%Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)


2.  Yes I have green ticks for each item.

3.   I have now applied a .htaccess redirect from the non www. version to the www. version of the site. I think this is working, fine now.
      I was not aware of keyword stuffing - I just added those headers because thats the areas the joiner covers and I needed them in there somewhere.

I have in the last 10 minutes:

a)   Uploaded a new sitemap made using Sitemap Plus by Loghound Software (I use RapidWeaver).
b)   Tested and submitted the sitemap -- Went ok.
c)   Performed a new F.A.G. which worked fine. Super fast actually. Then chose to resubmit all URLS etc to Google.

Do you think that might help the site to get ranked now? Any idea how long I should wait till I panic again??

Thanks,
Stuart


Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Benjamin.. 12/18/13 1:19 PM
It will help Google's understanding of the site and not get it confused. Now you have to see what you want to rank for and work on that, check out competitors sites and make yours better. It will take time to settle down now you've made a few changes, read the guides above and make sure your adhering to them. If its a new site you may get an artificial boost in the rankings for a while (when it goes don't panic!) 

I wouldn't just rely on Google for traffic

Ben 
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/18/13 1:46 PM
The crawl stats graph (top) makes me raise an eyebrow. I'd keep an eye on that. Crawling has dropped off quite drastically, though if there aren't many pages and they're not changing, maybe it's to be expected of a brand new site. Still... keep an eye on it. You might want to put something like pingdom on your site for a week or two and check out uptime/response times? 

Fetch as Google: You also need to look into the issue Thomas flagged up. That definitely sounds a bit weird and needing further investigation. I just tested the home page with Rex Swain's tool and Live http headers and got content length 62764 off both (plus 248 bytes of header).

I've also run a speed test (often a good basic diagnostic) and it shows this result:

It is quite slow. (This is tested from a Manchester server.)

there are two missing files (404 - red lines):
and some issues with webfonts. These might be causing Fetch issues? I don't know.

There's a lot you could do to improve that - and that will make some difference to your ranking (though it's less important than the site content).

Do you think that might help the site to get ranked now? Any idea how long I should wait till I panic again??

The Web is more competitive than it's ever been and it's only going to get more so. You'll just have to work really hard at promoting the site in non-spammy ways (as Ben said), such as social media, all the traditional forms of offline marketing... and so on. Ben's documents are the ones to read. Those are all packed with excellent advice.
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/21/13 6:38 AM
Thanks to everyone who has replied and contributed.

Unfortunately I'm not really that much further forwards in terms of getting this site re-indexed or ranked.

I found a single page in google search which was returning a 404 and have no idea about how google got hold of that page as the address was a development page while I was building the site. I've submitted a request to have that URL removed. The request is pending...... sigh....

I submitted the site to Bing webmaster tools. Here is my experience with that:

a)  Sitemap submitted, tested and accepted in about 6 minutes flat.
b)  Site URLS submitted and all accepted and crawled in about 2 hours.
c)  Stats and health all reporting just fine with no errors reported.

I submitted a ticket with my web hosting company and explained about the returned header length differing from the reported. They say that they have tested it and they think it's okay?? Thats Falcoda UK. They reckon the pages look fine and that Google should simply be reading the text/code thats returned and thats it.

I wanted to request a re-index and resubmission to Google in a last ditch attempt to salvage the relationship with the client - but the help docs for how to DO that stat that you need to do it via a button on the 'Manual Actions' page. I have no such button! I genuinely, despite searching for it, have no clear idea at all how to actually request a re-index or submission to Google.

The site appears not to have been ranked despite all my efforts. Every single attempt to "fetch as Google fails" never once do I get  reason why.Thats the home page and all the sub pages and directories. Exactly the same URLS were accepted and indexed by Bing first time.

If anyone has any other suggestions or ideas I'd be really grateful to hear them.

Thanks,
Stuart

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/21/13 6:43 AM
I just wanted to add that when carrying out a Fetch As Google the request immediately goes to "Pending' then I get a popup saying "Your Request Was Completed Successfully" then the request goes to Pending then another popup the same as the first, then the request goes to Pending..............

At the end the fetch request just says "Failed". And thats it......No way that I can see to engage or get a message to Google to ask whats going on.........Its just a simple website - why is it this difficult?

Regards,
Stuart
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Benjamin.. 12/21/13 7:55 AM
Hi Staurt

Sorry to hear you are still having problems, in your WMT have you got the www or the non www or both verified?

The url removal tool will say pending for a while and will action soon so dont worry about that, just make sure you delete the file from your folder or it may go back into the index again

You cant request the recosideration as you have no penalty and thats what that manual action section is for. The site is in Google's index and the home page was crawled on the 15th December http://webcache.googleusercontent.com/search?q=cache:0RkKyP_Z26UJ:stephenbryantjoinery.co.uk/+&cd=1&hl=en&ct=clnk&gl=uk

Google takes time to do things, you will soon work that out although we want things to happen immediately it doesnt, your client shouldn't be annoyed as you have built a good site and got it indexed in Google so you are doing the right thing its just a few bits that need tuning up

Ben
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/21/13 8:01 AM
To answer your most recent questions:

1. As I think we explained to you before, it takes time for a new website to rank. Are you clear on the difference between crawling (basic access to the site by Googlebot), indexing (Google creating a version of your site for search engine purposes), and ranking (achieving good results in actual searches)? You will have to work hard on this over a period of weeks, months, and years. You cannot expect brilliant results in a week or two. It's a competitive web world - and growing more so by the day.

2. As I explained to you, you have no links to the site. That is one reason why it doesn't rank. It will take time to build up natural links. Until then, the site will struggle to rank.

3. "I found a single page in google search which was returning a 404 and have no idea about how google got hold of that page as the address was a development page while I was building the site. I've submitted a request to have that URL removed. "

You should not use the emergency URL removal tool for that. If a page returns a 404, it is fine. Just leave it - and Google will clean it up by itself. See https://support.google.com/webmasters/answer/1269119

Cancel that removal and let Google clean the site up properly in its own time. 404s do not harm your site. Google understands what a 404 is and knows how to sort things out when it sees one.

4. "I wanted to request a re-index and resubmission to Google in a last ditch attempt to salvage the relationship with the client - but the help docs for how to DO that stat that you need to do it via a button on the 'Manual Actions' page. I have no such button! I genuinely, despite searching for it, have no clear idea at all how to actually request a re-index or submission to Google."

Again, you are confusing indexing and ranking. The site is already indexed (as we said before). There are no manual actions so you don't need to reindex or resubmit. What you need to do is be patient and work on your ranking.

5. Google is not Bing. That's all there is to say on that. It's like comparing an apple and a coconut. It gets you nowhere :)

6. You still haven't fixed various issues we pointed out above:

a) I mentioned a page where the content is entirely locked behind images and javascript; it still is - and any pages built that way will never rank until you fix that. They might look terrific but if they have no indexable content, they will never rank for anything. 

Pages like this:
need plenty of non-keyword-stuffed descriptive text or they will never rank for anything.

What do you think Google will make of a page like this with zero useful content?

b) You've still got the keyword stuffed locations on the home page.

and keyword stuffed META:

<meta name="keywords" content="east lothian joinery,edinburgh joinery,west lothian joinery,glasgow joinery,midlothian joinery,doors and windows,loft conversions,garage conversions,new builds,groundworks,Domestic Joinery,Commecial Joinery,Shopfitting" />

What on Earth is "Commecial Joinery"? Google does not use META keywords so delete that now.

and in addition...

c) You've got a previous version of the home page indexed as index.php. Now it returns a 404, whereas it should do a 301 redirect to your actual home page (the root). 

So:

This probably explains why your home page doesn't come up as the first result on a site: search.

Did you read the documents Ben pointed you to? You really must do so. I'm afraid you are making some very basic errors here. If you calmly and patiently work through those help documents, you'll get where you want to be with this site - but it will take time to get results. There are no shortcuts.

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Thomas P. 12/21/13 8:37 AM
Hi Stuart,
 
I did a little more checking into the truncation, and it seems that it's only happening with the web-sniffer tool.
Why the combination of that tool and your site is a showing truncated response - I don't know. (It's only the combination, of that tool with your site, which is exhibiting the anomaly).
So: I think you can let that issue rest. :-)
(If you want to provide feedback to your host, then ask them to check up on any servers responding from German IPs)
 
Besides that: I agree with both Ben & ets ;-)
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/21/13 8:47 AM
Ben and ets,

First and foremost - sincere thanks for the ongoing interest and support. I suspect that I'm trying your patience a little and for that I apologise.Where I have failed to address specific items that now arouse frustration its not from arrogance or any kind of disagreement with their validity - just me now knowing HOW to address them!

I have now:-

1.  Cancelled the URL removal request which was the dead link returned by Google to the index.php page.

2.  Set up a 301 redirect::  Redirect 301 /index.php http://www.stephenbryantjoinery.co.uk
     Which I now realise I should have done in the first place.

3.  Set up a 301 redirect from the non www. version of the site to the www. version and verified ownership of both sites to Googles satisfaction.

4.  Removed the 'keyword stuffing' section on the home page.

5.  Removed the typo in the keywords: "Commecial Joinery". Yes, what on earth is that!! What was I thinking...... :-(

This evening I'll add more descriptive text to the testimonials, portfolio and domestic joinery pages.

Despite how my pitiful messages here must read I do, believe it or not have (I think!) a fundamental understanding of the diffs between indexing, crawling and ranking. Though thank you for recapping on them for me. I accept that the site and it's constituent pages are indexed because I can see that when I search for site:stephenbryantjoinery.co.uk the pages are returned. I take  that to mean that they are indexed correctly. Unfortunately the site does not appear to have been crawled for ages, possibly the 13th December and from Googling around I was under the impression that a good way to facilitate a crawl is to Fetch As Google from WMT - which fails 100% of the time. Regarding rankings, I submit to your sagacious advice about links, time and quality content. Not sure I can do much about links sadly - but I can wait and continue to try to improve the content.

Again, apologies for taking up your time, but please accept my sincere thanks and gratitude for helping me through this sticky period. I've published around five sites now - every other one was added to WMT, add sitemap, FAG, Submit and within a week or two it was showing up in the rankings. This is the first time I've come off the rails. More luck than by design, I guess.

Thanks!
Stuart
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/21/13 8:50 AM
Hello Thomas,

Yes, I noticed the IP showing up in the sites logs and wondered... :-)

Thank you so much for following up with this - you have been extremely generous with your time and knowledge and I REALLY appreciate it. If there is ever anything I can do to return the good deed...... 

Just replied to Ben and ets.... I'll keep doing what I can to get this corrected.

Kind regards,
Stuart

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Benjamin.. 12/21/13 9:04 AM
Hi Stuart

Now in WMT you can select a preferred domain that should be the www version, once you have done that see what happens when you fetch (www) as Google

Glad you are making progress

Ben
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/21/13 9:16 AM
Hi Ben,

I should have mentioned that I did set the preferred domain to be the www version and received confirmation emails that it had "taken".

Bewilderingly, I still cannot Fetch As Google from the www instance of the site in WMT (doesn't work in the NON www version either).

It really is driving me potty, honestly. If I knew what I was doing wrong - I'd fix it without quarrel.

Thanks again, Ben,
Stuart

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/21/13 10:11 AM
Don't worry - we are here to help. I'm sorry if I sounded impatient! I've left home with the wrong Wi-Fi card and, consequently, am stuck using Windows XP on a 12-year-old laptop... which is enough to drive anyone potty.

Let's see if we can address the Fetch as Google issue. There is (I believe) a known issue where a Fetch as Google can hang as "Pending" for a long time (sometimes indefinitely). I saw Pelagic call it as a "known issue" not that long ago - and I've seen it reported a few times.


Up above, you said:

"Yes, indeed I do see entries in my site logs for the googlebot:
7    709    2.74%    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

So have you seen no entries like that since December 13? That would definitely be a concern.

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/21/13 12:07 PM
Hi ets,

Haha that gave me a laugh! I hate using someone else's 'puter too. 

Okay, so thanks for forcing me to think about the right things! I've downloaded the logs files and been through them more carefully than before. The various bots are there almost every day for sure. Here is a sample:-

stephenbryantjoinery.co.uk 157.55.32.52 - - [20/Dec/2013:17:24:32 +0000] "GET /robots.txt HTTP/1.1" 404 301 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
stephenbryantjoinery.co.uk 157.55.32.52 - - [20/Dec/2013:17:24:32 +0000] "GET / HTTP/1.1" 200 60668 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
stephenbryantjoinery.co.uk 157.56.229.189 - - [20/Dec/2013:18:29:16 +0000] "GET /robots.txt HTTP/1.1" 404 301 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
stephenbryantjoinery.co.uk 157.56.229.189 - - [20/Dec/2013:18:29:17 +0000] "GET / HTTP/1.1" 200 60668 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
stephenbryantjoinery.co.uk 66.249.64.4 - - [20/Dec/2013:19:37:33 +0000] "GET /Galleries/Recent/files/stacks_image_252.png HTTP/1.1" 301 378 "-" "Googlebot-Image/1.0"
stephenbryantjoinery.co.uk 66.249.64.2 - - [20/Dec/2013:19:37:36 +0000] "GET /robots.txt HTTP/1.1" 404 301 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
stephenbryantjoinery.co.uk 66.249.64.3 - - [20/Dec/2013:19:37:37 +0000] "GET /Galleries/Recent/files/stacks_image_252.png HTTP/1.1" 200 59913 "-" "Googlebot-Image/1.0"
stephenbryantjoinery.co.uk 66.249.64.4 - - [20/Dec/2013:19:37:37 +0000] "GET / HTTP/1.1" 200 60668 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

So the Bingbots and Googlebots are definitely there. I suppose this makes sense as there are no crawl errors reported in WMT.

The symptoms I'm seeing with F.A.G. are that I kick off the fetch of the base URL/homepage and the status changes to pending and I immediately see a 'your request was successful' popup and the stats then reverts to pending with another popup. That then repeats for between 2 to 4 hours and fails but with no reason given.

Combined with none of the pages showing up in any relevant searches (which might be down to impatience on my part!), I really thought/think something is up. This site, from a WMT perspective, is behaving very differently to the other 5 or 6 in my dashboard. Does the failure of the F.A.G. appear to be down to something on the site itself? I can't think of any components I used in that site that I have not used elsewhere with no ill effect.

Thanks again,
Stuart



Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/21/13 12:28 PM
The problem is that it's such a new site so we can't really tell from the graph you posted above what a normal level of crawling for this site actually is. You look at that graph and think "Wow, crawling has fallen off dramatically - there could be a serious issue". And yet, a new/changed site is bound to see a big spike in crawling to start with. If we look over the next 12 months, maybe 10-20 pages of crawling a day is actually a more sensible level.

I don't want to brush aside the "Fetch as Google" issue or suggest it's unimportant; it could be critically so. But the key thing is that Google is regularly visiting your site by itself - and the logs prove that.

I think I would focus most effort on the ranking side of things - and making sure each of your pages looks, smells, and tastes good to Google. As a first step, I would try switching off JS and images and browsing your way round the site. Note any pages that seem to have little or no text content and try to address that. I think you've made a visually very attractive site - a really nice job that I would be proud of as either a designer or a client - but you need to make sure you see it through Google's objective, algorithmic eyes.

You (or your client) also need to work out a marketing plan for the site. It's simply not the case that you bung your site onto Google and then wait for the customers to arrive. In some ways, it's the opposite; you have to market the site before it can market you. So I would give that some serious thought. The site needs links so you need to bring it to people's attention. Could you get articles in a local newspaper? What else could you do to get traffic to the site so that it will naturally build its own momentum?
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/22/13 3:30 AM
Hi ets,

Thanks for all the support, kind words and advice.

I've added additional copy to the various pages that seemed too scant. I'll continue to add more, in keeping with the guidelines set out in the various documents and sources that have been referred to in this thread. The site has benefitted from that.

Now that I've added that it would of course be nice to be able to do a Fetch As Google - but sadly that still fails 100% of the time taking between 2 and 4 hours to return a failure.

I appreciate that the bots are getting onto the site - as the logs proved - I just can't shake off that worry that because my Fetch as Google is broken, something is 'not quite right'. I cannot see anywhere to ask Google to tell me why its failing, which is a pity.

Thanks again!
Stuart

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/22/13 7:25 AM
Now that I've added that it would of course be nice to be able to do a Fetch As Google -

Yeah but no, as Vicky Pollard would say. Fetch as Google is primarily a diagnostic tool, not a tool for getting your updated pages reindexed. I would really just let that happen naturally and encourage it gently by submitting an up-to-date sitemap. It's a shame (and a concern) that it's not working, but regular crawling is the most important thing.

It's great that you're beefing up the pages. That will help with ranking.

And reassure your client that this is a long haul. SEO types that promise either "guaranteed" or immediate ranking results on Google are essentially fraudsters. Getting good, sustainable ranking in the long-term is the important thing - and I think if you lay solid foundations now, you will get there, slowly but surely.
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. JohnMu 12/23/13 4:23 AM
Hi Stuart

Looking at your site, I do see that we'd like to crawl more from the server, but we're holding back because we think the server might not be able to handle the load. This is the reason why the Fetch as Google requests aren't making it through. In particular, we're seeing a fairly high response-time for URLs from the server, which often signals that the server is pretty busy even without us crawling. On the one hand, this is something you could work to resolve by seeing if you could speed up your site's hosting (perhaps optimizing your server-side code or moving to a slightly faster server). On the other hand, your website currently isn't that large & not constantly changing, so it's not really necessary for crawl 1000's of URLs every day from there. From that point of view, while we'd like to crawl more, I don't see it as being a critical problem for your site in particular. If users are happy with the speed of your site, then we can definitely live with it too :).

Cheers
John
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. ets 12/23/13 9:15 AM
Thank you John! That's a great help.
Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Stuart Henderson 12/23/13 2:15 PM
Hello John,

Thank you for the precise and informative reply, its most appreciated.

I was not aware of the speed/response issues with the site or hosting. The site is hosted by Falcoda UK on one of their business plans. The hosting is not shared with any other site - its 100% dedicated to the www.stephenbryantjoinery.co.uk domain. As the site was made in RapidWeaver + Stacks I struggle to think how I might optimise the server-side code as the code is generated by these apps.

I've certainly not had any negative feedback about the speed of the site - though I will raise the matter with Falcoda themselves.

All I can really add is that from a Webmaster point-of-view how could I every have worked out for myself what you have said? If the inability to Fetch as Google is down to Google holding back then would a message to the Webmaster stating that be a viable option?

Thanks for the information, John!
Kind regards,
Stuart

Re: Unable to crawl site. Not appearing in searches. Completely confused. Please help. Thomas P. 12/23/13 2:29 PM
John & Stuart
Seeing how responsiveness seems to depend on where/who is making the requests, then: Could there be one specific router in (or bridging with) the LAN (DMOZ where the webserver is located in) which is suffering packet drops (or perhaps strange trafic throttling)?
More topics »