Categories: Crawling, indexing & ranking :

Can large number of internal broken links cause search penalty?

Showing 1-68 of 68 messages
Can large number of internal broken links cause search penalty? TheBigK 12/1/12 11:14 AM
I wish to ask if a website has several thousand internal URLs that are broken (point to non-existent page on the same domain), can it be a signal to Google to crawl the website less and reduce its trust in the domain? 


Can large number of internal broken links cause search penalty? cristina 12/1/12 11:21 AM
In principle errors 404 do not affect the way a site is crawled or indexed in search results. But it is a good idea to correct the broken links if possible.


Re: Can large number of internal broken links cause search penalty? TheBigK 12/1/12 11:30 AM
Just to confirm - Does it hold true even when the links that point to non-existent pages are originating on the same domain? Yes, of course it's a good idea to fix them, but I need confirmation on this from more people. 

I'm talking about a specific case where the urls on a domain point to non-existent pages on the same domain. Isn't it a big 'quality' issue for Google? Of course when external domain creates a bad-link to your domain, you shouldn't be hurt. 
Re: Can large number of internal broken links cause search penalty? cristina 12/1/12 11:34 AM
I think it applies to internal links as well. Many sites have broken internal links. But maybe give a bit more details, why there are so many broken links.


Re: Can large number of internal broken links cause search penalty? StevieD_Web 12/1/12 11:41 AM
how about a stupid quality issue, as in it is stupid to not fix an obvious and easy to fix problem.

Or a stupid search quality issue, as it is stupid for Google to send traffic to a page that no longer exists.

Of course if the page doesn't exist and fails to redirect to another page, then the searcher bounces (returns to Google to complete a similar search).... and bounce rate is reported to be a quality factor.


So do your site AND YOUR VISITORS a favor and fix the broken links.
Re: Can large number of internal broken links cause search penalty? TheBigK 12/1/12 11:43 AM
I've been researching the traffic drop on our own website and I've discussed the issues we've been facing in another thread. The general advise I got is that '404 errors don't cause traffic drop or search crawling'. BUT -

1. I think this advice applies to 404 errors created by other domains creating bad links to pages on your website. Of course you shouldn't be penalized for that. 

2. HOWEVER, when it's your own site that's generating bad - links to your own pages [because of a bug in the script], that'd create a serious quality issue for Google spiders. 

On our website, a Javascript bug created several thousand (~99k) bad URLs that point to unknown locations on our own domain. GWT reports this as 404 errors (obvious). But from the trend I observed in GWT was that the day this problem occurred, our crawl rate dropped and so did our rankings. 

Since Google crawls our website less, it takes more time to acknowledge that the errors have been fixed. 

So, I'm just confirming that a ton of bad links that point to bad locations on your own domain is a 'negative signal' for spiders. If that's the case, I'll just be patient and wait for Google's spider to do the job. 
Re: Can large number of internal broken links cause search penalty? TheBigK 12/1/12 11:54 AM
Hello Stevie,

Obviously the issues must be fixed. In our case, it was the Disqus plugin installed on our own site that created several thousand of those bad URLs pointing to non-existent locations on our own domain. Google used to crawl ~70k pages a day and suddenly it discovered that our site had a large number of broken links. It dropped the crawl rate (obvious) and our SERP (looks obvious, but I'm still researching). 

I fixed the issue by -

1. Creating proper redirects to all those bad urls to correct urls. So when Google revisits them - it's redirected to the correct URL.
2. Removed Disqus plugin from our website altogether. 

...and I've verified that the issue's gone. 

Google, however crawls our website at 10% of earlier speed, because it still thinks we've fixed only 50k of the 99k errors. Every day, it drops the count by about 1000 (as I mark them fixed). I just wish Google could crawl our website faster and recognize that the errors have been fixed. 
Re: Can large number of internal broken links cause search penalty? StevieD_Web 12/1/12 12:49 PM
1)   your marking the errors via WMT and Google acknowledging the pages are gone/fixed are two different events.... and Googlebot (index) is faster than WMT.   so continue to mark fixed but don't obsess over them.

2)  Googlebot crawls are dependent upon the stability of the site.  A stable, mature, seldom updated site is going to be crawled less often then an unstable site (frequent changes of pages, content etc) such as Amazon or CNN.   If you want frequent/deep scans, aggressively change the site.

Re: Can large number of internal broken links cause search penalty? cristina 12/1/12 1:50 PM
Check just in case that the redirects are correct. Look at the HTTP status and the Location field header of a sample of the URLs  with Fetch as Googlebot. Check that the URL in the location header is the correct absolute URL.
What type of redirect are you using? The correct type of redirect for broken links is permanent redirect HTTP status 301 from the incorrect URL to the corresponding good URL. If there is no specific corresponding good URL just leave the 404 as it is. The important thing is that the collection point of the broken link has been fixed.
Such a large number of errors do have some impact in the time of crawling, wasted bandwidth and processing resources etc. but it is quite common for large websites to have many internal broken links, especially if it is related to JavaScript code. It should not have in theory an impact on search results.
Check just in case that there are no other problems, look for example at Optimisation - HTML Improvements for possible duplicate issues, or at Content Keywords, etc.


Re: Can large number of internal broken links cause search penalty? Suzanneh 12/1/12 2:12 PM
Just to note, when I had those 1 million + 404s, these were on my site.  They were not links from another site pointing to non-existent URLs on my site.

One thing I don't understand is why you redirected the 404s?  You mean these wrong URLs actually had a real page to go to?

Suzanne
Re: Can large number of internal broken links cause search penalty? TheBigK 12/1/12 7:46 PM
@Christina & @Suzanneh,

Thank you for your responses. Allow me to explain the situation a bit in more detail and also write about the fix I applied. 

We've a popular WordPress blog with several thousand posts. We frequently updated our blog with several posts a day (multi-author blog). Google crawled us amazingly and each of our posts would appear in searches almost a minute after it was published. So far so good. 

We've had this 'Disqus Commenting Plugin' which is a popular plugin for WordPress, used by many websites. It worked fine and produced no error. I think around late August & early September the plugin got a JS error. And what it did was create several bad URLs versions for each of our posts-

Correct WordPress Post URL : sample.com/correct-url/
For each of such URL, it created following types of URLs -

sample.com/correct-url/random-number-1 -> This is a non-existent location, and therefore resulted in 404. 
sample.com/correct-url-/random-number-2 -> Same for this and several similar. 

So each of our correct URL got multiple bad URLs that Google discovered. This was an issue discussed on http://www.seroundtable.com/disqus-google-errors-15663.html

Of course this issue affected different websites in different ways. There are several webmasters who've said that they lost traffic. 

The Fix I applied -

For each of those bad URLs, I created a 301 redirect pointing to the correct URL. I did this about a month ago and since then Google's been dropping the 404s reported in GWT. The error count was at 99k by the time we discovered that it was 'Disqus' plugin, and now it's down to ~47k. We saw some improvement in traffic as the error count went down. 

The discovery of large number of 404 errors and drop in traffic & crawl rate are almost in sync which makes me believe that Google's lost trust in our website (for the time being, I hope). 
Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 3:16 AM
I do not know the URLs. Check if you have chains of redirects, for example if  a malformed URL without www redirects 301 to the malformed URL with www, that in turn redirects 301 to the good URL. It is better if both bad URLs with www and without www redirect 301 in one go to the good URL.
Did you submit a sitemap in GWT? Is it downloaded OK by Googlebot reasonably often?
Can you increase the crawl rate in GWT, in Configuration - Settings? It is better usually to leave the crawl rate settings at default, but see if you can increase the crawl reate settings temporarily and see how it goes.
Look just in case in GWT at both site URLs, with www and without www, see if there are more clues than the crawl errors.


Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 4:21 AM
No, there aren't chains of redirects. In fact, those BAD urls aren't even available on the pages anymore. Those have been removed after removal of Disqus. So currently those bad URLs exist in Google's log of errors. So the end user does not encounter the bad URLs anymore, only Google lands to correct pages when it tries to check whether those URLs have been fixed.

Yes, we do have sitemap and it's downloaded correct in the GWT. No errors on that front so far.

I had tried increasing crawl rate in GWT, but I was advised that I should better leave it for Google to decide. So currently, it's Google deciding how much to crawl. There's no improvement in the crawl rate though.

I've directed Google to use www versions of URLs and non-www redirect to the correct 'www' versions.

Question: Should I try increasing the crawl rate in GWT to direct Google to crawl more? Or should I just let Google decide?

PS: The site is getting updated just as frequently as it was before.
Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 6:04 AM
When you write that the crawl rate decreased, did it decrease for all pages, home page included? Where do you see that the crawl rate decreased, in GWT? Can you look in server access logs to have some approximate idea of how often Googlebot accesses the malformed URLs and the good URLs?


Re: Can large number of internal broken links cause search penalty? smallbiz-Bob 12/2/12 7:05 AM
My understanding is that an excessive number of broken links on a site does two things:

1) It signals Google that the site is poorly maintained.
2) It signals Google that visitors will likely encounter broken pages, leading to a poor visitor experience.

The site will be ranked lower than other sites with similar content, because they will provide visitors with a better experience.
Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 7:09 AM

Hello Cristina,

The crawl rate decrease is indicated by the graph. The posts are still indexed, but not as fast as they used to. Until August, Google crawled ~70-80 k pages per day and you can see the drop starting September (around 4-5 September when the crawler discovered large number of internal bad links). It took me a while to identify the issue and I fixed the issue towards the end of September. The error count reported in GWT is down under 47k from earlier 99k. I saw some improvement in traffic after the error count dropped below 55k. 



I think I'll have to talk to our server admins to find out how frequently is Google bot visiting the site. I've set the custom crawl rate to 'Fast' and will observe whether Google recognises the fixed errors faster. I'm 100% sure that there aren't any errors on the site as of now. All other error counts are shown '0'. 


Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 7:12 AM
Hello smallbiz-Bob - 

Yes, I think that's what we're experiencing though. I asked around several other forums and many webmasters told me that they've had similar observations. I do observed that many of our content rich pages now are outranked by other websites. I can provide examples, but I think it won't be proper to do so. 

My concern now is to tell Google that I've indeed fixed all the errors and Google can verify it through their own log. I downloaded all the 99k errors and can confirm that all the valid ones have been fixed. There are a small number of real 404s (deleted forum threads, profiles and blog posts) that should return 404. 

 
Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 9:30 AM
Did you look at Crawl Stats in GWT at both site URLs, with www, and without www?


Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 12:24 PM
Well, I verified the non-www version of the site, set-up the GWT setting to use 'www' and then deleted the 'non-www' version from GWT. So I'll have to add the non-www version again.

I'm not sure if that'd help, because the non-www URLs are already redirected to 'www' versions, in case Google goes to them. We've been using the www version of the site for several years now, and don't think Google would really bother going back to check non-www version and then slowing down the crawl rate.

The graph I attached above is for 'www' version of the site. It used to be very high until August end and the came down beginning September. 
Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 4:35 PM
Have a look just in case at the stats for the site URL without www, and by adding the site URL without www to GWT you could also set preferred domain (to with www). Adding the URL without www to GWT will not change the way Google crawls your site, it will possibly give you more information in the stats.

Re: Can large number of internal broken links cause search penalty? luzie 12/2/12 5:05 PM
>>> 1) It signals Google that the site is poorly maintained.
>>> 2) It signals Google that visitors will likely encounter broken pages, leading to a poor visitor experience.

Yes, this is exactly it.

I'd advocate to amend the Google webmastertools with a feature that tells webmasters whether 404s are coming from outside or are caused by site internal broken links.

-luzie-
Re: Can large number of internal broken links cause search penalty? zihara 12/2/12 5:16 PM
We know of one AdSense publisher who was making close to $10K per month when his account was wiped. It took him a month of pushing hard but finally a friend of a friend of a friend somehow connected him directly with a Google employee who told him it was primarily a 404 problem (because Google officially responds to absolutely nothing to do with a disabled account)... By the time he got the 404s fixed and was accepted back into the AdSense program (one of the extremely few who ever got an account re-enabled once it was disabled - and evidence that 404s are an actionable problem in Google's world because this publisher kept us appraised of what was happening every step of the way - and I saw the ads gone and saw them come back), along came Panda and wiped him again - his sites were all business directories and the 404s had come from businesses who'd listed themselves, then went out of business. Then Panda ravaged the whole "directory" niche.

So whatever folks are reading into the "official documentation," I'd suggest you read it with a large grain of salt. And keep in mind the Google AdSense team often recommends things that are totally contrary to what's recommended in Search and Search seems to like doing things that jerk the AdSense crew around.

Google user Beware!
Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 6:09 PM
All large sites have errors 404 at some time or other, from internal and external broken links. In this case the links were collected from JavaScript code. Google collects links by using various advanced algos that do more than collecting clear URLs from HTML <a> tags. If all sites that have errors 404 would be penalised in search results, the first page of search results would have mostly one or two-page sites with only one or two links, and that obviously is not the case.
The important thing is that the errors are fixed now.
But without the site URL it is impossible to see what is happening.


Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 6:18 PM
Zihara, it is possible that your friend had ads on 404 pages, pages that had no meaningful content but were visited a lot from various links. It is possible that he might have fixed the AdSense problem by simply not displaying ads on pages with HTTP status 404 (Not Found), so it might not have been about the 404s as such, but about the ads on them, but you have no details, so I cannot know what happened there.  

Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 10:33 PM
@Luzie: I think GWT reports the URL which are pointing resulting in 404. So webmasters can know that it's internal or external. I hope I got you right?
Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 10:35 PM
@zihara : In our case, it's the JavaScript error that's created all those bad internal links. Should we remove adsense from ourwebsite until the problems are fixed? 
Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 10:41 PM
Adsense is a totally different issue, I guess. In our case, the broken links exist only in Google's log; not on the site anymore. Disqus has been removed and unless Google drops the count from their log; it's going to crawl us lower and slower. 

I've an innovative idea of telling Google that I've fixed all those 404 errors!

While GWT only shows top 1000 errors, I have downloaded all the errors through API. I'm wondering if I should make a list of all those broken URLs and create a sitemap containing the broken links (which have been redirected to the correct URL). I will then submit this sitemap to Google. 

I think this way, Google will crawl those 'broken links' and through 301 redirect, land on correct URLs. I think it will make Google drop the error count faster and restore its faith in our website. 

Will this work?
Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 11:02 PM
No, do not submit URLs with 404 in the sitemap, only good URLs that you want indexed. Do you have an XML sitemap with the good URLs you want recrawled and with correct dates for lastmod? Is the sitemap in GWT downloaded OK by Google and often?


Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 11:26 PM
Of course I'm not submitting URLs with 404. What I have a list of URLs reported as 404 in GWT, but these have been fixed already. So each URL goes to a proper page. It's just that Google's not crawling them as fast as it should and acknowledge that they've been fixed.

So by submitting the links that Google thinks are 'still broken', I'll ask Google to crawl these URLs and see that they've been fixed. I do have a regular sitemap that has all the correct links that gets updated and submitted to Google every day. 


Re: Can large number of internal broken links cause search penalty? cristina 12/2/12 11:35 PM
You need recrawled the correct URLs, with HTTP status 200 (OK) that you want indexed in search results. Are the good URLs in your sitemap crawled OK by Googlebot?
What URLs do you want to add to your sitemap related to the URLs with 404? Without some real examples of URLs it is difficult to understand what you mean with the new sitemap question.


Re: Can large number of internal broken links cause search penalty? TheBigK 12/2/12 11:49 PM
Let me post examples. 

Correct URL: 

Disqus generated all following types of URLs, each pointing to non-existant location on our own domain.

...
...
and so on. 

I fixed this problem by setting up permanent redirect to respective correct URLs.

So now, 

Note that the bad URLs I listed above exist only in Google's error log reported in GWT. Not on the site anymore because I removed disqus plugin few weeks ago. 

My Innovative Plan -

Create a sitemap of URLs containing -

...
and so on, because that's the way Google will crawl those URLs faster (I hope) and recognise that they land up on correct pages. Google will therefore drop those errors from their log and recognise that the site indeed is error-free.

Does that sound like a good plan?
Re: Can large number of internal broken links cause search penalty? cristina 12/3/12 12:00 AM
No, you do not need Googlebot to spend the time on your site crawling garbage URLs, just to see the 301s and wasting even more time by procesimg tens of thousands of redirects 301 for phantom URLs. Google knows they are incorrect URLs because it got for them 404 that you can see in GWT. You just need Google to drop the garbage URLs from its crawling schedule. You need to concentrate on encouraging Googlebot to crawl the good URLs you want indexed in search results, by having a good sitemap and good site navigation structure.
Have a look if possible at your server access logs or log stats and try to get some detailed idea of how Googlebot is crawling your site.

Re: Can large number of internal broken links cause search penalty? TheBigK 12/3/12 12:05 AM
I just had a look at the server logs and found that Google's visiting the site almost every minute. Is that sufficient? It's crawling good URLs but the bad URLs are removed from their index only as I marke them as fixed. On 26 and 28 of November, Google dropped the error count by 4k and 3k respectively, a good diversion from regular 1k/day. 

I'm wondering how long will Google take to restore its faith and trust in our website once it recognises that errors are all gone. I'm really fed up chasing these errors, reading about SEO and not being able to dedicate time I should to users of our website. 
Re: Can large number of internal broken links cause search penalty? cristina 12/3/12 1:41 AM
Without the site URL it is difficult to see if there still is a problem.
As Stevie already wrote, you do not need to spend too much time on marking the errors 404 as fixed in GWT because this is just for you to have less cluttered stats. If you think that you fixed all causes for the errors on your site, just concentrate on improving the content of your site. Keep an eye on the dates in GWT for the errors 404 to check that there are not many new dates. Some of the dates might still be quite recent if the JavaScript code causing the errors is still in the Google cache. Have a look at the Google cache of some of the URLs that appear in GWT as the page where the links to the URLs with 404 are, for recent discovery dates in GWT of the 404s, check the date of the Google cache.
If Googlebot crawls your site every minute, and crawls OK mostly good URLs that you want indexed in search results, this is good.

Re: Can large number of internal broken links cause search penalty? luzie 12/3/12 3:13 AM
Yes, you're right, but it's kind of tedious to sift through the list of 404s in order to see where they're coming from, so basically what I'm asking for is the webmastertool to have two lists, one of the internal and one of the external 404s.
Re: Can large number of internal broken links cause search penalty? TheBigK 12/3/12 3:24 AM
Well, I guess that I'll just wait and watch. I think it'd be *really* great if Google could restore the 'mark everything as fixed' option through UI or through API. We need a way to tell Google that we take the errors seriously and want them to crawl us faster. Marking only top 1000 as fixed is just too time consuming. 

I had initiated a new thread for the request, but it just got lost in the sea of threads here. 

I'm wondering if any Googler can read this post and respond. I'm sure several webmasters would love to have the option back.
Re: Can large number of internal broken links cause search penalty? JohnMu 12/3/12 3:03 PM
Hi TheBigK

The number of crawl errors on your site generally doesn't affect the rest of your site's crawling, indexing, or ranking. It's completely normal for a site to have URLs that are invalid and which return 404 (or 410, etc). That wouldn't be something which we would count against a site -- on the contrary, it's something which tells us that the site is configured correctly. 

For more information about 404's in particular, I'd also check out our blog post at http://googlewebmastercentral.blogspot.com/2011/05/do-404s-hurt-my-site.html 

When it comes to marking crawl errors as "fixed" in Webmaster Tools, this is something which is not used for the rest of our crawling / indexing systems, it's only used to simplify things in your UI. We try to bubble up more important crawl errors (such as pages that recently existed), so if you're not seeing any important URLs there (such as some that you thought were valid URLs), then I wouldn't worry about those crawl errors. 

Cheers
John
Re: Can large number of internal broken links cause search penalty? TheBigK 12/3/12 8:39 PM
John, are you saying that even if a domain has a ton of broken links that point to non-existent locations on itself - it's still not a problem? Doesn't that tell Google that the site has a 'bad' URL structure that the admin needs to fix? 

404s can be caused by two types of URLs -

1. Outside domain pointing to non-existent locations on your domain. [ external.com/broken-linkto-my-domain.com
2. Internal broken links pointing to non-existent locations on the same domain [ mydomain.com/bad-linkto-mydomain ]

The first type shouldn't be a problem because the webmaster can't do much about it. HOWEVER, when there are several thousand internal links that are broken - doesn't Google get a bad signal about the overall 'quality of website' ? 

So, are you suggesting that it's ok not to fix those internal broken links because they don't affect Google in any way? 

Is quality of website not an important factor for Google to rank it?

PS: Why is it that the decline in Google Crawl Rate & Traffic Drop almost in sync with rise in the count of internal broken links for our website? 
 
Re: Can large number of internal broken links cause search penalty? webado 12/3/12 9:22 PM
Not a problem unless thsoe broken urls  appear in the index. This means that they were once good urls, responding with 200 and have become no longer available lately.

Those urls you're talking about would not have been  indexed as they were broken all along.
Re: Can large number of internal broken links cause search penalty? TheBigK 12/3/12 9:35 PM
@Webado : Good to see your response. As I said, the rise of the broken links was in the short period of time  ( 11k - 17k - 26k - 44k ...) because of Javascript bug. Our blog has several thousand posts and by the time we figured out it was disqus plugin that was creating all those bad links that pointed to non-existent locations on our own domain - the error count was > 99k. 

Looking at the crawl rate and traffic graph, which declines almost in sync with the rise of those bad links - I figured out that once I fixed them and Google acknowledges them as 'fixed' - I should be on my way to recovery. And the traffic trend I'm observing currently is in sync with what I believed. 

I've been asking on several webmaster forums whether those broken internal links would make Google crawl us slower and drop our traffic - and majority of them have responded with 'yes'. It's quite logical. Given that our site is high quality that never participated in any bad practises (and I've checked this 10 times), the only thing that seems to be affecting us is these bad links that originate on our own domain and terminate on non-existing locations on the same domain. 

I think John's talking about 404 errors in general. Of course when those are generated by external domains pointing to your domain - they shouldn't affect your rankings. 

My head's about to explode. :-( 
Re: Can large number of internal broken links cause search penalty? webado 12/3/12 9:43 PM
If those urls which were reported as 404 were never indexed then they do not hurt, regardless how many there are.

Since you're redirecting them now they will eventually (after a long time) disappear from the report. But whether they do or don't it doesn't really matter. 

Putting all the redirected urls in a sitemap will only result in as many sitemap errors being reported since having redirected urls in a sitemap is also an error.
Re: Can large number of internal broken links cause search penalty? luzie 12/4/12 1:16 AM
>>> I've been asking on several webmaster forums whether those
>>> broken internal links would make Google crawl us slower and
>>> drop our traffic - and majority of them have responded with 'yes'.

Ok, I need to be more specific regarding the definition of internal broken links. What I was referring to is broken internal hyperlinks on existing pages that lead to no-where (user can still click on an existing link but lands on a 404). This is the kind of "broken links" that should cause problems because of the bad user experience involved. (It would seem that the OP's broken javascript "links" would more or less fall under this category)
Re: Can large number of internal broken links cause search penalty? TheBigK 12/4/12 1:25 AM
The more I type to explain the situation - the lesser it gets read. I'll try once again, but keeping things as short as possible. 

Situation: Javascript bug lead to creation of several thousand links on our domain that point to unknown location on our own domain. Here's an example -

Correct URL (Returns 200, has useful, unique original content)

Javascript bug created URLs (only discovered by Googlebot, not humans) :
(Bad URL) mydomain.com/my-correct-URL/13452444 -> Note that this URL is pointing to an unknown location on my own domain! But when Google tries to crawl it, it throws 404. 

Because our website has thousands of posts, the number of the automatically created bad URLs is ~99k. 

I THINK : Google bot comes to our website. Encounters this javascript bug and starts following all these virtual URLs which point to non-existent locations and throw 404. Google says "Hmm, this website has so many URLs that point to their own domain's non-existent locations. This website has a ton of errors. It gives users bad experience. Let me rank them down". 

Have I clearly described the situation? 
Re: Can large number of internal broken links cause search penalty? TheBigK 12/4/12 1:40 AM
Okay, here's a screenshot from our webmaster tools account. Check the reported URLs -

Correct URL: /nano-sim-coming-soon-1266/
Javascript Created URLs: /nano-sim-coming-soon-1266/<random number string> (See below)

Note that all these URLs point to non-existent locations on our own website. Is it not the reason for traffic drop? 
Re: Can large number of internal broken links cause search penalty? Free2Write 12/4/12 1:59 AM
Sounds like old news.
http://www.seroundtable.com/disqus-google-errors-15663.html

Re: Can large number of internal broken links cause search penalty? Tiggerito2 12/4/12 2:45 AM
That problem with disqus seems to have been fixed. Mark the URLs with those numbers as fixed and they will go away (they have for me).
Re: Can large number of internal broken links cause search penalty? David Coton 12/4/12 2:50 AM
This is a good tool that i use http://www.brokenlinkcheck.com/ and its fairly cheap.

Kind Regards
David
Re: Can large number of internal broken links cause search penalty? Suzanneh 12/4/12 3:24 AM
A does not always cause B.  Just because the drop happened around the time of the 404s does not mean it was because of the 404.

Did you read the Google answer that John pointed to (I'm pretty sure others have too)?

Q: Do the 404 errors reported in Webmaster Tools affect my site’s ranking?
A: 404s are a perfectly normal part of the web; the Internet is always changing, new content is born, old content dies, and when it dies it (ideally) returns a 404 HTTP response code. Search engines are aware of this; we have 404 errors on our own sites, as you can see above, and we find them all over the web. In fact, we actually prefer that, when you get rid of a page on your site, you make sure that it returns a proper 404 or 410 response code (rather than a “soft 404”).

You didn't even need to redirect those bad URLs.

You've had a Google employee tell you that the 404s won't effect you.  You believe it did; you redirected the URLs.  Not much else to be done then...?  Continue working on the site. :-)

Suzanne
Re: Can large number of internal broken links cause search penalty? TheBigK 12/4/12 4:17 AM
@Suzanneh: I'm not challenging anyone, but it's bit surprising that Google totally ignores your website's errors. I'm concerned because for the past 2 - 2.5 months, I'm finding out whether I've really solved the problem. If internal broken links caused the problem, then I can go back to my routine assuming rankings will come back as Google crawls those bad URLs and removes them. Else, I will have to spend time finding out what the 'real' problem is.

@Free2Write: Yes, pretty old problem. Several webmasters said that it caused them traffic.
@Tiggerito : Yes, yesterday saw a drop of 2.5k errors, from regular 1k. Good news! 
@David Coton: Checked our website and it shows only 4 errors which I can't do anything about, because they are from external domains. 

It's really strange that there are so many mixed opinions on this issue. Can someone spend a few minutes actually 'reading' my post, understanding the real situation and give me their opinion on whether I'm on the right direction? Should I go back to my routine of creating content or investigate more time finding out the problem that caused traffic drop? 
Re: Can large number of internal broken links cause search penalty? luzie 12/4/12 4:27 AM
>>> It's really strange that there are so many mixed opinions on this issue.

Not that I could see, there's just some confusion on what kind of 404s we're talking about.

Let's consider the following three different cases:

1. 404s from outside - no harm done

2. 404s from non-existing URLs, not internally linked anymore - no harm done

3. 404s caused by existing broken hyperlinks lurking somewhere on site - potentially dangerous
Re: Can large number of internal broken links cause search penalty? webado 12/4/12 4:57 AM
https://www.google.com/search?q=inurl%3A%22nano-sim-coming-soon-1266%22&rls=com.microsoft:en-US:%7Breferrer:source?%7D&ie=UTF-8&oe=UTF-8&sourceid=ie7&rlz=1I7GGHP_enCA455

There's only one url indexed for that page. None of the broken urls generated by Disqus are. So there's no bad user experience involved.
Re: Can large number of internal broken links cause search penalty? webado 12/4/12 5:01 AM
Xenu Link Sleuth is free not just merely cheap.
Re: Can large number of internal broken links cause search penalty? TheBigK 12/4/12 5:14 AM
Yes, there aren't any broken URLs because I fixed the problem and even removed Disqus. In fact, it was Google that encountered those broken URLs, not people. But sudden rise of broken links was the problem. 
Re: Can large number of internal broken links cause search penalty? webado 12/4/12 5:23 AM
What you're missing is the fact that none of those broken urls were ever indexed. 

They could not have been since they have always responded with 404 (unless your site was not responding properly for a non-existent url, which in itself would have been the single biggest technical problem for which you as site owner were responsible, not the fact that Disqus caused broken urls to exist).
Re: Can large number of internal broken links cause search penalty? JohnMu 12/4/12 7:11 AM
Hi TheBigK
This thread is moving too quickly :-) - let me respond to your questions:

The number of broken links -- assuming the links point to URLs that don't exist -- generally does not affect your site's crawling, indexing or ranking at all, regardless if it's a handful or millions of them. This does not make us assume that a website is of lower quality (personally, it's more like a sign that the website is technically handling these invalid URLs correctly, which would be a good sign). The number of 404/410 crawl errors would also not negatively affect the crawl rate of the website -- it might even increase the crawl rate since the server can likely respond to these requests a bit faster than to normal requests.

The caveat that Luzie mentioned still applies though -- if these links were meant to point to legitimate content, then of course those links won't work, and that can make it harder for us to find that legitimate content. In other words, if your website is linking internally with broken URLs instead of correct URLs, then that would be worth fixing. In your case, these links appear to be pointing to invalid URLs, so that's not a worry here. 

The reason for the "generally" is somewhat technical and not something that most websites would need to worry about. In particular, we try to limit our crawling on a per-server basis to avoid overloading the server and its websites. If we were to crawl invalid URLs instead of useful URLs, then it could take longer for us to recrawl the useful URLs. Since we try to prioritize normal URLs that we know about over URLs that we're just double-checking to see if they exist, this wouldn't be an issue; we'd still crawl your normal URLs normally and just try to squeeze these extra URLs in on the side. Even if we were to crawl some normal URLs a bit less frequently, that would generally not affect their indexing or ranking.

Hope this helps!
John
Re: Can large number of internal broken links cause search penalty? TheBigK 12/4/12 8:21 AM
Hello John,

Thanks a lot for your reply. One last question before I get back to making my site rock again (and attract some Google love). The links originated from my own site and pointed to non-existant locations on my own site (same domain). This definitely is a technical fault on the website, no matter what was the reason for creation of those broken links. These bad links shouldn't have existed in first place. 

I'd appreciate inputs on why the rise of errors reported in GWT and drop in crawl rate & traffic are in sync. Is it just a coincidence?

PS: May I request you to look at our GWT account and tell us what caused the traffic drop around 4-5 September? I'm almost on the brink of giving up! 
Re: Can large number of internal broken links cause search penalty? Bewenched 12/4/12 6:09 PM
You might want to look into using Google custom search for your 404 page. There's a setting in their tool that suggests corrected pages.
We use it just in case someone posts a forum link to a page that's now long gone.... I do hate visitors landing on a 404 page with no options... they usually just bounce away losing potential traffic and/or sales.

And yes it could potentially signal poor website design.

With any new code always test and test and test and always use canatonical url tags in your header.
Re: Can large number of internal broken links cause search penalty? mark111 12/5/12 3:56 AM
Sorry Suzanne, I have to disagree. Unfortunately I changed the navigation of my site 3 times in quick succession and hence ended up with over 11,000 404 errors. I have gone from being no1 page1 for my desired keyword phrase and demoted to page 40 or worse. I have read exactly what you have said  and watched a video with Matt Cutts confirming this but I know it is for this reason I'm being penalised. I could use the removal tool though Matt says that it is not a good idea to do this for thousands of URL's so my only other alternative would be to do 1000's of URL directs which will take forever. 

I can understand why google would penalise me for this, obviously thousands of indexed urls which don't exist is not good for the search engines. Problem is that there doesn't seem to be a clear way to deal with this. The only advice available from google is just leave the 404's and they will go away. When? How long? 

I decided to test with another website and created a load of 404 errors and then removed the website and closed down the webmaster tools account. The indexed pages vanished within a month. This is an option which can work, make a list of all your inbound links and when you rebuild the site make sure that these links have URL directs.

My problem is that a lot of my traffic comes from google product search, it would be too damaging for me to close down the site and then start all over again. There has to be an easier way to go about this and I'm not quite sure what it is???
Re: Can large number of internal broken links cause search penalty? mark111 12/5/12 3:59 AM
@Luzie

2. 404s from non-existing URLs, not internally linked anymore - no harm done

Err, this is exactly what has harmed me in a big way! :(
Re: Can large number of internal broken links cause search penalty? Suzanneh 12/5/12 4:09 AM
Mark, I posted in your other thread.  Changing URL structure and getting 404s is an entirely different matter. If you change a URL structure, you need to redirect from the old page to the new page. When you have a minute can you provide more details in your thread (what you did, sample URLs).

Suzanne.
Re: Can large number of internal broken links cause search penalty? mark111 12/5/12 4:10 AM
Hi John,

It seems that you are giving great advice.

How would I go about fixing my problem? I have over 11,000 404 errors due to me up changing navigation and up loading many different products in quick succession. Matt Cutts advises not to use the removal tool for this and to do URL directs for this many URL's would take a long time and not sure that it would even be the right thing to do.

My ranking has suffered dramatically and it perfectly makes sense as to why this has happened. 1. I have learnt my lesson and wont do it again but this still doesn't help me with solving the problem. Can you offer any advice?
Re: Can large number of internal broken links cause search penalty? mark111 12/5/12 4:12 AM
Thanks
Re: Can large number of internal broken links cause search penalty? Suzanneh 12/5/12 5:01 AM
Just to confirm to those reading this thread:  mark111's problem is completely different.  He confirmed in his thread that he changed the URL structure but did not redirect the old URL to the new URL.  Therefore, the old URL is returning a 404.

Suzanne
Re: Can large number of internal broken links cause search penalty? mark111 12/5/12 5:29 AM
I know the problem, know what I have done wrong and understand why I am being penalised by google so not learning anything here. What I'm asking is: Do I have to do over 11,000 URL redirects for every single 404 error or is there something else I can do?
Re: Can large number of internal broken links cause search penalty? JohnMu 12/5/12 5:31 AM
Hi TheBigK

Links to invalid URLs, even if they're from your side, regardless of whether they're due to technical issues or not, would generally not affect our algorithms' view of the quality of your website. Our algorithms are really good at dealing with this kind of issue, and wouldn't get swayed by that when evaluating the quality of a website. 

Regarding the overall crawl rate, I'd recommend double-checking the specifics. From what I'm seeing, we've become more optimized in our crawling of your site, which could definitely result in seeing an overall lower crawl rate, while keeping the same coverage & freshness. One thing I did was to double-check the unique URLs that we crawled in late August over a period of 4 days, and compared that with the unique URLs that we crawled in the last 4 days. Looking at that, I saw that we were not re-crawling URLs that had a noindex (eg http://goo.gl/20U5z ), URLs that changed & redirect (eg http://goo.gl/6kAgphttp://goo.gl/MG8gQ ), and similar changes. I didn't check all URLs that we didn't recrawl, but out of those that I checked, most appeared to not require a recent recrawl. 

Cheers
John
Re: Can large number of internal broken links cause search penalty? Tiggerito2 12/5/12 6:12 AM
On Wednesday, 5 December 2012 23:59:42 UTC+10:30, mark111 wrote:
I know the problem, know what I have done wrong and understand why I am being penalised by google so not learning anything here. What I'm asking is: Do I have to do over 11,000 URL redirects for every single 404 error or is there something else I can do?

Your not being penalised by Google, they are just doing what they can.

They indexed and ranked all your old pages and you showed up in search engines because of that. 

You then removed them all, without telling Google a thing. What would you do if something you referred to just disappeared? Stop referring to it!

All the websites that linked to those old pages now contain broken links. You are now annoying all the visitors that come from the websites that endorsed you (Nothing to do with Google).

All Google wants is that you tell them if you change things, so they can update their data and continue to send people to your website like they did before.

And I'm sure those websites that endorsed you would prefer you didn't embarrass them by showing an error page on their referrals.

Google will eventually find your new pages but all your history/ranking from the old pages will be lost if you don't help Google know what you have done.

You don't have to do 11,000 URL redirects, but for everyone you miss (a puppy will die) some ranking and user experience may be lost.

If it's too hard a job. research into the pages you have that had value and redirect them. Which had traffic or which have backlinks.




Re: Can large number of internal broken links cause search penalty? mark111 12/5/12 6:24 AM
Hi Tiggerito,

100% correct! I have started first by looking at which pages have external links and the pages which have the most internal links and doing redirects for these. Doing redirects for all the products urls will be a complete bore but vital at the same time.

I penalised myself through my own stupid actions, have got that. I was just asking if there was an easier way to go about rectifying the issue and all everyone said was 404 errors don't harm your ranking which is obviously not true.

Thanks for the detailed reply and have a nice day. :)

Re: Can large number of internal broken links cause search penalty? Tiggerito2 12/5/12 7:11 AM
Your in a big bucket of stupid people. It's a very common issue that few know about until after the fact.

Some other tips:

Make sure your 404 page helps people. One trick is to take the referring search term or destination URL and create a default search term for the 404 page to display results for.

Is there a pattern between the old URLs and the new ones? It's possible to do bulk redirects if you have a pattern.

Track the URLs that cause 404s and fix them. 
More topics »