Google's Webmaster blog had a great web developer article today about 404 Pages and how to optimize them for Google.
I was making a mistake - I had a custom 404 page, but I was blocking the page in robots.txt . Apparently, you are not supposed to block the 404 page - if you do, Google won't crawl it and won't see the 404 response code (Also keep in mind you have to send the 404 response code on custom 404 pages).
The article also has some great info explaining some of the weird links you may see in Webmaster Tools and some info on 301 redirects.
Read it!![]()
I was making a mistake - I had a custom 404 page, but I was blocking the page in robots.txt . Apparently, you are not supposed to block the 404 page - if you do, Google won't crawl it and won't see the 404 response code (Also keep in mind you have to send the 404 response code on custom 404 pages).
The article also has some great info explaining some of the weird links you may see in Webmaster Tools and some info on 301 redirects.
Read it!
