Archive for posts tagged ‘Crunch Course’
Create a robots.txt if you haven’t yet
Kevin Day, February 5th, 2010I have anecdotal evidence that Google is more likely to crawl and index your site if you have a robots.txt (set to allow googlebot) than if you don’t have any robots.txt file.
For Crunch Course, I’ve taken my time setting up a robots.txt file. Partly because I didn’t think it would help much and partly because it takes a couple steps to do it in Django.
As a result, Google’s cache of Crunch Course is about a month old.
I finally got around to adding a robots.txt yesterday, and I’m now getting a bunch of 404 errors from the googlebot for old links that I’ve since changed the structure of. That’s indicating to me that it checks for a robots.txt frequently, but it is much more likely to actually crawl the site if it has the green light to do so. Of course it could also just be a coincidence, but that doesn’t make for a good blog post.
So there you have it, indisputable evidence from one data point that Google is more likely to crawl your site if you have a robots.txt than if you don’t.
Bulletproof Web Design Now on Crunch Course
Kevin Day, February 2nd, 2010If you want to get serious about HTML and CSS, you have to read Bulletproof Web Design, by Dan Cederholm. It’s a must-read for any web professional.
We’re just starting to go through the book chapter by chapter on Crunch Course. There’s more than 40 of us now, so you’ll have a lot of people to learn with.
