What Robots.txt Should Block: A List Of What Shouldn't Be Crawled?

[jcohen]

Here's a partial list of what maybe shouldn't be crawled that I put in my robots.txt.

Is there any thing else that should be added that shouldn't be crawled?

User-agent: *
Disallow: /blog/wp-admin
Disallow: /blog/wp-includes
Disallow: /blog/wp-content/plugins
Disallow: /blog/wp-content/cache
Disallow: /blog/wp-content/themes
Disallow: /blog/trackback
Disallow: /cgi-bin
Disallow: /search
Disallow: /blog/feed
Disallow: /blog/rss
Disallow: /blog/comments/feed
Disallow: /blog/feed/$
Disallow: /blog/*/feed/$
Disallow: /blog/*/feed/rss/$
Disallow: /blog/*/trackback/$
We generally like to put blogs in a

[jcohen]

Disregard  the last line:  "We generally like to put blogs in a.."

[jcohen]

Any chance on any help with this?

Thanks.

Joel

[Author]

Posts