Robocop

Robocop is a simple Rack middleware that inserts the X-Robots-Tag into the headers of all your responses.

The X-Robots-Tag can be used in place of a robots.txt file or meta tags to tell crawlers what they're allowed to do with your content. See this article for more information.

Tested against Ruby 1.8.7, 1.9.2, 1.9.3, and Rubinius & JRuby in both 1.8 and 1.9 mode.

The simplest way to install Robocop is to use Bundler.

Add Robocop to your Gemfile:

gem 'robocop'

To use Robocop in your Rails application, add the following line to your application config file (config/application.rb for Rails 3, config/environment.rb for Rails 2):

config.middleware.use Robocop::Middleware do
  directives :all
end

Simple add the following to your config.ru:

use Robocop::Middleware do
  directives :all
end

The following directives can be passed in to Robocop's configuration:

• all
• noindex
• nofollow
• none
• noarchive
• nosnippet
• noodp
• notranslate
• noimageindex

If you just want to specify a list of directives for all useragents to follow, simply pass in a list of directives with the directive method:

config.middleware.use Robocop::Middleware do
  directives :noindex, :nofollow
end

If you want to give specific user agents unique sets of directives, you can do so by using the useragent method:

config.middleware.use Robocop::Middleware do
  useragent :googlebot do
    directive :all
  end

  directives :noindex, :nofollow
end

It should be noted that if both the useragents and directives options are passed in, the useragents are output first in the header, followed by the generic directives.

• Re-factor & DRY up code
• Directive validation
• Add support for unavailable_after directive.
• Sanity checks for directives that are passed in. e.g. passing all, noindex, nofollow doesn't make any sense and should not be allowed.

• Fork the project.
• Code your feature addition or bug fix.
• Add specs for it. This is important so we don't break it in a future version unintentionally.
• Commit, do not mess with Rakefile or version number. If you want to have your own version, that's fine but bump version in a commit by itself so we can ignore it when merging.
• Send a pull request. Bonus points for topic branches.

Copyright (c) 2012 PJ Kelly. See LICENSE for details.