RobotsDotText

Supply rails-environment specific robots.txt files.

NOTE: This will not work with applications mounted under a sub-directory.

gem install robots_dot_text
# or
bundle install robots_dot_text

There in an included generator that will move your existing robots.txt file to the robots.default.txt file that this gem will fallback to if an environment specific robots.txt file is not found.

rails g robots_dot_text:move_existing_robots_file

If you don't run the generator the robots.txt file in your public directory will get picked up first.

You can create a file for every rails environment using the naming convention robots.rails environment.txt in your public directory and they will be rendered when requesting /robots.txt under that particular environment.

robots.production.txt
robots.development.txt

If you would like to customize the robots.txt file name to something other than the rails environment you can do so by overriding the RobotsDotTextController#robots_file_descriptor method in your app.

class RobotsDotTextController < ActionController::Base
  include RobotsDotText::Controller
  private
  def robots_file_descriptor
    request.env["SERVER_NAME"]
  end
end

The above would load robots.localhost.txt when running under localhost:3000 or robots.example.txt when running under example.com.