NOTE: This repo is still under construction, please use at your own risk. :-)

This is a unofficial Ruby gem that provides a wrapper around Google Robotstxt Parser C++ library.

Add this line to your application's Gemfile:

gem 'google_robotstxt_parser'

And then execute:

bundle

Or install it yourself as:

gem install google_robotstxt_parser

At the time there is only one method available which is demonstrated below:

require './robotstxt'
include Robotstxt

robotstxt_content = "# robotstxt.org/\n\nUser-agent: *\nDisallow: /\n\nSitemap: https://www.bqst.fr/sitemap.xml"
user_agent = "GoogleBot"
url = "https://www.bqst.fr"

allowed_by_robots(robotstxt_content, user_agent, url)
# return true if user_agent is allowed to access url

If you're using Heroku, you'll need CMAKE buildpacks to build this gems. To add it to your app :

$ heroku buildpacks:remove heroku/ruby
$ heroku buildpacks:add starkast/cmake
$ heroku buildpacks:add heroku/ruby
$ heroku buildpacks 
=== mytool Buildpack URLs
1. starkast/cmake
2. heroku/ruby
$ git push heroku master

There are quite a few outstanding tasks:

• Some tests :-)

Bastien Montois @ la-revanche-des-sites.fr

Bug reports and pull requests are welcome on GitHub at https://github.com/larevanchedessites/google-robotstxt-ruby. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

View the changelog. This gem follows Semantic Versioning.

The gem is available as open source under the terms of the MIT License. See LICENSE for details.

Everyone interacting in the project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.