0.02
Low commit activity in last 3 years
No release in over a year
middleman-robots create robots.txt includes Allow or Disallow and sitemap path.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

>= 0.14.3
>= 1.16
>= 2.18.0
>= 3.1.0
>= 12.3
>= 0
>= 0.52.1

Runtime

 Project Readme

Middleman::Robots

Gem Version GitHub Actions CI

middleman-robots is an extension of Middleman. This can create robots.txt.

This plugin support Middleman v3-stable and v4.

Installation

Add this line to your application's Gemfile:

gem 'middleman-robots'

And then execute:

$ bundle

Or install it yourself as:

$ gem install middleman-robots
# config.rb
activate :robots, 
  rules: [
    { user_agent: '*', allow: %w[/] }
  ],
  sitemap: 'https://example.com/sitemap.xml'

Created robots.txt:

User-Agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

You can use options, rules [ {user_agent(string), allow(array), disallow(array)}] and sitemap. Like this:

# config.rb
activate :robots,
  rules: [
    {
      user_agent: 'Googlebot',
      disallow: %w[tmp/ /something/dir/file_disallow.html],
      allow: %w[allow/ /something/dir/file_allow.html]
    },
    {
      user_agent: 'Googlebot-Image',
      disallow: %w[tmp/ /something/dir/file_disallow.html],
      allow: %w[allow/ /something/dir/file_allow.html]
    }
  ],
  sitemap: 'https://example.com/sitemap.xml'

Created robots.txt:

User-Agent: Googlebot
Disallow: /tmp/
Disallow: /something/dir/file_disallow.html
Allow: /allow/
Allow: /something/dir/file_allow.html

User-Agent: Googlebot-Image
Disallow: /tmp/
Disallow: /something/dir/file_disallow.html
Allow: /allow/
Allow: /something/dir/file_allow.html

Sitemap: https://example.com/sitemap.xml

Contributing

  1. Fork it ( https://github.com/yterajima/middleman-robots/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request