StupidCrawler

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.

Add this line to your application's Gemfile:

gem 'stupid_crawler'

And then execute:

$ bundle

Or install it yourself as:

$ gem install stupid_crawler

stupid-crawler --help

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/buren/stupid_crawler.

The gem is available as open source under the terms of the MIT License.