Clownfish

Helper for Anemone. Makes common crawls easier to repeat.

Add this line to your application's Gemfile:

gem 'clownfish'

And then execute:

$ bundle

Or install it yourself as:

$ gem install clownfish

require 'clownfish'

clownfish = MyClownfish.new

Anemone.crawl_with_clownfish(start_url, clownfish)

# query clownfish for data from crawl

A clownfish is an object that has one or more of the following instance methods:

Reference: Anemone RDocs

Returns a Hash of Symbol to values. See Anemone::Core::DEFAULT_OPTS for available options. This is forwarded as the second argument to Anemone.crawl (rdoc). Invoked once before crawl.

Returns a single Regexp or Array of Regexp. Urls matching any of these will not be crawled. Invoked once before crawl.

Takes one argument, an Anemone::Page (rdoc). Invoked once per page during crawl.

Takes one argument, an Anemone::Page (rdoc). Returns the links (Array of URI) on that page that should be crawled. See Anemone::Page#links for a starting point. Invoked once per page during crawl.

Takes one argument, an Anemone::PageStore (rdoc). Invoked once after crawl is done.

See wiki for examples.

Lists every page that has links, the links and the status code when following those links.

Record every url and it's response time.

Count pages.

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request