Project

recluse

0.0
No commit activity in last 3 years
No release in over 3 years
Friendly, neighborhood web crawler for quality assurance.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

~> 1.12
~> 5.10.1
~> 10.0
~> 0.47.1

Runtime

~> 2.4.0
~> 0.7.7
~> 2.7.5
~> 1.7.0.1
~> 0.19.1
~> 0.0.4
 Project Readme

Recluse

Recluse is a web crawler meant to ease quality assurance. Currently, it has three crawling tests:

  • Status—checks the HTTP status codes of links on the site. Good for detecting broken links.
  • Find—finds pages with links matching the pattern. Good for ensuring that references to a page are removed or renamed.
  • Assert—checks pages for the existence of HTML elements. Good for asserting that things are consistent across pages.

Installation

Add this line to your application's Gemfile:

gem 'recluse'

And then execute:

$ bundle

Or install it yourself as:

$ gem install recluse

Profiles

Recluse depends on creating profiles for your sites. This way, the configuration can be reusable for frequent quality assurance checks. Profiles are saved as YAML files (.yaml) in ~/.recluse/ and have the following format:

---
name: profile_name
roots:
- http://example.com/
- http://anotherroot.biz/subdir
email: email@domain.com
blacklist:
- http://example.com/dontgohere/*
whitelist:
- http://example.com/dontgohere/unlessitshere/*
internal_only: false
scheme_squash: false
redirect: false

Profile options

Name Required Type Default Description
name Yes String The name of your profile for identification. Should also match the filename (i.e., site has filename site.yaml).
roots Yes Array of URLs The roots to start from for spidering. Will spider all subdirectories and files.
email Yes String Your email. This is for identification of who is crawling a web page in case a system administrator has issues with it.
blacklist No Array of globs Empty array Glob patterns of sites not to spider. Useful to keep Recluse focused only on the important stuff.
whitelist No Array of globs Empty array Glob patterns of sites to spider, even if they are blacklisted.
internal_only No Boolean false If true, Recluse will not follow external links. If false, it will follow for the status mode.
scheme_squash No Boolean false Treats "http" URLs the same as "https". This way, Recluse will not redundantly spider secure and nonsecure duplicates of the same page.
redirect No Boolean false Follow the redirect to the resulting page if true.

Use

After installation, the recluse executable should be available for your command line.

Tests

Status

Spiders through the profile and reports the HTTP status codes of the links. If the profile is not internal only, external links will also have their statuses checked.

$ recluse status csv_path profile1 [profile2] ... [options]
Argument Alias Required Type Default Description
csv_path Yes String The path of where to save results. Results are saved as CSV (comma-separated values).
profiles Yes Array of profile names List of profiles to check. More than one profile can be checked in one run.
group_by --group-by
-g
No One of none or url none What to group by in the result output. If none, there will be a row for each pair of checked URL and the page it was found on. If url, there will be one row for each URL, and the page cell will have a list of every page the URL was found on.
include --include
-i
No Array of status codes Include all Include these status codes in the results. Can be a specific number (ex: 200) or a wildcard (ex: 2xx). You can also include idk for pages that result in errors that prevent status code detection.
exclude --exclude
-x
No Array of status codes Exclude none Exclude these status codes from the results. Same format as including.
Output format
Status code,URL,On page,With error

Find

Spiders through the profiles and checks if a link matching one of the provided patterns is found. Will only go over internal pages.

$ recluse find csv_path profile1 [profile2] ... --globs pattern1 [pattern2] ... [options]
Argument Alias Required Type Default Description
csv_path Yes String The path of where to save results. Results are saved as CSV (comma-separated values).
profiles Yes Array of profile names List of profiles to check. More than one profile can be checked in one run.
globs --globs
-G
Yes Array of globs Glob patterns to find as URLs of links on the page.
group_by --group-by
-g
No One of none, url, or page none What to group by in the result output. If none, there will be a row for each pair of checked URL and the page it was found on. If url, there will be one row for each URL, and the page cell will have a list of every page the URL was found on. If page, there will be one row for each page, and the URL cell will list every matching URL found on the page.
Output format
Group by none or url
Matching URLs,Pages
Group by page
Page,Matching URLs

Assert

Asserts the existence of an HTML element using CSS-style selectors. Will only check internal pages.

$ recluse assert csv_path profile1 [profile2] ... --exists selector1 [selector2] ...
Argument Alias Required Type Default Description
csv_path Yes String The path of where to save results. Results are saved as CSV (comma-separated values).
profiles Yes Array of profile names List of profiles to check. More than one profile can be checked in one run.
true --true
--report-true-only
No Boolean false Report only true assertions. Reports both true and false assertions by default.
false --false
--report-false-only
No Boolean false Report only false assertions. Reports both true and false assertions by default.
exists --exists
-e
Yes Array of CSS selectors CSS selectors to assert the existence of on each spidered page.
Output format
Selector,Exists,On page

Profile management

Where

Path where the profiles are stored for manual edits.

$ recluse where

Creation

Create a profile.

$ recluse profile create [options] name email root1 [root2] ...

For further description of the arguments, check the Profile options section.

Argument Alias Required Type Default
name Yes String
email Yes String
roots Yes Array of strings
blacklist --blacklist No Array of globs Empty array
whitelist --whitelist No Array of globs Empty array
internal_only --internal_only
--no-internal-only
No Boolean false
scheme_squash --scheme-squash
--no-scheme-squash
No Boolean false
redirect --redirect
--no-redirect
No Boolean false

Edit

Edit profile options. Any option not provided will stay as it was.

$ recluse profile edit name [options]
Argument Alias Required Type
name Yes String
email --email No String
roots --roots No Array of strings
blacklist --blacklist No Array of globs
whitelist --whitelist No Array of globs
internal_only --internal_only
--no-internal-only
No Boolean
scheme_squash --scheme-squash
--no-scheme-squash
No Boolean
redirect --redirect
--no-redirect
No Boolean
Blacklist, whitelist, and roots

More powerful blacklist and whitelist editing. All examples are interchangeable between the three list types. However, if the profile has no roots, it will not run.

Add

Add patterns/roots to the profile's list.

$ recluse profile blacklist add name new_thing1 [new_thing2] ...
Remove

Remove patterns/roots from the profile's list.

$ recluse profile blacklist remove name thing1 [thing2] ...
Clear

Remove all patterns/roots from the profile's list.

$ recluse profile blacklist clear name
List

List the patterns/roots in the profile's list.

$ recluse profile blacklist list name

Remove

Delete a profile.

$ recluse profile remove name

Rename

Rename a profile.

$ recluse profile rename old_name new_name

List

List all profiles.

$ recluse profile list

Info

List the YAML info of the profile.

$ recluse profile info name

Contributing

Bug reports and pull requests are welcome on GitHub.

Extending

Recluse is modular so you can add tasks if you want. Below is an example of adding your own task to Recluse.

require 'recluse'

module MyModule
  ##
  # Create a task object
  class MyTask < Recluse::Tasks::Task
    ##
    # First argument must be the profile. The rest are hash arguments specific for the task.
    def initialize(profile, option1: false, option2: true, results: nil)
      # Sets up everything based on the profile, queue-specific options, and can also prepopulate results.
      super(profile, queue_options, results: results)
      @queue.run_if do |link|
      	# Run a link if this function returns true.
      	# Link is a Recluse::Link object.
      end
      @queue.on_complete do |link, response|
        # Run this function after the page has either successfully been retrieved, or failed to be retrieved.
        # Link is a Recluse::Link object.
        # Response is a Recluse::Response object.
      end
    end
  end
end

# Add your task to the task list under the key 'my_task'.
Recluse::Tasks.add_task(:my_task, MyModule::MyTask)

# You can now access 'my_task' like you would the default Recluse tasks.
my_profile = Recluse::Profile.load('my_profile')
my_profile.test(:my_task, option1: true, option2: true)
results = my_profile.results[:my_task]

License

The gem is available as open source under the terms of the MIT License.