Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp.

Post URLs to Wayback Machine (Internet Archive), using a crawler, from Sitemap(s) or a list of URLs.

validate-website is a web crawler for checking the markup validity with XML Schema / DTD and not found urls.

Hacker-gems installs most needed gems for hackers.

A Gem to produce competitive intelligence data

A library to search for emails in search engines

Monet is a web UI change comparer.

Crawler that auto generates SitePrism Page Objects

ronin-web-spider is a collection of common web spidering routines using the spidr gem.

Crawl a site and check various health indicators, such as: HTTP 4XX, 5XX status, valid HTML/XML/JSON. Missing image alt attributes/missing HTML title/description.

Bunch of Hacking tools

Command Line Interface (CLI) for the spidr gem.

Stupid crawler that looks for URLs on a given site. Result is saved as two CSV files one with found URLs and another with failed URLs.