Wraith is a screenshot comparison tool, created by developers at BBC News.

Photish is a simple, convention based (but configurable) static photo site generator.

https://github.com/0dayinc/pwn/README.md

A command-line front-end for the anemone web-spider. Generates reports for seo, http errors and an xml sitemap. Extensible page handler.

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

A little website crawler.

Chef Key/value stash cache hash objects store.

Anemone helper making common crawls easier to repeat.

Script para pre-calentar servidores escenic

A very personal search index for any site you like

Find hotlinks on a given domain

Kit is DSC's Community and Content Management System (CCMS) built as a Rails engine for Rails 3.1 and above. It provides an entire application's worth of CMS functions including in-place WYSIWYG editing with versioning, flexible layouts and templates, CSS and JS all managed within your browser, drag and drop image/file uploading, modules for sophisticated forums (with in place moderation) asset management, calendars, advertising, menus, RSS feeds, re-useable components, full audit trail of editing, integration with mailchimp, Google Analytics and lots more.

A Ruby gem for checking the links in a web site. Can either scan files or crawl pages. Multi-threaded, with red/green colored output, support for SSL, and support for following redirects. Works great with Octopress, Jekyll, or any collection of static HTML files. With 100% RSpec coverage.

During the first stage of a security test, it's useful to enumerate website urls without making too much noise. Links can help in this using robots.txt or link in a web page telling you the website contents.

Dynamically generate Load Impact user scenarios.

Get image urls from mangareader(http://www.mangareader.net)

ページをスクレイピングしたり、スクリーンショットを撮ったり(scraping web page, screen shot web page .... and more)

This is web comic crawler that Akihabara otaku.

A simply web crawler.

A working website link checker