redback

Redback is a Ruby spider (geddit?). Pass it a website, and it will begin its many-legged crawl, scurrying across the site to pull out all the unique URLs it can find.

Just like a terrifying real-life spider, redback aims to be fast: in particular, it sends requests in parallel so one slow page won't slow down your crawl.

$ gem install redback

$ redback http://example.com/

…in which case it will print all the URLs it finds within the site http://example.com/.

You can output he results to a file like this:

$ redback http://example.com > output.txt

Or feed them to another command line tool like this:

$ redback http://xkcd.com | grep xml

It can also be used as a library:

require 'redback'

Redback.new "http://example.com" { |url| puts url }

The Redback.new method accepts a URL and a block; the block will be executed for each URL found.