0.01
Repository is archived
No commit activity in last 3 years
No release in over 3 years
Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by (http|file|etc) using EventMachine.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

~> 1.3
>= 0

Runtime

 Project Readme

Cosmicrawler

Cosmicrawler is crawler library for Ruby. It provides scalable asynchronous crawling by http, file, etc using EventMachine.

Build Status

Installation

Add this line to your application's Gemfile:

gem 'cosmicrawler'

And then execute:

$ bundle

Or install it yourself as:

$ gem install cosmicrawler

Usage

http

require 'cosmicrawler'

Cosmicrawler.http_crawl(%w(http://example.com/1 http://example.com/2)) {|request|
  get = request.get
  puts get.response if get.response_header.status == 200
}
require 'cosmicrawler'
require 'em-http-request'

Cosmicrawler.each(%w(http://example.com/1 http://example.com/2)) {|item|
  request = EM::HttpRequest.new(item)
  get = request.get
  puts get.response if get.response_header.status == 200
}      

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

License

Ruby's License