Leggy

Leggy is a simple Ruby wrapper for the 80Legs API. Sign up for a new account at http://80legs.com/ or view the API docs at http://datafiniti.github.io/80docs/. Leggy is built on top of Resource Kit and Kartograph and accepts custom Faraday connections.

Add this line to your application's Gemfile:

gem 'leggy'

And then execute:

$ bundle

Or install it yourself as:

$ gem install leggy

Before you can use the 80Legs client, you need to configure the API token. You can use the .configure method to do so:

Leggy.configure do |config|
  config.api_token = 'api_token'
end

Or you can set it as as the 80_LEGS_API_TOKEN ENV variable:

$ export 80_LEGS_API_TOKEN=your_80legs_api_token

Sign up for an account and API token at 80legs.

See http://datafiniti.github.io/80docs/?ruby#users for complete documentation.

user = Leggy.users.find(api_token: Leggy.configuration.api_token)
user.first_name 
#=> "Matt" 

See http://datafiniti.github.io/80docs/?ruby#apps for complete documentation.

apps = apps = Leggy.apps.all
apps.first.name
#=> "TextFromURLListOnly.js"

file = File.read("/path/to/my/eighty_app.js")
Leggy.apps.create(name: 'eighty_app', body: file)
#=> true 

For more information on creating an 80app, see: https://80legs.groovehq.com/knowledge_base/categories/80apps.

app = Leggy.apps.find(name: 'TextFromURLListOnly.js')
app.name
#=> "TextFromURLListOnly.js"

CAUTION This action cannot be undone.

Leggy.apps.delete(name: 'sample_delete')
#=> true

See http://datafiniti.github.io/80docs/?ruby#url-lists for complete documentation.

With the list.json file as follows:

[
  "http://example.com",
  "https://example.org"
]

You can now upload the list.json url list to 80legs:

file = File.read(/path/to/my/urls/list.json)
url = Leggy.urls.create(name: 'sample', body: file)
urls.name
#=> "sample"

urls = Leggy.urls.all
urls.collect(&:name)
#=> ["sample"]

Given your list.json file uploaded previously includes http://example.com && http://example.org:

Leggy.urls.find(name: 'sample')
#=> ["http://example.com", "https://example.org"]

CAUTION This action cannot be undone.

Leggy.urls.delete(name: 'sample')
#=> true

See http://datafiniti.github.io/80docs/?ruby#crawls for complete documentation.

crawls = Leggy.crawls.all
crawls.first.status
#=> "COMPLETED"

Querying by status is not yet available.

options = { 
  name: 'crawl_test',
  urllist: 'sample_crawl_list',
  app: "HeaderData.js",
  max_depth: 1,
  max_urls: 10  
}
Leggy.crawls.start(options)
#=> true

crawl = Leggy.crawls.status(name: 'crawl_test')
crawl.status 
#=> "QUEUED"

Leggy.crawls.cancel(name: 'crawl_test')
#=> true

Once cancelled, the crawl cannot be restarted.

See http://datafiniti.github.io/80docs/?ruby#results for complete documentation.

results = Leggy.results.all(name: 'testing_crawl_results')
results.first
#=> "http://datafiniti-voltron-results.s3.amazonaws.com/<TOKEN>/167627_1.txt"

See http://datafiniti.github.io/80docs/?ruby#errors for complete documentation.

Leggy raises the following errors:

• 400 (Bad Request): Leggy::Exception::BadRequest
• 401 (Unauthorized): Leggy::Exception::Unauthorized
• 404 (Not Found): Leggy::Exception::NotFound
• 422 (Unprocessable Entity): Leggy::Exception::UnprocessableEntity
• 523 (Service Unavailable): Leggy::Exception::ServiceUnavailable
• 5xx (Server Error): Leggy::Exception::ServerError

All exceptions inherit from Leggy::Error, so you can use that rescue from any exceptions this client library raises. This does not take into account errors external dependencies may raise, such as Faraday or http adapter errors.

1. Fork it ( https://github.com/activefx/leggy/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request