Juknife
A Ruby library for Web scraping. It allows to describe scraping nodes and text structurally and declaratively.
For example, scraping on Google search can be written as follows:
class GoogleSearchAgent < Juknife::Agent
request do
get 'https://www.google.co.jp/search'
user_agent 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)'
query do
{
q: params[:query]
}
end
end
scraping do
items :results, '#ires .g' do
item :title, 'h3'
item :url, '.kv > cite'
item :description, '.st'
end
end
end
agent = GoogleSearchAgent.new
agent.scrape(query: 'test')
# =>
# {:results=>
# [
# {:title=>"testの意味・用例|英辞郎 on the WEB:アルク",
# :url=>"https://eow.alc.co.jp/search?q=test",
# :description=>
# "test 【1自動】 試験[検査]を受ける 《医・化学》検査[分析]を行う 〔試験で... - アルクがお\n" +
# "届けする進化するオンライン英和・和英辞書データベース。一般的な単語や連語から、\n" +
# "イディオム、専門用語、スラングまで幅広く収録。"}
# }
# ]
# }
Installation
Add this line to your application's Gemfile:
gem 'juknife'
And then execute:
$ bundle
Or install it yourself as:
$ gem install juknife
Usage
[TODO]
Development
After checking out the repo, run bin/setup
to install dependencies. Then, run rake spec
to run the tests. You can also run bin/console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
. To release a new version, update the version number in version.rb
, and then run bundle exec rake release
, which will create a git tag for the version, push git commits and tags, and push the .gem
file to rubygems.org.
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/juknife. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.