Project

husc

0.0
No commit activity in last 3 years
No release in over 3 years
This project enables site crawling and data extraction with xpath and css selectors. You can also send forms such as text data, files, and checkboxes.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Runtime

 Project Readme

husc

Build Status Gem Version MIT License

A simple crawling utility for Ruby.

Description

This project enables site crawling and data extraction with xpath and css selectors. You can also send forms such as text data, files, and checkboxes.

Requirement

  • Ruby 2.3 or above

Usage

Description of Instance Methods

name Description
send Set the value you want to submit to the form.
submit Submit form.
css Get node by css selector.
xpath Get node by xpath.
attr Get node's attribute.
inner_text Get node's inner text.

Simple Example

require 'husc'

url = 'http://www.example.com/'
doc = Husc.new(url)

# access another url
doc.get('another url')

# get current url
doc.url

# get current site's html
doc.html

# get <table> tags as dict
doc.tables

Scraping Example

# search for nodes by css selector
# tag   : css('name')
# class : css('.name')
# id    : css('#name')
doc.css('div')
doc.css('.main-text')
doc.css('#tadjs')

# search for nodes by xpath
doc.xpath('//*[@id="top"]/div[1]')

# other example
doc.css('div').css('a')[2].attr('href') # => string object
doc.css('p').inner_text() # => string object
# You do not need to specify "[]" to access the first index

Submitting Form Example

  1. Specify target node's attribute
  2. Specify value(int or str) / check(bool) / file_name(str)
  3. Call submit() with form attribute specified
# login
doc.send(id:'id attribute', value:'value to send')
doc.send(id:'id attribute', value:'value to send')
doc.submit(id:'id attribute') # submit

# post file
doc.send(id:'id attribute', file_name:'target file name')

# checkbox
doc.send(id:'id attribute', check:true)  # check
doc.send(id:'id attribute', check:false) # uncheck

# button click
doc.send(id:'id attribute', click:true)  # click
doc.send(id:'id attribute', click:false) # unclick


# example of specify other attribute
doc.send(name:'name attribute', value:'hello')
doc.send(class:'class attribute', value:100)

Installation

$ gem install husc

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/AjxLab/husc.