Ruby Semantic Crawler Library¶ ↑

This project encapsulates data gathering from different sources. It simplifies the extension of internal data with public available knowledge. The major strength is the use of semantic technology to bypass complex NLP (natural language processing).

• Geonames

• CIA Factbook RDF Dump

• FAO - Food and Agriculture Organization of the United Nations

• LinkedGeoData - LGD

• GDACS

• Freebase

• Microdata on Websites, e.g. annotated with schema.org vocabulary

• DBPedia

• Different Government Sources

$ gem install semantic-crawler

Or from source:

$ git clone git://github.com/obale/semantic_crawler.git
$ cd semantic_crawler
$ bundle install
$ rake build
$ rake install pkg/semantic-crawler-*.gem

You can add this library also as dependency in your Gemfile:

gem "semantic-crawler"

Or from source:

gem "semantic-crawler", :git => "git://github.com/obale/semantic_crawler.git"                       # for the master branch or
gem "semantic-crawler", :git => "git://github.com/obale/semantic_crawler.git", :branch => "develop" # for the developer branch

This examples are only a short outline how to use the library. For more information read the documentation or look into the source code. To use the library include or execute the following line:

>> require "semantic_crawler"

Extract in an easy way microdata, e.g. with schema.org vocabulary, from websites.

>>> microdata = SemanticCrawler::Websites::MicroData.new("https://www.alex-oberhauser.com")
>>> org = microdata.to_s['http://schema.org/Organization'][1]
>>> puts org['name']
"Sigimera Ltd."
>>> puts org['url']
"http://www.sigimera.com"
>>> puts org['employee']['http://schema.org/Person'].first['name']
"Alex Oberhauser"
>>> puts org['employee']['http://schema.org/Person'].first['jobTitle']
"Co-Founder & CEO"

The GeoNames module is able to return a Factbook::Country and Fao::Country module on the base of input GPS coordinates (lat/long).

>> innsbruck = SemanticCrawler::GeoNames::Country.new(47.271338, 11.395333)
>> articles = innsbruck.get_wikipedia_articles
>> articles.each do |article|
>>      puts article.wikipedia_url
>> end
>> factbook_obj = innsbruck.get_factbook_country
>> fao_obj = innsbruck.get_fao_country

Fetch Factbook information about Austria:

>> austria = SemanticCrawler::Factbook::Country.new("austria")
>> puts austria.background
>> puts austria.climate

Parse crisis information feed from GDACS.org:

>> feed = SemanticCrawler::Gdacs::Feed.new
>> puts feed.title.to_s
>> puts feed.description.to_s
>> feed.items.each do |item|
>>      puts item.title.to_s
>>      puts item.eventtype.to_s
>>      item.resources.each do |resource|
>>          puts resource.url.to_s
>>      end
>> end

Get information from the the GDACS.org emergency feed:

>> emergency_feed = SemanticCrawler::Gdacs::EmergencyFeed.new
>> puts emergency_feed.title.to_s
>> items = @emergency_feed.items
>> items.each do |item|
>>      puts item.title.to_s
>>      puts item.link.to_s
>> end

Country information from FAO:

>> austria = SemanticCrawler::Fao::Country.new("Austria")
>> puts austria.name_currency("en")
>> puts austria.official_name("es")

Geo information from LinkedGeoData:

>> # All nodes around the center of dresden, in a radius of 1000m
>> @dresden = SemanticCrawler::LinkedGeoData::RelevantNodes.new(51.033333, 13.733333, 1000)
>> # Optional with type: SemanticCrawler::LinkedGeoData::RelevantNodes.new(51.033333, 13.733333, 1000, "TrafficSignals")
>> nodes = @dresden.relevant_nodes
>> nodes.each do |item|
>>      puts item.latitude
>>      puts item.longitude
>>      puts item.type
>> end

Freebase.com country information:

>> country = SemanticCrawler::Freebase::Country.new("Austria")
>> links = country.same_as
>> links.each do |link|
>>      be_valid link.start_with?("http")
>> end
>> puts country.website

• Ruby 1.9.3-p125 and Rails 3.2

• Ruby 2.0.0-p0 and Rails 3.2

• RubyGems

• RubyDoc

• GitHub

• Travis CI

• Bugtracker

• Wiki

© 2012 - 2013 by Alex Oberhauser for Sigimera Ltd., published under MIT license.

This software is provided “as is” and without any express or implied warranties, including, without limitation, the implied warranties of merchantibility and fitness for a particular purpose.