DBpedia entity sumarization

Ruby tool to entity sumarization. Sumarization results.

Tool description

Tool was developed as part of diploma thesis. Tool generate .json knowledge base. Knowledge base is based on outward resource links from Wikipedia abstracts, global link relations of entity classes and global literal relations of entity classes. Strict links are collected from dataset of NLP Interchange Format (NIF). In next step find all relations in which are used collected links for entities of given class (eg. City, Artist, ...). Linked based relations are expanded with entity class global relations that point to literal. Tool can reduce basic duplicates of predicates. At the end calculate total score for every predicate and generate knowledge base.

RubyDoc

browser_web_data_entity_sumarization-1.1.0

Basic usage

Requires installed JRuby on machine.

This project is mainly based on usage of dataset in NIF format. For example can be download from http://wiki.dbpedia.org/nif-abstract-datasets

Install

CMD:

gem install browser_web_data_entity_sumarization

How to initialize tool in project

Gemfile:

gem 'browser_web_data_entity_sumarization'

How to initialize tool in script

Script:

require 'browser_web_data_entity_sumarization'

Use in script

Initialize main instance of statistic

 nif_dataset_path = 'nif-text-links_en.ttl'
 resource_dir = 'statisticts/'
 statistic = BrowserWebData::EntitySumarization::Statistic.new(nif_dataset_path, resource_dir, true)

Start generate complete knowledge base for all entity class types

nif_dataset is required. Knowledge base is generate from link statistics and literal statistics

params = {
  entity_types: statistic.get_all_classes, # [City, Person, ... ]
  entity_count: 10, # count of best ranked resource for every entity type
  demand_reload: false, # resource is skipped if already has stored in result dir
  console_output: true, # allow to display some information from process
  identify_identical_predicates: false # try to find identical predicates and grouped them as one item
}

statistic.create_complete_knowledge_base(params)

Warning: Parameter identify_identical_predicates: true can increased process time identification is based on verifying every combination of predicates list. Method reduce list to max 250 predicates. All resource are completed by one iteration from NIF dataset file.

Start generate only global literal statistic

nif_dataset is no required

entity_class_type = 'http://www.w3.org/2002/07/owl#Thing'
resources_limit = 1000

statistic.generate_literal_statistics(entity_class_type, resources_limit)

and alternatively for all classes:

resources_limit = 1000

statistic.get_all_classes.each{|entity_class_type|
  statistic.generate_literal_statistics(entity_class_type, resources_limit)
}

Refresh resource statistic files

nif_dataset is no required. If links has already extracted from NIF dataset and need only reload predicates from DBpedia, can use:

best_ranked_resource_count = 100

statistic.refresh_statistics_in_files(statistic.get_all_classes, resources_limit)

(currently is no allowed to specify which files might be refreshed, only by define entity classes and count of resources to find by best rank.)

Recalculate knowledge base

nif_dataset is no required. In case you need to recalculate predicates count and update knowledge base, can use:

entity_class_type = 'http://www.w3.org/2002/07/owl#Thing'
identify_identical_predicates = false

statistic.generate_knowledge_base(entity_class_type, identify_identical_predicates)

and alternatively for all classes:

identify_identical_predicates = false

statistic.get_all_classes.each{|entity_class_type|
  statistic.generate_knowledge_base(entity_class_type, identify_identical_predicates)
}

Results description

Results published in this project Sumarization results was generated from NIF dataset: nif-text-links_en 2016-04/ext/nif-abstracts/en/

Results contains:

knowledge_base.json
global_statistics.json
identical_predicates.json
statistics.zip

Global statistic

Statistics contains predicates with their total count. Predicates are assign to every entity class. This project requires only predicates that point to literals.

File data example:

{
  "Entity_type_1": {
       "predicate_1": 200,
       "predicate_2": 4321
  },
  "Entity_type_2": {
      "predicate_1": 3000,
      "predicate_3": 4321
   }
}

Link statistic

process_time - relative time to find links in nif dataset and time to find relations with that link.
resource_uri - resource
nif_data - contains hashes of found links, anchor for link, indexes of anchor, paragraph information and relations by entity class.
- anchor - text from abstract
- indexes - position of anchor in paragraphs
- section - identifier of section where anchored text is included
- weight - computed by relative position anchored text in abstract
- link
- strict_properties
  - contains properties grouped by entity class type
  - contains properties by strict relation between resource and link
  - contains also total count of occurrence in resources by entity class type
- properties - contains properties
  - contains properties grouped by entity class type
  - contains properties by relation with ignore resource uri
  - contains also total count of occurrence in resources by entity class type

Strict predicates/properties

Base relation:
<resource> ?property <link>

Predicates/properties

Base relation:

?subject a <entity_class>
?subject ?property <link>