Saxy
Memory-efficient XML parser. Finds object definitions in XML and translates them into Ruby hashes.
It uses SAX parser (provided by Nokogiri gem) under the hood, which means that it doesn't load the whole XML file into memory. It goes once through it and yields hashes along the way.
In result the memory footprint of the parser remains small and more or less constant irrespective of the size of the XML file, be it few KB or hundreds of GB.
Installation
Add this line to your application's Gemfile:
gem 'saxy'
And then execute:
$ bundle
Or install it yourself as:
$ gem install saxy
Requirements
As of 0.5.0
version saxy
requires ruby 1.9.3 or higher. Previous versions of the gem work with ruby 1.8 and 1.9.2 (see below), but they are not maintained anymore.
Ruby 1.8 support
See ruby-1.8
branch. Install with:
gem 'saxy', '~> 0.3.0'
Ruby 1.9.2 support
See ruby-1.9.2
branch. Install with:
gem 'saxy', '~> 0.4.0'
Changelog
See CHANGELOG.md
file.
Usage
You instantiate the parser by passing path to XML file or an IO-like object, object-identifying tag name and options hash (optionally) as its arguments.
parser = Saxy.parse(path_or_io, object_tag, options = {})
Then iterate over it using each
(or any of convenient methods provided by Enumerable
mix-in).
parser.each do |object|
...
end
Options
-
encoding
- Forces the parser to work in given encoding -
recovery
- Should this parser recover from structural errors? It will not stop processing file on structural errors if set totrue
. -
replace_entities
- Should this parser replace entities?&
will get converted to&
if set totrue
. -
error_handler
- If set to a callable, parser will call it with any error it encounters instead of raising exceptions.
Combination of error_handler
and recovery
options allows for continued processing when encountering recoverable errors (e.g. unescaped predefined entities).
error_handler = proc { |e| $stderr.puts "#{e.message} at line #{e.context.line}, column #{e.context.column}." }
Saxy.parse(path_or_io, object_tag, error_handler: error_handler, recovery: true) { ... }
Example
Assume the XML file (an imaginary product feed):
<?xml version='1.0' encoding='UTF-8'?>
<webstore>
<name>Amazon</name>
<products>
<product>
<name>Kindle - The world's best-selling e-reader.</name>
<images>
<thumbSize width="80" height="60">http://amazon.com/kindle_thumb.jpg</thumbSize>
</images>
</product>
<product>
<name>Kindle Touch - Simple-to-use touchscreen with built-in WIFI.</name>
<images>
<thumbSize width="120" height="90">http://amazon.com/kindle_touch_thumb.jpg</thumbSize>
</images>
</product>
</products>
</webstore>
The following will parse the XML, find product definitions (inside <product>
and </product>
tags), build Hash
es and yield them inside the block.
Usage with a file path:
Saxy.parse("filename.xml", "product").each do |product|
puts product["name"]
puts product["images"]["thumb_size"]["contents"]
puts "#{product["images"]["thumb_size"]["width"]}x#{product["images"]["thumb_size"]["height"]}"
end
# =>
"Kindle - The world's best-selling e-reader."
"http://amazon.com/kindle_thumb.jpg"
"80x60"
"Kindle Touch - Simple-to-use touchscreen with built-in WIFI."
"http://amazon.com/kindle_touch_thumb.jpg"
"120x90"
Usage with an IO-like object ARGF
or $stdin
:
# > cat filename.xml | ruby this_script.rb
Saxy.parse(ARGF, "product").each do |product|
puts product["name"]
end
# =>
"Kindle - The world's best-selling e-reader."
Saxy supports Enumerable, so you can use its goodies to your comfort without building intermediate arrays:
Saxy.parse("filename.xml", "product").map do |object|
# map yielded Hash to ActiveRecord instances, etc.
end
You can also grab an Enumerator for external use (e.g. lazy evaluation, etc.):
enumerator = Saxy.parse("filename.xml", "product").each
lazy = Saxy.parse("filename.xml", "product").lazy # Ruby 2.0
Multiple definitions of child objects are grouped in arrays:
webstore = Saxy.parse("filename.xml", "webstore").first
webstore["products"]["product"].size # => 2
Debugging
Invalid XML files happen a lot and error messages are not always extremely helpful. In case of a parsing error, some additional information can be retrieved from parser's context.
begin
Saxy.parse(...) { ... }
rescue e => Saxy::ParsingError
puts "#{e.message} at #{e.context.line} line and #{e.context.column}"
end
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Added some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request
License
See LICENSE.txt
file.