data_resurrection¶ ↑

Bring your data, buried in decrepit formats, back to life! Convert data from old formats to modern ones. Currently supports DBF.

Just run:

$ gem install data_resurrection

If you have a PostgreSQL database and a DBF file called ‘decrepit.dbf’, you must call:

r = DataResurrection::Resuscitator.new(:dbf, :postgresql => :settings)
r.resurrect('decrepit.dbf', :target => 'new_beautiful_table',
  :from => 'WINDOWS-1252', :to => 'UTF-8')

and you will have a table called ‘new_beautiful_table’ in your PostgreSQL database.

The hash parameter to constructor should be the your database settings for ActiveRecord (i.e., those definitions within config/database.yml in a Rails application).

The method “resurrect” accepts “:target”, “:from” and “:to” as options. Target is the name of the table to be created in the pointed database, and :from and :to (both optional) forces a encoding conversion for all fields. Parameter :from may be a list in case of multiple encodings for the same table (yes, this kind of freaky thing really exists).

In some cases, field types in original tables are not compatible with ones in the target table. By example, in a real case, a DBF table has an integer field containing a 12-digit value, what caused an overflow when trying to copy the value to an integer field on a PostGreSQL table. This can be handled with an additional option to the method “resurrect”:

@data_resurrection.resurrect(@dbf_file_path, :target => 'nationality',
  :from => ['WINDOWS-1252', 'CP850'], :to => 'UTF-8',
  :field_types => {:nr => :string },
  :replacement => { '╟' => 'Ã' })

The option :field_types is a hash in which each key is the field name and the value is the field type in the target table.

The option :replacement allows defining arbitrary character replacement, after running all encodings. In the example, all existing weird “╟” in all strings will be replaced by “Ô.

If a field name equals to SQL reserved words or core Ruby methods like “class”, the field name is appended with an underscore in the new table.

Data resurrection heavily uses this amazing gem internally, but currently it doesn’t support encoding conversion, field type conversion and does not handle fields whose name equals to SQL reserved words and Ruby methods.

This project was built for my own use, and it works for me. Remember that database migrations are a very critical issue. Anything you do with this software is at your own risk.