TypedData
TypedData is a library that converts hash objects managed by an Avro schema so that the objects can be loaded into BigQuery.
Installation
Add this line to your application's Gemfile:
gem 'typed_data'And then execute:
$ bundle install
Or install it yourself as:
$ gem install typed_data
Usage
Use as Ruby library
require "typed_data"
schema = {
"name" => "Record",
"type" => "record",
"fields" => [
{
"name" => "int_field",
"type" => "int",
},
{
"name" => "int_or_string_field",
"type" => ["int", "string"],
},
{
"name" => "array_field",
"type" => {
"type" => "array",
"items" => "int",
},
},
{
"name" => "union_type_array_field",
"type" => {
"type" => "array",
"items" => ["int", "string"],
},
},
{
"name" => "nested_map_field",
"type" => {
"type" => "map",
"values" => {
"type" => "map",
"values" => ["int", "string"],
},
},
},
],
}
converter = TypedData::Converter.new(schema)
converter.convert({
"int_field" => 1,
"int_or_string_field" => "string",
"array_field" => [1, 2],
"union_type_array_field" => [1, "2"],
"nested_map_field" => {
"nested_map" => {
"key1" => 1,
"key2" => "2",
},
},
})
#=> {"int_field"=>1,
# "int_or_string_field"=>{"string_value"=>"string"},
# "array_field"=>[1, 2],
# "union_type_array_field"=>[{"int_value"=>1}, {"string_value"=>"2"}],
# "nested_map_field"=>
# [{"key"=>"nested_map",
# "value"=>
# [{"key"=>"key1", "value"=>{"int_value"=>1}},
# {"key"=>"key2", "value"=>{"string_value"=>"2"}}]}]}You can specify the formatter for union type keys. The default formatter is :bigquery, which is used for BigQuery tables created by loading Avro data for the first time.
The other formatter is :avro, the formatter for the Avro JSON encoding, which is used in tables managed by Google BigQuery Sink Connector:
converter = TypedData::Converter.new(schema, key_formatter: :avro)
converter.convert({
"int_field" => 1,
"int_or_string_field" => "string",
"array_field" => [1, 2],
"union_type_array_field" => [1, "2"],
"nested_map_field" => {
"nested_map" => {
"key1" => 1,
"key2" => "2",
},
},
})
#=> {"int_field"=>1,
# "int_or_string_field"=>{"string"=>"string"},
# "array_field"=>[1, 2],
# "union_type_array_field"=>[{"int"=>1}, {"string"=>"2"}],
# "nested_map_field"=>
# [{"key"=>"nested_map",
# "value"=>
# [{"key"=>"key1", "value"=>{"int"=>1}},
# {"key"=>"key2", "value"=>{"string"=>"2"}}]}]}TypedData::Restorer enables you to restore the converted data:
restorer = TypedData::Restorer.new(schema)
restorer.restore({
"int_field" => 1,
"int_or_string_field" => { "string_value" => "string" },
"array_field" => [1, 2],
"union_type_array_field" => [
{ "int_value" => 1 },
{ "string_value" => "2" },
],
"nested_map_field" => [
{
"key" => "nested_map",
"value" =>[
{
"key" => "key1",
"value" => { "int_value" => 1 }
},
{
"key" => "key2",
"value" => { "string_value" => "2"}
},
],
},
],
})
#=> {"int_field"=>1,
# "int_or_string_field"=>"string",
# "array_field"=>[1, 2],
# "union_type_array_field"=>[1, "2"],
# "nested_map_field"=>{"nested_map"=>{"key1"=>1, "key2"=>"2"}}}Use as CLI
$ typed-data help
Commands:
typed-data convert [file] --schema=SCHEMA # Convert data in an encoding similar to Avro JSON encoding
typed-data help [COMMAND] # Describe available commands or one specific command
typed-data restore [file] --schema=SCHEMA # Restore converted data
$ typed-data help convert
Usage:
typed-data convert [file] --schema=SCHEMA
Options:
--schema=SCHEMA # Path to Avro schema file
[--key-format=FORMAT] # Format for union type key
# Default: bigquery
# Possible values: bigquery, avro
Description:
This command converts data in an encoding similar to Avro JSON encoding. You can specify the file in
JSON format or JSON Lines format. If the file option is ommited, the command read data from stdin.
$ typed-data help restore
Usage:
typed-data restore [file] --schema=SCHEMA
Options:
--schema=SCHEMA # Path to Avro schema file
[--key-format=FORMAT] # Format for union type key
# Default: bigquery
# Possible values: bigquery, avro
Description:
This command restores converted data. You can specify the file in JSON format or JSON Lines format. If
the file option is ommited, the command read data from stdin.
For example, you can restore the data loaded into a BigQuery table like below:
$ bq query --format json 'SELECT * FROM <table>' | typed-data restore --schema /path/to/avsc
Development
After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/abicky/typed_data.
License
The gem is available as open source under the terms of the MIT License.