parquet-ruby
This project is a Ruby library wrapping the parquet-rs rust crate.
Usage
This library provides high-level bindings to parquet-rs with two primary APIs for reading Parquet files: row-wise and column-wise iteration. The column-wise API generally offers better performance, especially when working with subset of columns.
Row-wise Iteration
The each_row method provides sequential access to individual rows:
require "parquet"
# Basic usage with default hash output
Parquet.each_row("data.parquet") do |row|
puts row.inspect # {"id"=>1, "name"=>"name_1"}
end
# Array output for more efficient memory usage
Parquet.each_row("data.parquet", result_type: :array) do |row|
puts row.inspect # [1, "name_1"]
end
# Select specific columns to reduce I/O
Parquet.each_row("data.parquet", columns: ["id", "name"]) do |row|
puts row.inspect
end
# Reading from IO objects
File.open("data.parquet", "rb") do |file|
Parquet.each_row(file) do |row|
puts row.inspect
end
endColumn-wise Iteration
The each_column method reads data in column-oriented batches, which is typically more efficient for analytical queries:
require "parquet"
# Process columns in batches of 1024 rows
Parquet.each_column("data.parquet", batch_size: 1024) do |batch|
# With result_type: :hash (default)
puts batch.inspect
# {
# "id" => [1, 2, ..., 1024],
# "name" => ["name_1", "name_2", ..., "name_1024"]
# }
end
# Array output with specific columns
Parquet.each_column("data.parquet",
columns: ["id", "name"],
result_type: :array,
batch_size: 1024) do |batch|
puts batch.inspect
# [
# [1, 2, ..., 1024], # id column
# ["name_1", "name_2", ...] # name column
# ]
endArguments
Both methods accept these common arguments:
-
input: Path string or IO-like object containing Parquet data -
result_type: Output format (:hashor:array, defaults to:hash) -
columns: Optional array of column names to read (improves performance)
Additional arguments for each_column:
-
batch_size: Number of rows per batch (defaults to implementation-defined value)
When no block is given, both methods return an Enumerator.