SalesforceChunker

The salesforce_chunker gem is a ruby library for interacting with the Salesforce Bulk API. It was primarily designed as an extractor to handle queries using batching and Primary Key Chunking.

Currently, only querying is built into SalesforceChunker::Client, but non-query jobs can be created with SalesforceChunker::Job.

Add this line to your application's Gemfile:

gem 'salesforce_chunker'

And then execute:

$ bundle

Or install it yourself as:

$ gem install salesforce_chunker

client = SalesforceChunker::Client.new(
  username: "username", 
  password: "password", 
  security_token: "security_token",
)

names = client.query(query: "Select Name From User", object: "User").map { |result| result["Name"] }

client = SalesforceChunker::Client.new(
  username:             "username",
  password:             "password",
  security_token:       "security_token",
  domain:               "login",
  salesforce_version:   "42.0",
)

options = {
  query:            "Select Name from Account",
  object:           "Account",
  batch_size:       100000,
  retry_seconds:    10,
  timeout_seconds:  3600,
  logger:           nil,
  log_output:       STDOUT,
  job_type:         "primary_key_chunking",
  include_deleted:  true,
}

client.query(options) do |result|
  process(result)
end

query can either be called with a block, or will return an enumerator:

names = client.query(query, object, options).map { |result| result["Name"] }

One of the advantages of the Salesforce Bulk API over the other Salesforce APIs is the ability for Salesforce to process a number of requests (either queries or uploads) in parallel on their servers. The request chunks are referred to as batches.

In a single batch query, one SOQL statement is executed as a single batch. This works best if the total number of records to return is fewer than around 100,000 depending on memory usage and the number of fields being returned.

In Primary Key Chunking, the internal Salesforce PK chunking flag is used. Salesforce will create a number of batches automatically based on an internal Id index. See https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm

This approach is called "Manual" Chunking because it is our own implementation of PK Chunking in this gem. The gem downloads a CSV ordered list of all Ids it needs to download, and then uses this list to generate breakpoints that it uses to create batches.

Advantages of Manual Chunking:

• Manual Chunking takes into account the where clause in the SOQL statement. For example, if you are filtering a small number of a large object count, say 250k out of 20M Objects, then Manual Chunking will split this into 3 batches of max 100k while PK chunking will split this into 200 batches, which will use up batches and API requests against your account and take a longer amount of time.
• Any object can use Manual Chunking (according to Salesforce, PK chunking is supported for the following objects: Account, Asset, Campaign, CampaignMember, Case, CaseHistory, Contact, Event, EventRelation, Lead, LoginHistory, Opportunity, Task, User, and custom objects.)

Advantages of Primary Key Chunking:

• Primary Key Chunking appears to be slightly faster, if using a PK Chunking eligible object and no where clause.
• Primary Key Chunking may be less buggy because many more people depend on the Salesforce API than this gem.

Using SalesforceChunker::Job, you have more direct access to the Salesforce Bulk API functions, such as create_batch, get_batch_statuses, and retrieve_batch_results. This can be used to perform custom tasks, such as upserts or multiple batch queries.

This should be used in coordination with SalesforceChunker::Connection, which has the same initialization process as SalesforceChunker::Client.

connection = SalesforceChunker::Connection.new(
  username: "username",
  password: "password",
  security_token: "security_token",
)

job = SalesforceChunker::Job.new(
  connection: connection,
  object: "Account",
  operation: "query",
  log_output: STDOUT,
)

job.create_batch("Select Id From Account Order By Id Desc Limit 1")
job.create_batch("Select Id From Account Order By Id Asc Limit 1")
job.close

job.instance_variable_set(:@batches_count, 2)
ids = job.download_results.to_a

Also, SalesforceChunker::SingleBatchJob can be used to create a Job with only a single batch. This automatically handles the batch creation, closing, and setting @batches_count.

job = SalesforceChunker::SingleBatchJob.new(
  connection: connection,
  object: "Account",
  operation: "upsert",
  payload: [{ "Name" => "Random Account", "IdField__c" => "123456" }],
  external_id: "IdField__c",
  log_output: STDOUT,
)

loop do
  batch = job.get_batch_statuses.first
  if batch["state"] == "Completed"
    break
  elsif batch["state"] == "Failed"
    raise "batch failed"
  end
  sleep 5
end

After checking out the repo,

• run bin/setup to install dependencies.
• run rake test to run the tests.
• run bin/console for an interactive prompt that will allow you to experiment.

Bug reports and pull requests are welcome on GitHub at https://github.com/Shopify/salesforce_chunker.

The gem is available as open source under the terms of the MIT License.