Faraday Highly Available Retries

An extension for the Faraday::Retry middleware allowing retries to failover across multiple resolved endpoints.

At the time a request is made, the list of hosts pulled from the connection and request is resolved to an array of IP addresses. This list of IP addresses is then shuffled, and if a request fails to connect to the first IP address, it will try the next one. And so on and so forth, until the retries are exhausted. If all of the IPs fail, then we cycle back to the first one and try again.

The reason this is impactful and should be used in conjunction with the retry middleware is that the retry middleware will leave DNS resolution to the OS, which has the potential of two pitfalls:

1. Caching results and not try resolving to a different IP address.
2. Leaving resolution to a roll of the dice allowing the same IP to be tried multiple times.

The issue with the first pitfall is rather obvious, but the second is a bit more subtle. If you have a DNS entry that resolves to 2 IP addresses [A, B] and 1 attempt, the OS resolution will result in one of the following list of attempts:

1. [A, A]
2. [A, B]
3. [B. B]
4. [B, A]

This gives you a 50% chance of hitting the same IP twice, and a 50% chance of hitting the other IP twice, meaning the retry has no effect. By using this gem, you can ensure that the list of IPs is resolved ahead of time and cycled through to ensure a different IP is tried on each attempt.

Add this line to your application's Gemfile:

gem 'faraday-highly_available_retries'

And then execute:

bundle install

Or install it yourself as:

gem install faraday-highly_available_retries

This extension can be used on its own but is ultimately meant to be used in tandem with the Faraday::Retry middleware. You should make sure that this middleware is added in the request stack after the retry middleware to ensure it can alter the environment on retries.

When you configure the connection, you can do it a few different ways for the failover to work.

This simplest way is to make a generic Faraday connection, and pass the full URI in the request. When this is done, the failover middleware will parse the URI and use the host and port to determine the failover endpoints using DNS resolution.

require 'faraday/retry/failover'

conn = Faraday.new do |f|
  f.request :retry, { max: 2 } # This makes sure we retry 2, resulting in 3 attempts total before failing finally
  f.request :highly_available_retries
end

conn.get('https://api.invoca.net/api/2020-10-01/transaction/33.json')

This is the same as the previous example, but the base URL is set in the connection. This is useful when the base URL is always the same, and you have a DNS entry that will resolve to multiple IPs.

require 'faraday/retry/failover'

conn = Faraday.new('https://api.invoca.net') do |f|
  f.request :retry, { max: 2 }
  f.request :highly_available_retries
end

conn.get('/api/2020-10-01/transaction/33.json')

This is the most useful when you already have multiple IPs or separate hostnames that you want to use for failover. The hosts list provided can contain hostnames and IPs both with and without ports.

conn = Faraday.new do |f|
  f.request :retry, { max: 2 }
  f.request :highly_available_retries, { hosts: ['api.invoca.net', 'api.invoca.com'] }
end

conn.get('/api/2020-10-01/transaction/33.json')

After checking out the repo, run bin/setup to install dependencies.

Then, run bin/test to run the tests.

To install this gem onto your local machine, run rake build.

To release a new version, make a commit with a message such as "Bumped to 0.0.2" and then run rake release. See how it works here.

Bug reports and pull requests are welcome on GitHub.

The gem is available as open source under the terms of the MIT License.