Grape::Idempotency 🍇🔁

Gem for supporting idempotency in your Grape APIs.

Implementation based on the Active Internet-Draft draft-ietf-httpapi-idempotency-key-header-04

Topics covered in this README:

• Installation
• Basic Usage
• How it works
  • Making idempotency key header mandatory
  • Redis Storage Connectivity Issue
• Configuration
• Changelog
• Contributing

Ruby 2.6 or newer is required. Grape 1 or newer is required.

grape-idempotency is available as a gem, to install it run:

gem install grape-idempotency

Configure the Grape::Idempotency class with a Redis instance.

require 'redis'
require 'grape-idempotency'

redis = Redis.new(host: 'localhost', port: 6379)
Grape::Idempotency.configure do |c|
  c.storage = redis
end

Now you can wrap your code inside the idempotent method:

require 'grape'
require 'grape-idempotency'

class API < Grape::API
    post '/payments' do
      idempotent do
        status 201
        Payment.create!({
          amount: params[:amount]
        })
      end
    end
  end
end

That's all! 🚀

Once you've set up the gem and enclosed your endpoint code within the idempotent method, your endpoint will exhibit idempotent behavior, but this will only occur if the consumer of the endpoint includes an idempotency key in their request. (If you want to make the idempotency key header mandatory for your endpoint, check How to make idempotency key header mandatory section)

This key allows your consumer to make the same request again in case of a connection error, without the risk of creating a duplicate object or executing the update twice.

To execute an idempotent request, simply request your user to include an extra Idempotency-Key: <key> header as part of his request.

This gem operates by storing the initial request's status code and response body, regardless of whether the request succeeded or failed, using a specific idempotency key. Subsequent requests with the same key will consistently yield the same result, even if there were 500 errors.

Keys are automatically removed from the system if they are at least 24 hours old, and a new request is generated when a key is reused after the original has been removed. The idempotency layer compares incoming parameters to those of the original request and returns a 422 - Unprocessable Entity status code if they don't match, preventing accidental misuse. If a request is received while another one with the same idempotency key is still being processed the idempotency layer returns a 409 - Conflict status

Results are only saved if an API endpoint begins its execution. If incoming parameters fail validation or if the request conflicts with another one executing concurrently, no idempotent result is stored because no API endpoint has initiated execution. In such cases, retrying these requests is safe.

Additionally, this gem automatically appends the Original-Request header and the Idempotency-Key header to your API's response, enabling you to trace back to the initial request that generated that specific response.

For some endpoints, you want to enforce your consumers to provide idempotency key. So, when wrapping the code inside the idempotent method, you can mark it as required:

require 'grape'
require 'grape-idempotency'

class API < Grape::API
    post '/payments' do
      idempotent(required: true) do
        status 201
        Payment.create!({
          amount: params[:amount]
        })
      end
    end
  end
end

If the Idempotency-Key request header is missing for a idempotent operation requiring this header, the gem will reply with an HTTP 400 status code with the following body:

{
  "title": "Idempotency-Key is missing",
  "detail": "This operation is idempotent and it requires correct usage of Idempotency Key.",
}

If you want to change the error message returned in this scenario, check How to configure idempotency key missing error message section.

By default, Redis exceptions are not handled by the grape-idempotency gem.

Therefore, if an exception arises while attempting to read, write or delete data from the Redis storage, the gem will re-raise the identical exception to your application. Thus, you will be responsible for handling it within your own code, such as:

require 'grape'
require 'grape-idempotency'

class API < Grape::API
    post '/payments' do
      begin
        idempotent do
          status 201
          Payment.create!({
            amount: params[:amount]
          })
        end
      rescue Redis::BaseError => e
        error!("Redis error! Idempotency is very important here and we cannot continue.", 500)
      end
    end
  end
end

If you want to avoid this functionality, and you want the gem handles the potential Redis exceptions, you have the option to configure the gem for handling these Redis exceptions. Please refer to the manage_redis_exceptions configuration property.

🚨 WARNING: If a Redis exception appears AFTER performing the wrapped code, nothing will be re-raised. The process will continue working and the response will be returned to the consumer of your API. However, a 409 Conflict response can be returned to your consumer if it retried the same call with the same idempotency key. This is because the gem was not able to associate the response of the original request to the original idempotency key because those connectivity issues.

In addition to the storage aspect, you have the option to supply additional configuration details to tailor the gem to the specific requirements of your project.

As we have mentioned in the How it works section, keys are automatically removed from the system if they are at least 24 hours old. However, a 24-hour timeframe may not be suitable for your project. To accommodate your specific needs, you can adjust this duration by using the expires_in parameter for configuration:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.expires_in = 1800
end

So with the cofiguration above, the keys will expire in 30 minutes.

As outlined in the How it works section, in order to perform an idempotent request, you need to instruct your users to include an additional Idempotency-Key: <key> header with their request. However, if this header format doesn't align with your project requirements, you have the flexibility to configure the specific header that the gem will examine to determine idempotent behavior:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.idempotency_key_header = "x-custom-idempotency-key"
end

Given the previous configuration, the gem will examine the X-Custom-Idempotency-Key: <key> for determine the idempotent behavior.

By default, this gem stores a random hex value as identifier when storing the original request and returns it in all the subsequent requests that use the same idempotency-key as Original-Request header in the response.

This value can be also provided by your consumer using the X-Request-Id: <request-id> header when performing the request to your API.

However, if you prefer to use a different format for getting the request identifier, you can configure the header to check using the request_id_header parameter:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.request_id_header = "x-trace-id"
end

In the case above, you request your consumers to use the X-Trace-Id: <trace-id> header when requesting your API.

By default, the logger used by the gem is configured like Logger.new(STDOUT) and INFO level. As this gem does not log any message with INFO level, only ERROR messages will be logged.

If you want to provide your own logger, you want to change the level to DEBUG or you want to provide your own prefix, you can configure the gem like:

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.logger = Infrastructure::MyLogger.new
  c.logger_level = :debug
  c.logger_prefix = '[my-own-prefix]'
end

An example of the logged information when changing the level of the log to DEBUG and customizing the logger_prefix:

I, [2023-11-23T22:41:39.148163 #1]  DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
I, [2023-11-23T22:41:39.148176 #1]  DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
I, [2023-11-23T22:41:39.148192 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key" => "fd77c9d6-b7da-4966-aac8-40ee258f24aa"
I, [2023-11-23T22:41:39.148210 #1]  DEBUG -- : [my-own-prefix] Previous request information has NOT been found for the provided idempotency key.
I, [2023-11-23T22:41:39.148248 #1]  DEBUG -- : [my-own-prefix] Request stored as processing.
I, [2023-11-23T22:41:39.148261 #1]  DEBUG -- : [my-own-prefix] Performing the provided block.
I, [2023-11-23T22:41:39.148268 #1]  DEBUG -- : [my-own-prefix] Block has been performed.
I, [2023-11-23T22:41:39.148287 #1]  DEBUG -- : [my-own-prefix] Storing response.
I, [2023-11-23T22:41:39.148317 #1]  DEBUG -- : [my-own-prefix] Response stored.
I, [2023-11-23T22:41:39.148473 #1]  DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
I, [2023-11-23T22:41:39.148486 #1]  DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
I, [2023-11-23T22:41:39.148502 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key" => "fd77c9d6-b7da-4966-aac8-40ee258f24aa"
I, [2023-11-23T22:41:39.148523 #1]  DEBUG -- : [my-own-prefix] Request has been found for the provided idempotency key => {"path"=>"/payments", "params"=>{"locale"=>"undefined", "{\"amount\":10000}"=>nil}, "status"=>500, "original_request"=>"wadus", "response"=>"{\"error\":\"Internal Server Error\"}"}
I, [2023-11-23T22:41:39.148537 #1]  DEBUG -- : [my-own-prefix] Returning the response from the original request.

By default, the grape-idempotency gem is configured to re-raise Redis exceptions.

If you want to delegate the Redis exception management into the gem, you can configure it using the manage_redis_exceptions configuration property.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.manage_redis_exceptions = true
end

However, this approach carries a certain level of risk. In the case that Redis experiences an outage, the idempotent functionality will be lost, the endpoint will behave as no idempotent, and this issue may go unnoticed.

When providing a Idempotency-Key: <key> header, this gem compares incoming parameters to those of the original request (if exists) and returns a 409 - Conflict status code if they don't match, preventing accidental misuse. The response body returned by the gem looks like:

{

  "title": "Idempotency-Key is already used",
  "detail": "This operation is idempotent and it requires correct usage of Idempotency Key. Idempotency Key MUST not be reused across different payloads of this operation."
}

You have the option to specify the desired response body to be returned to your users when this error occurs. This allows you to align the error format with the one used in your application.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.conflict_error_response = {
    "type": "about:blank",
    "status": 409,
    "title": "Conflict",
    "detail": "You are using the same idempotent key for two different requests"
}
end

In the configuration above, the error is following the RFC-7807 format.

When a request with a Idempotency-Key: <key> header is performed while a previous one still on going with the same idempotency value, this gem returns a 409 - Conflict status. The response body returned by the gem looks like:

{

  "title": "A request is outstanding for this Idempotency-Key",
  "detail": "A request with the same idempotent key for the same operation is being processed or is outstanding."
}

You have the option to specify the desired response body to be returned to your users when this error occurs. This allows you to align the error format with the one used in your application.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.processing_response = {
    "type": "about:blank",
    "status": 409,
    "title": "A request is still being processed",
    "detail": "A request with the same idempotent key is being procesed"
}
end

In the configuration above, the error is following the RFC-7807 format.

If the Idempotency-Key request header is missing for a idempotent operation requiring this header, the gem will reply with an HTTP 400 status code with the following body:

{
  "title": "Idempotency-Key is missing",
  "detail": "This operation is idempotent and it requires correct usage of Idempotency Key.",
}

You have the option to specify the desired response body to be returned to your users when this error occurs. This allows you to align the error format with the one used in your application.

Grape::Idempotency.configure do |c|
  c.storage = @storage
  c.mandatory_header_response = {
    "type": "about:blank",
    "status": 400,
    "title": "Idempotency-Key is missing",
    "detail": "Please, provide a valid idempotent key in the headers for performing this operation"
}
end

In the configuration above, the error is following the RFC-7807 format.

If you're interested in seeing the changes and bug fixes between each version of grape-idempotency, read the Changelog.

We welcome and appreciate contributions from the open-source community. Before you get started, please take a moment to review the guidelines below.

1. Fork the repository.
2. Clone the repository to your local machine.
3. Create a new branch for your contribution.
4. Make your changes and ensure they meet project standards.
5. Commit your changes with clear messages.
6. Push your branch to your GitHub repository.
7. Open a pull request in our repository.
8. Participate in code review and address feedback.
9. Once approved, your changes will be merged.

This project is dockerized, so be sure you have docker installed in your machine.

Once you clone the repository, you can use the Make commands to build the project.

make build

You can pass the tests running:

make test

Open issues on the GitHub issue tracker with clear information.

• Juan Carlos García - Creator - https://github.com/jcagarcia
• Carlos Cabanero - Contributor - https://github.com/Flip120