TokenEstimator

TokenEstimator is a Rails gem that provides functionality to count tokens in various file formats and text inputs using different tokenizers.

Add this line to your application's Gemfile:

gem "token_counter"

And then execute:

bundle install

Count tokens from a given text.

    require "token_estimator"

    tokenizer_name = "gpt2"
    estimator = TokenEstimator::Estimator.new(tokenizer_name)

    text = "Your sample text here."
    token_estimation = estimator.count_tokens_from_text(text)

    puts "Token estimation: #{token_estimation}"

Count tokens from a file. The file type is determined by the file extension.

    require "token_estimator"

    file_path = "spec/fixtures/files/lorem.pdf"
    tokenizer_name = "gpt2"
    estimator = TokenEstimator::Estimator.new(tokenizer_name)

    token_estimation = estimator.count_tokens_from_file(file_path)

    puts "Token estimation: #{token_estimation}"

Counts tokens from an Excel (.xlsx) file.

Counts tokens from a CSV file.

Counts tokens from a PDF file.

Counts tokens from a plain text (.txt) file.

Counts tokens from a Markdown (.md) file.

Counts tokens from a JSON file.

Counts tokens from an HTML file.

Counts tokens from a JSON object.

Counts tokens from an HTML string.

Return the supported file types.

Here is a checklist of the formats we currently support for token counting and those we plan to support in the future:

• PDF
• Markdown (.md)
• CSV
• Excel (XLSX)
• JSON
• Plain Text
• HTML
• DOCX (Word Documents)
• XML
• RTF (Rich Text Format)
• PNG
• JPG

If you try to count tokens from an unsupported file type, the gem will raise an UnsupportedFileTypeError

begin
  token_count = estimator.count_tokens_from_file("path/to/your/file.unsupported")
rescue TokenEstimator::UnsupportedFileTypeError => e
  puts e.message
end

Contribution directions go here. You can fork the repository, create a new branch, and submit a pull request for review. Please make sure to write tests for your contributions and follow the coding standards set in the project.

The gem is available as open source under the terms of the MIT License.