Author

Martin Velez

Copyright

Copyright © 2011 Martin Velez

License

Distributed under the same terms as Ruby

ruby_ngrams is an extension of Ruby’s core String class. It provides a String object with the capability to produce n-grams.

From Wikipedia,

"In the fields of computational linguistics and probability, an n-gram is a 
contiguous sequence of n items from a given sequence of text or speech. The 
items in question can be phonemes, syllables, letters, words or base pairs 
according to the application. n-grams are collected from a text or speech corpus.

An n-gram of size 1 is referred to as a "unigram"; size 2 is a "bigram" 
(or, less commonly, a "digram"); size 3 is a "trigram"; size 4 is a "four-gram" 
and size 5 or more is simply called an "n-gram"."

Instead of creating another namespace, this task seemed simple enough to merit extending the String class. A string is a sequence of characters.

It can be words, binary code, sentences, paragraphs, etc. In short, anything that you can store in a Ruby String object can be parsed into n-grams of length n.

The main method being added to the String class is ngrams(). It produces an array of n-grams from a Ruby String object.

For example, let s be a Ruby String object. Then s.ngrams() returns array of n-grams of from s.

Tokenization of s is set to single characters by default. For example, if

s = "Hello World!",

then the tokens of s are

["H","e","l","l","o"," ","W","o","r","l","d","!"].

By specifying a regular expression, you can tokenize the string s in many different and useful ways.

If you set n = 4, then

s.ngrams = [["H", "e", "l", "l"], 
["e", "l", "l", "o"], 
["l", "l", "o", " "], 
["l", "o", " ", "W"], 
["o", " ", "W", "o"], 
[" ", "W", "o", "r"], 
["W", "o", "r", "l"], 
["o", "r", "l", "d"], 
["r", "l", "d", "!"]].

Each item in the s.ngrams array can joined but doesn’t need to be. If you want to join them, normally you can do so easily if it is text.

Be careful if you are trying to join n-grams with non-printable characters.

You can google “n-grams” to get more information about how n-grams are useful.

gem install ruby_ngrams

This is another tool I found but which did too much. I only wanted to produce n-grams from a string.

• raingrams

You can simply run the executable and provide input via STDIN.

ruby_ngrams

You can also provide input via one or more filenames

ruby_ngrams [FILES]

• Ruby 1.9.1 or greater

• ruby_cli to run the gem executable

• Test to determine limits of current approach which parses and stores n-grams in memory.

https://github.com/martinvelez/ruby_ngrams