MojiBake¶ ↑

• github.com/dekellum/mojibake

• <img src=“https://travis-ci.org/dekellum/mojibake.png” />

Mojibake occurs in English most frequently due to misinterpreting and bad-transcoding between Windows-1252, ISO-8859-1, and UTF-8. This module provides a mojibake sequence to original character mapping table, and utility to recover mojibake’d text.

Testing has been with English but other Latin based languages, where Windows-1252 is in the wild, should also benefit.

Mojibake mappings generation requires the String encoding support in ruby 1.9 as provided by:

• ruby >= 1.9.2 (tested 1.9.2p290, 1.9.3p392, 2.0.0p247 Linux)

• jruby ~> 1.6.5 or ~> 1.7.5 (tested 1.6.8, 1.7.5 Linux)

• rubinius >= 2.0.0 (tested via Travis CI)

Note: jruby versions 1.7.0-1.7.4 have various encoding support regressions. Recovery of text with default settings is supported by using the pre-generated table.json when in ruby 1.8 mode, or for other jruby versions.

gem install mojibake

require 'mojibake'
mapper = MojiBake::Mapper.new
mapper.recover( '“quotedâ€�' ) #=> '“quoted”'

Or via cli:

mojibake -h

List the mojibake mapping table (output in UTF-8):

mojibake -t

Recover from a text file:

mojibake input.txt

Copyright © 2011-2015 David Kellum

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at:

www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.