Indy: A Log Archaeology Tool

Synopsis

Log files are often searched for particular strings but are not often treated as data structures. Indy attempts to deliver log content via more powerful features by allowing the ability to collect segments of a log from particular time; find a particular event; or monitor/reflect on a log to see if a particular event occurred (or not occurred).

Installation

To install Indy use the following command:

$ gem install indy

Usage

Basic Example (default log pattern)

log = Indy.search(:file => './data.log').all.first.message
# => "Entering APPLICATION."

Custom Example

# Given this log file:
#
# 2015/09/08 INFO #index Indexing application.
# 2015/09/11 ERROR #search Searching application.
# ...

log_contents = File.open('./example.log', 'r').read
pattern = /^(\d{4}\/\d{2}\/\d{2})\s+(INFO|DEBUG|WARN|ERROR)\s+(#\S+)\s+(.+)$/
fields = [:time, :severity, :method, :message]
custom_time_format = '%Y/%m/%d'

indy = Indy.search(  :source => log_contents,
                     :entry_regexp => pattern,
                     :entry_fields => fields,
                     :time_format => custom_time_format
           )
indy.after(:time => '2015/09/10').like(:severity => 'ERROR') do |entry|
    puts "#{entry.time} (#{entry.method}) #{entry.message}"
end

Specify your Source

As a process or command

Indy.search( {:cmd => 'ssh user@system "bash --login -c \"cat /var/log/standard.log\" "'} ).for(:severity => 'INFO')

As a file

Indy.search(file_object).for(:application => 'MyApp')
Indy.search(:file => file_object).for(:application => 'MyApp')
Indy.search(:file => '/log/data.log').for(:application => 'MyApp')

As a string

log_string = %{
    2000-09-07 14:07:41 INFO  MyApp - Entering application.
    2000-09-07 14:07:41 INFO  MyApp - Exiting application. }

Indy.search(log_string).for(:message => 'Entering application')

Log Pattern

Default Log Format

The default log format follows this form:

YYYY-MM-DD HH:MM:SS SEVERITY APPLICATION_NAME - MESSAGE

which uses this regular expression:

/^(\d{4}.\d{2}.\d{2}\s+\d{2}.\d{2}.\d{2})\s+(TRACE|DEBUG|INFO|WARN|ERROR|FATAL)\s+(\w+)\s+-\s+(.+)$/

and specifies these fields:

[:time, :severity, :application, :message]

allowing searches like so:

Indy.search(log_file).for(:severity => 'INFO')
Indy.search(log_file).for(:application => 'MyApp', :severity => 'DEBUG')

Custom Log Format

Brew your own log format! To do so, specify an :entry_regexp pattern that captures each field you want to reference. Also, specify an :entry_fields array of symbols that name the fields captured by your regexp. If your date/time format differs from the default (%Y-%m-%d %H:%M:%S), you will need to specify a :time_format parameter.

# If your log format is:
# YYYY-MM-DD HH:MM:SS SEVERITY APPLICATION#METHOD - MESSAGE

# Build a regexp pattern
pattern = /^(\d{4}.\d{2}.\d{2}\s+\d{2}.\d{2}.\d{2})\s*(INFO|DEBUG|WARN|ERROR)\s*([^#]+)#([^\s]+)\s*-\s*(.+)$/
# List the log fields
fields = [:time,:severity,:application,:method,:message]

# Use Indy#with to define your format
Indy.search(source).with(:entry_regexp => pattern, :entry_fields => fields)

Predefined Log Format

Several log formats have been predefined for ease of configuration. See indy/formats.rb

# Indy::COMMON_LOG_FORMAT
# Indy::COMBINED_LOG_FORMAT
# Indy::LOG4R_DEFAULT_FORMAT
#
# Example (Log4r)
#  INFO mylog: This is a message with level INFO
Indy.new(:source => log_file).with(Indy::LOG4R_DEFAULT_FORMAT).for(:application => 'mylog')

Multiline log entries

By default, Indy assumes that log lines are separated by new lines. Any lines that don't match the active pattern are ignored. To enable multiline log entries you MUST do these things:

Use Indy.new() and include the :multiline => true parameter
Use a log entry regexp that does not use $ and/or \n to define the end of the entry.
Add a capture group that surrounds one full log entry.

Multiline Regexp tips

Use non-greedy matching when needed: .*? instead of .*
Assuming your log entries do not include a unique line ending, you can use a zero-width positive lookahead assertion to verify that each line is followed by the start of a valid log entry, or the end of the string. e.g.: if 'foo' starts each entry, use this assertion (?=^foo|\z) at the end of your :entry_regexp

Check out Regexp Extensions

Example:

# Given this log containing three entries:
#
# INFO MyApp - Multiline message begins here...
# and ends here
# DEBUG MyOtherApp - Single line message.
# WARN MyOtherApp - A third entry.

severity_string = 'DEBUG|INFO|WARN|ERROR|FATAL'

multiline_regexp = /^((#{severity_string}) (\w+) - (.*?)(?=^#{severity_string}|\z))/

# For reference, a single line regexp would be:
#                  /^(#{severity_string}) (\w+) - (.*)$/


Indy.new( :multiline => true,
          :entry_regexp => multiline_regexp,
          :entry_fields => [:severity, :application, :message],
          :source => MY_LOG
        )

Explicit Time Format

If not specified, Indy tries to guess your time format (courtesy of DateTime#parse). If you supply an explicit time format, it will use DateTime#strptime. If that fails, it will then guess with DateTime#parse.

This is required when log data uses a non-standard date format, e.g.: U.S. format 12-31-2000, and must be used in conjunction with :entry_regexp and :entry_fields parameters.

# 12-31-2011 Application starting
Indy.new(   :time_format => '%m-%d-%Y',
            :source => LOG_FILE,
            :entry_regexp => /\d\d-\d\d-\d\d\d\d .*?/,
            :entry_fields => [:time, :message]
        ).all

Format directives are documented in DateTime#strftime.

Match Criteria

Exact Match

Indy.search(source).for(:message => 'Entering Application')
Indy.search(source).for(:severity => 'INFO')

Exact Match with multiple parameters

Indy.search(source).for(:message => 'Entering Application', :application => 'MyApp')
Indy.search(source).for(:severity => 'INFO', :application => 'MyApp')

Partial Match

Indy.search(source).like(:message => 'Memory')

Partial Match with multiple parameters

Indy.search(source).like(:severity => '(?:INFO|DEBUG)', :message => 'Memory')

Log Scopes

Multiple scope methods can be called on an instance. Use #reset_scope to remove scope constraints on the instance.

Time Scope

# After Dec 1
Indy.search(source).after(:time => '2010-12-01 23:59:59').all

# 20 minutes Around New Year's eve
Indy.search(source).around(:time => '2011-01-01 00:00:00', :span => 20).all

# After Jan 1 but Before Feb 1
@log = Indy.search(source)
@log.after(:time => '2011-01-01 00:00:00').before(:time => '2011-02-01 00:00:00')
@log.all

# Within Jan 1 and Feb 1 (same time scope as above)
Indy.search(source).within(:start_time => '2011-01-01 00:00:00', :end_time =>'2011-02-01 00:00:00').all

# After Jan 1
@log = Indy.search(source)
@log.after(:time => '2011-01-01 00:00:00')
@log.all)
# Reset the time scope to include entries before Jan 1
@log.reset_scope
# Before Feb 1
@log.before(:time => '2011-02-01 00:00:00')
@log.all

Process the Results

An Array is returned by #for and #like, containing a Struct::Entry for each log entry. The full entry is available with entry.raw_entry.

entries = Indy.search(source).for(:message => 'Entering Application')
entries.first.members
# => ["time", "severity", "application", "message", "raw_entry"]

Indy.search(source).for(:message => 'Entering Application').each do |entry|
  puts "[#{entry.time}] #{entry.message}: #{entry.application}"
end

Contributing

To get your improvements included, please fork and submit a pull request. Bugs and/or failing tests are very appreciated.

Any suggestions about log formats to support, and/or intelligent defaults would be great.

To create a report in /coverage, run gem install simplecov COVERAGE=true rake test

Compatibility

Indy was developed on MacOS, and should support MS Windows. Travis builds run the following ruby flavors:

2.4.3
2.3.6
2.2.9
1.9.3
jruby

LICENSE

(The MIT License)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

indy

Development

Runtime