Bind Log Analyzer

Simple analysis and SQL storage for Bind DNS server's logs. The gem includes a web interface to analyze the data collected from the analyzed logs.

Requirements

This gem was tested with:

ruby-2.1.x
rubygem (1.8.15)
bundler (1.0.21)
activerecord (3.2.2)

Installation

Just install the gem:

gem install bind_log_analyzer

The gem requires active_record but you probably need to install the right adapter. As example, if you'll use MySQL, install the mysql2 gem.

Configuration

Bind

To configure Bind add these lines to /etc/bind/named.conf.options (or whatever your s.o. and bind installation require)

logging{
    channel "querylog" {
            file "/var/log/bind/query.log";
            print-time yes;
    };

    category queries { querylog; };
};

Restart bind and make sure than the query.log file contains lines as this:

28-Mar-2012 16:48:19.694 client 192.168.10.38#58767: query: www.github.com IN A + (192.168.10.1)

or the regexp will fail :(

Database

To store the logs you can use every database supported by ActiveRecord. Just create a database and a user with the right privileges. You can provide the -s flag to BindLogAnalyzer to make it create the table. Otherwise create it by yourself. This is the MySQL CREATE TABLE syntax:

CREATE TABLE `logs` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `date` datetime NOT NULL,
  `client` varchar(255) NOT NULL,
  `query` varchar(255) NOT NULL,
  `q_type` varchar(255) NOT NULL,
  `server` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=latin1;

Usage

Use the provided --help to get various options available. This is the default help:

-h, --help                       Display this screen
-v, --verbose LEVEL              Enables verbose output. Use level 1 for WARN, 2 for INFO and 3 for DEBUG
-w, --webserver [HTTP_PORT]      Launches the Sinatra web server on specified port, or 4567 if omitted
-s, --setup                      Creates the needed tables in the database.
-f, --file FILE                  Indicates the log file to parse. It's mandatory if you don't specify the --webserver option.
-c, --config CONFIG              A yaml file containing the database configurations under the "database" entry
-a, --adapter ADAPTER            The database name to save the logs
-d, --database DATABASE          The database name to save the logs
-H, --host HOST                  The address (IP, hostname or path) of the database
-P, --port PORT                  The port of the database
-u, --user USER                  The username to be used to connect to the database
-p, --password PASSWORD          The password of the user

The first time you launch BindLogAnalyzer you can use the -s|--setup flag to make it create the table (using ActiveRecord::Migration). The database credentials can be provided using the needed flags or creating a YAML file containing all the informations under the database key. This is an example:

database:
  adapter: mysql2
  database: bindloganalyzer
  host: localhost
  port: 3306
  username: root
  password:

There are two usage of the gem:

Use --file FILE flag to pass to BindLogAnalyzer the file to analyze.
Use -w, --webserver [HTTP_PORT] to start the Sinatra webserver and watch the stats on the collected data.

Automatization

A good way to use this script is to let it be launched by logrotate so create the /etc/logrotate.d/bind file with this content:

/var/log/named/query.log {
    weekly
    missingok
    rotate 8
    compress
    delaycompress
    notifempty
    create 644 bind bind
    postrotate
        if [ -e /var/log/named/query.log.1 ]; then
            exec su - YOUR_USER -c '/usr/local/bin/update_bind_log_analyzer.sh /var/log/named/query.log.1'
        fi
    endscript
}

The script /usr/local/bin/update_bind_log_analyzer.sh can be wherever you prefer. Its typical content if you use RVM and a dedicated gemset for BindLogAnalyzer, can be:

#!/bin/bash

# *************************** #
#       EDIT THESE VARS       #
# *************************** #
BLA_RVM_GEMSET="1.9.3-p125@bind_log_analyzer"
BLA_USER="my_username"
BLA_DB_FILE="/etc/bind_log_analyzer/database.yml"

# *************************** #
# DO NOT EDIT BELOW THIS LINE #
# *************************** #
. /home/$BLA_USER/.rvm/scripts/rvm && source "/home/$BLA_USER/.rvm/scripts/rvm"
rvm use $BLA_RVM_GEMSET
bind_log_analyzer --config $BLA_DB_FILE --file $1

Performance

On a 1.6 Ghz Intel Core i5 with SSD SATA2 disk, using Ruby-1.9.3-p125 and MySQL 5.5.15, this is the performance:

~$ time bind_log_analyzer -f query.log -c database.yml
Analyzed 319758 lines and correctly stored 319758 logs
bind_log_analyzer -f query.log -c database.yml  322,44s user 22,90s system 76% cpu 7:33,17 total

which is equivalent to ±706 query/sec.

Development

First, create a database and add its credentials in the database.yml file.

Then create the logs table with the following query:

CREATE TABLE `logs` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `date` datetime NOT NULL,
  `client` varchar(255) NOT NULL,
  `query` varchar(255) NOT NULL,
  `q_type` varchar(255) NOT NULL,
  `server` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=latin1;

Run tests

bundle exec rspec

Changelog

###0.2.4

Support both old and new Bind log versions Add --bind option to the cli to bind Sinatra on specified IP Reverse the order of the last queries in the GUI Fix tests

0.2.3

Added the -u|--uniqueness flag to check if a record exists in the db before inserting a new one

0.2.2

Added the new BindLogAnalyzer::LogUtils module and the Logger support in various parts of the code

0.2.1

If the -c, --config CONFIG parameter is not specified, it tries to use a database.yml file in the directory from where the gem was launched

0.2.0

First version including the web interface

0.1.0

First stable version