Counters

Easily record any metrics from anywhere within your code, using a very simple interface:

• ping: When’s the last time we saw this thing?
• hit: Increments a counter
• magnitude: Measures numerical values
• latency: Measures time intervals

Let’s say you have a web crawler. There are a ton of things you can measure about your crawler: how many pages it processed (#hit), how many bytes you read (#magnitude), how long did it take to download the page (#latency), how long did it take to parse the raw HTML to a useable format (#latency).

require "uri"
require "open-uri"
require "counters"
require "nokogiri"

Counter = Counters::Redis.new(Redis.new, :namespace => "crawler", :base_key => "counters")

urls_to_crawl = ["http://blog.teksol.info/", "http://techcrunch.com/", "http://www.google.com/"]

while url = urls_to_crawl.pop
  Counter.ping "crawler"

  begin
    Counter.hit "urls_popped"

    puts "Fetching #{url}"
    raw_html = Counter.latency "download" do
      URI.parse(url).read
    end

    Counter.magnitude "bytes_in", raw_html.length

    parsed_html = Counter.latency "html_parsing" do
      Nokogiri::HTML(raw_html)
    end
  rescue
    Counter.hit "error"
  end
end

For testing purposes, there also exists a Counters::Memory. This would be good in test mode, for example. The counters are exposed through accessor methods returning a Hash.

You may log to a file, but be advised the file’s size grows very quickly. Counters are stored in the file, one per line, in an easily readable format.

$ irb -r counters
> Counter = Counters::File.new("counters.log")
 => #<Counters::File:0x00000101a18f18 @logger=#<Logger:0x00000101a18ef0 @progname=nil, @level=0, @default_formatter=#<Logger::Formatter:0x00000101a18ea0 @datetime_format=nil>, @formatter=#<Proc:0x00000101a18bd0@/Users/francois/Projects/counters/lib/counters/file.rb:15 (lambda)>, @logdev=#<Logger::LogDevice:0x00000101a18e28 @shift_size=1048576, @shift_age=0, @filename="counters.log", @dev=#<File:counters.log>, @mutex=#<Logger::LogDevice::LogDeviceMutex:0x00000101a18e00 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Mutex:0x00000101a18d88>>>>> 
> Counter.hit "crawler.page_read"
 => true
> Counter.magnitude "crawler.bytes_in", 9_921
 => true
> Counter.latency "crawler.processing" do sleep 0.3 ; end
 => true
> Counter.ping "crawler.alive"
 => true
> exit

$ cat counters.log
2011-02-21T09:46:21.296326000 - hit: crawler.page_read
2011-02-21T09:46:24.280388000 - magnitude: crawler.bytes_in 9921
2011-02-21T09:46:27.989183000 - latency: crawler.processing 0.3001821041107178s
2011-02-21T09:46:31.031969000 - ping: crawler.alive
2011-02-21T09:46:21.296326000 - hit: crawler.page_read
2011-02-21T09:46:24.280388000 - magnitude: crawler.bytes_in 13291
2011-02-21T09:46:27.989183000 - latency: crawler.processing 0.3123122982101s

You may also output your counters to StatsD. The only change that might be surprising is magnitudes are output as timer events. Magnitudes are used to record values (such as process RSS, packet sizes, etc).

Instantiate a StatsD instance:

Counter = Counters::StatsD.new("127.0.0.1", 8125, :namespace => "analyzer")

# Alternatively, use a URI:
Counter = Counters::StatsD.new(:url => "udp://127.0.0.1:8125", :namespace => "crawler")

See the file samples/crawler.rb for a more detailed example.

(The MIT License)

Copyright © 2008-2009 François Beausoleil (francois@teksol.info)

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
‘Software’), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.