0.0
No commit activity in last 3 years
No release in over 3 years
Parse a directory full of JSON files from puppet and cherry pick keys we are interested in.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

~> 1.13
~> 10.0

Runtime

= 0.4.0
 Project Readme

PeLogdump2csv

Turn JSON dumps from PE into CSV!

Give a monster directory of files, eg:

$ ls
total 377M
   0 127.0.0.1-10_28_16_09:05.json
 68K 127.0.0.1-10_28_16_09:10.json
 68K 127.0.0.1-10_28_16_09:15.json
 68K 127.0.0.1-10_28_16_09:20.json
 68K 127.0.0.1-10_28_16_09:25.json
 76K 127.0.0.1-10_28_16_09:30.json
 68K 127.0.0.1-10_28_16_09:35.json
 68K 127.0.0.1-10_28_16_09:40.json

...

Scan each file for specific JSON values and output a cohesive set of values to STDOUT:

timestamp,pe-jruby-metrics.status.experimental.metrics.average-borrow-time,pe-jruby-metrics.status.experimental.metrics.num-free-jrubies,code-manager-service.status.file_sync
10_28_16_09:10,1518,1,enabled
10_28_16_09:15,1790,4,enabled
10_28_16_09:20,1422,0,enabled
10_28_16_09:25,1682,4,enabled
10_28_16_09:30,1464,0,enabled

This way, data can be loaded into Excel to draw graphs

Installation

Add this line to your application's Gemfile:

gem 'pe_logdump2csv'

And then execute:

$ bundle

Or install it yourself as:

$ gem install pe_logdump2csv

Usage

  1. cd into a directory of log files
  2. pe_logdump2csv csv --pick xxx,yyy,zzz
  • This will output CSV on the commandline for each of the listed JSON keys
  • Redirect output to a file and it can then be opened in Excel
  • A header line will be printed, the timestamp is obtained from the filename
  • Files must be called SOMETHING-date.json - we throw away the SOMETHING and the .json to obtain the timestamp
  • You can pick as many fields as you like, just delimit them by commas with no spaces
  • See below for how to figure out field names

Field names

Field names are computed buy combining JSON keys with periods, eg given the following JSON, if you were interested in the value of file_sync, you would pass code-manager-service.status.file_sync to --pick

{
    "code-manager-service": {
        "detail_level": "debug",
        "service_status_version": 1,
        "service_version": "1.4.2",
        "state": "running",
        "status": {
            "file_sync": "enabled",
            "services": {
                "file-sync-storage": "running"

Example commands

pe_logdump2csv csv --pick pe-jruby-metrics.status.experimental.metrics.average-borrow-time,pe-jruby-metrics.status.experimental.metrics.num-free-jrubies,code-manager-service.status.file_sync

Collect:

  • average-borrow-time
  • num-free-jrubies
  • file sync status

Output:

timestamp,pe-jruby-metrics.status.experimental.metrics.average-borrow-time,pe-jruby-metrics.status.experimental.metrics.num-free-jrubies,code-manager-service.status.file_sync
10_28_16_09:10,1518,1,enabled
10_28_16_09:15,1790,4,enabled
10_28_16_09:20,1422,0,enabled
10_28_16_09:25,1682,4,enabled
10_28_16_09:30,1464,0,enabled
10_28_16_09:35,1253,2,enabled
10_28_16_09:40,1259,2,enabled

Obtaining logs

TODO: PR gratefully accepted :)

Development

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/GeoffWilliams/pe_logdump2csv.

License

The gem is available as open source under the terms of the Apache 2.