logstash-output-monasca_log_api

This module is a logstash-output-plugin for the Monasca Log Api.

Compatible logstash version: Logstash 7.x

Get latest stable version

https://rubygems.org/gems/logstash-output-monasca_log_api

gem install logstash-output-monasca_log_api

Build from source

Requirements

JRuby (I recommend to use Ruby Version Manager (RVM) How to install? -> https://rvm.io/rvm/install)
JDK
Git
bundler

Install requirements

RVM

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash
source /home/vagrant/.rvm/scripts/rvm

JRuby

rvm install jruby

Clone project

git clone https://github.com/logstash-plugins/logstash-output-monasca_log_api.git

Use jruby

rvm use jruby

Install bundler

gem install bundler

Fetch dependencies

bundle install

Unit tests

Run unit tests with:

bundle exec rspec

Run specified test

bundle exec rspec spec/outputs/monasca/monasca_log_api_client_spec.rb

Run coverage

JRUBY_OPTS="-Xcli.debug=true --debug" bundle exec rspec

Coverage report can be found in ./coverage

Code style check

rubocop lib/

Deploy Plugin to logstash

Source: https://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_output_plugin.html#_building_and_testing_4

Build Gemfile

First we need to create a Gemfile.

What is a Gemfile? Gems are packages for the Ruby programming language. A Gemfile defines all dependencies which are neccessary to build the product.

How to build a Gemfile? Run this:

gem build logstash-output-monasca_log_api.gemspec

Deploy Gemfile to logstash

Download logstash (>=2.3.4)
Extract logstash and navigate into the folder

Run this command from logstash folder to install the plugin

bin/logstash-plugin install /my/logstash/plugins/logstash-output-monasca_log_api-1.0.0.gem

Verify installed plugins: With bin/plugin list --verbose you can check installed plugins. There should be logstash-output-monasca_log_api.

Start logstash output plugin

Configuration

Plugin name: monasca_log_api

Save the configfile wherever you like. For example ~/logstash.conf

name	description	type	required	default	example
monasca_log_api_url	monasca log api url	string	true		https://192.168.10.4:5607/v3.0
monasca_log_api_insecure	set to true if monasca-log-api is using an insecure ssl certificate	boolean	false	false
keystone_api_url	keystone api url	string	true		http://192.168.10.5:35357/v3
keystone_api_insecure	set to true if keystone is using an insecure ssl certificate	boolean	false	false
project_name	Keystone user credentials: project name	string	true		monasca
username	Keystone user credentials: username	string	true		admin-agent
password	Keystone user credentials: password	string	true		password
user_domain_name	Keystone user credentials: user domain name	string	true		default
project_domain_name	Keystone user credentials: project domain name	string	true		default
dimensions	global array dimensions in form of key-value pairs to describe the monitored node	array	false		['app_type:kafka', 'priority:high']
num_of_logs	maximum number of logs that are send by one request to monasca-log-api	number	false	125
elapsed_time_sec	send logs if the maximum elapsed time in seconds is reached	number	false	30
delay	delay time in seconds to wait before checking the elapsed_time_sec again	number	false	10
max_data_size_kb	maximum size in kb of logs that are send by one request to monasca-log-api	number	false	5120

Example configuration files

Simple

output {
  monasca_log_api {
    monasca_log_api_url => "http://192.168.10.4:5607/v3.0"
    keystone_api_url => "http://192.168.10.5:35357/v3"
    project_name => "cmm"
    project_domain_name => "Default"
    username => "cmm-operator"
    user_domain_name => "Default"
    password => "admin"
  }
}

Complete

output {
  monasca_log_api {
    monasca_log_api_url => "https://192.168.10.4:5607/v3.0"
    monasca_log_api_insecure => true
    keystone_api_url => "http://192.168.10.5:35357/v3"
    keystone_api_insecure => false
    project_name => "cmm"
    project_domain_name => "Default"
    username => "cmm-operator"
    user_domain_name => "Default"
    password => "admin"
    dimensions => ["hostname: monasca", "ip:192.168.10.4"]
    num_of_logs => 125
    delay => 10
    elapsed_time_sec => 30
    max_data_size_kb => 5120
  }
}

Run

bin/logstash -f ~/logstash.conf

Run in debug mode

bin/logstash -f ~/logstash.conf --debug

Specify log output file

bin/logstash -f ~/logstash.conf -l /var/log/monasca/log/agent/test-log-agent.log

Logstash File Input plugin

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html

Configuration

Local dimensions can be added with add_field setting

input {
  file {
    add_field => { "dimensions" => { "service" => "monasca-api" }}
    add_field => { "dimensions" => { "language" => "java" }}
    add_field => { "dimensions" => { "log_level" => "error" }}
    path => "/var/log/monasca/api/error.log"
    }
  }

Cross tenant functionality

The output plugin is able to post data on behalf of another tenant to the monasca-log-api. For example you can use a user of tenant X to send logs to the monasca-log-api for another tenant Y. The logs will be stored in the index of tenant Y.

To use this functionality you need to pass through another field called cross_tenant to each incoming event. This can be done by using the add_field for file input plugin. Example configuration:

input {
  file {
    add_field => { "dimensions" => { "service" => "neutron" "component" => "firewall" }}
    add_field => { "cross_tenant" => "903ac629d8424dc39ae928a5fff338e7" }
    path => "/var/log/neutron/firewall.log"
  }
}

Important: The user which is sending the data on behalf of other tenants requires a specific role. This role is defined in the configuration file of the monasca-log-api as delegate_roles.

Exception handling

This output plugin will sleep for maximum 60 seconds after the request to Monasca API fails and will retry the request until successful.

Open tasks

Language translations (Replace hardcoded String messages with a configuration/language file)
Unit tests for failed requests handling