Logstash Sumo Logic Output Plugin

This is an output plugin for Logstash. It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

Getting Started

This guide is for the users just want to download the binary and make the plugin work. For the developer, please refer to the Developer Guide

1. Create a Sumo Logic HTTP source

Create a Sumo Logic free account if you currently don't have one.

Create a HTTP source in your account and get the URL for this source. It should be something like: https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX

2. Install LogStash on your machine

Following this instruction to download and install LogStash. This plugin requires Logstash 2.3 or higher to work.

3. Install latest Logstash Sumo Logic Output plugin from RubyGems

bin/logstash-plugin install logstash-output-sumologic

4. Start Logstash and send log

In the Logstash home, running:

bin/logstash -e "input{stdin{}}output{sumologic{url=>'<URL from step 1>'}}"

This will send any input from console to Sumo Logic cloud service.

5. Try out samples

Send Log lines

Set the URL got from step 1 as environment variable:

export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX

Launch sample with:

bin/logstash -f samples/log.conf

The input from console will be sent to Sumo Logic cloud service as log lines.

Send Metrics

Set the URL got from step 1 as environment variable:

export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX

Install plugins-filters-metrics plugin

Launch sample with:

bin/logstash -f samples/metrics.conf

Mocked events will be sent to Sumo Logic server as 1 minute and 15 minutes rate metrics.

6. Get result from Sumo Logic web app

Logon to Sumo Logic web app and run

What's Next

Try it with different input/filter/codec plugins
Start LogStash as a service/daemon in your production environment
Report any issue or idea through Git Hub

Parameters of Plugin

Parameter	Type	Required?	Default value	Description
`url`	string	Yes		HTTP Source URL
`source_category`	string	No	`Logstash`	Source category to appear when searching in Sumo Logic by `_sourceCategory`. Using empty string if want keep source category of the HTTP source.
`source_name`	string	No	`logstash-output-sumologic`	Source name to appear when searching in Sumo Logic by `_sourceName`. Using empty string if want keep source name of the HTTP source.
`source_host`	string	No	machine name	Source host to appear when searching in Sumo Logic by `_sourceHost`. Using empty string if want keep source host of the HTTP source.
`extra_headers`	hash	No		Extra fields need to be send in HTTP headers.
`compress`	boolean	No	`false`	Enable or disable compression.
`compress_encoding`	string	No	`"deflate"`	Encoding method of comressing, can only be `"deflate"` or `"gzip"`.
`interval`	number	No	`0`	The maximum time for waiting before sending the message pile, in seconds.
`pile_max`	number	No	`102400`	The maximum size of message pile, in bytes.
`queue_max`	number	No	`4096`	The maximum message piles can be hold in memory.
`sender_max`	number	No	`100`	The maximum HTTP senders working in parallel.
`format`	string	No	`"%{@timestamp} %{host} %{message}"`	For log only, the formatter of log lines. Use `%{@json}` as the placeholder for whole event json.
`json_mapping`	hash	No		Override the structure of `{@json}` tag with the given key value pairs. For example: `json_mapping => { "foo" => "%{@timestamp}" "bar" => "%{message}" }` will create messages as: `{"foo":"2016-07-27T18:37:59.460Z","bar":"hello world"}` `{"foo":"2016-07-27T18:38:01.222Z","bar":"bye!"}`
`metrics`	hash	No		If defined, the event will be sent as metrics. Keys will be the metrics name and values will be the metrics value.
`metrics_format`	string	No	`"cabon2"`	Metrics format, can only be `"graphite"` or `"carbon2"`.
`metrics_name`	string	No	`"*"`	Define the metric name looking, the placeholder "*" will be replaced with the actual metric name.
`intrinsic_tags`	hash	No		For carbon2 format only, send extra intrinsic key-value pairs other than `metric` (which is the metric name).
`meta_tags`	hash	No		For carbon2 format only, send metadata key-value pairs.
`fields_as_metrics`	boolean	No	`false`	If `true`, all fields in logstash event with number value will be sent as a metrics (with filtering by `fields_include` and `fields_exclude` ; the `metics` parameter is ignored.
`fields_include`	array	No	all fields	Working with `fields_as_metrics` parameter, only the fields which full name matching these RegEx pattern(s) will be included in metrics.
`fields_exclude`	array	No	none	Working with `fields_as_metrics` parameter, the fields which full name matching these RegEx pattern(s) will be ignored in metrics.
`sleep_before_requeue`	number	No	`30`	The message failed to send to server will be retried after (x) seconds. Not retried if negative number set
`stats_category`	string	No	`Logstash.stats`	The source category this plugin's stats will be sent with when `stats_enabled` is `true`
`stats_enabled`	boolean	No	`false`	If `true`, stats of this plugin will be sent as metrics
`stats_interval`	number	No	`60`	The stats will be sent every (x) seconds

This plugin is based on logstash-mixin-http_client thus also supports all HTTP layer parameters like proxy, authentication, timeout etc.

Trouble Shooting

Enable plugin logging

Logstash is using log4j2 framework for logging. Starting with 5.0, each individual plugin can configure the logging strategy. Here is a sample log4j2.properties to print plugin log to console and a rotating file.

Optimize throughput

The throughput can be tuning with following parameters:

Messages will be piled before sending if both interval and pile_max are larger than 0. (e.g. multiple messages will sent in single HTTP request); The maximum size of pile is defined in pile_max and if there is no more message comes in, piled message will be sent out every interval seconds. A higher number of these parameters normally means more messages will be piled together so overall reduce the overhead in transmission and benefit for compressing efficiency; but it may make a larger latency because messages may be hold in plugin for longer before sending;
Message piles will be cached before sending in a memory queue. The maximum piles can stay in queue is defined with queue_max. A larger setting may be helpful if input is blocked by the plugin consuming speed, but may also consume more RAM (which can be set in JVM options)
The plugin will use up to sender_max HTTP senders in parallel for talking to Sumo Logic server. This number is also limited by the max TCP connections
Depends on the content pattern, adjusting compress/compress_encoding for balancing between the CPU consumption and package size

On the other side, this version is marked as thread safe so if necessary, multiple plugins can work in parallel as workers

Monitor throughput in metrics

If your Sumo Logic account supports metrics feature, you can enable the stats monitor of this plugin with configuring stats_enabled to true. For every stats_interval seconds, a batch of metrics data points will be sent to Sumo Logic with source category specified by stats_category parameter:

Metric	Description
`total_input_events`	Total number of events handled from the plugin startup
`total_input_bytes`	Total bytes of inputs after encoded to payload
`total_metrics_datapoints`	Total metrics data points generated from input
`total_log_lines`	Total log lines generated from input
`total_output_requests`	Total number of HTTP requests sent to Sumo Logic server
`total_output_bytes`	Total bytes of payloads sent to Sumo Logic server
`total_output_bytes_compressed`	Total bytes of payloads sent to Sumo Logic server (after compressing)
`total_response_times`	Total number of responses acknowledged by Sumo Logic server
`total_response_success`	Total number of accepted(200) acknowledged by Sumo Logic server

NOTE: The data points will consume DPM quota

TLS 1.2 Requirement

Sumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.

hshek-logstash-output-sumologic

Development

Runtime