No commit activity in last 3 years
No release in over 3 years
Fluentd output plugin for remote syslog. This is meant for processing kubernetes annotated messages.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

Runtime

 Project Readme

fluent-plugin-kubernetes_tagged_remote_syslog

Fluentd plugin for output to remote syslog service (e.g. Papertrail). It is meant to use in Kubernetes environment with kubernetes_metadata_filter plugin annotating the messages.

This plugin was created because regular remote syslog Fluentd plugins doesn't work well with kubernetes annotated messages. hostname/program fields have not helpful values localhost/fluentd and actual message is unreadable compact JSON.

Installation

 fluent-gem install fluent-plugin-kubernetes_tagged_remote_syslog

Example input/output

Default input should be JSON message annotated with kubernetes info:

{
  "log": "log me\n",
  "time": "2015-05-05T19:54:41.248307+00:00",
  "stream": "stderr",
  "docker": {
    "id": "df14e0d5ae4c07284fa636d739c8fc2e6b52bc344658de7d3f08c36a2e804115",
  }
  "kubernetes": {
    "host": "jimmi-redhat.localnet",
    "pod_name":"fabric8-console-controller-98rqc",
    "pod_id": "c76927af-f563-11e4-b32d-54ee7527188d",
    "container_name": "fabric8-console-container",
    "namespace_name": "default",
    "namespace_id": "23437884-8e08-4d95-850b-e94378c9b2fd",
    "labels": {
      "component": "fabric8Console"
    }
  }
}

This will become:

default fabric8-console-controller-98rqc: log me\n

And at the message receive Papertrail will add a timestamp in front of the message:

May 05 19:54:45 default fabric8-console-controller-98rqc: log me\n

Usage

<match foo>
  type kubernetes_tagged_remote_syslog
  host example.com
  port 514
  program 'pod_name'
  hostname 'namespace_name'
</match>

hostname value will look for the key with that name in message and adds it as hostname. If no key is found it will use the value of the key itself. program value works in the same way as hostname.

Dropping messages from namespaces you don't need

Parameter skip_namespaces will drop messages from selected namespaces. This should be technically in its own filter plugin, but that would require installing and configuring two pluggins so I just dropped it here.

Default configuration will drop messages from kube-system and deis namespaces.

Special Use Case - Deis Workflow

I am running this plugin myself as part of Deis Workflow Fluentd DaemonSet. This requires editing values.yaml:

...
fluentd:
  syslog:
    # Configure the following ONLY if using Fluentd to send log messages to both
    # the Logger component and external syslog endpoint
    # external syslog endpoint url
    host: ""
    # external syslog endpoint port
    port: ""
  daemon_environment:
    FLUENTD_PLUGIN_1: "fluent-plugin-kubernetes_tagged_remote_syslog"
    CUSTOM_STORE_1: "<store> \n
                       @type kubernetes_tagged_remote_syslog \n
                       host logs5.papertrailapp.com \n
                       port REDACTED \n
                     </store>"
...

Plugin will be installed with fluent-gem from rubygems.com.

License

Copyright (c) 2017 Jakub Kvita. See LICENSE for details.