fluent-plugin-http-pull

Fluentd input plugin to pull log from rest api.

Many of modern server application offer status reporting API via http (even 'fluentd' too). This plugin will help to gathering status log from these status api.

• Installation
• Usage
  • Basic Usage
  • Monitor Status Code
  • Override User Agent
  • HTTP Basic Auth
  • HTTP Proxy
  • Logging Response Header
  • Custom Request Header
  • HTTPS Support
• Configuration
  • tag
  • url
  • agent
  • interval
  • format
  • status_only
  • http_method
  • timeout
  • proxy
  • user
  • password
  • response_header
  • request_header
  • verify_ssl
  • ca_path
  • ca_file
• Version Tested
• Copyright

$ gem install fluent-plugin-http-pull

Add following line to your Gemfile:

gem "fluent-plugin-http-pull"

And then execute:

$ bundle

In your Fluentd configuration, use @type http_pull. tag, url, interval, format is mandatory configuration.

<source>
	@type http_pull

	tag status
	url http://your-infrastructure/api/status.json
	interval 1s

	format json
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

In this example, a response of your infrastructure parsed as json. A result contains url, status, message. message is a response of your infrastructure.

{
	"url": "http://your-infrastructure/api/status.json",
	"status": 200,
	"message": {
		// response of your infra structure
	}
}

You can found more examples in this document.

If you need only http status code, not response body, You can turn off response body parser to set status_only is true. Remember that format is mandatory. In this case, you must set format is none.

<source>
	@type http_pull

	tag fluentd.status
	url http://your-infrastructure/healthcheck
	interval 1s

	status_only true
	format none
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/healthcheck","status":200}

You can set the User Agent by specifying the agent option.

<source>
	@type http_pull

	tag status
	url http://your-infrastructure/api/status.json
	interval 1s

	format json
	agent infrastructure_monitor
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

If your infrastructure is protected by HTTP basic auth, You can use user, password options to provide authentication information.

<source>
	@type http_pull

	tag status
	url http://your-infrastructure/api/status.json
	interval 1s

	format json
	user foo
	password bar
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

You can send your requests via proxy server.

<source>
	@type http_pull

	tag status
	url http://your-infrastructure/api/status.json
	proxy http://your-proxy-server:3128
	interval 1s

	format json
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

If you wish to monitoring not only response body but also response header, provide name of header in response_header sections.

<source>
	@type http_pull

	tag status
	url http://your-infrastructure/api/status.json
	interval 1s

	format json

	<response_header>
		header Content-Type
	</response_header>

	<response_header>
		header Content-Length
	</response_header>
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

The custom request header also supported. This will help you when your infrastructure needs authentication headers or something like that.

<source>
	@type http_pull

	tag status
	url http://your-infrastructure/api/status.json
	interval 1s

	format json

	<request_header>
		header API_ACCESS_KEY
		value hatsune
	</request_header>

	<response_header>
		header API_ACCESS_KEY_SECRET
		value miku
	</response_header>
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

If your infrastructure has https endpoints, just use https url for monitoring them.

If your infrastructure has https endpoints secured by self signed certification, you can provide custom certification file via ca_path, ca_file option.

<source>
	@type http_pull

	tag status
	url https://your-infrastructure/api/status.json
	interval 1s
	ca_path /somewhere/ca/stored
	ca_file /somewhere/ca/stored/server.crt

	format json
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

And, disable SSL verification also supported. (not recommended)

<source>
	@type http_pull

	tag status
	url https://your-infrastructure/api/status.json
	interval 1s
	verify_ssl false

	format json
</source>

# 2017-05-17 21:41:47.872951000 +0900 status: {"url":"http://yourinfrastructure/api/status.json","status":200,"message":{ ... }}

The tag of the event.

The url of remote server.

The user agent string of request.

The interval time between periodic request.

The format of the response body. Due to limitation of current implement it is always required regardless status_only option.

http_pull uses parse plugin to parse the response body. See parser article for more detail.

If status_only is true, body is not parsed.

The http request method for each requests. Avaliable options are listed below.

• get
• post
• delete

If status_only is true, http_method was override to head

The timeout of each request.

The HTTP proxy URL to use for each requests

The user for basic auth

The password for basic auth

The name of response header for capture.

The name, value pair of custom reuqest header.

When false, SSL verification is ignored.

The absolute path of directory where ca_file stored. Should be used with ca_file.

The absolute path of ca_file. Should be used with ca_path.

This plugin is tested with the following version combination of ruby and fluentd.

• Copyright(c) 2017- filepang
• License
  • Apache License, Version 2.0