No commit activity in last 3 years
No release in over 3 years
This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

Runtime

 Project Readme
<title>Logstash for SCAPI - output scacsv</title>

scacsv

Synopsis

Receives a stream of events and outputs files complying with the SCAPI requirements related to header and file naming. Essentially provides a match between Logstash's 'streaming' approach and SCAPI's file-based input reqmts. This is what it might look like in your config file:
output {
scacsv {
  fields => ... # array (required)
  header => ... # array (optional), default: {}
  path => ... # string (required)
  group => ... # string (required)
  max_size => ... # number (optional), default: 0 (not used)
  flush_interval => ... # number (optional), default: 60
  file_interval_width => ... # string (optional), default: ""
  time_field => ... # string (optional), default: 'timestamp'
  time_field_format => ... # string (required)
  timestamp_output_format => ... # string (optional), default: Uses values from time_field_format
  force_GMT_filenames => ... # boolean (optional), default: false
  increment_time => ... # boolean (optional), default: false
  }
}

Details

  • Value type is Array
  • There is no default for this setting

Specify which fields from the incoming event you wish to output, and which order

  • Value type is Array
  • Default value is {}

Used to specify a string to put as the header (first) line in the file. Useful if you want to override the default ones which are determined from the fields setting

  • Value type is string
  • Default value is ""

Path of temporary output file. Output will be written to this file until it is time to close the file. Then it will be renamed to SCAPI file convention. The temporary output file path will be then reused for the next set of output. For example, if output data for a CPU group, we might define the following path

path => "./cpu.csv".

  • Value type is string
  • There is no default value for this setting.

SCAPI input filenames must have a group identifier as part of the name. The filename generally has this format <group>__<starttime>__<endtime>.csv. This group parameter is used to specify that group name and it will be used as a prefix when the file is renamed from path. For example

path => "./cpu".

  • Value type is number
  • Default value is 0 (meaning it is not used)

This will closing and rename a file if there have been max_size events received. This is to limit the size of a file, and sometimes can be useful when 'chopping' a stream into chunks for use in SCAPI

  • Value type is number
  • Default value is 60

Amount of time (seconds) to wait before flushing, closing and renaming a file, if there have been no events received. This is to ensure that after a period of idleness, we will output a SCAPI file.

  • Value type is string
  • Default value is "" (meaning it is not used). Allowed values are "MINUTE", "HOUR", "DAY"

Setting this enables files to be closed on specified boundaries. This is useful to break incoming stream up on PI preferred boundaries. If HOUR was set for example, then all incoming data for a particular hour would be put in a file for that hour, and when new data in the next hour arrives, the file is close and a new one opened

  • Value type is string
  • Default value is "timestamp"

Specify which field to use as the 'timestamp' when determining filename times. Values from the 'timestamp' field will be used for starttime (first value seen) and endtime (last value seen) in the file name <group>__<starttime>__<endtime>.csv

  • Value type is string
  • There is no default value for this setting

A format string, in java SimpleDateFormat format, to specify how to interpret the timefield values e.g. "yyyy-MM-dd HH:mm:ss".

  • Value type is string
  • If not specified, it uses the format declared by time_field_format

A format string, in java SimpleDateFormat format, to specify how to output filename timestamps

  • Value type is boolean
  • Default value is false

By default, the supplied timestamp will be left as is. If set to true, then the timestamp will be incremented by 1. This is to ensure that the end time is greater than the last event time in the file - per PI datafile requirements

  • Value type is boolean
  • Default value is false

When set to true, this will ensure that the filenames are rendered with GMT timestamps, independent of what the the locale for the host is set to. This is primarily useful when ISO8601 format timestamps are used


Hello! I'm your friendly footer. If you're actually reading this, I'm impressed.

<script src="/js/patch.js?1.4.2"></script>