No commit activity in last 3 years
No release in over 3 years
There's a lot of open issues
Google Cloud Pub/Sub input/output plugin for Fluentd event collector
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Development

Runtime

 Project Readme

fluent-plugin-gcloud-pubsub

Build Status Gem Version

Overview

Cloud Pub/Sub Input/Output plugin for Fluentd with gcloud gem

  • Publish BufferedOutput chunk into Cloud Pub/Sub with batch publishing
  • Pull messages from Cloud Pub/Sub

Preparation

  • Create a project on Google Developer Console
  • Add a topic of Cloud Pub/Sub to the project
  • Add a pull style subscription to the topic
  • Download your credential (json)

Publish messages

Use out_gcloud_pubsub.

Configuration

publish dummy json data like {"message": "dummy", "value": 0}\n{"message": "dummy", "value": 1}\n ....

<source>
  type dummy
  tag example.publish
  rate 100 
  auto_increment_key value
</source>

<match example.publish>
  type gcloud_pubsub
  project <YOUR PROJECT>
  topic <YOUR TOPIC>
  key <YOUR KEY>
  flush_interval 10
  autocreate_topic false
</match>
  • autocreate_topic (optional, default: false)
    • If set to true, specified topic will be created when it doesn't exist.

Pull messages

Use in_gcloud_pubsub.

Configuration

Pull json data from Cloud Pub/Sub

<source>
  type gcloud_pubsub
  tag example.pull
  project <YOUR PROJECT>
  topic <YOUR TOPIC>
  subscription <YOUR SUBSCRIPTION>
  key <YOUR KEY>
  max_messages 1000
  return_immediately true
  pull_interval 2
  format json
</source>

<match example.pull>
  type stdout
</match>