fluent-plugin-gcloud-pubsub
Overview
Cloud Pub/Sub Input/Output plugin for Fluentd with gcloud gem
- Publish BufferedOutput chunk into Cloud Pub/Sub with batch publishing
- Pull messages from Cloud Pub/Sub
Preparation
- Create a project on Google Developer Console
- Add a topic of Cloud Pub/Sub to the project
- Add a pull style subscription to the topic
- Download your credential (json)
Publish messages
Use out_gcloud_pubsub
.
Configuration
publish dummy json data like {"message": "dummy", "value": 0}\n{"message": "dummy", "value": 1}\n ...
.
<source>
type dummy
tag example.publish
rate 100
auto_increment_key value
</source>
<match example.publish>
type gcloud_pubsub
project <YOUR PROJECT>
topic <YOUR TOPIC>
key <YOUR KEY>
flush_interval 10
autocreate_topic false
</match>
-
autocreate_topic
(optional, default:false
)- If set to
true
, specified topic will be created when it doesn't exist.
- If set to
Pull messages
Use in_gcloud_pubsub
.
Configuration
Pull json data from Cloud Pub/Sub
<source>
type gcloud_pubsub
tag example.pull
project <YOUR PROJECT>
topic <YOUR TOPIC>
subscription <YOUR SUBSCRIPTION>
key <YOUR KEY>
max_messages 1000
return_immediately true
pull_interval 2
format json
</source>
<match example.pull>
type stdout
</match>
-
max_messages
-
see maxMessages on https://cloud.google.com/pubsub/subscriber
-
return_immediately
-
see returnImmediately on https://cloud.google.com/pubsub/subscriber
-
When
return_immediately
is true, this plugin ignore pull_interval