Fluentd input plugin for AWS RDS Aurora

This fluent input plugin collects RDS Aurora slowquery log with ParameterGroup option log_output=FILE. This plugin fetches only the difference between the latest and previous fetched slow query log.

There are a lot of RDS mysql slowlog input plugins that collect mysql slowquery logs with ParameterGroup option log_output=TABLE. However if you use RDS Aurora with the option, there are two following problems:

mysql.slow_log table is operated with Engine=CSV and we cannot add index to the table. So if tons of slow queries are registered to mysql.slow_log, your SELECT * FROM mysql.slow_log WHERE condition queries also become slow.

mysql.slow_log table on RDS mysql can be purged by using CALL mysql.rds_rotate_slow_log. On the other hand, the stored procedure executed on Aurora makes following error.

> CALL mysql.rds_rotate_slow_log;
ERROR 1289 (HY000): The 'CSV' feature is disabled; you need MySQL built with 'CSV' to have it working

According to the above reasons, I have implemented input plugin with log_output=FILE for Aurora.

$ fluent-gem install fluent-plugin-aurora-slowquerylog --no-document

TBW

<source>
  @type aurora_slowquerylog
  tag  aurora.slowlog
  db_instance_identifier  aurora_node_id
  region  us-east-1
  log_file_name  slowquery/mysql-slowquery.log
  aurora_state_file  /var/run/fluentd/aurora_state
  log_fetch_interval  30 #optionnal
  aws_access_key_id  your_aws_access_key_id #optionnal
  aws_secret_access_key  your_aws_secret_access_key #optionnal
  filename_contains mysql-slowquery.log # default 'mysql-slowquery.log'
</source>

• tag tag name of events
• db_instance_identifier AWS Aurora node id
• region AWS region name
• log_file_name RDS slowlog name. Currently we cannot change file name from 'slowquery/mysql-slowquery.log'
• aurora_state_file state file that keeps maker information and current & last slowquery log name
• log_fetch_interval interval time(second) for log fetch (optional)
• aws_access_key_id AWS access key id. For AWS user IAM instance profile is recommended without using this option (optional)
• aws_secret_access_key AWS secret access key. For AWS user IAM instance profile is recommended without using this option (optional)
• filename_contains filter condition for fetching slow query log (optional)

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/fluent-plugin-aurora-slowquerylog.

The gem is available as open source under the terms of the MIT License.