fluent-plugin-ec2-metadata

Fluentd plugin to add Amazon EC2 metadata fields to a event record

Requirements

fluent-plugin-ec2-metadata	fluentd	ruby
>= 0.1.0	v0.14.x	>= 2.1
0.0.15 <=	v0.12.x	>= 1.9

Installation

Use RubyGems:

gem install fluent-plugin-ec2-metadata

Configuration

Example:

<match foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds
  imdsv2 true                  # Optional, default false

  output_tag ${instance_id}.${tag}
  <record>
    hostname      ${tagset_name}
    instance_id   ${instance_id}
    instance_type ${instance_type}
    az            ${availability_zone}
    private_ip    ${private_ip}
    vpc_id        ${vpc_id}
    ami_id        ${image_id}
    account_id    ${account_id}
  </record>
</match>

Assume following input is coming:

foo.bar {"message":"hello ec2!"}

then output becomes as below (indented):

i-28b5ee77.foo.bar {
  "hostname"      : "web0001",
  "instance_id"   : "i-28b5ee77",
  "instance_type" : "m1.large",
  "az"            : "us-west-1b",
  "private_ip     : "10.21.34.200",
  "vpc_id"        : "vpc-25dab194",
  "account_id"    : "123456789",
  "image_id"      : "ami-123456",
  "message"       : "hello ec2!"
}

Or you can use filter version:

<filter foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID      
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds
  imdsv2 true                  # Optional, default false

  <record>
    hostname      ${tagset_name}
    instance_id   ${instance_id}
    instance_type ${instance_type}
    private_ip    ${private_ip}
    az            ${availability_zone}
    vpc_id        ${vpc_id}
    ami_id        ${image_id}
    account_id    ${account_id}
  </record>
</filter>

Placeholders

The following placeholders are always available:

${tag} input tag
${tag_parts} input tag splitted by '.'. you can use it like ${tag_parts[0]} or ${tag_parts[-1]}
${instance_id} instance id
${instance_type} instance type
${availability_zone} availability zone
${region} region
${private_ip} private ip
${mac} MAC address
${vpc_id} vpc id
${subnet_id} subnet id
${account_id} account id
${image_id} ami image id

The followings are available when you define aws_key_id and aws_sec_key(or define IAM Policy):

${tagset_xxx} EC2 tag (e.g. tagset_name is replaced by the value of Key = Name)

The following is an example for a minimal IAM policy needed to ReadOnlyAccess to EC2.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "elasticloadbalancing:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:Describe*"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "autoscaling:Describe*",
      "Resource": "*"
    }
  ]
}

Refer to the AWS documentation for example policies. Using IAM roles with a properly configured IAM policy are preferred over embedding access keys on EC2 instances.

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request