fluent-plugin-s3 1.5.1 → 1.7.0

data/VERSION

@@ -1 +1 @@
-1.5.1
+1.7.0

data/docs/credentials.md

@@ -0,0 +1,171 @@
+# Configuration: credentials
+
+Both S3 input/output plugin provide several credential methods for authentication/authorization.
+
+## AWS key and secret authentication
+
+These parameters are required when your agent is not running on EC2 instance with an IAM Role. When using an IAM role, make sure to configure `instance_profile_credentials`. Usage can be found below.
+
+### aws_key_id
+
+AWS access key id.
+
+### aws_sec_key
+
+AWS secret key.
+
+## \<assume_role_credentials\> section
+
+Typically, you use AssumeRole for cross-account access or federation.
+
+    <match *>
+      @type s3
+
+      <assume_role_credentials>
+        role_arn          ROLE_ARN
+        role_session_name ROLE_SESSION_NAME
+      </assume_role_credentials>
+    </match>
+
+See also:
+
+*   [Using IAM Roles - AWS Identity and Access
+    Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
+*   [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
+*   [Aws::AssumeRoleCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/AssumeRoleCredentials.html)
+
+### role_arn (required)
+
+The Amazon Resource Name (ARN) of the role to assume.
+
+### role_session_name (required)
+
+An identifier for the assumed role session.
+
+### policy
+
+An IAM policy in JSON format.
+
+### duration_seconds
+
+The duration, in seconds, of the role session. The value can range from
+900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value
+is set to 3600 seconds.
+
+### external_id
+
+A unique identifier that is used by third parties when assuming roles in
+their customers' accounts.
+
+## \<web_identity_credentials\> section
+
+Similar to the assume_role_credentials, but for usage in EKS.
+
+    <match *>
+      @type s3
+
+      <web_identity_credentials>
+        role_arn          ROLE_ARN
+        role_session_name ROLE_SESSION_NAME
+        web_identity_token_file AWS_WEB_IDENTITY_TOKEN_FILE
+      </web_identity_credentials>
+    </match>
+
+See also:
+
+*   [Using IAM Roles - AWS Identity and Access
+    Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html)
+*   [IAM Roles For Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
+*   [Aws::STS::Client](http://docs.aws.amazon.com/sdkforruby/api/Aws/STS/Client.html)
+*   [Aws::AssumeRoleWebIdentityCredentials](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/AssumeRoleWebIdentityCredentials.html)
+
+### role_arn (required)
+
+The Amazon Resource Name (ARN) of the role to assume.
+
+### role_session_name (required)
+
+An identifier for the assumed role session.
+
+### web_identity_token_file (required)
+
+The absolute path to the file on disk containing the OIDC token
+
+### policy
+
+An IAM policy in JSON format.
+
+### duration_seconds
+
+The duration, in seconds, of the role session. The value can range from
+900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value
+is set to 3600 seconds.
+
+
+## \<instance_profile_credentials\> section
+
+Retrieve temporary security credentials via HTTP request. This is useful on
+EC2 instance.
+
+    <match *>
+      @type s3
+
+      <instance_profile_credentials>
+        ip_address IP_ADDRESS
+        port       PORT
+      </instance_profile_credentials>
+    </match>
+
+See also:
+
+*   [Aws::InstanceProfileCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/InstanceProfileCredentials.html)
+*   [Temporary Security Credentials - AWS Identity and Access
+    Management](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
+*   [Instance Metadata and User Data - Amazon Elastic Compute
+    Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
+
+### retries
+
+Number of times to retry when retrieving credentials. Default is 5.
+
+### ip_address
+
+Default is 169.254.169.254.
+
+### port
+
+Default is 80.
+
+### http_open_timeout
+
+Default is 5.
+
+### http_read_timeout
+
+Default is 5.
+
+## \<shared_credentials\> section
+
+This loads AWS access credentials from local ini file. This is useful for
+local developing.
+
+    <match *>
+      @type s3
+
+      <shared_credentials>
+        path         PATH
+        profile_name PROFILE_NAME
+      </shared_credentials>
+    </match>
+
+See also:
+
+*   [Aws::SharedCredentials](http://docs.aws.amazon.com/sdkforruby/api/Aws/SharedCredentials.html)
+
+### path
+
+Path to the shared file. Defaults to "#{Dir.home}/.aws/credentials".
+
+### profile_name
+
+Defaults to 'default' or `[ENV]('AWS_PROFILE')`.

data/docs/howto.md

@@ -0,0 +1,92 @@
+# Object Metadata Added To Records
+
+If the [`add_object_metadata`](input.md#add_object_metadata) option is set to true, then the name of the bucket
+and the key for a given object will be added to each log record as [`s3_bucket`](input.md#s3_bucket)
+and [`s3_key`](input.md#s3_key), respectively. This metadata can be used by filter plugins or other
+downstream processors to better identify the source of a given record.
+
+# IAM Policy
+
+The following is an example for a IAM policy needed to write to an s3 bucket (matches my-s3bucket/logs, my-s3bucket-test, etc.).
+
+    {
+      "Version": "2012-10-17",
+      "Statement": [
+        {
+          "Effect": "Allow",
+          "Action": [
+            "s3:ListBucket"
+          ],
+          "Resource": "arn:aws:s3:::my-s3bucket"
+        },
+        {
+          "Effect": "Allow",
+          "Action": [
+            "s3:PutObject",
+            "s3:GetObject"
+          ],
+          "Resource": "arn:aws:s3:::my-s3bucket/*"
+        }
+      ]
+    }
+
+Note that the bucket must already exist and **[`auto_create_bucket`](output.md#auto_create_bucket)** has no effect in this case.
+
+`s3:GetObject` is needed for object check to avoid object overwritten.
+If you set `check_object false`, `s3:GetObject` is not needed.
+
+Refer to the [AWS
+documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/ExampleIAMPolicies.html) for example policies.
+
+Using [IAM
+roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html)
+with a properly configured IAM policy are preferred over embedding access keys
+on EC2 instances.
+
+## Example when `check_bucket false` and `check_object false`
+
+When the mentioned configuration will be made, fluentd will work with the
+minimum IAM poilcy, like:
+
+
+    "Statement": [{
+      "Effect": "Allow",
+      "Action": "s3:PutObject",
+      "Resource": ["*"]
+    }]
+
+
+# Use your (de)compression algorithm
+
+s3 plugin has pluggable compression mechanizm like Fluentd's input / output
+plugin. If you set 'store_as xxx', `out_s3` plugin searches
+`fluent/plugin/s3_compressor_xxx.rb` and `in_s3` plugin searches
+`fluent/plugin/s3_extractor_xxx.rb`. You can define your (de)compression with
+'S3Output::Compressor'/`S3Input::Extractor` classes. Compressor API is here:
+
+    module Fluent # Since fluent-plugin-s3 v1.0.0 or later, use Fluent::Plugin instead of Fluent
+      class S3Output
+        class XXXCompressor < Compressor
+          S3Output.register_compressor('xxx', self)
+
+          # Used to file extension
+          def ext
+            'xxx'
+          end
+
+          # Used to file content type
+          def content_type
+            'application/x-xxx'
+          end
+
+          # chunk is buffer chunk. tmp is destination file for upload
+          def compress(chunk, tmp)
+            # call command or something
+          end
+        end
+      end
+    end
+
+`Extractor` is similar to `Compressor`
+See bundled `Compressor`/`Extractor` classes for more detail.
+

data/docs/input.md

@@ -0,0 +1,98 @@
+# Input: Setup
+
+1. Create new [SQS](https://aws.amazon.com/documentation/sqs/) queue (use same region as S3)
+2. Set proper permission to new queue
+3. [Configure S3 event notification](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html)
+4. Write configuration file such as fluent.conf
+5. Run fluentd
+
+# Configuration: Input
+
+See also [Configuration: credentials](credentials.md) for common comprehensive parameters.
+
+    <source>
+      @type s3
+
+      aws_key_id YOUR_AWS_KEY_ID
+      aws_sec_key YOUR_AWS_SECRET_KEY
+      s3_bucket YOUR_S3_BUCKET_NAME
+      s3_region ap-northeast-1
+      add_object_metadata true
+
+      <sqs>
+        queue_name YOUR_SQS_QUEUE_NAME
+      </sqs>
+    </source>
+
+## add_object_metadata
+
+Whether or not object metadata should be added to the record. Defaults to `false`. See below for details.
+
+## s3_bucket (required)
+
+S3 bucket name.
+
+## s3_region
+
+S3 region name. For example, US West (Oregon) Region is
+"us-west-2". The full list of regions are available here. >
+http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We
+recommend using `s3_region` instead of `s3_endpoint`.
+
+## store_as
+
+archive format on S3. You can use serveral format:
+
+* gzip (default)
+* json
+* text
+* lzo (Need lzop command)
+* lzma2 (Need xz command)
+* gzip_command (Need gzip command)
+  * This compressor uses an external gzip command, hence would result in utilizing CPU cores well compared with `gzip`
+
+See [Use your compression algorithm](howto.md#use-your-compression-algorithm) section for adding another format.
+
+## format
+
+Parse a line as this format in the S3 object. Supported formats are
+"apache_error", "apache2", "syslog", "json", "tsv", "ltsv", "csv",
+"nginx" and "none".
+
+## check_apikey_on_start
+
+Check AWS key on start. Default is true.
+
+## proxy_uri
+
+URI of proxy environment.
+
+## \<sqs\> section
+
+### queue_name (required)
+
+SQS queue name. Need to create SQS queue on the region same as S3 bucket.
+
+### queue_owner_aws_account_id
+
+SQS Owner Account ID
+
+### aws_key_id
+
+Alternative aws key id for SQS
+
+### aws_sec_key
+
+Alternative aws key secret for SQS
+
+### skip_delete
+
+When true, messages are not deleted after polling block. Default is false.
+
+### wait_time_seconds
+
+The long polling interval. Default is 20.
+
+### retry_error_interval
+
+Interval to retry polling SQS if polling unsuccessful, in seconds. Default is 300.