logstash-integration-aws 0.1.0.pre

data/docs/input-sqs.asciidoc

@@ -0,0 +1,287 @@
+// :integration: aws
+:plugin: sqs
+:type: input
+:default_codec: json
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== Sqs input plugin
+
+// include::{include_path}/plugin_header-integration.asciidoc[]
+
+==== Description
+
+
+Pull events from an Amazon Web Services Simple Queue Service (SQS) queue.
+
+SQS is a simple, scalable queue system that is part of the
+Amazon Web Services suite of tools.
+
+Although SQS is similar to other queuing systems like AMQP, it
+uses a custom API and requires that you have an AWS account.
+See http://aws.amazon.com/sqs/ for more details on how SQS works,
+what the pricing schedule looks like and how to setup a queue.
+
+To use this plugin, you *must*:
+
+ * Have an AWS account
+ * Setup an SQS queue
+ * Create an identity that has access to consume messages from the queue.
+
+The "consumer" identity must have the following permissions on the queue:
+
+ * `sqs:ChangeMessageVisibility`
+ * `sqs:ChangeMessageVisibilityBatch`
+ * `sqs:DeleteMessage`
+ * `sqs:DeleteMessageBatch`
+ * `sqs:GetQueueAttributes`
+ * `sqs:GetQueueUrl`
+ * `sqs:ListQueues`
+ * `sqs:ReceiveMessage`
+
+Typically, you should setup an IAM policy, create a user and apply the IAM policy to the user.
+A sample policy is as follows:
+[source,json]
+    {
+      "Statement": [
+        {
+          "Action": [
+            "sqs:ChangeMessageVisibility",
+            "sqs:ChangeMessageVisibilityBatch",
+            "sqs:DeleteMessage",
+            "sqs:DeleteMessageBatch",
+            "sqs:GetQueueAttributes",
+            "sqs:GetQueueUrl",
+            "sqs:ListQueues",
+            "sqs:ReceiveMessage"
+          ],
+          "Effect": "Allow",
+          "Resource": [
+            "arn:aws:sqs:us-east-1:123456789012:Logstash"
+          ]
+        }
+      ]
+    }
+
+See http://aws.amazon.com/iam/ for more details on setting up AWS identities.
+
+
+[id="plugins-{type}s-{plugin}-options"]
+==== Sqs Input Configuration Options
+
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-access_key_id>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-additional_settings>> |<<hash,hash>>|No
+| <<plugins-{type}s-{plugin}-aws_credentials_file>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-endpoint>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-id_field>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-md5_field>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-polling_frequency>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-proxy_uri>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-queue>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-queue_owner_aws_account_id>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-region>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-role_arn>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-role_session_name>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-secret_access_key>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-sent_timestamp_field>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-session_token>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-threads>> |<<number,number>>|No
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+input plugins.
+
+&nbsp;
+
+[id="plugins-{type}s-{plugin}-access_key_id"]
+===== `access_key_id` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:
+
+1. Static configuration, using `access_key_id` and `secret_access_key` params in logstash plugin config
+2. External credentials file specified by `aws_credentials_file`
+3. Environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
+4. Environment variables `AMAZON_ACCESS_KEY_ID` and `AMAZON_SECRET_ACCESS_KEY`
+5. IAM Instance Profile (available when running inside EC2)
+
+[id="plugins-{type}s-{plugin}-additional_settings"]
+===== `additional_settings`
+
+* Value type is <<hash,hash>>
+* Default value is `{}`
+
+Key-value pairs of settings and corresponding values used to parametrize
+the connection to SQS. See full list in https://docs.aws.amazon.com/sdk-for-ruby/v2/api/Aws/SQS/Client.html[the AWS SDK documentation]. Example:
+
+[source,ruby]
+    input {
+      sqs {
+        access_key_id => "1234"
+        secret_access_key => "secret"
+        queue => "logstash-test-queue"
+        additional_settings => {
+          force_path_style => true
+          follow_redirects => false
+        }
+      }
+    }
+
+[id="plugins-{type}s-{plugin}-aws_credentials_file"]
+===== `aws_credentials_file` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Path to YAML file containing a hash of AWS credentials.
+This file will only be loaded if `access_key_id` and
+`secret_access_key` aren't set. The contents of the
+file should look like this:
+
+[source,ruby]
+----------------------------------
+    :access_key_id: "12345"
+    :secret_access_key: "54321"
+----------------------------------
+
+[id="plugins-{type}s-{plugin}-endpoint"]
+===== `endpoint`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The endpoint to connect to. By default it is constructed using the value of `region`.
+This is useful when connecting to S3 compatible services, but beware that these aren't
+guaranteed to work correctly with the AWS SDK.
+
+[id="plugins-{type}s-{plugin}-id_field"]
+===== `id_field` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Name of the event field in which to store the SQS message ID
+
+[id="plugins-{type}s-{plugin}-md5_field"]
+===== `md5_field` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Name of the event field in which to store the SQS message MD5 checksum
+
+[id="plugins-{type}s-{plugin}-polling_frequency"]
+===== `polling_frequency` 
+
+  * Value type is <<number,number>>
+  * Default value is `20`
+
+Polling frequency, default is 20 seconds
+
+[id="plugins-{type}s-{plugin}-proxy_uri"]
+===== `proxy_uri` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+URI to proxy server if required
+
+[id="plugins-{type}s-{plugin}-queue"]
+===== `queue` 
+
+  * This is a required setting.
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Name of the SQS Queue name to pull messages from. Note that this is just the name of the queue, not the URL or ARN.
+
+[id="plugins-{type}s-{plugin}-queue_owner_aws_account_id"]
+===== `queue_owner_aws_account_id` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+ID of the AWS account owning the queue if you want to use a https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-basic-examples-of-sqs-policies.html#grant-two-permissions-to-one-account[cross-account queue] with embedded policy. Note that AWS SDK only support numerical account ID and not account aliases.
+
+[id="plugins-{type}s-{plugin}-region"]
+===== `region` 
+
+  * Value type is <<string,string>>
+  * Default value is `"us-east-1"`
+
+The AWS Region
+
+[id="plugins-{type}s-{plugin}-role_arn"]
+===== `role_arn`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The AWS IAM Role to assume, if any.
+This is used to generate temporary credentials, typically for cross-account access.
+See the https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html[AssumeRole API documentation] for more information.
+
+[id="plugins-{type}s-{plugin}-role_session_name"]
+===== `role_session_name`
+
+  * Value type is <<string,string>>
+  * Default value is `"logstash"`
+
+Session name to use when assuming an IAM role.
+
+[id="plugins-{type}s-{plugin}-secret_access_key"]
+===== `secret_access_key` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The AWS Secret Access Key
+
+[id="plugins-{type}s-{plugin}-sent_timestamp_field"]
+===== `sent_timestamp_field` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Name of the event field in which to store the SQS message Sent Timestamp
+
+[id="plugins-{type}s-{plugin}-session_token"]
+===== `session_token` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The AWS Session token for temporary credential
+
+[id="plugins-{type}s-{plugin}-threads"]
+===== `threads` 
+
+  * Value type is <<number,number>>
+  * Default value is `1`
+
+
+
+
+
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]
+
+:default_codec!:

data/docs/output-cloudwatch.asciidoc

@@ -0,0 +1,321 @@
+// :integration: aws
+:plugin: cloudwatch
+:type: output
+:default_codec: plain
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== Cloudwatch output plugin
+
+// include::{include_path}/plugin_header-integration.asciidoc[]
+
+==== Description
+
+This output lets you aggregate and send metric data to AWS CloudWatch
+
+==== Summary:
+This plugin is intended to be used on a logstash indexer agent (but that
+is not the only way, see below.)  In the intended scenario, one cloudwatch
+output plugin is configured, on the logstash indexer node, with just AWS API
+credentials, and possibly a region and/or a namespace.  The output looks
+for fields present in events, and when it finds them, it uses them to
+calculate aggregate statistics.  If the `metricname` option is set in this
+output, then any events which pass through it will be aggregated & sent to
+CloudWatch, but that is not recommended.  The intended use is to NOT set the
+metricname option here, and instead to add a `CW_metricname` field (and other
+fields) to only the events you want sent to CloudWatch.
+
+When events pass through this output they are queued for background
+aggregation and sending, which happens every minute by default.  The
+queue has a maximum size, and when it is full aggregated statistics will be
+sent to CloudWatch ahead of schedule. Whenever this happens a warning
+message is written to logstash's log.  If you see this you should increase
+the `queue_size` configuration option to avoid the extra API calls.  The queue
+is emptied every time we send data to CloudWatch.
+
+Note: when logstash is stopped the queue is destroyed before it can be processed.
+This is a known limitation of logstash and will hopefully be addressed in a
+future version.
+
+==== Details:
+There are two ways to configure this plugin, and they can be used in
+combination: event fields & per-output defaults
+
+Event Field configuration...
+You add fields to your events in inputs & filters and this output reads
+those fields to aggregate events.  The names of the fields read are
+configurable via the `field_*` options.
+
+Per-output defaults...
+You set universal defaults in this output plugin's configuration, and
+if an event does not have a field for that option then the default is
+used.
+
+Notice, the event fields take precedence over the per-output defaults.
+
+At a minimum events must have a "metric name" to be sent to CloudWatch.
+This can be achieved either by providing a default here OR by adding a
+`CW_metricname` field. By default, if no other configuration is provided
+besides a metric name, then events will be counted (Unit: Count, Value: 1)
+by their metric name (either a default or from their `CW_metricname` field)
+
+Other fields which can be added to events to modify the behavior of this
+plugin are, `CW_namespace`, `CW_unit`, `CW_value`, and
+`CW_dimensions`.  All of these field names are configurable in
+this output.  You can also set per-output defaults for any of them.
+See below for details.
+
+Read more about http://aws.amazon.com/cloudwatch/[AWS CloudWatch],
+and the specific of API endpoint this output uses,
+http://docs.amazonwebservices.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html[PutMetricData]
+
+[id="plugins-{type}s-{plugin}-options"]
+==== Cloudwatch Output Configuration Options
+
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-access_key_id>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-aws_credentials_file>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-batch_size>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-dimensions>> |<<hash,hash>>|No
+| <<plugins-{type}s-{plugin}-field_dimensions>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-field_metricname>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-field_namespace>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-field_unit>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-field_value>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-metricname>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-namespace>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-proxy_uri>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-queue_size>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-region>> |<<string,string>>, one of `["us-east-1", "us-east-2", "us-west-1", "us-west-2", "eu-central-1", "eu-west-1", "eu-west-2", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "ap-northeast-2", "sa-east-1", "us-gov-west-1", "cn-north-1", "ap-south-1", "ca-central-1"]`|No
+| <<plugins-{type}s-{plugin}-secret_access_key>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-session_token>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-timeframe>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-unit>> |<<string,string>>, one of `["Seconds", "Microseconds", "Milliseconds", "Bytes", "Kilobytes", "Megabytes", "Gigabytes", "Terabytes", "Bits", "Kilobits", "Megabits", "Gigabits", "Terabits", "Percent", "Count", "Bytes/Second", "Kilobytes/Second", "Megabytes/Second", "Gigabytes/Second", "Terabytes/Second", "Bits/Second", "Kilobits/Second", "Megabits/Second", "Gigabits/Second", "Terabits/Second", "Count/Second", "None"]`|No
+| <<plugins-{type}s-{plugin}-value>> |<<string,string>>|No
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+output plugins.
+
+&nbsp;
+
+[id="plugins-{type}s-{plugin}-access_key_id"]
+===== `access_key_id` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:
+
+1. Static configuration, using `access_key_id` and `secret_access_key` params in logstash plugin config
+2. External credentials file specified by `aws_credentials_file`
+3. Environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
+4. Environment variables `AMAZON_ACCESS_KEY_ID` and `AMAZON_SECRET_ACCESS_KEY`
+5. IAM Instance Profile (available when running inside EC2)
+
+[id="plugins-{type}s-{plugin}-aws_credentials_file"]
+===== `aws_credentials_file` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Path to YAML file containing a hash of AWS credentials.
+This file will only be loaded if `access_key_id` and
+`secret_access_key` aren't set. The contents of the
+file should look like this:
+
+[source,ruby]
+----------------------------------
+    :access_key_id: "12345"
+    :secret_access_key: "54321"
+----------------------------------
+
+
+[id="plugins-{type}s-{plugin}-batch_size"]
+===== `batch_size` 
+
+  * Value type is <<number,number>>
+  * Default value is `20`
+
+How many data points can be given in one call to the CloudWatch API
+
+[id="plugins-{type}s-{plugin}-dimensions"]
+===== `dimensions` 
+
+  * Value type is <<hash,hash>>
+  * There is no default value for this setting.
+
+The default dimensions [ name, value, ... ] to use for events which do not have a `CW_dimensions` field
+
+[id="plugins-{type}s-{plugin}-field_dimensions"]
+===== `field_dimensions` 
+
+  * Value type is <<string,string>>
+  * Default value is `"CW_dimensions"`
+
+The name of the field used to set the dimensions on an event metric
+The field named here, if present in an event, must have an array of
+one or more key & value pairs, for example...
+    `add_field => [ "CW_dimensions", "Environment", "CW_dimensions", "prod" ]`
+or, equivalently...
+    `add_field => [ "CW_dimensions", "Environment" ]`
+    `add_field => [ "CW_dimensions", "prod" ]`
+
+[id="plugins-{type}s-{plugin}-field_metricname"]
+===== `field_metricname` 
+
+  * Value type is <<string,string>>
+  * Default value is `"CW_metricname"`
+
+The name of the field used to set the metric name on an event
+The author of this plugin recommends adding this field to events in inputs &
+filters rather than using the per-output default setting so that one output
+plugin on your logstash indexer can serve all events (which of course had
+fields set on your logstash shippers.)
+
+[id="plugins-{type}s-{plugin}-field_namespace"]
+===== `field_namespace` 
+
+  * Value type is <<string,string>>
+  * Default value is `"CW_namespace"`
+
+The name of the field used to set a different namespace per event
+Note: Only one namespace can be sent to CloudWatch per API call
+so setting different namespaces will increase the number of API calls
+and those cost money.
+
+[id="plugins-{type}s-{plugin}-field_unit"]
+===== `field_unit` 
+
+  * Value type is <<string,string>>
+  * Default value is `"CW_unit"`
+
+The name of the field used to set the unit on an event metric
+
+[id="plugins-{type}s-{plugin}-field_value"]
+===== `field_value` 
+
+  * Value type is <<string,string>>
+  * Default value is `"CW_value"`
+
+The name of the field used to set the value (float) on an event metric
+
+[id="plugins-{type}s-{plugin}-metricname"]
+===== `metricname` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The default metric name to use for events which do not have a `CW_metricname` field.
+Beware: If this is provided then all events which pass through this output will be aggregated and
+sent to CloudWatch, so use this carefully.  Furthermore, when providing this option, you
+will probably want to also restrict events from passing through this output using event
+type, tag, and field matching
+
+[id="plugins-{type}s-{plugin}-namespace"]
+===== `namespace` 
+
+  * Value type is <<string,string>>
+  * Default value is `"Logstash"`
+
+The default namespace to use for events which do not have a `CW_namespace` field
+
+[id="plugins-{type}s-{plugin}-proxy_uri"]
+===== `proxy_uri` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+URI to proxy server if required
+
+[id="plugins-{type}s-{plugin}-queue_size"]
+===== `queue_size` 
+
+  * Value type is <<number,number>>
+  * Default value is `10000`
+
+How many events to queue before forcing a call to the CloudWatch API ahead of `timeframe` schedule
+Set this to the number of events-per-timeframe you will be sending to CloudWatch to avoid extra API calls
+
+[id="plugins-{type}s-{plugin}-region"]
+===== `region` 
+
+  * Value can be any of: `us-east-1`, `us-east-2`, `us-west-1`, `us-west-2`, `eu-central-1`, `eu-west-1`, `eu-west-2`, `ap-southeast-1`, `ap-southeast-2`, `ap-northeast-1`, `ap-northeast-2`, `sa-east-1`, `us-gov-west-1`, `cn-north-1`, `ap-south-1`, `ca-central-1`
+  * Default value is `"us-east-1"`
+
+The AWS Region
+
+[id="plugins-{type}s-{plugin}-secret_access_key"]
+===== `secret_access_key` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The AWS Secret Access Key
+
+[id="plugins-{type}s-{plugin}-session_token"]
+===== `session_token` 
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The AWS Session token for temporary credential
+
+[id="plugins-{type}s-{plugin}-timeframe"]
+===== `timeframe` 
+
+  * Value type is <<string,string>>
+  * Default value is `"1m"`
+
+Constants
+aggregate_key members
+Units
+How often to send data to CloudWatch
+This does not affect the event timestamps, events will always have their
+actual timestamp (to-the-minute) sent to CloudWatch.
+
+We only call the API if there is data to send.
+
+See the Rufus Scheduler docs for an https://github.com/jmettraux/rufus-scheduler#the-time-strings-understood-by-rufus-scheduler[explanation of allowed values]
+
+[id="plugins-{type}s-{plugin}-unit"]
+===== `unit` 
+
+  * Value can be any of: `Seconds`, `Microseconds`, `Milliseconds`, `Bytes`, `Kilobytes`, `Megabytes`, `Gigabytes`, `Terabytes`, `Bits`, `Kilobits`, `Megabits`, `Gigabits`, `Terabits`, `Percent`, `Count`, `Bytes/Second`, `Kilobytes/Second`, `Megabytes/Second`, `Gigabytes/Second`, `Terabytes/Second`, `Bits/Second`, `Kilobits/Second`, `Megabits/Second`, `Gigabits/Second`, `Terabits/Second`, `Count/Second`, `None`
+  * Default value is `"Count"`
+
+The default unit to use for events which do not have a `CW_unit` field
+If you set this option you should probably set the "value" option along with it
+
+[id="plugins-{type}s-{plugin}-value"]
+===== `value` 
+
+  * Value type is <<string,string>>
+  * Default value is `"1"`
+
+The default value to use for events which do not have a `CW_value` field
+If provided, this must be a string which can be converted to a float, for example...
+    "1", "2.34", ".5", and "0.67"
+If you set this option you should probably set the `unit` option along with it
+
+
+
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]
+
+:default_codec!: