fluent-plugin-cloudwatch-ingest 0.2.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 679ae50f9e4e67e437ebffc139bd0809f7a4d06f
4
- data.tar.gz: 2d99a123ed73c8b739e40194d4d0337361177494
3
+ metadata.gz: cb6e47ba22bacca8648c6f4f607a5c225d7336b4
4
+ data.tar.gz: 198af146c4862e1f75837cdf35f5721958a3482b
5
5
  SHA512:
6
- metadata.gz: 940a0fe5c366ad41d233c3284da8eb0ba5098628e46cb85d44038e5f305d344460e4c0dc7d05ae5f2b929c03839521cdab1de2a92d07c896170a133c46c38e9c
7
- data.tar.gz: 83d7c29387185268964db5d50db307dce7953017650992b96b995abf807a9e2c22fc5740ba9da612ab3b8f17636a48be72088f13c0f1d721de2ec279e99ac15e
6
+ metadata.gz: 2a072d31bb053ae4b869246552efd4c546e6b191995d4efa1b4a091b90e282d5da7b7cceac2cd7a239da1a387dc64b92e3266c88d52b9560f75b5760df35f2c0
7
+ data.tar.gz: 55b8fdfbc69b1908a364115ae305a01c3ba51d2d25330ee2dc344a9e2faa17640039eff945308bf750f993636d49084adc7640012e346b852af2b06ff533d2ff
data/README.md CHANGED
@@ -66,6 +66,57 @@ When the state file is located on a shared filesystem an exclusive write lock wi
66
66
  As such it is safe to run multiple instances of this plugin consuming from the same CloudWatch logging source without fear of duplication, as long as they share a state file.
67
67
  In a properly configured auto-scaling group this provides for uninterrupted log ingestion in the event of a failure of any single node.
68
68
 
69
+ ### Sub-second timestamps
70
+ When using `event_time true` the `@timestamp` field for the record is taken from the time recorded against the event by Cloudwatch. This is the most common mode to run in as it's an easy path to normalization: all of your Lambdas or other AWS service need not have the same, valid, `time_format` nor a regex that matches every case.
71
+
72
+ If your output plugin supports sub-second precision (and you're running fluentd 0.14.x) you'll "enjoy" sub-second precision.
73
+
74
+ #### Elasticsearch
75
+ It is a common pattern to use fluentd alongside the [fluentd-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch) plugin, either directly or via [fluent-plugin-aws-elasticsearch-service](https://github.com/atomita/fluent-plugin-aws-elasticsearch-service), to ingest logs into Elasticsearch.
76
+
77
+ At present there is a bug within this plugin that, via an unwise cast, causes records without a named timestamp field to be cast to `DateTime`, losing the precision. This PR: https://github.com/uken/fluent-plugin-elasticsearch/pull/249 fixes that issue. If you need this functionality then I would urge you to comment and express interest over there.
78
+
79
+ Failing that I maintain my own fork of that repository with the fix in place: https://github.com/sampointer/fluent-plugin-elasticsearch/tree/add_configurable_time_precision_when_timestamp_missing
80
+
81
+ ### IAM
82
+ IAM is a tricky and often bespoke subject. Here's a starter that will ingest all of the logs for all of your Lambdas in the account in which the plugin is running:
83
+
84
+ ```json
85
+ {
86
+ "Version": "2012-10-17",
87
+ "Statement": [
88
+ {
89
+ "Effect": "Allow",
90
+ "Action": [
91
+ "logs:DescribeLogGroups",
92
+ "logs:DescribeLogStreams",
93
+ "logs:DescribeMetricFilters",
94
+ "logs:FilterLogEvents",
95
+ "logs:GetLogEvents"
96
+ ],
97
+ "Resource": [
98
+ "arn:aws:logs:eu-west-1:123456789012:log-group:/aws/lambda/*:*"
99
+ ]
100
+ },
101
+ {
102
+ "Effect": "Allow",
103
+ "Action": [
104
+ "logs:DescribeLogGroups",
105
+ ],
106
+ "Resource": [
107
+ "arn:aws:logs:eu-west-1:123456789012:log-group:*:*"
108
+ ]
109
+ }
110
+ ]
111
+ }
112
+ ```
113
+
114
+ ### Cross-account authentication
115
+ Is a tricky subject that probably cannot be described here. Broadly speaking the IAM instance role of the host on which the plugin is running
116
+ needs to be able to `sts:AssumeRole` the `sts_arn` (and obviously needs `sts_enabled` to be true).
117
+
118
+ The assumed role should look more-or-less like that above in terms of the actions and resource combinations required.
119
+
69
120
  ## Development
70
121
 
71
122
  After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -9,8 +9,8 @@ Gem::Specification.new do |spec|
9
9
  spec.authors = ['Sam Pointer']
10
10
  spec.email = ['san@outsidethe.net']
11
11
 
12
- spec.summary = 'Fluentd plugin to ingest cloudwatch logs'
13
- spec.description = 'Alternative to ryotarai/fluent-plugin-cloudwatch-logs'
12
+ spec.summary = 'Fluentd plugin to ingest AWS Cloudwatch logs'
13
+ spec.description = 'Fluentd plugin to ingest AWS Cloudwatch logs'
14
14
  spec.homepage = 'https://github.com/sampointer/fluent-plugin-cloudwatch-ingest'
15
15
 
16
16
  # Prevent pushing this gem to RubyGems.org by setting 'allowed_push_host', or
@@ -2,7 +2,7 @@ module Fluent
2
2
  module Plugin
3
3
  module Cloudwatch
4
4
  module Ingest
5
- VERSION = '0.2.0'.freeze
5
+ VERSION = '0.2.1'.freeze
6
6
  end
7
7
  end
8
8
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-cloudwatch-ingest
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.2.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sam Pointer
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2017-04-05 00:00:00.000000000 Z
11
+ date: 2017-04-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -94,7 +94,7 @@ dependencies:
94
94
  - - "~>"
95
95
  - !ruby/object:Gem::Version
96
96
  version: 2.8.4
97
- description: Alternative to ryotarai/fluent-plugin-cloudwatch-logs
97
+ description: Fluentd plugin to ingest AWS Cloudwatch logs
98
98
  email:
99
99
  - san@outsidethe.net
100
100
  executables: []
@@ -139,8 +139,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
139
139
  version: '0'
140
140
  requirements: []
141
141
  rubyforge_project:
142
- rubygems_version: 2.2.5
142
+ rubygems_version: 2.6.11
143
143
  signing_key:
144
144
  specification_version: 4
145
- summary: Fluentd plugin to ingest cloudwatch logs
145
+ summary: Fluentd plugin to ingest AWS Cloudwatch logs
146
146
  test_files: []