fluent-plugin-cloudwatch-ingest 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +51 -0
- data/fluent-plugin-cloudwatch-ingest.gemspec +2 -2
- data/lib/fluent/plugin/cloudwatch/ingest/version.rb +1 -1
- metadata +5 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: cb6e47ba22bacca8648c6f4f607a5c225d7336b4
|
|
4
|
+
data.tar.gz: 198af146c4862e1f75837cdf35f5721958a3482b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2a072d31bb053ae4b869246552efd4c546e6b191995d4efa1b4a091b90e282d5da7b7cceac2cd7a239da1a387dc64b92e3266c88d52b9560f75b5760df35f2c0
|
|
7
|
+
data.tar.gz: 55b8fdfbc69b1908a364115ae305a01c3ba51d2d25330ee2dc344a9e2faa17640039eff945308bf750f993636d49084adc7640012e346b852af2b06ff533d2ff
|
data/README.md
CHANGED
|
@@ -66,6 +66,57 @@ When the state file is located on a shared filesystem an exclusive write lock wi
|
|
|
66
66
|
As such it is safe to run multiple instances of this plugin consuming from the same CloudWatch logging source without fear of duplication, as long as they share a state file.
|
|
67
67
|
In a properly configured auto-scaling group this provides for uninterrupted log ingestion in the event of a failure of any single node.
|
|
68
68
|
|
|
69
|
+
### Sub-second timestamps
|
|
70
|
+
When using `event_time true` the `@timestamp` field for the record is taken from the time recorded against the event by Cloudwatch. This is the most common mode to run in as it's an easy path to normalization: all of your Lambdas or other AWS service need not have the same, valid, `time_format` nor a regex that matches every case.
|
|
71
|
+
|
|
72
|
+
If your output plugin supports sub-second precision (and you're running fluentd 0.14.x) you'll "enjoy" sub-second precision.
|
|
73
|
+
|
|
74
|
+
#### Elasticsearch
|
|
75
|
+
It is a common pattern to use fluentd alongside the [fluentd-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch) plugin, either directly or via [fluent-plugin-aws-elasticsearch-service](https://github.com/atomita/fluent-plugin-aws-elasticsearch-service), to ingest logs into Elasticsearch.
|
|
76
|
+
|
|
77
|
+
At present there is a bug within this plugin that, via an unwise cast, causes records without a named timestamp field to be cast to `DateTime`, losing the precision. This PR: https://github.com/uken/fluent-plugin-elasticsearch/pull/249 fixes that issue. If you need this functionality then I would urge you to comment and express interest over there.
|
|
78
|
+
|
|
79
|
+
Failing that I maintain my own fork of that repository with the fix in place: https://github.com/sampointer/fluent-plugin-elasticsearch/tree/add_configurable_time_precision_when_timestamp_missing
|
|
80
|
+
|
|
81
|
+
### IAM
|
|
82
|
+
IAM is a tricky and often bespoke subject. Here's a starter that will ingest all of the logs for all of your Lambdas in the account in which the plugin is running:
|
|
83
|
+
|
|
84
|
+
```json
|
|
85
|
+
{
|
|
86
|
+
"Version": "2012-10-17",
|
|
87
|
+
"Statement": [
|
|
88
|
+
{
|
|
89
|
+
"Effect": "Allow",
|
|
90
|
+
"Action": [
|
|
91
|
+
"logs:DescribeLogGroups",
|
|
92
|
+
"logs:DescribeLogStreams",
|
|
93
|
+
"logs:DescribeMetricFilters",
|
|
94
|
+
"logs:FilterLogEvents",
|
|
95
|
+
"logs:GetLogEvents"
|
|
96
|
+
],
|
|
97
|
+
"Resource": [
|
|
98
|
+
"arn:aws:logs:eu-west-1:123456789012:log-group:/aws/lambda/*:*"
|
|
99
|
+
]
|
|
100
|
+
},
|
|
101
|
+
{
|
|
102
|
+
"Effect": "Allow",
|
|
103
|
+
"Action": [
|
|
104
|
+
"logs:DescribeLogGroups",
|
|
105
|
+
],
|
|
106
|
+
"Resource": [
|
|
107
|
+
"arn:aws:logs:eu-west-1:123456789012:log-group:*:*"
|
|
108
|
+
]
|
|
109
|
+
}
|
|
110
|
+
]
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### Cross-account authentication
|
|
115
|
+
Is a tricky subject that probably cannot be described here. Broadly speaking the IAM instance role of the host on which the plugin is running
|
|
116
|
+
needs to be able to `sts:AssumeRole` the `sts_arn` (and obviously needs `sts_enabled` to be true).
|
|
117
|
+
|
|
118
|
+
The assumed role should look more-or-less like that above in terms of the actions and resource combinations required.
|
|
119
|
+
|
|
69
120
|
## Development
|
|
70
121
|
|
|
71
122
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
|
@@ -9,8 +9,8 @@ Gem::Specification.new do |spec|
|
|
|
9
9
|
spec.authors = ['Sam Pointer']
|
|
10
10
|
spec.email = ['san@outsidethe.net']
|
|
11
11
|
|
|
12
|
-
spec.summary = 'Fluentd plugin to ingest
|
|
13
|
-
spec.description = '
|
|
12
|
+
spec.summary = 'Fluentd plugin to ingest AWS Cloudwatch logs'
|
|
13
|
+
spec.description = 'Fluentd plugin to ingest AWS Cloudwatch logs'
|
|
14
14
|
spec.homepage = 'https://github.com/sampointer/fluent-plugin-cloudwatch-ingest'
|
|
15
15
|
|
|
16
16
|
# Prevent pushing this gem to RubyGems.org by setting 'allowed_push_host', or
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fluent-plugin-cloudwatch-ingest
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Sam Pointer
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2017-04-
|
|
11
|
+
date: 2017-04-07 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: bundler
|
|
@@ -94,7 +94,7 @@ dependencies:
|
|
|
94
94
|
- - "~>"
|
|
95
95
|
- !ruby/object:Gem::Version
|
|
96
96
|
version: 2.8.4
|
|
97
|
-
description:
|
|
97
|
+
description: Fluentd plugin to ingest AWS Cloudwatch logs
|
|
98
98
|
email:
|
|
99
99
|
- san@outsidethe.net
|
|
100
100
|
executables: []
|
|
@@ -139,8 +139,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
139
139
|
version: '0'
|
|
140
140
|
requirements: []
|
|
141
141
|
rubyforge_project:
|
|
142
|
-
rubygems_version: 2.
|
|
142
|
+
rubygems_version: 2.6.11
|
|
143
143
|
signing_key:
|
|
144
144
|
specification_version: 4
|
|
145
|
-
summary: Fluentd plugin to ingest
|
|
145
|
+
summary: Fluentd plugin to ingest AWS Cloudwatch logs
|
|
146
146
|
test_files: []
|