fluent-plugin-cloudwatch-logs 0.10.0 → 0.11.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 105e96fa516c4f972fc2b01f83c1daa30d39ff4b827b46b811b9f30514231516
4
- data.tar.gz: 85741a1b0ccad0f9e207c837e7417b786ef68d1fc1552cbe80b73030e5b4191b
3
+ metadata.gz: e50538afdcec35527089c0d1767dff70ddc7f2b5873523d596f78a9e7d344a88
4
+ data.tar.gz: c8da30c9f0286bca9dca9aab693224f2165f7fdced0ceb8de07db8fb8a259b3a
5
5
  SHA512:
6
- metadata.gz: 84a8458e9704102609b382b9cb8fe4c332a1c1b9c9562b89b959961bb3624958cbffc1f965eba2a502c10a0e45f1ceb427a987681d5420c35663bc6df88295ed
7
- data.tar.gz: d6e22ad22c4eef5cb6c8fab7690ca75585e7ad99756926181f50c43fef70df0e52e12722e14809b793947e56ed8199fc393a318f42308f05fd6d23d6aa9a4b13
6
+ metadata.gz: 4b623676f4b5baec31da710b5446a189d58f063a7fe2f1f3fe721d54a32bcb619c16b45416167a24a61d68f416b2202d13451f900e7038e084600924330ef9ed
7
+ data.tar.gz: 1519461663014d5234ec69088a1b14a4dd345006c4ec8eebdefa87612a15563a6f65c8effacfee5edd7d0c8672765769d5e08d35d761388ac2fde0ac9669a823
@@ -0,0 +1,12 @@
1
+ name: Autocloser
2
+ on: [issues]
3
+ jobs:
4
+ autoclose:
5
+ runs-on: ubuntu-latest
6
+ steps:
7
+ - name: Autoclose issues that did not follow issue template
8
+ uses: roots/issue-closer-action@v1.1
9
+ with:
10
+ repo-token: ${{ secrets.GITHUB_TOKEN }}
11
+ issue-close-message: "@${issue.user.login} this issue was automatically closed because it did not follow the issue template."
12
+ issue-pattern: "(.*Problem.*)|(.*Expected Behavior or What you need to ask.*)|(.*Using Fluentd and CloudWatchLogs plugin versions.*)"
data/README.md CHANGED
@@ -43,6 +43,46 @@ Create IAM user with a policy like the following:
43
43
  }
44
44
  ```
45
45
 
46
+ More restricted IAM policy for `out_cloudwatch_logs` is:
47
+
48
+ ```json
49
+ {
50
+ "Version": "2012-10-17",
51
+ "Statement": [
52
+ {
53
+ "Action": [
54
+ "logs:PutLogEvents",
55
+ "logs:CreateLogGroup",
56
+ "logs:PutRetentionPolicy",
57
+ "logs:CreateLogStream",
58
+ "logs:DescribeLogGroups",
59
+ "logs:DescribeLogStreams"
60
+ ],
61
+ "Effect": "Allow",
62
+ "Resource": "*"
63
+ }
64
+ ]
65
+ }
66
+ ```
67
+
68
+ Also, more restricted IAM policy for `in_cloudwatch_logs` is:
69
+
70
+ ```json
71
+ {
72
+ "Version": "2012-10-17",
73
+ "Statement": [
74
+ {
75
+ "Action": [
76
+ "logs:GetLogEvents",
77
+ "logs:DescribeLogStreams"
78
+ ],
79
+ "Effect": "Allow",
80
+ "Resource": "*"
81
+ }
82
+ ]
83
+ }
84
+ ```
85
+
46
86
  ## Authentication
47
87
 
48
88
  There are several methods to provide authentication credentials. Be aware that there are various tradeoffs for these methods,
@@ -120,6 +160,11 @@ Fetch sample log from CloudWatch Logs:
120
160
  #endpoint http://localhost:5000/
121
161
  #json_handler json
122
162
  #log_rejected_request true
163
+ #<web_identity_credentials>
164
+ # role_arn "#{ENV['AWS_ROLE_ARN']}"
165
+ # role_session_name ROLE_SESSION_NAME
166
+ # web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
167
+ #</web_identity_credentials>
123
168
  </match>
124
169
  ```
125
170
 
@@ -154,6 +199,14 @@ Fetch sample log from CloudWatch Logs:
154
199
  * `retention_in_days_key`: use specified field of records as retention period
155
200
  * `use_tag_as_group`: to use tag as a group name
156
201
  * `use_tag_as_stream`: to use tag as a stream name
202
+ * `<web_identity_credentials>`: For EKS authentication.
203
+ * `role_arn`: The Amazon Resource Name (ARN) of the role to assume. This parameter is required when using `<web_identity_credentials>`.
204
+ * `role_session_name`: An identifier for the assumed role session. This parameter is required when using `<web_identity_credentials>`.
205
+ * `web_identity_token_file`: The absolute path to the file on disk containing the OIDC token. This parameter is required when using `<web_identity_credentials>`.
206
+ * `policy`: An IAM policy in JSON format. (default `nil`)
207
+ * `duration_seconds`: The duration, in seconds, of the role session. The value can range from
208
+ 900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value
209
+ is set to 3600 seconds (1 hour). (default `nil`)
157
210
 
158
211
  **NOTE:** `retention_in_days` requests additional IAM permission `logs:PutRetentionPolicy` for log_group.
159
212
  Please refer to [the PutRetentionPolicy column in documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/permissions-reference-cwl.html) for details.
@@ -181,6 +234,11 @@ Please refer to [the PutRetentionPolicy column in documentation](https://docs.aw
181
234
  #<storage>
182
235
  # @type local # or redis, memcached, etc.
183
236
  #</storage>
237
+ #<web_identity_credentials>
238
+ # role_arn "#{ENV['AWS_ROLE_ARN']}"
239
+ # role_session_name ROLE_SESSION_NAME
240
+ # web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
241
+ #</web_identity_credentials>
184
242
  </source>
185
243
  ```
186
244
 
@@ -209,6 +267,14 @@ Please refer to [the PutRetentionPolicy column in documentation](https://docs.aw
209
267
  * `format`: specify CloudWatchLogs' log format. (default `nil`)
210
268
  * `<parse>`: specify parser plugin configuration. see also: https://docs.fluentd.org/v/1.0/parser#how-to-use
211
269
  * `<storage>`: specify storage plugin configuration. see also: https://docs.fluentd.org/v/1.0/storage#how-to-use
270
+ * `<web_identity_credentials>`: For EKS authentication.
271
+ * `role_arn`: The Amazon Resource Name (ARN) of the role to assume. This parameter is required when using `<web_identity_credentials>`.
272
+ * `role_session_name`: An identifier for the assumed role session. This parameter is required when using `<web_identity_credentials>`.
273
+ * `web_identity_token_file`: The absolute path to the file on disk containing the OIDC token. This parameter is required when using `<web_identity_credentials>`.
274
+ * `policy`: An IAM policy in JSON format. (default `nil`)
275
+ * `duration_seconds`: The duration, in seconds, of the role session. The value can range from
276
+ 900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value
277
+ is set to 3600 seconds (1 hour). (default `nil`)
212
278
 
213
279
  ## Test
214
280
 
@@ -2,7 +2,7 @@ module Fluent
2
2
  module Plugin
3
3
  module Cloudwatch
4
4
  module Logs
5
- VERSION = "0.10.0"
5
+ VERSION = "0.11.2"
6
6
  end
7
7
  end
8
8
  end
@@ -17,6 +17,7 @@ module Fluent::Plugin
17
17
  config_param :aws_use_sts, :bool, default: false
18
18
  config_param :aws_sts_role_arn, :string, default: nil
19
19
  config_param :aws_sts_session_name, :string, default: 'fluentd'
20
+ config_param :aws_sts_endpoint_url, :string, default: nil
20
21
  config_param :region, :string, default: nil
21
22
  config_param :endpoint, :string, default: nil
22
23
  config_param :tag, :string
@@ -35,6 +36,13 @@ module Fluent::Plugin
35
36
  config_param :time_range_format, :string, default: "%Y-%m-%d %H:%M:%S"
36
37
  config_param :throttling_retry_seconds, :time, default: nil
37
38
  config_param :include_metadata, :bool, default: false
39
+ config_section :web_identity_credentials, multi: false do
40
+ config_param :role_arn, :string
41
+ config_param :role_session_name, :string
42
+ config_param :web_identity_token_file, :string, default: nil #required
43
+ config_param :policy, :string, default: nil
44
+ config_param :duration_seconds, :time, default: nil
45
+ end
38
46
 
39
47
  config_section :parse do
40
48
  config_set_default :@type, 'none'
@@ -75,10 +83,29 @@ module Fluent::Plugin
75
83
 
76
84
  if @aws_use_sts
77
85
  Aws.config[:region] = options[:region]
78
- options[:credentials] = Aws::AssumeRoleCredentials.new(
86
+ credentials_options = {
79
87
  role_arn: @aws_sts_role_arn,
80
88
  role_session_name: @aws_sts_session_name
81
- )
89
+ }
90
+ credentials_options[:sts_endpoint_url] = @aws_sts_endpoint_url if @aws_sts_endpoint_url
91
+ if @region and @aws_sts_endpoint_url
92
+ credentials_options[:client] = Aws::STS::Client.new(:region => @region, endpoint: @aws_sts_endpoint_url)
93
+ elsif @region
94
+ credentials_options[:client] = Aws::STS::Client.new(:region => @region)
95
+ end
96
+ options[:credentials] = Aws::AssumeRoleCredentials.new(credentials_options)
97
+ elsif @web_identity_credentials
98
+ c = @web_identity_credentials
99
+ credentials_options = {}
100
+ credentials_options[:role_arn] = c.role_arn
101
+ credentials_options[:role_session_name] = c.role_session_name
102
+ credentials_options[:web_identity_token_file] = c.web_identity_token_file
103
+ credentials_options[:policy] = c.policy if c.policy
104
+ credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
105
+ if @region
106
+ credentials_options[:client] = Aws::STS::Client.new(:region => @region)
107
+ end
108
+ options[:credentials] = Aws::AssumeRoleWebIdentityCredentials.new(credentials_options)
82
109
  else
83
110
  options[:credentials] = Aws::Credentials.new(@aws_key_id, @aws_sec_key) if @aws_key_id && @aws_sec_key
84
111
  end
@@ -7,6 +7,8 @@ module Fluent::Plugin
7
7
  class CloudwatchLogsOutput < Output
8
8
  Fluent::Plugin.register_output('cloudwatch_logs', self)
9
9
 
10
+ class TooLargeEventError < Fluent::UnrecoverableError; end
11
+
10
12
  helpers :compat_parameters, :inject
11
13
 
12
14
  DEFAULT_BUFFER_TYPE = "memory"
@@ -17,6 +19,7 @@ module Fluent::Plugin
17
19
  config_param :aws_use_sts, :bool, default: false
18
20
  config_param :aws_sts_role_arn, :string, default: nil
19
21
  config_param :aws_sts_session_name, :string, default: 'fluentd'
22
+ config_param :aws_sts_endpoint_url, :string, default: nil
20
23
  config_param :region, :string, :default => nil
21
24
  config_param :endpoint, :string, :default => nil
22
25
  config_param :log_group_name, :string, :default => nil
@@ -44,6 +47,13 @@ module Fluent::Plugin
44
47
  config_param :remove_retention_in_days_key, :bool, default: false
45
48
  config_param :json_handler, :enum, list: [:yajl, :json], :default => :yajl
46
49
  config_param :log_rejected_request, :bool, :default => false
50
+ config_section :web_identity_credentials, multi: false do
51
+ config_param :role_arn, :string
52
+ config_param :role_session_name, :string
53
+ config_param :web_identity_token_file, :string, default: nil #required
54
+ config_param :policy, :string, default: nil
55
+ config_param :duration_seconds, :time, default: nil
56
+ end
47
57
 
48
58
  config_section :buffer do
49
59
  config_set_default :@type, DEFAULT_BUFFER_TYPE
@@ -92,10 +102,29 @@ module Fluent::Plugin
92
102
 
93
103
  if @aws_use_sts
94
104
  Aws.config[:region] = options[:region]
95
- options[:credentials] = Aws::AssumeRoleCredentials.new(
105
+ credentials_options = {
96
106
  role_arn: @aws_sts_role_arn,
97
107
  role_session_name: @aws_sts_session_name
98
- )
108
+ }
109
+ credentials_options[:sts_endpoint_url] = @aws_sts_endpoint_url if @aws_sts_endpoint_url
110
+ if @region and @aws_sts_endpoint_url
111
+ credentials_options[:client] = Aws::STS::Client.new(:region => @region, endpoint: @aws_sts_endpoint_url)
112
+ elsif @region
113
+ credentials_options[:client] = Aws::STS::Client.new(:region => @region)
114
+ end
115
+ options[:credentials] = Aws::AssumeRoleCredentials.new(credentials_options)
116
+ elsif @web_identity_credentials
117
+ c = @web_identity_credentials
118
+ credentials_options = {}
119
+ credentials_options[:role_arn] = c.role_arn
120
+ credentials_options[:role_session_name] = c.role_session_name
121
+ credentials_options[:web_identity_token_file] = c.web_identity_token_file
122
+ credentials_options[:policy] = c.policy if c.policy
123
+ credentials_options[:duration_seconds] = c.duration_seconds if c.duration_seconds
124
+ if @region
125
+ credentials_options[:client] = Aws::STS::Client.new(:region => @region)
126
+ end
127
+ options[:credentials] = Aws::AssumeRoleWebIdentityCredentials.new(credentials_options)
99
128
  else
100
129
  options[:credentials] = Aws::Credentials.new(@aws_key_id, @aws_sec_key) if @aws_key_id && @aws_sec_key
101
130
  end
@@ -130,6 +159,9 @@ module Fluent::Plugin
130
159
  def write(chunk)
131
160
  log_group_name = extract_placeholders(@log_group_name, chunk) if @log_group_name
132
161
  log_stream_name = extract_placeholders(@log_stream_name, chunk) if @log_stream_name
162
+ aws_tags = @log_group_aws_tags.each {|k, v|
163
+ @log_group_aws_tags[extract_placeholders(k, chunk)] = extract_placeholders(v, chunk)
164
+ } if @log_group_aws_tags
133
165
 
134
166
  queue = Thread::Queue.new
135
167
 
@@ -182,7 +214,7 @@ module Fluent::Plugin
182
214
  #as we create log group only once, values from first record will persist
183
215
  record = rs[0][2]
184
216
 
185
- awstags = @log_group_aws_tags
217
+ awstags = aws_tags
186
218
  unless @log_group_aws_tags_key.nil?
187
219
  if @remove_log_group_aws_tags_key
188
220
  awstags = record.delete(@log_group_aws_tags_key)
@@ -319,8 +351,7 @@ module Fluent::Plugin
319
351
  while event = events.shift
320
352
  event_bytesize = event[:message].bytesize + EVENT_HEADER_SIZE
321
353
  if MAX_EVENT_SIZE < event_bytesize
322
- log.warn "Log event in #{group_name} is discarded because it is too large: #{event_bytesize} bytes exceeds limit of #{MAX_EVENT_SIZE}"
323
- break
354
+ raise TooLargeEventError, "Log event in #{group_name} is discarded because it is too large: #{event_bytesize} bytes exceeds limit of #{MAX_EVENT_SIZE}"
324
355
  end
325
356
 
326
357
  new_chunk = chunk + [event]
@@ -399,7 +430,13 @@ module Fluent::Plugin
399
430
  raise err
400
431
  end
401
432
  rescue Aws::CloudWatchLogs::Errors::ThrottlingException => err
402
- if !@put_log_events_disable_retry_limit && @put_log_events_retry_limit < retry_count
433
+ if @put_log_events_retry_limit < 1
434
+ log.warn "failed to PutLogEvents and discard logs because put_log_events_retry_limit is less than 1", {
435
+ "error_class" => err.class.to_s,
436
+ "error" => err.message,
437
+ }
438
+ return
439
+ elsif !@put_log_events_disable_retry_limit && @put_log_events_retry_limit < retry_count
403
440
  log.error "failed to PutLogEvents and discard logs because retry count exceeded put_log_events_retry_limit", {
404
441
  "error_class" => err.class.to_s,
405
442
  "error" => err.message,
@@ -630,6 +630,8 @@ class CloudwatchLogsInputTest < Test::Unit::TestCase
630
630
  end
631
631
 
632
632
  test "emit with today's log stream" do
633
+ omit "This testcase is unstable in CI." if ENV["CI"] == "true"
634
+
633
635
  config = <<-CONFIG
634
636
  tag test
635
637
  @type cloudwatch_logs
@@ -492,6 +492,47 @@ class CloudwatchLogsOutputTest < Test::Unit::TestCase
492
492
  assert_equal("value2", awstags.fetch("tag2"))
493
493
  end
494
494
 
495
+ def test_log_group_aws_tags_with_placeholders
496
+ clear_log_group
497
+
498
+ config = {
499
+ "@type" => "cloudwatch_logs",
500
+ "auto_create_stream" => true,
501
+ "use_tag_as_stream" => true,
502
+ "log_group_name_key" => "group_name_key",
503
+ "log_group_aws_tags" => '{"tag1": "${tag}", "tag2": "${namespace_name}"}',
504
+ }
505
+ config.merge!(config_elementify(aws_key_id)) if aws_key_id
506
+ config.merge!(config_elementify(aws_sec_key)) if aws_sec_key
507
+ config.merge!(config_elementify(region)) if region
508
+ config.merge!(config_elementify(endpoint)) if endpoint
509
+
510
+ d = create_driver(
511
+ Fluent::Config::Element.new('ROOT', '', config, [
512
+ Fluent::Config::Element.new('buffer', 'tag, namespace_name', {
513
+ '@type' => 'memory',
514
+ }, [])
515
+ ])
516
+ )
517
+
518
+ records = [
519
+ {'cloudwatch' => 'logs1', 'message' => 'message1', 'group_name_key' => log_group_name, "namespace_name" => "fluentd"},
520
+ {'cloudwatch' => 'logs2', 'message' => 'message1', 'group_name_key' => log_group_name, "namespace_name" => "fluentd"},
521
+ {'cloudwatch' => 'logs3', 'message' => 'message1', 'group_name_key' => log_group_name, "namespace_name" => "fluentd"},
522
+ ]
523
+
524
+ time = Time.now
525
+ d.run(default_tag: fluentd_tag) do
526
+ records.each_with_index do |record, i|
527
+ d.feed(time.to_i + i, record)
528
+ end
529
+ end
530
+
531
+ awstags = get_log_group_tags
532
+ assert_equal(fluentd_tag, awstags.fetch("tag1"))
533
+ assert_equal("fluentd", awstags.fetch("tag2"))
534
+ end
535
+
495
536
  def test_retention_in_days
496
537
  clear_log_group
497
538
 
@@ -713,6 +754,32 @@ class CloudwatchLogsOutputTest < Test::Unit::TestCase
713
754
  assert_equal({'cloudwatch' => 'logs2', 'message' => 'message2'}, JSON.parse(events[1].message))
714
755
  end
715
756
 
757
+ def test_retrying_on_throttling_exception_with_put_log_events_retry_limit_as_zero
758
+ client = Aws::CloudWatchLogs::Client.new
759
+ @called = false
760
+ stub(client).put_log_events(anything) {
761
+ raise(Aws::CloudWatchLogs::Errors::ThrottlingException.new(nil, "error"))
762
+ }.once.ordered
763
+
764
+ d = create_driver(<<-EOC)
765
+ #{default_config}
766
+ log_group_name #{log_group_name}
767
+ log_stream_name #{log_stream_name}
768
+ @log_level debug
769
+ put_log_events_retry_limit 0
770
+ EOC
771
+ time = event_time
772
+ d.instance.instance_variable_set(:@logs, client)
773
+ d.run(default_tag: fluentd_tag) do
774
+ d.feed(time, {'message' => 'message1'})
775
+ end
776
+
777
+ logs = d.logs
778
+ assert_equal(0, logs.select {|l| l =~ /Called PutLogEvents API/ }.size)
779
+ assert_equal(1, logs.select {|l| l =~ /failed to PutLogEvents/ }.size)
780
+ assert_equal(0, logs.select {|l| l =~ /retry succeeded/ }.size)
781
+ end
782
+
716
783
  def test_retrying_on_throttling_exception
717
784
  resp = Object.new
718
785
  mock(resp).rejected_log_events_info {}
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-cloudwatch-logs
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.10.0
4
+ version: 0.11.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ryota Arai
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-06-23 00:00:00.000000000 Z
11
+ date: 2020-11-20 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd
@@ -108,13 +108,14 @@ dependencies:
108
108
  - - ">="
109
109
  - !ruby/object:Gem::Version
110
110
  version: '0'
111
- description:
111
+ description:
112
112
  email:
113
113
  - ryota.arai@gmail.com
114
114
  executables: []
115
115
  extensions: []
116
116
  extra_rdoc_files: []
117
117
  files:
118
+ - ".github/workflows/issue-auto-closer.yml"
118
119
  - ".gitignore"
119
120
  - ".travis.yml"
120
121
  - Gemfile
@@ -135,7 +136,7 @@ homepage: https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-log
135
136
  licenses:
136
137
  - MIT
137
138
  metadata: {}
138
- post_install_message:
139
+ post_install_message:
139
140
  rdoc_options: []
140
141
  require_paths:
141
142
  - lib
@@ -150,8 +151,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
150
151
  - !ruby/object:Gem::Version
151
152
  version: '0'
152
153
  requirements: []
153
- rubygems_version: 3.0.3
154
- signing_key:
154
+ rubygems_version: 3.1.2
155
+ signing_key:
155
156
  specification_version: 4
156
157
  summary: CloudWatch Logs Plugin for Fluentd
157
158
  test_files: