fluent-diagtool 1.0.2 → 1.0.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 44527fe77df5bcf3b4888b5a697f796460a69e3873561c88b278e318f5c83cd8
4
- data.tar.gz: 6052777c8a6930f4dde4bc1fa40bd3a62eba4541253488d419f21f02ab11568a
3
+ metadata.gz: 36f2a315c0601942f4d58cd452f14f070af79e67cbfdf3c87cdbf910678381c2
4
+ data.tar.gz: 960056d096bba41112ab9dc3d08a10a8af2693500d79fc8a6b4f4736ccb77d3c
5
5
  SHA512:
6
- metadata.gz: 1d03c3277e64b402db14aaa6f530c02ceb640ff57b53e170fc2633e63d23a53b4b8f604a292b4a958e74035ccb6dac4f02b782ee58e1ea579e916cf3ba050012
7
- data.tar.gz: b2d8da326cab4181b391758e9e7fa7cfb8682580726b0921afcc08a6fc8788db818bd325c8a24dcc8ee83975d69d058f306bc8d852b8417a2a4d9f3341e38fd0
6
+ metadata.gz: ca7a0047228b91ae650ba9db3fb2a9e63589b45a7594a52ab62f59d02110da68b4ec94f11a9e3a6b9f507aa12dba7bf03f1366ab63b7c169805bee0f4304771d
7
+ data.tar.gz: a01cc78efb7e024a56619d30fdc959439cb07deccd67dd6f0faa94c026f0a054f2640c2bd796f76012ac8a0887b7ebc6c7d766c997288f3a30f78c6df4061162
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- fluent-diagtool (1.0.1)
4
+ fluent-diagtool (1.0.3)
5
5
  fileutils
6
6
  json
7
7
 
@@ -9,8 +9,8 @@ GEM
9
9
  remote: https://rubygems.org/
10
10
  specs:
11
11
  diff-lcs (1.3)
12
- fileutils (1.4.1)
13
- json (2.3.1)
12
+ fileutils (1.7.2)
13
+ json (2.6.3)
14
14
  rake (12.3.3)
15
15
  rspec (3.9.0)
16
16
  rspec-core (~> 3.9.0)
data/README.md CHANGED
@@ -21,7 +21,8 @@ The scope of data collection:
21
21
  <br>
22
22
 
23
23
  ## Prerequisite
24
- Diagtool has been developed for Fluentd(td-agent) and FluentBit(td-agent-bit) running on Linux OS and Diatool does not work for Windows.
24
+ Diagtool has been developed for Fluentd(td-agent, fluent-package) and FluentBit(td-agent-bit) running on Linux OS, mainly.
25
+ On Windows, it only supports the `installed td-agent-gem list` collection for Fluentd, currently (since v1.0.3).
25
26
  Diagtool is written in Ruby and Ruby version should be higher than 2.3 for the installation.
26
27
  The supported Linux OS is described in the following page:
27
28
  https://docs.fluentd.org/quickstart/td-agent-v2-vs-v3-vs-v4
@@ -31,9 +32,9 @@ https://docs.fluentd.org/quickstart/td-agent-v2-vs-v3-vs-v4
31
32
  When you are using td-agent, you can install Diagtool easily with "/usr/sbin/td-agent-gem" command.
32
33
  ```
33
34
  # /usr/sbin/td-agent-gem install fluent-diagtool
34
- Successfully installed fluent-diagtool-1.0.0
35
- Parsing documentation for fluent-diagtool-1.0.0
36
- Installing ri documentation for fluent-diagtool-1.0.0
35
+ Successfully installed fluent-diagtool-1.0.3
36
+ Parsing documentation for fluent-diagtool-1.0.3
37
+ Installing ri documentation for fluent-diagtool-1.0.3
37
38
  Done installing documentation for fluent-diagtool after 0 seconds
38
39
  1 gem installed
39
40
  ```
@@ -42,9 +43,9 @@ When using /usr/sbin/td-agent-gem command, fluent-diagtool is installed under "/
42
43
  Otherwise, you can install Diagtool with common gem command. In this case, Ruby version higher than 2.3 might be required to install.
43
44
  ```
44
45
  # gem install fluent-diagtool
45
- Successfully installed fluent-diagtool-1.0.0
46
- Parsing documentation for fluent-diagtool-1.0.0
47
- Installing ri documentation for fluent-diagtool-1.0.0
46
+ Successfully installed fluent-diagtool-1.0.3
47
+ Parsing documentation for fluent-diagtool-1.0.3
48
+ Installing ri documentation for fluent-diagtool-1.0.3
48
49
  Done installing documentation for fluent-diagtool after 0 seconds
49
50
  1 gem installed
50
51
  ```
@@ -64,7 +65,13 @@ Usage: fluent-diagtool -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {lis
64
65
  -c, --conf config_file provide a full path of td-agent configuration file (Optional : Default=None)
65
66
  -l, --log log_file provide a full path of td-agent log file (Optional : Default=None)
66
67
  ```
68
+
69
+ On Windows, only the `-o, --output DIR` option is supported.
70
+
67
71
  ### Precheck
72
+
73
+ (Not supported on Windows)
74
+
68
75
  In order to run Diagtool correctly, it is required to ensure that Diagtool can obtain the fundamental information of Fluentd. Basically, Diagtool automatically parses the required information from the running Fluentd processes. The precheck option is useful to confirm if Diagtool certainly collects the information as expected.
69
76
  The following output example shows the case where Diatool properly collects the required information.
70
77
 
@@ -162,6 +169,21 @@ Once the pre-check is completed, you are ready to run the tool. The “-o” is
162
169
  2020-10-07 21:29:30 +0000: [Diagtool] [INFO] [Collect] Generate tar file /tmp/diagout-20201007212928.tar.gz
163
170
  ```
164
171
 
172
+ fluent-package (td-agent) on Windows: Fluent Package Command Prompt (Td-agent Command Prompt) with Administrator privilege
173
+
174
+ ```
175
+ $ fluent-diagtool -o /opt
176
+ 2023-11-21 12:46:08 +0900: [Diagtool] [INFO] Parsing command options...
177
+ 2023-11-21 12:46:08 +0900: [Diagtool] [INFO] Option : Output directory = /opt
178
+ 2023-11-21 12:46:08 +0900: [Diagtool] [INFO] Initializing parameters...
179
+ 2023-11-21 12:46:08 +0900: [Diagtool] [INFO] [Collect] Collecting fluent-package gem information...
180
+ 2023-11-21 12:46:10 +0900: [Diagtool] [INFO] [Collect] fluent-package gem information is stored in /opt/20231121124608/output/tdgem_list.output
181
+ 2023-11-21 12:46:10 +0900: [Diagtool] [INFO] [Collect] fluent-package gem information (bundled by default) is stored in /opt/20231121124608/output/gem_bundled_list.output
182
+ 2023-11-21 12:46:10 +0900: [Diagtool] [INFO] [Collect] fluent-package manually installed gem information is stored in /opt/20231121124608/output/gem_local_list.output
183
+ 2023-11-21 12:46:10 +0900: [Diagtool] [INFO] [Collect] fluent-package manually installed gems:
184
+ 2023-11-21 12:46:10 +0900: [Diagtool] [INFO] [Collect] * fluent-plugin-forest
185
+ ```
186
+
165
187
  #### The "@include" directive in td-agent configuration file
166
188
  The "@include" directive is a function to reuse configuration defined in other configuration files. Diagtool reads Fluentd configuration and gathers the files described in "@include" directive as well. The details of "@include" directive are described in followed page:
167
189
  https://docs.fluentd.org/configuration/config-file#6-re-use-your-config-the-include-directive
@@ -203,6 +225,8 @@ The diagtool provides a hash-seed option with '-s'. When hash-seed is specified,
203
225
  ```
204
226
 
205
227
  ## Tested Environment
206
- - OS : CentOS 8.1 / Ubuntu 20.04
228
+ - OS : CentOS 8.1 / Ubuntu 20.04 / Windows Home 10
207
229
  - Fluentd : td-agent version 3/4
208
- https://docs.fluentd.org/quickstart/td-agent-v2-vs-v3
230
+ https://docs.fluentd.org/quickstart/td-agent-v2-vs-v3-vs-v4
231
+ - Fluentd : fluent-package version 5
232
+ https://docs.fluentd.org/quickstart/fluent-package-v5-vs-td-agent
data/exe/fluent-diagtool CHANGED
@@ -17,9 +17,6 @@
17
17
  #
18
18
 
19
19
  require 'optparse'
20
- require 'fluent/diagtool/collectutils'
21
- require 'fluent/diagtool/maskutils'
22
- require 'fluent/diagtool/validutils'
23
20
  require 'fluent/diagtool/diagutils'
24
21
  include Diagtool
25
22
 
@@ -19,9 +19,12 @@ require 'open3'
19
19
  require 'logger'
20
20
  require 'net/http'
21
21
  require 'uri'
22
+ require 'fluent/diagtool/windows/collectutils' if Diagtool.windows?
22
23
 
23
24
  module Diagtool
24
25
  class CollectUtils
26
+ prepend Windows::PlatformSpecificCollectUtils if Diagtool.windows?
27
+
25
28
  def initialize(conf, log_level)
26
29
  @logger = Logger.new(STDOUT, level: log_level, formatter: proc {|severity, datetime, progname, msg|
27
30
  "#{datetime}: [Collectutils] [#{severity}] #{msg}\n"
@@ -34,8 +37,9 @@ module Diagtool
34
37
  @outdir = conf[:outdir]
35
38
  @tdenv = {
36
39
  'FLUENT_CONF' => '',
37
- 'TD_AGENT_LOG_FILE' => ''
38
- }
40
+ 'TD_AGENT_LOG_FILE' => '',
41
+ 'FLUENT_PACKAGE_LOG_FILE' => '',
42
+ }
39
43
  @package_name = conf[:package_name]
40
44
  @service_name = conf[:service_name]
41
45
 
@@ -403,16 +407,68 @@ module Diagtool
403
407
  return output
404
408
  end
405
409
 
410
+ def fluent_package?
411
+ @package_name == "fluent-package"
412
+ end
413
+
414
+ def match_platform?(platforms_option)
415
+ if Diagtool.windows?
416
+ platforms_option == "windows_platforms"
417
+ else
418
+ platforms_option == "not_windows_platforms"
419
+ end
420
+ end
421
+
422
+ def collect_bundled_plugins
423
+ gemfile_path = if fluent_package?
424
+ "/opt/fluent/share/Gemfile"
425
+ else
426
+ "/opt/td-agent/share/Gemfile"
427
+ end
428
+ File.read(gemfile_path).lines(chomp: true).grep(/\Agem "fluent-plugin-/).collect do |line|
429
+ gem_name = line[/\Agem "(fluent-plugin-.+?)"/, 1]
430
+ platforms_option = line[/platforms: (\w+)/, 1]
431
+ next gem_name, platforms_option
432
+ end.select do |gem_name, platforms_option|
433
+ platforms_option.nil? || match_platform?(platforms_option)
434
+ end.collect do |gem_name, platforms_option|
435
+ gem_name
436
+ end
437
+ end
438
+
439
+ def collect_manually_installed_gems(gemlist_path)
440
+ installed_gems = File.read(gemlist_path).lines(chomp: true).collect do |line|
441
+ line.split.first
442
+ end
443
+ bundled_plugins = collect_bundled_plugins
444
+ bundled_gem_path = File.join(@outdir, "gem_bundled_list.output")
445
+ File.open(bundled_gem_path, 'w') do |f|
446
+ f.puts(bundled_plugins.join("\n"))
447
+ end
448
+ local_gem_path = File.join(@outdir, "gem_local_list.output")
449
+ # the folloing gems are not collected as bundled plugins
450
+ masked_gems = ["fluentd", "fluent-config-regexp-type", "fluent-logger", "fluent-diagtool"]
451
+ File.open(local_gem_path, 'w') do |f|
452
+ f.puts((installed_gems - bundled_plugins - masked_gems).join("\n"))
453
+ end
454
+ return {:bundled => bundled_gem_path, :local => local_gem_path}
455
+ end
456
+
406
457
  def collect_tdgems()
407
458
  output = @outdir+'/tdgem_list.output'
408
- command = if @package_name == "fluent-package"
459
+ command = if fluent_package?
409
460
  "fluent-gem"
410
461
  else
411
462
  "td-agent-gem"
412
463
  end
413
- stdout, stderr, status = Open3.capture3("#{command} list | grep fluent")
464
+ stdout, stderr, status = Open3.capture3("#{command} list")
465
+ gems = stdout.each_line(chomp: true).select do |line|
466
+ line.include?("fluent")
467
+ end.collect do |line|
468
+ line.split.first
469
+ end
414
470
  File.open(output, 'w') do |f|
415
- f.puts(stdout)
471
+ f.puts(gems.join("\n"))
416
472
  end
417
473
  return output
418
474
  end
@@ -14,15 +14,31 @@
14
14
  # limitations under the License.
15
15
  #
16
16
 
17
+ module Diagtool
18
+ ON_WINDOWS = /mingw/.match?(RUBY_PLATFORM)
19
+ def self.windows?
20
+ ON_WINDOWS
21
+ end
22
+ end
23
+
17
24
  require 'logger'
18
25
  require 'fileutils'
19
26
  require 'fluent/diagtool/collectutils'
20
27
  require 'fluent/diagtool/maskutils'
21
28
  require 'fluent/diagtool/validutils'
29
+ require 'fluent/diagtool/windows/diagutils' if Diagtool.windows?
22
30
  include Diagtool
23
31
 
24
32
  module Diagtool
25
33
  class DiagUtils
34
+ # TODO: Consider making the logic of this class more abstract and
35
+ # cutting out the unix-specific logic into a separate module as well.
36
+ # (Currently, very limited features are supported for Windows.
37
+ # In order to reduce impact on the existing logic for Unix-like.
38
+ # only Windows-specific logic is separated into the module, for now.
39
+ # In the future, the implementation of this class should be more abstract.)
40
+ prepend Windows::PlatformSpecificDiagUtils if Diagtool.windows?
41
+
26
42
  def initialize(params)
27
43
  time = Time.new
28
44
  @time_format = time.strftime("%Y%m%d%0k%M%0S")
@@ -127,6 +143,17 @@ module Diagtool
127
143
  tdgem = c.collect_tdgems()
128
144
  diaglogger_info("[Collect] #{@conf[:package_name]} gem information is stored in #{tdgem}")
129
145
 
146
+ gem_info = c.collect_manually_installed_gems(tdgem)
147
+ diaglogger_info("[Collect] #{@conf[:package_name]} gem information (bundled by default) is stored in #{gem_info[:bundled]}")
148
+ diaglogger_info("[Collect] #{@conf[:package_name]} manually installed gem information is stored in #{gem_info[:local]}")
149
+ local_gems = File.read(gem_info[:local]).lines(chomp: true)
150
+ unless local_gems == [""]
151
+ diaglogger_info("[Collect] #{@conf[:package_name]} manually installed gems:")
152
+ local_gems.each do |gem|
153
+ diaglogger_info("[Collect] * #{gem}")
154
+ end
155
+ end
156
+
130
157
  diaglogger_info("[Collect] Collecting config file of OS log...")
131
158
  oslog = c.collect_oslog()
132
159
  if @conf[:mask] == 'yes'
@@ -1,5 +1,5 @@
1
1
  module Fluent
2
2
  module Diagtool
3
- VERSION = "1.0.2"
3
+ VERSION = "1.0.3"
4
4
  end
5
5
  end
@@ -0,0 +1,34 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ module Diagtool
18
+ module Windows
19
+ module PlatformSpecificCollectUtils
20
+ def _find_fluentd_info()
21
+ # Not supported yet.
22
+ end
23
+
24
+ def _find_fluentbit_info()
25
+ # Not supported yet.
26
+ end
27
+
28
+ def _find_os_info()
29
+ # Not supported yet.
30
+ return {}
31
+ end
32
+ end
33
+ end
34
+ end
@@ -0,0 +1,97 @@
1
+ #
2
+ # Fluentd
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+
17
+ require 'logger'
18
+ require 'fileutils'
19
+ require 'fluent/diagtool/collectutils'
20
+
21
+ module Diagtool
22
+ module Windows
23
+ module PlatformSpecificDiagUtils
24
+ def run_precheck()
25
+ raise "[Precheck] Precheck feature is not supported on Windows."
26
+ end
27
+
28
+ def run_diagtool()
29
+ @conf[:time] = @time_format
30
+ @conf[:workdir] = @conf[:basedir] + '/' + @time_format
31
+ @conf[:outdir] = @conf[:workdir] + '/output'
32
+ FileUtils.mkdir_p(@conf[:workdir])
33
+ FileUtils.mkdir_p(@conf[:outdir])
34
+ diaglog = @conf[:workdir] + '/diagtool.output'
35
+
36
+ @logger = Logger.new(STDOUT, formatter: proc {|severity, datetime, progname, msg|
37
+ "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
38
+ })
39
+ @logger_file = Logger.new(diaglog, formatter: proc {|severity, datetime, progname, msg|
40
+ "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
41
+ })
42
+ diaglogger_info("Parsing command options...")
43
+ diaglogger_info(" Option : Output directory = #{@conf[:basedir]}")
44
+
45
+ loglevel = 'WARN'
46
+ diaglogger_info("Initializing parameters...")
47
+ c = CollectUtils.new(@conf, loglevel)
48
+
49
+ diaglogger_info("[Collect] Collecting #{@conf[:package_name]} gem information...")
50
+ tdgem = c.collect_tdgems()
51
+ diaglogger_info("[Collect] #{@conf[:package_name]} gem information is stored in #{tdgem}")
52
+
53
+ gem_info = c.collect_manually_installed_gems(tdgem)
54
+ diaglogger_info("[Collect] #{@conf[:package_name]} gem information (bundled by default) is stored in #{gem_info[:bundled]}")
55
+ diaglogger_info("[Collect] #{@conf[:package_name]} manually installed gem information is stored in #{gem_info[:local]}")
56
+ local_gems = File.read(gem_info[:local]).lines(chomp: true)
57
+ unless local_gems == [""]
58
+ diaglogger_info("[Collect] #{@conf[:package_name]} manually installed gems:")
59
+ local_gems.each do |gem|
60
+ diaglogger_info("[Collect] * #{gem}")
61
+ end
62
+ end
63
+ end
64
+
65
+ def parse_diagconf(params)
66
+ options = {
67
+ :precheck => '', :basedir => '', :type =>'', :mask => '', :words => [], :wfile => '', :seed => '', :tdconf =>'', :tdlog => ''
68
+ }
69
+
70
+ supported_options = [:type, :output]
71
+
72
+ unless params[:type] == nil || params[:type] == 'fluentd'
73
+ raise "fluentd type '-t' only supports 'fluentd' on Windows."
74
+ end
75
+ options[:type] = 'fluentd'
76
+
77
+ if params[:output] != nil
78
+ if Dir.exist?(params[:output])
79
+ options[:basedir] = params[:output]
80
+ else
81
+ raise "output directory '#{params[:output]}' does not exist"
82
+ end
83
+ else
84
+ raise "output directory '-o' must be specified"
85
+ end
86
+
87
+ params.keys.each do |option|
88
+ unless supported_options.include?(option)
89
+ raise "#{option} is not supported on Windows."
90
+ end
91
+ end
92
+
93
+ return options
94
+ end
95
+ end
96
+ end
97
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-diagtool
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.2
4
+ version: 1.0.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - kubotat
8
- autorequire:
8
+ autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2023-08-30 00:00:00.000000000 Z
11
+ date: 2023-11-22 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fileutils
@@ -66,11 +66,13 @@ files:
66
66
  - lib/fluent/diagtool/maskutils.rb
67
67
  - lib/fluent/diagtool/validutils.rb
68
68
  - lib/fluent/diagtool/version.rb
69
+ - lib/fluent/diagtool/windows/collectutils.rb
70
+ - lib/fluent/diagtool/windows/diagutils.rb
69
71
  homepage: https://github.com/fluent/diagtool/tree/dev
70
72
  licenses:
71
73
  - Apache-2.0
72
74
  metadata: {}
73
- post_install_message:
75
+ post_install_message:
74
76
  rdoc_options: []
75
77
  require_paths:
76
78
  - lib
@@ -85,8 +87,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
85
87
  - !ruby/object:Gem::Version
86
88
  version: '0'
87
89
  requirements: []
88
- rubygems_version: 3.4.10
89
- signing_key:
90
+ rubygems_version: 3.3.26
91
+ signing_key:
90
92
  specification_version: 4
91
93
  summary: Diagnostic Tool for Fluentd
92
94
  test_files: []