fluent-diagtool 0.1.9 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e5be26bf333469e8c3c4eed5be22a33d05f8a37306d5e93067c10134a9415cb3
-  data.tar.gz: 128d6aa7ff697b5fcae2669a8e14dffb96f52ab360d4585313942343432a89b8
+  metadata.gz: 8c852c088190fa51d4232c45aa7afd4013473c8089e194511a0910a8ca4794e5
+  data.tar.gz: 96b031de365ad1d47c71b7ffe25dfbd53f4909fe07be484944f7814cfead4939
 SHA512:
-  metadata.gz: 4165a154d754ac8d7d338bd4adaa681059579c6f7c7410badefdb34d9ab260211e5807a54dcd602bb3f765a8265aa40a2ff8b95d5ecb82c5ccb8931be198c6a3
-  data.tar.gz: c0f8916a1399438aa50a15deb90f59321e5e02fc5902c04d04ed1ac98701f8f1bc109de5fa79ff9167169a9e848310f7988a6a521af95bb74dade68ddbdbc4e0
+  metadata.gz: c6c96dd92c4c8db2975c84395288817b5053b3cdb4b0897970a5e253fc2714886baf0799dd5bd4161439e043c4827a65c178adefde241e2e974de53265bebef8
+  data.tar.gz: e7d24f7ca1d450e70bc037c74bf1ff8a88e2d174cc77c04211ba164bf7e7e97961b0d2970c8e3e00e0ab57f357c6c13aa8640afc1ffa8b9d7b63d5867630c9e4

data/README.md

@@ -1,6 +1,6 @@
 # Fluentd Diagnostic Tool
 
-The diagtool enable users to automate the date collection which is required for trouble shooting. The data collected by diagtool include the configuration and log files of the td-agent and diagnostic information of operating system such as network and memory status and stats. In some cases, configuration and log files contains the security sensitive information, such as IP addresses and Hostname. The diagtool also provides the functions to generate mask on IP addresses, Hostname(in FQDN style) and user defined keywords described in the collected data.   
+The diagtool enables users to automate the date collection which is required for troubleshooting. The data collected by diagtool include the configuration and log files of the td-agent and diagnostic information from an operating system such as network and memory status and stats. In some cases, configuration and log files contain security sensitive information, such as IP addresses and Hostname. The diagtool also provides the functions to generate masks on IP addresses, Hostname(in FQDN style) and user defined keywords described in the collected data.   
 The scope of data collection:  
 - TD Agent information
   - configuration files (*)
@@ -15,7 +15,7 @@ The scope of data collection:
     - maximum number of file descriptor(ulimit -n)
     - kernel network parameters(sysctl)
   - snapshot of current process(ps)
-  - network conectivity status/stats(netstat -plan/netstat -s)
+  - network connectivity status/stats(netstat -plan/netstat -s)
   - memory information(/proc/meminfo)
 <br>   
 
@@ -33,16 +33,28 @@ Successfully installed fileutils-1.0.2
 Fetching: json-2.1.0.gem (100%)
 Building native extensions. This could take a while...
 Successfully installed json-2.1.0
-Fetching: fluent-diagtool-0.1.2.gem (100%)
-Successfully installed fluent-diagtool-0.1.2
+Fetching: fluent-diagtool-1.0.0.gem (100%)
+Successfully installed fluent-diagtool-1.0.0
 3 gems installed
 ```
+When you are using td-agent, fluent-adiagtool should be installed using /usr/sbin/td-agent-gem command instead of gem command.
+```
+# /usr/sbin/td-agent-gem install fluent-diagtool
+Fetching fluent-diagtool-1.0.0.gem
+Successfully installed fluent-diagtool-1.0.0
+Parsing documentation for fluent-diagtool-1.0.0
+Installing ri documentation for fluent-diagtool-1.0.0
+Done installing documentation for fluent-diagtool after 0 seconds
+1 gem installed
+```
 
 ## Usage
+There are a few options in Diagtool. You can check the options of Diagtool with "--help" options. Diagtool performs the validation function in the process by default but you can turn on/off the mask function depending on the use cases.
 ```
 # diagtool --help
 Usage: /usr/local/bin/diagtool -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}
         --precheck                   Run Precheck (Optional)
+    -t, --type fluentd|fluentbit     Select the type of Fluentd (Mandatory)
     -o, --output DIR                 Output directory (Mandatory)
     -m, --mask yes|no                Enable mask function (Optional : Default=no)
     -w, --word-list word1,word2      Provide a list of user-defined words which will to be masked (Optional : Default=None)
@@ -52,10 +64,11 @@ Usage: /usr/local/bin/diagtool -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]}
     -l, --log log_file               provide a full path of td-agent log file (Optional : Default=None)
 ```
 ### Pre-check
-The diagtool automatically extract the path of td-agent configuration and log files from td-agent daemon and use them during data collection if the td-agent is managed as daemon. The precheck options provides the function to confirm if the diagtool could gather the td-agent information as expected. 
+The diagtool automatically parses the path of Fluentd configuration and log files from running Fluentd processes and daemon. The precheck options provides the function to confirm if the diagtool could gather the fluentd information as expected. 
 The following command output shows the case when the diagtool successfully gather information from daemon.
+You need to specify the type of Fluentd, "fluentd" or "fluentbit".
 ```
-# diagtool --precheck
+# diagtool --precheck -t fluentd
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck] Check OS parameters...
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    operating system = CentOS Linux 8 (Core)
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    kernel version = Linux 4.18.0-147.el8.x86_64
@@ -66,10 +79,10 @@ The following command output shows the case when the diagtool successfully gathe
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    td-agent log = td-agent.log
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck] Precheck completed. You can run diagtool command without -c and -l options
 ```
-In some cases, users do not manage td-agent as daemon but use own script to run td-agent with command line options. In that cases, users need to speccify the path of td-agent configuration and log files with -c and -l options respectively.  
+In some cases, users do not manage td-agent as daemon but use their own scripts to run td-agent with command line options. In that cases, users need to specify the path of td-agent configuration and log files with -c and -l options respectively.  
 The following example shows the precheck results when the diagtool is not able to extract the path of td-agent configuration and log files.
 ```
-# diagtool --precheck
+# diagtool --precheck -t fluentd
 2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck] Check OS parameters...
 2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    operating system = CentOS Linux 8 (Core)
 2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    kernel version = Linux 4.18.0-147.5.1.el8_1.x86_64
@@ -85,7 +98,7 @@ The following example shows the precheck results when the diagtool is not able t
 ### Run diagtool
 
 #### The "@include" directive in td-agent configuration file
-The "@include" directive is a function to reuse configuration defined in another configuration files. The diagtool read the td-agent configuration and collect the files described in "@include" directive as well. The details of "@include" directive are described in followed url:  
+The "@include" directive is a function to reuse configuration defined in other configuration files. The diagtool reads the td-agent configuration and collects the files described in "@include" directive as well. The details of "@include" directive are described in followed url:  
 https://docs.fluentd.org/configuration/config-file#6-re-use-your-config-the-include-directive
 
 #### User defined words to be masked 
@@ -100,7 +113,7 @@ NOTE: When user specified the keywork, only the exact match words will be masked
 
 #### Command sample:
 ```
-# diagtool -o /tmp/work1 -w passwd1,passwd2 -f word_list_sample -m yes
+# diagtool -t fluentd -o /tmp/work1 -w passwd1,passwd2 -f word_list_sample -m yes
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO] Parsing command options...
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO]    Option : Output directory = /tmp/work1
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO]    Option : Mask = yes
@@ -157,8 +170,8 @@ NOTE: When user specified the keywork, only the exact match words will be masked
 2020-05-12 18:21:22 -0400: [Diagtool] [INFO] [Collect] Generate tar file /tmp/work1/diagout-20200512182119.tar.gz
 ```
 #### Mask Function
-When run diagtool with mask option, the log of mask is also created in 'mask_{timestamp}.json' file. Users are able to confirm how the mask was generated on each files.  
-The diagtool provides hash-seed option with '-s'. When hash-seed is specified, the mask will be generated with original word and hash-seed so that users could use unique mask value.
+When run diagtool with the mask option, the log of mask is also created in 'mask_{timestamp}.json' file. Users are able to confirm how the mask was generated on each file.  
+The diagtool provides a hash-seed option with '-s'. When hash-seed is specified, the mask will be generated with the original word and hash-seed so that users could use a unique mask value.
 #### Mask sample - IP address: IPv4_{md5hash}
 ```
     "Line112-8": {
@@ -184,7 +197,7 @@ The diagtool provides hash-seed option with '-s'. When hash-seed is specified, t
 
 ## Tested Environment
 - OS : CentOS 8.1
-- Fluentd : td-agent version 3  
+- Fluentd : td-agent version 3/4 
   https://docs.fluentd.org/quickstart/td-agent-v2-vs-v3
-
+- Fluentbit : td-agent-bit
 

data/exe/{diagtool → fluent-diagtool}

@@ -27,6 +27,7 @@ params = {}
 OptionParser.new do |opt|
   opt.banner = "Usage: #{$0} -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}"
   opt.on('--precheck', 'Run Precheck (Optional)')
+  opt.on('-t','--type fluentd|fluentbit', String, 'Select the type of Fluentd (Mandatory)')
   opt.on('-o','--output DIR', String, 'Output directory (Mandatory)')
   opt.on('-m','--mask yes|no', String, 'Enable mask function (Optional : Default=no)')
   opt.on('-w','--word-list word1,word2', Array, 'Provide a list of user-defined words which will to be masked (Optional : Default=None)')

data/lib/fluent/diagtool/collectutils.rb

@@ -17,6 +17,8 @@
 require 'fileutils'
 require 'open3'
 require 'logger'
+require 'net/http'
+require 'uri'
 
 module Diagtool
   class CollectUtils
@@ -25,18 +27,29 @@ module Diagtool
         "#{datetime}: [Collectutils] [#{severity}] #{msg}\n"
       })
       @precheck = conf[:precheck]
+      @type = conf[:type]
       @time_format = conf[:time]
       @basedir = conf[:basedir]
       @workdir = conf[:workdir]
-      @outdir = conf[:outdir]			
-       
-      @tdenv = gen_tdenv()
+      @outdir = conf[:outdir]
+      @tdenv = {
+        'FLUENT_CONF' => '',
+        'TD_AGENT_LOG_FILE' => ''
+      }			
+      
+      case @type
+      when 'fluentd'
+        _find_fluentd_info()
+      when 'fluentbit'
+        _find_fluentbit_info()
+      end
+
       if not conf[:tdconf].empty?
-	@tdconf = conf[:tdconf].split('/')[-1]
+        @tdconf = conf[:tdconf].split('/')[-1]
         @tdconf_path = conf[:tdconf].gsub(@tdconf,'')
       elsif
-	if not @tdenv['FLUENT_CONF'].empty?
-      	  @tdconf = @tdenv['FLUENT_CONF'].split('/')[-1]
+        if not @tdenv['FLUENT_CONF'].empty?
+          @tdconf = @tdenv['FLUENT_CONF'].split('/')[-1]
       	  @tdconf_path = @tdenv['FLUENT_CONF'].gsub(@tdconf,'')
 	else
 	  raise "The path of td-agent configuration file need to be specified."  if conf[:precheck] == false
@@ -50,15 +63,20 @@ module Diagtool
           @tdlog =  @tdenv['TD_AGENT_LOG_FILE'].split('/')[-1]
           @tdlog_path = @tdenv['TD_AGENT_LOG_FILE'].gsub(@tdlog,'')
         else
-          raise "The path of td-agent log file need to be specified." if conf[:precheck] == false
-	end
+          case @type
+          when 'fluentd'
+            raise "The path of td-agent log file need to be specified." if conf[:precheck] == false
+          when 'fluentbit'
+            @logger.warn("FluentBit logs are redirected to the standard output interface ")
+          end
+	      end
       end 
-      @osenv = gen_osenv()
+      @osenv = _find_os_info()
       @oslog_path = '/var/log/'
       @oslog = 'messages'
       @syslog = 'syslog'
       @sysctl_path = '/etc/'
-      @sysctl = 'sysctl.conf'        
+      @sysctl = 'sysctl.conf'
 
       @logger.info("Loading the environment parameters...")
       @logger.info("    operating system = #{@osenv['Operating System']}")
@@ -69,7 +87,7 @@ module Diagtool
       @logger.info("    td-agent log = #{@tdlog}")
     end
     
-    def gen_osenv()
+    def _find_os_info()
       stdout, stderr, status = Open3.capture3('hostnamectl')
       os_dict = {}
       stdout.each_line { |l|
@@ -84,50 +102,102 @@ module Diagtool
       return os_dict
     end
     
-    def gen_tdenv()
+    def _find_fluentd_info()
+      ### check if the td-agent is run as daemon
       stdout, stderr, status = Open3.capture3('systemctl cat td-agent')
-      env_dict = {}
       if status.success?
-	if @precheck == false  # SKip if precheck is true
+        if @precheck == false  # SKip if precheck is true
           File.open(@outdir+'/td-agent_env.output', 'w') do |f|
             f.puts(stdout)
           end
-	end  
+        end  
         stdout.split().each do | l |
           if l.include?('Environment')
-            env_dict[l.split('=')[1]] = l.split('=')[2]
+            @tdenv[l.split('=')[1]] = l.split('=')[2]
           end
-      	end
+        end
       else
-        exe = 'fluentd'
-        stdout, stderr, status = Open3.capture3("ps aux | grep #{exe} | grep -v grep")
-        line = stdout.split(/\n/)
-	log_path = ''
-        conf_path = ''
-        line.each { |l|
-          cmd = l.split.drop(10)
-          i = 0
-          log_pos = 0
-          conf_pos = 0
-          if cmd[-1] != '--under-supervisor'
-            cmd.each { |c|
-              if c.include?("--log") || c.include?("-l")
-                log_pos = i + 1
-                log_path = cmd[log_pos]
-              elsif c.include?("--conf") || c.include?("-c")
-                conf_pos = i + 1
-                conf_path = cmd[conf_pos]
+        ### check if the td-agent is not run as daemon or run Fluentd with customized script
+        stdout, stderr, status = Open3.capture3('ps aux | grep fluentd | grep -v ".*\(grep\|diagtool\)"')
+        if status.success?
+          line = stdout.split(/\n/)
+          line.each do |l|
+            cmd = l.split.drop(10)
+            i = 0
+            if cmd[-1] != '--under-supervisor'
+              cmd.each do |c|
+                case
+                when c == "-c"
+                  @tdenv['FLUENT_CONF'] = cmd[i+1]
+                when c == "-l"
+                  @tdenv['TD_AGENT_LOG_FILE'] = cmd[i+1]
+                when c.include?("--conf")
+                  @tdenv['FLUENT_CONF'] = c.split("=")[1]
+                when c.include?("--log")
+                  @tdenv['TD_AGENT_LOG_FILE'] = c.split("=")[1]
+                end
+                i+=1
               end
-              i+=1
-            }
-	  end
-	}
-        env_dict['FLUENT_CONF'] = conf_path
-        env_dict['TD_AGENT_LOG_FILE'] = log_path
+            end
+          end
+        else
+          @logger.warn("No Fluentd daemon or proccess running") 
+        end
       end
-      return env_dict
     end
     
+    def _find_fluentbit_info()
+      ### check if the td-agent-bit is run as daemon
+      stdout, stderr, status = Open3.capture3('systemctl cat td-agent-bit')
+      if status.success?
+        if @precheck == false  # SKip if precheck is true
+          File.open(@outdir+'/td-agent-bit_env.output', 'w') do |f|
+            f.puts(stdout)
+          end
+        end
+        stdout.split(/\n/).each do | line |
+          if line.start_with?("ExecStart")
+            cmd = line.split("=")[1]
+            i =0
+            cmd.split().each do | c |
+              case
+              when c == "-c"
+                @tdenv['FLUENT_CONF'] = cmd.split()[i+1]
+              when c == "-l"
+                @tdenv['TD_AGENT_LOG_FILE'] = cmd.split()[i+1]
+              when c.include?("--conf")
+                @tdenv['FLUENT_CONF'] = c.split("=")[1]
+              when c.include?("--log")
+                @tdenv['TD_AGENT_LOG_FILE'] = c.split("=")[1]
+              end
+              i+=1
+            end
+          end
+        end
+      else
+        ### check if the td-agent-bit is not run as daemon or run FluentdBit with customized script 
+        stdout, stderr, status = Open3.capture3('ps aux | grep fluent-bit | grep -v ".*\(grep\|diagtool\)"')
+        if status.success?
+          i = 0
+          stdout.split().each do | line |
+            case
+            when line.include?("--conf")
+              @tdenv['FLUENT_CONF'] = line.split("=")[1]
+            when line.include?("--log")
+              @tdenv['TD_AGENT_LOG_FILE'] = line.split("=")[1]
+            when line == "-c"
+              @tdenv['FLUENT_CONF'] = stdout.split()[i+1]
+            when line == "-l"
+              @tdenv['TD_AGENT_LOG_FILE'] = stdout.split()[i+1]
+            end
+            i+=1
+          end
+        else
+          @logger.warn("No FluentBit daemon or proccess running")
+        end
+      end
+    end
+
     def export_env()
       env = {
         :os => @osenv['Operating System'],
@@ -146,39 +216,134 @@ module Diagtool
       FileUtils.cp(@tdconf_path+@tdconf, target_dir)
       conf = @workdir+@tdconf_path+@tdconf
       conf_list = []
-      conf_list.push target_dir + @tdconf
-      File.readlines(conf).each { |line|
-      if line.include? '@include'
-        f = line.split()[1]
-        if f.start_with?(/\//)  # /tmp/work1/b.conf
-          if f.include?('*')
-            Dir.glob(f).each { |ff|
-              conf_inc = target_dir + ff.gsub(/\//,'__')
-              FileUtils.cp(ff, conf_inc)
-              conf_list.push conf_inc
-             }
-	  else 
-	    conf_inc = target_dir+f.gsub(/\//,'__')
-            FileUtils.cp(f, conf_inc)
-            conf_list.push  conf_inc
-	  end
-        else
-	  f = f.gsub('./','') if f.include?('./')
-          if f.include?('*')
-            Dir.glob(@tdconf_path+f).each{ |ff|
-              conf_inc = target_dir + ff.gsub(@tdconf_path,'').gsub(/\//,'__')
-              FileUtils.cp(ff, conf_inc)
-              conf_list.push conf_inc
-            }
-	  else
-            conf_inc = target_dir+f.gsub(/\//,'__')
-            FileUtils.cp(@tdconf_path+f, conf_inc)
-            conf_list.push  conf_inc
-	  end
+      conf_list.push conf
+      case @type
+      when 'fluentd'
+        conf_list = conf_list + _collect_tdconf_include(conf)
+      when 'fluentbit'
+        conf_list = conf_list + _collect_tdconf_include(conf) + _collect_tdbit_parser(conf) + _collect_tdbit_plugins(conf) 
+      end   
+      return conf_list
+    end
+
+    def _collect_tdconf_include(conf)
+      target_dir = @workdir+@tdconf_path
+      inc_list = []
+      File.readlines(conf).each do |line|
+        if line.start_with?('@include')
+          l = line.split()[1]
+          if l.start_with?('http')
+            uri = URI(l)
+            inc_http = target_dir + 'http' + uri.path.gsub('/','_')
+            File.open(inc_http, 'w') do |f|
+              f.puts(Net::HTTP.get(uri))
+            end
+            inc_list.push inc_http
+          else
+            if l.start_with?(/\//)  # /tmp/work1/b.conf
+              if l.include?('*')
+                Dir.glob(l).each { |ll|
+                  inc_conf = target_dir + ll.gsub(/\//,'-')
+                  FileUtils.cp(ll, inc_conf)
+                  inc_list.push inc_conf
+                }
+              else 
+                inc_conf = target_dir+l.gsub(/\//,'-')
+                FileUtils.cp(l, inc_conf)
+                inc_list.push inc_conf
+              end
+            else
+              l = l.gsub('./','') if l.include?('./')
+              if l.include?('*')
+                Dir.glob(@tdconf_path+f).each{ |ll|
+                  inc_conf = target_dir + ll.gsub(@tdconf_path,'').gsub(/\//,'-')
+                  FileUtils.cp(ll, inc_conf)
+                  inc_list.push inc_conf
+                }
+              else
+                inc_conf = target_dir+l.gsub(/\//,'-')
+                FileUtils.cp(@tdconf_path+l, inc_conf)
+                inc_list.push inc_conf
+              end
+            end
+          end
+        end
+      end
+      return inc_list
+    end
+
+    def _collect_tdbit_parser(conf)
+      target_dir = @workdir+@tdconf_path
+      parser_conf = []
+      File.readlines(conf).each do |line|
+        if line.strip.start_with?('parsers_file') || line.strip.start_with?('Parsers_File')
+          l = line.split()[1]
+          if l.start_with?(/\//)  # /tmp/work1/b.conf
+            if l.include?('*')
+              Dir.glob(l).each { |ll|
+                pconf = target_dir + ll.gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                parser_conf.push(pconf)
+              }
+            else 
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(l, pconf)
+              parser_conf.push(pconf)
+            end
+          else
+            l = l.gsub('./','') if l.include?('./')
+            if l.include?('*')
+              Dir.glob(@tdconf_path+f).each{ |ll|
+                pconf = target_dir + ll.gsub(@tdconf_path,'').gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                parser_conf.push(pconf)
+              }
+            else
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(@tdconf_path+l, pconf)
+              parser_conf.push(pconf)
+            end
+          end
+        end
+      end  
+      return parser_conf
+    end
+
+    def _collect_tdbit_plugins(conf)
+      target_dir = @workdir+@tdconf_path
+      plugins_conf = []
+      File.readlines(conf).each do |line|
+        if line.strip.start_with?('plugins_file') || line.strip.start_with?('Plugins_File')
+          l = line.split()[1]
+          if l.start_with?(/\//)  # /tmp/work1/b.conf
+            if l.include?('*')
+              Dir.glob(l).each { |ll|
+                pconf = target_dir + ll.gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                plugins_conf.push(pconf)
+              }
+            else 
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(l, pconf)
+              plugins_conf.push(pconf)
+            end
+          else
+            l = l.gsub('./','') if l.include?('./')
+            if l.include?('*')
+              Dir.glob(@tdconf_path+f).each{ |ll|
+                pconf = target_dir + ll.gsub(@tdconf_path,'').gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                plugins_conf.push(pconf)
+              }
+            else
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(@tdconf_path+l, pconf)
+              plugins_conf.push(pconf)
+            end
+          end
         end
       end
-     }
-     return conf_list
+      return plugins_conf
     end
 
     def collect_tdlog()
@@ -200,20 +365,35 @@ module Diagtool
         FileUtils.cp(@oslog_path+@syslog, target_dir)
         return target_dir+@syslog
       else
-	@logger.warn("Can not find OS log file in #{oslog} or #{syslog}")
+        @logger.warn("Can not find OS log file in #{oslog} or #{syslog}")
+      end
+    end
+
+    def collect_ntp(command)
+      output = @outdir+'/ntp_info.output'
+      stdout_date, stderr_date, status_date = Open3.capture3("date")
+      stdout_ntp, stderr_ntp, status_ntp = Open3.capture3("chronyc sources") if command == "chrony"
+      stdout_ntp, stderr_ntp, status_ntp = Open3.capture3("ntpq -p") if command == "ntp"
+      File.open(output, 'w') do |f|
+        f.puts(stdout_date)
+        f.puts(stdout_ntp)
       end
     end
 
     def collect_cmd_output(cmd)
-      cmd_name = cmd.gsub(/\s/,'_').gsub(/\//,'-').gsub(',','_')
-      output = @outdir+'/'+cmd_name+'.txt'
-      stdout, stderr, status = Open3.capture3(cmd)
-      if status.success?
-	File.open(output, 'w') do |f|
-          f.puts(stdout)
+      if system(cmd + '> /dev/null 2>&1')
+        cmd_name = cmd.gsub(/\s/,'_').gsub(/\//,'-').gsub(',','_')
+        output = @outdir+'/'+cmd_name+'.txt'
+        stdout, stderr, status = Open3.capture3(cmd)
+        if status.success?
+          File.open(output, 'w') do |f|
+            f.puts(stdout)
+          end
+        else
+          @logger.warn("Command #{cmd} failed due to the following message -  #{stderr.chomp}")
         end
       else
-        @logger.warn("Command #{cmd} failed due to the following message -  #{stderr.chomp}")
+        @logger.warn("Command #{cmd} does not exist -  skip collecting #{cmd} output")
       end
       return output
     end

data/lib/fluent/diagtool/diagutils.rb

@@ -27,11 +27,11 @@ module Diagtool
       time = Time.new
       @time_format = time.strftime("%Y%m%d%0k%M%0S")
       @conf = parse_diagconf(params)
-      @cmd_list = [
-      	"ps -eo pid,ppid,stime,time,%mem,%cpu,cmd",
-	"cat /proc/meminfo",
-	"netstat -plan",
-	"netstat -s",
+      @cmd_list = [ 
+        "ps -eo pid,ppid,stime,time,%mem,%cpu,cmd",
+        "cat /proc/meminfo",
+	      "netstat -plan",
+	      "netstat -s",
       ]
     end
     
@@ -42,6 +42,7 @@ module Diagtool
       loglevel = 'WARN'
       c = CollectUtils.new(@conf, loglevel)
       c_env = c.export_env()
+      prechecklog.info("[Precheck] Fluentd Type = #{@conf[:type]}")
       prechecklog.info("[Precheck] Check OS parameters...")
       prechecklog.info("[Precheck]    operating system = #{c_env[:os]}")
       prechecklog.info("[Precheck]    kernel version = #{c_env[:kernel]}")
@@ -51,13 +52,13 @@ module Diagtool
       prechecklog.info("[Precheck]    td-agent log path = #{c_env[:tdlog_path]}")
       prechecklog.info("[Precheck]    td-agent log = #{c_env[:tdlog]}")
       if c_env[:tdconf_path] == nil || c_env[:tdconf] == nil
-	prechecklog.warn("[Precheck]    can not find td-agent conf path: please run diagtool command with -c /path/to/<td-agent conf file>")
+        prechecklog.warn("[Precheck]    can not find td-agent conf path: please run diagtool command with -c /path/to/<td-agent conf file>")
       end
       if c_env[:tdlog_path] == nil || c_env[:tdlog] == nil
         prechecklog.warn("[Precheck]    can not find td-agent log path: please run diagtool command with -l /path/to/<td-agent log file>")
       end
       if c_env[:tdconf_path] != nil && c_env[:tdconf] != nil && c_env[:tdlog_path] != nil && c_env[:tdlog] != nil
-	 prechecklog.info("[Precheck] Precheck completed. You can run diagtool command without -c and -l options")
+        prechecklog.info("[Precheck] Precheck completed. You can run diagtool command without -c and -l options")
       end
     end
 
@@ -97,8 +98,19 @@ module Diagtool
       v = ValidUtils.new(loglevel)
 							
       diaglogger_info("[Collect] Collecting log files of td-agent...")
-      tdlog = c.collect_tdlog()
-      diaglogger_info("[Collect] log files of td-agent are stored in #{tdlog}")
+      case @type
+      when 'fluentd'
+        tdlog = c.collect_tdlog()
+        diaglogger_info("[Collect] log files of td-agent are stored in #{tdlog}")
+      when 'fleuntbit'
+        if tdlog.empty?
+          diaglogger_info("FluentBit logs are redirected to the standard output interface ")
+          tdlog = ''
+        else
+          tdlog = c.collect_tdlog()
+          diaglogger_info("[Collect] log files of td-agent are stored in #{tdlog}")
+        end
+      end
 
       diaglogger_info("[Collect] Collecting config file of td-agent...")
       tdconf = c.collect_tdconf()
@@ -119,10 +131,10 @@ module Diagtool
       diaglogger_info("[Collect] Collecting date/time information...")
       if system('which chronyc > /dev/null 2>&1')
         ntp = c.collect_cmd_output(command="chronyc sources")
-	diaglogger_info("[Collect] date/time information is stored in #{ntp}")
+        diaglogger_info("[Collect] date/time information is stored in #{ntp}")
       elsif system('which ntpq > /dev/null 2>&1')
-        ntp = c.collect_ntp(command="ntpq -p")
-	diaglogger_info("[Collect] date/time information is stored in #{ntp}")
+        ntp = c.collect_cmd_output(command="ntpq -p")
+        diaglogger_info("[Collect] date/time information is stored in #{ntp}")
       else
         diaglogger_warn("[Collect] chrony/ntp does not exist. skip collectig date/time information")
       end
@@ -131,13 +143,15 @@ module Diagtool
       #  Correct OS information
       ###
       @cmd_list.each { |cmd|
-	diaglogger_info("[Collect] Collecting command output : command = #{cmd}")
-	out = c.collect_cmd_output(cmd)
-	if @conf[:mask] == 'yes'
-          diaglogger_info("[Mask] Masking netstat file : #{out}...")
-          out = m.mask_tdlog(out, clean = true)
+        diaglogger_info("[Collect] Collecting command output : command = #{cmd}")
+        if system(cmd + '> /dev/null 2>&1')
+          out = c.collect_cmd_output(cmd)
+          if @conf[:mask] == 'yes'
+            diaglogger_info("[Mask] Masking command output file : #{out}...")
+            out = m.mask_tdlog(out, clean = true)
+          end
+          diaglogger_info("[Collect] Collecting command output #{cmd.split[0]} stored in #{out}")
         end
-	diaglogger_info("[Collect] Collecting command output #{cmd.split[0]} stored in #{out}")
       }
 			
       ###
@@ -171,19 +185,24 @@ module Diagtool
       end
 
       if @conf[:mask] == 'yes'
-	tdconf.each { | file |
-	  diaglogger_info("[Mask] Masking td-agent config file : #{file}...")
-	  m.mask_tdlog(file, clean = true)
-	}
-        tdlog.each { | file |
-          diaglogger_info("[Mask] Masking td-agent log file : #{file}...")
-          filename = file.split("/")[-1]
-          if filename.include?(".gz")
-            m.mask_tdlog_gz(file, clean = true)
-          elsif
-            m.mask_tdlog(file, clean = true)
-          end
-	}
+        tdconf.each { | file |
+          diaglogger_info("[Mask] Masking td-agent config file : #{file}...")
+          m.mask_tdlog(file, clean = true)
+        }
+      end
+
+      if @conf[:mask] == 'yes'
+        if tdlog != nil
+          tdlog.each { | file |
+            diaglogger_info("[Mask] Masking td-agent log file : #{file}...")
+            filename = file.split("/")[-1]
+            if filename.include?(".gz")
+              m.mask_tdlog_gz(file, clean = true)
+            elsif
+              m.mask_tdlog(file, clean = true)
+            end
+          }
+        end
       end
       
       if @conf[:mask] == 'yes'
@@ -197,8 +216,9 @@ module Diagtool
 
     def parse_diagconf(params)
       options = {
-        :precheck => '', :basedir => '', :mask => '', :words => [], :wfile => '', :seed => '', :tdconf =>'', :tdlog => ''
+        :precheck => '', :basedir => '', :type =>'', :mask => '', :words => [], :wfile => '', :seed => '', :tdconf =>'', :tdlog => ''
       }
+      ### Parse precheck flag
       if params[:precheck]
         options[:precheck] = params[:precheck]
       else
@@ -215,6 +235,13 @@ module Diagtool
           raise "output directory '-o' must be specified"
         end
       end
+      ### Parse fluent type
+      if params[:type] == 'fluentd' || params[:type] == 'fluentbit'
+        options[:type] = params[:type]
+      else
+        raise "fluentd type '-t' must be specified (fluentd or fluentbit)"
+      end
+      ### Parse mask flag
       if params[:mask] == nil
         options[:mask] = 'no'
       else
@@ -224,7 +251,11 @@ module Diagtool
           raise "invalid arguments '#{params[:mask]}' : input of '-m|--mask' should be 'yes' or 'no'"
         end
       end
+     
+      ### Parse uder-defined keyword list which will be used in the mask function 
       options[:words] = params[:"word-list"] if params[:"word-list"] != nil
+
+      ### Parse uder-defined keyword file which will be used in the mask function 
       if params[:"word-file"] != nil
         f = params[:"word-file"]
         if File.exist?(f)
@@ -236,8 +267,11 @@ module Diagtool
         end
       end
       options[:words] = options[:words].uniq 
+
+      ### Parse hash seed which will be used in the mask function 
       options[:seed] = params[:"hash-seed"] if params[:"hash-seed"] != nil
-      
+
+      ### Parse the path of fluentd config file
       if params[:conf] != nil
         f = params[:conf]
         if File.exist?(f)
@@ -247,6 +281,7 @@ module Diagtool
         end
       end
 
+      ### Parse the path of fluentd log file
       if params[:log] != nil
         f = params[:log]
         if File.exist?(f)

data/lib/fluent/diagtool/version.rb

@@ -1,5 +1,5 @@
 module Fluent
   module Diagtool
-    VERSION = "0.1.9"
+    VERSION = "1.0.0"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-diagtool
 version: !ruby/object:Gem::Version
-  version: 0.1.9
+  version: 1.0.0
 platform: ruby
 authors:
 - kubotat
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-08-13 00:00:00.000000000 Z
+date: 2020-10-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fileutils
@@ -44,7 +44,7 @@ description: Bringing productivity of trouble shooting to the next level  by aut
 email:
 - tkubota@ctc-america.com
 executables:
-- diagtool
+- fluent-diagtool
 extensions: []
 extra_rdoc_files: []
 files:
@@ -56,11 +56,10 @@ files:
 - README.md
 - Rakefile
 - bin/console
-- bin/diagtool.rb
 - bin/setup
 - bin/word_list_sample
 - exclude_list01
-- exe/diagtool
+- exe/fluent-diagtool
 - fluent-diagtool.gemspec
 - lib/fluent/diagtool/collectutils.rb
 - lib/fluent/diagtool/diagutils.rb
@@ -86,8 +85,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.6.2
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: Diagnostic Tool for Fluentd

data/bin/diagtool.rb

@@ -1,87 +0,0 @@
-require 'optparse'
-require 'logger'
-require '../lib/diagutils'
-include Diagtool
-
-logger = Logger.new(STDOUT, formatter: proc {|severity, datetime, progname, msg|
-  "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
-})
-
-output_dir = '../output'
-mask = 'yes'
-exlist= Array.new
-
-opt = OptionParser.new
-opt.banner = "Usage: #{$0} -o OUTPUT_DIR -m {yes | no} -e {word1,[word2...]} -f {listfile}"
-opt.on('-o','--output DIR', String, 'Output directory (Default=./output)') { |o|
-	output_dir = o 
-}
-opt.on('-m','--mask YES|NO', String, 'Enable mask function (Default=True)') { |m| 
-	if m == 'yes' || m == 'no'
-		mask = m
-	else
-		logger.error("Invalid value '#{m}' : -m | --mask should be yes or no")
-		exit!
-	end
-}
-opt.on('-e','--exclude-list LIST', Array, 'Provide a list of exclude words which will to be masked (Default=None)') { |e| exlist += e } 
-opt.on('-f','--exclude-file FILE', String, 'provide a file which describes a List of exclude words (Default=None)') { |f|
-	if File.exist?(f)
-		File.readlines(f).each do  |l|
-			exlist.append(l.gsub(/\n/,''))
-		end
-	else
-		logger.error("No such file or directory")
-		exit!
-	end
-}
-opt.parse(ARGV)
-exlist = exlist.uniq
-
-logger.info("Parsing command options...")
-logger.info("   Option : Output directory = #{output_dir}")
-logger.info("   Option : Mask = #{mask}")
-logger.info("   Option : Exclude list = #{exlist}")
-
-logger.info("Initializing parameters...")
-node1 = Diagutils.new(output_dir,exlist, 'INFO')
-
-logger.info("Collecting log files of td-agent...")
-tdlog = node1.collect_tdlog()
-logger.info("log files of td-agent are stored in #{tdlog}")
-
-logger.info("Collecting config file of td-agent...")
-tdconf = node1.collect_tdconf()
-logger.info("config file is stored in #{tdconf}")
-
-logger.info("Collecting systctl information...")
-sysctl = node1.collect_sysctl()
-logger.info("sysctl information is stored in #{sysctl}")
-
-logger.info("Collecting date/time information...")
-ntp = node1.collect_ntp()
-logger.info("date/time information is stored in #{ntp}")
-
-logger.info("Collecting ulimit information...")
-ulimit = node1.collect_ulimit()
-logger.info("ulimit information is stored in #{ulimit}")
-
-if mask == 'yes'
-	logger.info("Masking td-agent config file : #{tdconf}...")
-	node1.mask_tdconf(tdconf)
-	tdlog.each do | file |
-		logger.info("Masking td-agent log file : #{file}...")
-      		filename = file.split("/")[-1]
-		if filename.include?(".gz")
-               		node1.mask_tdlog_gz(file)
-       		elsif
-               		node1.mask_tdlog(file)
-       		end
-	end
-end
-
-tar_file = node1.compress_output()
-logger.info("Generate tar file #{tar_file}")
-
-
-