fluent-diagtool 0.1.5 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 614c1b026136b0743f9c6439b6dd2349adbe9edf3cdb2d77e1ab78698a9d9e49
-  data.tar.gz: a2228adc3509363ba6bf8f4ddee4d6b1f3feaef2088452a5c69e72057eac2336
+  metadata.gz: 8c852c088190fa51d4232c45aa7afd4013473c8089e194511a0910a8ca4794e5
+  data.tar.gz: 96b031de365ad1d47c71b7ffe25dfbd53f4909fe07be484944f7814cfead4939
 SHA512:
-  metadata.gz: af815f59c3ed5492175d5fe988eb6eb33fae556b4a2fcd6a20cf89e7e71aa1db92438f7ec1002c08e7f00d59099f91b2d8764f91b18cfb52593c81c5ae89f8e4
-  data.tar.gz: c7adc36937ba9e1b74a2914a6c2fdf41a6f44dfbd155aa3eb99cf59126022280461a38972eb45f0f486e639b701db6ea7d1c3ec2de73893dd6c982ea9dde8f0c
+  metadata.gz: c6c96dd92c4c8db2975c84395288817b5053b3cdb4b0897970a5e253fc2714886baf0799dd5bd4161439e043c4827a65c178adefde241e2e974de53265bebef8
+  data.tar.gz: e7d24f7ca1d450e70bc037c74bf1ff8a88e2d174cc77c04211ba164bf7e7e97961b0d2970c8e3e00e0ab57f357c6c13aa8640afc1ffa8b9d7b63d5867630c9e4

data/README.md

@@ -1,6 +1,6 @@
 # Fluentd Diagnostic Tool
 
-The diagtool enable users to automate the date collection which is required for trouble shooting. The data collected by diagtool include the configuration and log files of the td-agent and diagnostic information of operating system such as network and memory status and stats. In some cases, configuration and log files contains the security sensitive information, such as IP addresses and Hostname. The diagtool also provides the functions to generate mask on IP addresses, Hostname(in FQDN style) and user defined keywords described in the collected data.   
+The diagtool enables users to automate the date collection which is required for troubleshooting. The data collected by diagtool include the configuration and log files of the td-agent and diagnostic information from an operating system such as network and memory status and stats. In some cases, configuration and log files contain security sensitive information, such as IP addresses and Hostname. The diagtool also provides the functions to generate masks on IP addresses, Hostname(in FQDN style) and user defined keywords described in the collected data.   
 The scope of data collection:  
 - TD Agent information
   - configuration files (*)
@@ -15,7 +15,7 @@ The scope of data collection:
     - maximum number of file descriptor(ulimit -n)
     - kernel network parameters(sysctl)
   - snapshot of current process(ps)
-  - network conectivity status/stats(netstat -plan/netstat -s)
+  - network connectivity status/stats(netstat -plan/netstat -s)
   - memory information(/proc/meminfo)
 <br>   
 
@@ -33,16 +33,28 @@ Successfully installed fileutils-1.0.2
 Fetching: json-2.1.0.gem (100%)
 Building native extensions. This could take a while...
 Successfully installed json-2.1.0
-Fetching: fluent-diagtool-0.1.2.gem (100%)
-Successfully installed fluent-diagtool-0.1.2
+Fetching: fluent-diagtool-1.0.0.gem (100%)
+Successfully installed fluent-diagtool-1.0.0
 3 gems installed
 ```
+When you are using td-agent, fluent-adiagtool should be installed using /usr/sbin/td-agent-gem command instead of gem command.
+```
+# /usr/sbin/td-agent-gem install fluent-diagtool
+Fetching fluent-diagtool-1.0.0.gem
+Successfully installed fluent-diagtool-1.0.0
+Parsing documentation for fluent-diagtool-1.0.0
+Installing ri documentation for fluent-diagtool-1.0.0
+Done installing documentation for fluent-diagtool after 0 seconds
+1 gem installed
+```
 
 ## Usage
+There are a few options in Diagtool. You can check the options of Diagtool with "--help" options. Diagtool performs the validation function in the process by default but you can turn on/off the mask function depending on the use cases.
 ```
 # diagtool --help
 Usage: /usr/local/bin/diagtool -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}
         --precheck                   Run Precheck (Optional)
+    -t, --type fluentd|fluentbit     Select the type of Fluentd (Mandatory)
     -o, --output DIR                 Output directory (Mandatory)
     -m, --mask yes|no                Enable mask function (Optional : Default=no)
     -w, --word-list word1,word2      Provide a list of user-defined words which will to be masked (Optional : Default=None)
@@ -52,10 +64,11 @@ Usage: /usr/local/bin/diagtool -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]}
     -l, --log log_file               provide a full path of td-agent log file (Optional : Default=None)
 ```
 ### Pre-check
-The diagtool automatically extract the path of td-agent configuration and log files from td-agent daemon and use them during data collection if the td-agent is managed as daemon. The precheck options provides the function to confirm if the diagtool could gather the td-agent information as expected. 
+The diagtool automatically parses the path of Fluentd configuration and log files from running Fluentd processes and daemon. The precheck options provides the function to confirm if the diagtool could gather the fluentd information as expected. 
 The following command output shows the case when the diagtool successfully gather information from daemon.
+You need to specify the type of Fluentd, "fluentd" or "fluentbit".
 ```
-# diagtool --precheck
+# diagtool --precheck -t fluentd
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck] Check OS parameters...
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    operating system = CentOS Linux 8 (Core)
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    kernel version = Linux 4.18.0-147.el8.x86_64
@@ -66,10 +79,10 @@ The following command output shows the case when the diagtool successfully gathe
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    td-agent log = td-agent.log
 2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck] Precheck completed. You can run diagtool command without -c and -l options
 ```
-In some cases, users do not manage td-agent as daemon but use own script to run td-agent with command line options. In that cases, users need to speccify the path of td-agent configuration and log files with -c and -l options respectively.  
+In some cases, users do not manage td-agent as daemon but use their own scripts to run td-agent with command line options. In that cases, users need to specify the path of td-agent configuration and log files with -c and -l options respectively.  
 The following example shows the precheck results when the diagtool is not able to extract the path of td-agent configuration and log files.
 ```
-# diagtool --precheck
+# diagtool --precheck -t fluentd
 2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck] Check OS parameters...
 2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    operating system = CentOS Linux 8 (Core)
 2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    kernel version = Linux 4.18.0-147.5.1.el8_1.x86_64
@@ -81,6 +94,14 @@ The following example shows the precheck results when the diagtool is not able t
 2020-05-28 05:45:14 +0000: [Diagtool] [WARN] [Precheck]    can not find td-agent conf path: please run diagtool command with -c /path/to/<td-agent conf file>
 2020-05-28 05:45:14 +0000: [Diagtool] [WARN] [Precheck]    can not find td-agent log path: please run diagtool command with -l /path/to/<td-agent log file>
 ```
+
+### Run diagtool
+
+#### The "@include" directive in td-agent configuration file
+The "@include" directive is a function to reuse configuration defined in other configuration files. The diagtool reads the td-agent configuration and collects the files described in "@include" directive as well. The details of "@include" directive are described in followed url:  
+https://docs.fluentd.org/configuration/config-file#6-re-use-your-config-the-include-directive
+
+#### User defined words to be masked 
 The user-defined words can be specified both -e option and -f option and the words are merged when both options are selected.
 The format of user-defined words list file specified in -f option should be followed format.
 ```
@@ -90,10 +111,9 @@ centos8102
 ```
 NOTE: When user specified the keywork, only the exact match words will be masked. For instance, when users like to mask words like "nginx1" and "nginx2", users need to specify "nginx1" and "nginx2" respectively and "nginx*" should not work in the tool.
 
-### Run diagtool
 #### Command sample:
 ```
-# diagtool -o /tmp/work1 -w passwd1,passwd2 -f word_list_sample -m yes
+# diagtool -t fluentd -o /tmp/work1 -w passwd1,passwd2 -f word_list_sample -m yes
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO] Parsing command options...
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO]    Option : Output directory = /tmp/work1
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO]    Option : Mask = yes
@@ -149,9 +169,9 @@ NOTE: When user specified the keywork, only the exact match words will be masked
 2020-05-12 18:21:22 -0400: [Diagtool] [INFO] [Mask] Export mask log file : ./mask_20200512182119.json
 2020-05-12 18:21:22 -0400: [Diagtool] [INFO] [Collect] Generate tar file /tmp/work1/diagout-20200512182119.tar.gz
 ```
-## Mask Function
-When run diagtool with mask option, the log of mask is also created in 'mask_{timestamp}.json' file. Users are able to confirm how the mask was generated on each files.  
-The diagtool provides hash-seed option with '-s'. When hash-seed is specified, the mask will be generated with original word and hash-seed so that users could use unique mask value.
+#### Mask Function
+When run diagtool with the mask option, the log of mask is also created in 'mask_{timestamp}.json' file. Users are able to confirm how the mask was generated on each file.  
+The diagtool provides a hash-seed option with '-s'. When hash-seed is specified, the mask will be generated with the original word and hash-seed so that users could use a unique mask value.
 #### Mask sample - IP address: IPv4_{md5hash}
 ```
     "Line112-8": {
@@ -177,7 +197,7 @@ The diagtool provides hash-seed option with '-s'. When hash-seed is specified, t
 
 ## Tested Environment
 - OS : CentOS 8.1
-- Fluentd : td-agent version 3  
+- Fluentd : td-agent version 3/4 
   https://docs.fluentd.org/quickstart/td-agent-v2-vs-v3
-
+- Fluentbit : td-agent-bit
 

data/exclude_list01

@@ -0,0 +1,2 @@
+centos8101
+centos8102

data/exe/{diagtool → fluent-diagtool}

@@ -27,6 +27,7 @@ params = {}
 OptionParser.new do |opt|
   opt.banner = "Usage: #{$0} -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}"
   opt.on('--precheck', 'Run Precheck (Optional)')
+  opt.on('-t','--type fluentd|fluentbit', String, 'Select the type of Fluentd (Mandatory)')
   opt.on('-o','--output DIR', String, 'Output directory (Mandatory)')
   opt.on('-m','--mask yes|no', String, 'Enable mask function (Optional : Default=no)')
   opt.on('-w','--word-list word1,word2', Array, 'Provide a list of user-defined words which will to be masked (Optional : Default=None)')

data/lib/fluent/diagtool/collectutils.rb

@@ -17,26 +17,39 @@
 require 'fileutils'
 require 'open3'
 require 'logger'
+require 'net/http'
+require 'uri'
 
 module Diagtool
   class CollectUtils
     def initialize(conf, log_level)
       @logger = Logger.new(STDOUT, level: log_level, formatter: proc {|severity, datetime, progname, msg|
-        "#{datetime}: [Diagutils] [#{severity}] #{msg}\n"
+        "#{datetime}: [Collectutils] [#{severity}] #{msg}\n"
       })
       @precheck = conf[:precheck]
+      @type = conf[:type]
       @time_format = conf[:time]
       @basedir = conf[:basedir]
       @workdir = conf[:workdir]
-      @outdir = conf[:outdir]			
-       
-      @tdenv = get_tdenv()
+      @outdir = conf[:outdir]
+      @tdenv = {
+        'FLUENT_CONF' => '',
+        'TD_AGENT_LOG_FILE' => ''
+      }			
+      
+      case @type
+      when 'fluentd'
+        _find_fluentd_info()
+      when 'fluentbit'
+        _find_fluentbit_info()
+      end
+
       if not conf[:tdconf].empty?
-	@tdconf = conf[:tdconf].split('/')[-1]
+        @tdconf = conf[:tdconf].split('/')[-1]
         @tdconf_path = conf[:tdconf].gsub(@tdconf,'')
       elsif
-	if not @tdenv['FLUENT_CONF'].empty?
-      	  @tdconf = @tdenv['FLUENT_CONF'].split('/')[-1]
+        if not @tdenv['FLUENT_CONF'].empty?
+          @tdconf = @tdenv['FLUENT_CONF'].split('/')[-1]
       	  @tdconf_path = @tdenv['FLUENT_CONF'].gsub(@tdconf,'')
 	else
 	  raise "The path of td-agent configuration file need to be specified."  if conf[:precheck] == false
@@ -50,14 +63,20 @@ module Diagtool
           @tdlog =  @tdenv['TD_AGENT_LOG_FILE'].split('/')[-1]
           @tdlog_path = @tdenv['TD_AGENT_LOG_FILE'].gsub(@tdlog,'')
         else
-          raise "The path of td-agent log file need to be specified." if conf[:precheck] == false
-	end
+          case @type
+          when 'fluentd'
+            raise "The path of td-agent log file need to be specified." if conf[:precheck] == false
+          when 'fluentbit'
+            @logger.warn("FluentBit logs are redirected to the standard output interface ")
+          end
+	      end
       end 
-      @osenv = get_osenv()
+      @osenv = _find_os_info()
       @oslog_path = '/var/log/'
       @oslog = 'messages'
+      @syslog = 'syslog'
       @sysctl_path = '/etc/'
-      @sysctl = 'sysctl.conf'        
+      @sysctl = 'sysctl.conf'
 
       @logger.info("Loading the environment parameters...")
       @logger.info("    operating system = #{@osenv['Operating System']}")
@@ -68,7 +87,7 @@ module Diagtool
       @logger.info("    td-agent log = #{@tdlog}")
     end
     
-    def get_osenv()
+    def _find_os_info()
       stdout, stderr, status = Open3.capture3('hostnamectl')
       os_dict = {}
       stdout.each_line { |l|
@@ -83,50 +102,102 @@ module Diagtool
       return os_dict
     end
     
-    def get_tdenv()
+    def _find_fluentd_info()
+      ### check if the td-agent is run as daemon
       stdout, stderr, status = Open3.capture3('systemctl cat td-agent')
-      env_dict = {}
       if status.success?
-	if @precheck == false  # SKip if precheck is true
+        if @precheck == false  # SKip if precheck is true
           File.open(@outdir+'/td-agent_env.output', 'w') do |f|
             f.puts(stdout)
           end
-	end  
+        end  
         stdout.split().each do | l |
           if l.include?('Environment')
-            env_dict[l.split('=')[1]] = l.split('=')[2]
+            @tdenv[l.split('=')[1]] = l.split('=')[2]
           end
-      	end
+        end
       else
-        exe = 'fluentd'
-        stdout, stderr, status = Open3.capture3("ps aux | grep #{exe} | grep -v grep")
-        line = stdout.split(/\n/)
-	log_path = ''
-        conf_path = ''
-        line.each { |l|
-          cmd = l.split.drop(10)
-          i = 0
-          log_pos = 0
-          conf_pos = 0
-          if cmd[-1] != '--under-supervisor'
-            cmd.each { |c|
-              if c.include?("--log") || c.include?("-l")
-                log_pos = i + 1
-                log_path = cmd[log_pos]
-              elsif c.include?("--conf") || c.include?("-c")
-                conf_pos = i + 1
-                conf_path = cmd[conf_pos]
+        ### check if the td-agent is not run as daemon or run Fluentd with customized script
+        stdout, stderr, status = Open3.capture3('ps aux | grep fluentd | grep -v ".*\(grep\|diagtool\)"')
+        if status.success?
+          line = stdout.split(/\n/)
+          line.each do |l|
+            cmd = l.split.drop(10)
+            i = 0
+            if cmd[-1] != '--under-supervisor'
+              cmd.each do |c|
+                case
+                when c == "-c"
+                  @tdenv['FLUENT_CONF'] = cmd[i+1]
+                when c == "-l"
+                  @tdenv['TD_AGENT_LOG_FILE'] = cmd[i+1]
+                when c.include?("--conf")
+                  @tdenv['FLUENT_CONF'] = c.split("=")[1]
+                when c.include?("--log")
+                  @tdenv['TD_AGENT_LOG_FILE'] = c.split("=")[1]
+                end
+                i+=1
               end
-              i+=1
-            }
-	  end
-	}
-        env_dict['FLUENT_CONF'] = conf_path
-        env_dict['TD_AGENT_LOG_FILE'] = log_path
+            end
+          end
+        else
+          @logger.warn("No Fluentd daemon or proccess running") 
+        end
       end
-      return env_dict
     end
     
+    def _find_fluentbit_info()
+      ### check if the td-agent-bit is run as daemon
+      stdout, stderr, status = Open3.capture3('systemctl cat td-agent-bit')
+      if status.success?
+        if @precheck == false  # SKip if precheck is true
+          File.open(@outdir+'/td-agent-bit_env.output', 'w') do |f|
+            f.puts(stdout)
+          end
+        end
+        stdout.split(/\n/).each do | line |
+          if line.start_with?("ExecStart")
+            cmd = line.split("=")[1]
+            i =0
+            cmd.split().each do | c |
+              case
+              when c == "-c"
+                @tdenv['FLUENT_CONF'] = cmd.split()[i+1]
+              when c == "-l"
+                @tdenv['TD_AGENT_LOG_FILE'] = cmd.split()[i+1]
+              when c.include?("--conf")
+                @tdenv['FLUENT_CONF'] = c.split("=")[1]
+              when c.include?("--log")
+                @tdenv['TD_AGENT_LOG_FILE'] = c.split("=")[1]
+              end
+              i+=1
+            end
+          end
+        end
+      else
+        ### check if the td-agent-bit is not run as daemon or run FluentdBit with customized script 
+        stdout, stderr, status = Open3.capture3('ps aux | grep fluent-bit | grep -v ".*\(grep\|diagtool\)"')
+        if status.success?
+          i = 0
+          stdout.split().each do | line |
+            case
+            when line.include?("--conf")
+              @tdenv['FLUENT_CONF'] = line.split("=")[1]
+            when line.include?("--log")
+              @tdenv['TD_AGENT_LOG_FILE'] = line.split("=")[1]
+            when line == "-c"
+              @tdenv['FLUENT_CONF'] = stdout.split()[i+1]
+            when line == "-l"
+              @tdenv['TD_AGENT_LOG_FILE'] = stdout.split()[i+1]
+            end
+            i+=1
+          end
+        else
+          @logger.warn("No FluentBit daemon or proccess running")
+        end
+      end
+    end
+
     def export_env()
       env = {
         :os => @osenv['Operating System'],
@@ -143,12 +214,140 @@ module Diagtool
       target_dir = @workdir+@tdconf_path
       FileUtils.mkdir_p(target_dir)
       FileUtils.cp(@tdconf_path+@tdconf, target_dir)
-      return target_dir+@tdconf
+      conf = @workdir+@tdconf_path+@tdconf
+      conf_list = []
+      conf_list.push conf
+      case @type
+      when 'fluentd'
+        conf_list = conf_list + _collect_tdconf_include(conf)
+      when 'fluentbit'
+        conf_list = conf_list + _collect_tdconf_include(conf) + _collect_tdbit_parser(conf) + _collect_tdbit_plugins(conf) 
+      end   
+      return conf_list
     end
-    
+
+    def _collect_tdconf_include(conf)
+      target_dir = @workdir+@tdconf_path
+      inc_list = []
+      File.readlines(conf).each do |line|
+        if line.start_with?('@include')
+          l = line.split()[1]
+          if l.start_with?('http')
+            uri = URI(l)
+            inc_http = target_dir + 'http' + uri.path.gsub('/','_')
+            File.open(inc_http, 'w') do |f|
+              f.puts(Net::HTTP.get(uri))
+            end
+            inc_list.push inc_http
+          else
+            if l.start_with?(/\//)  # /tmp/work1/b.conf
+              if l.include?('*')
+                Dir.glob(l).each { |ll|
+                  inc_conf = target_dir + ll.gsub(/\//,'-')
+                  FileUtils.cp(ll, inc_conf)
+                  inc_list.push inc_conf
+                }
+              else 
+                inc_conf = target_dir+l.gsub(/\//,'-')
+                FileUtils.cp(l, inc_conf)
+                inc_list.push inc_conf
+              end
+            else
+              l = l.gsub('./','') if l.include?('./')
+              if l.include?('*')
+                Dir.glob(@tdconf_path+f).each{ |ll|
+                  inc_conf = target_dir + ll.gsub(@tdconf_path,'').gsub(/\//,'-')
+                  FileUtils.cp(ll, inc_conf)
+                  inc_list.push inc_conf
+                }
+              else
+                inc_conf = target_dir+l.gsub(/\//,'-')
+                FileUtils.cp(@tdconf_path+l, inc_conf)
+                inc_list.push inc_conf
+              end
+            end
+          end
+        end
+      end
+      return inc_list
+    end
+
+    def _collect_tdbit_parser(conf)
+      target_dir = @workdir+@tdconf_path
+      parser_conf = []
+      File.readlines(conf).each do |line|
+        if line.strip.start_with?('parsers_file') || line.strip.start_with?('Parsers_File')
+          l = line.split()[1]
+          if l.start_with?(/\//)  # /tmp/work1/b.conf
+            if l.include?('*')
+              Dir.glob(l).each { |ll|
+                pconf = target_dir + ll.gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                parser_conf.push(pconf)
+              }
+            else 
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(l, pconf)
+              parser_conf.push(pconf)
+            end
+          else
+            l = l.gsub('./','') if l.include?('./')
+            if l.include?('*')
+              Dir.glob(@tdconf_path+f).each{ |ll|
+                pconf = target_dir + ll.gsub(@tdconf_path,'').gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                parser_conf.push(pconf)
+              }
+            else
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(@tdconf_path+l, pconf)
+              parser_conf.push(pconf)
+            end
+          end
+        end
+      end  
+      return parser_conf
+    end
+
+    def _collect_tdbit_plugins(conf)
+      target_dir = @workdir+@tdconf_path
+      plugins_conf = []
+      File.readlines(conf).each do |line|
+        if line.strip.start_with?('plugins_file') || line.strip.start_with?('Plugins_File')
+          l = line.split()[1]
+          if l.start_with?(/\//)  # /tmp/work1/b.conf
+            if l.include?('*')
+              Dir.glob(l).each { |ll|
+                pconf = target_dir + ll.gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                plugins_conf.push(pconf)
+              }
+            else 
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(l, pconf)
+              plugins_conf.push(pconf)
+            end
+          else
+            l = l.gsub('./','') if l.include?('./')
+            if l.include?('*')
+              Dir.glob(@tdconf_path+f).each{ |ll|
+                pconf = target_dir + ll.gsub(@tdconf_path,'').gsub(/\//,'-')
+                FileUtils.cp(ll, pconf)
+                plugins_conf.push(pconf)
+              }
+            else
+              pconf = target_dir+l.gsub(/\//,'-')
+              FileUtils.cp(@tdconf_path+l, pconf)
+              plugins_conf.push(pconf)
+            end
+          end
+        end
+      end
+      return plugins_conf
+    end
+
     def collect_tdlog()
       target_dir = @workdir+@tdlog_path
-      p target_dir
       FileUtils.mkdir_p(target_dir)
       Dir.glob(@tdlog_path+@tdlog+'*').each{ |f| 
         FileUtils.cp(f, target_dir)
@@ -156,65 +355,20 @@ module Diagtool
       return Dir.glob(target_dir+@tdlog+'*')
     end
     
-    def collect_sysctl()
-      target_dir = @workdir+@sysctl_path
-      FileUtils.mkdir_p(target_dir)
-      FileUtils.cp(@sysctl_path+@sysctl, target_dir)
-      return target_dir+@sysctl
-    end
-    
     def collect_oslog()
       target_dir = @workdir+@oslog_path
       FileUtils.mkdir_p(target_dir)
-      FileUtils.cp(@oslog_path+@oslog, target_dir)
-      return target_dir+@oslog
-    end
-    
-    def collect_ulimit()
-      output = @outdir+'/ulimit_n.output'
-      stdout, stderr, status = Open3.capture3("ulimit -n")
-      File.open(output, 'w') do |f|
-        f.puts(stdout)
-      end
-      return output
-    end
-   
-    def collect_ps_eo()
-      output = @outdir+'/ps_eo.output'
-      stdout, stderr, status = Open3.capture3("ps -eo pid,ppid,stime,time,%mem,%cpu,cmd")
-      File.open(output, 'w') do |f|
-        f.puts(stdout)
+      if File.exist? @oslog_path+@oslog
+      	FileUtils.cp(@oslog_path+@oslog, target_dir)
+      	return target_dir+@oslog
+      elsif File.exist? @oslog_path+@syslog
+        FileUtils.cp(@oslog_path+@syslog, target_dir)
+        return target_dir+@syslog
+      else
+        @logger.warn("Can not find OS log file in #{oslog} or #{syslog}")
       end
-      return output
     end
 
-    def collect_meminfo()
-      output = @outdir+'/meminfo.output'
-      stdout, stderr, status = Open3.capture3("cat /proc/meminfo")
-      File.open(output, 'w') do |f|
-        f.puts(stdout)
-      end
-      return output
-    end
-    
-    def collect_netstat_plan()
-      output = @outdir+'/netstat_plan.output'
-      stdout, stderr, status = Open3.capture3("netstat -plan")
-      File.open(output, 'w') do |f|
-        f.puts(stdout)
-      end
-      return output
-    end
-    
-    def collect_netstat_s()
-      output = @outdir+'/netstat_s.output'
-      stdout, stderr, status = Open3.capture3("netstat -s")
-      File.open(output, 'w') do |f|
-        f.puts(stdout)
-      end
-      return output
-    end
-    
     def collect_ntp(command)
       output = @outdir+'/ntp_info.output'
       stdout_date, stderr_date, status_date = Open3.capture3("date")
@@ -224,9 +378,26 @@ module Diagtool
         f.puts(stdout_date)
         f.puts(stdout_ntp)
       end
+    end
+
+    def collect_cmd_output(cmd)
+      if system(cmd + '> /dev/null 2>&1')
+        cmd_name = cmd.gsub(/\s/,'_').gsub(/\//,'-').gsub(',','_')
+        output = @outdir+'/'+cmd_name+'.txt'
+        stdout, stderr, status = Open3.capture3(cmd)
+        if status.success?
+          File.open(output, 'w') do |f|
+            f.puts(stdout)
+          end
+        else
+          @logger.warn("Command #{cmd} failed due to the following message -  #{stderr.chomp}")
+        end
+      else
+        @logger.warn("Command #{cmd} does not exist -  skip collecting #{cmd} output")
+      end
       return output
     end
-    
+
     def collect_tdgems()
       output = @outdir+'/tdgem_list.output'
       stdout, stderr, status = Open3.capture3("td-agent-gem list | grep fluent")

data/lib/fluent/diagtool/diagutils.rb

@@ -27,26 +27,12 @@ module Diagtool
       time = Time.new
       @time_format = time.strftime("%Y%m%d%0k%M%0S")
       @conf = parse_diagconf(params)
-      #@conf[:time] = @time_format
-      #@conf[:workdir] = @conf[:basedir] + '/' + @time_format
-      #@conf[:outdir] = @conf[:workdir] + '/output'
-
-      #FileUtils.mkdir_p(@conf[:workdir])
-      #FileUtils.mkdir_p(@conf[:outdir])
-      
-      #diaglog = @conf[:workdir] + '/diagtool.output'
-      #@masklog = './mask_' + @time_format + '.json'
-      #@logger = Logger.new(STDOUT, formatter: proc {|severity, datetime, progname, msg|
-      #  "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
-      #})
-      #@logger_file = Logger.new(diaglog, formatter: proc {|severity, datetime, progname, msg|
-      #  "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
-      #})
-      #diaglogger_info("Parsing command options...")
-      #diaglogger_info("   Option : Output directory = #{@conf[:basedir]}")
-      #diaglogger_info("   Option : Mask = #{@conf[:mask]}")
-      #diaglogger_info("   Option : Word list = #{@conf[:words]}")
-      #diaglogger_info("   Option : Hash Seed = #{@conf[:seed]}")
+      @cmd_list = [ 
+        "ps -eo pid,ppid,stime,time,%mem,%cpu,cmd",
+        "cat /proc/meminfo",
+	      "netstat -plan",
+	      "netstat -s",
+      ]
     end
     
     def run_precheck()
@@ -56,6 +42,7 @@ module Diagtool
       loglevel = 'WARN'
       c = CollectUtils.new(@conf, loglevel)
       c_env = c.export_env()
+      prechecklog.info("[Precheck] Fluentd Type = #{@conf[:type]}")
       prechecklog.info("[Precheck] Check OS parameters...")
       prechecklog.info("[Precheck]    operating system = #{c_env[:os]}")
       prechecklog.info("[Precheck]    kernel version = #{c_env[:kernel]}")
@@ -65,13 +52,13 @@ module Diagtool
       prechecklog.info("[Precheck]    td-agent log path = #{c_env[:tdlog_path]}")
       prechecklog.info("[Precheck]    td-agent log = #{c_env[:tdlog]}")
       if c_env[:tdconf_path] == nil || c_env[:tdconf] == nil
-	prechecklog.warn("[Precheck]    can not find td-agent conf path: please run diagtool command with -c /path/to/<td-agent conf file>")
+        prechecklog.warn("[Precheck]    can not find td-agent conf path: please run diagtool command with -c /path/to/<td-agent conf file>")
       end
       if c_env[:tdlog_path] == nil || c_env[:tdlog] == nil
         prechecklog.warn("[Precheck]    can not find td-agent log path: please run diagtool command with -l /path/to/<td-agent log file>")
       end
       if c_env[:tdconf_path] != nil && c_env[:tdconf] != nil && c_env[:tdlog_path] != nil && c_env[:tdlog] != nil
-	 prechecklog.info("[Precheck] Precheck completed. You can run diagtool command without -c and -l options")
+        prechecklog.info("[Precheck] Precheck completed. You can run diagtool command without -c and -l options")
       end
     end
 
@@ -111,8 +98,19 @@ module Diagtool
       v = ValidUtils.new(loglevel)
 							
       diaglogger_info("[Collect] Collecting log files of td-agent...")
-      tdlog = c.collect_tdlog()
-      diaglogger_info("[Collect] log files of td-agent are stored in #{tdlog}")
+      case @type
+      when 'fluentd'
+        tdlog = c.collect_tdlog()
+        diaglogger_info("[Collect] log files of td-agent are stored in #{tdlog}")
+      when 'fleuntbit'
+        if tdlog.empty?
+          diaglogger_info("FluentBit logs are redirected to the standard output interface ")
+          tdlog = ''
+        else
+          tdlog = c.collect_tdlog()
+          diaglogger_info("[Collect] log files of td-agent are stored in #{tdlog}")
+        end
+      end
 
       diaglogger_info("[Collect] Collecting config file of td-agent...")
       tdconf = c.collect_tdconf()
@@ -130,39 +128,37 @@ module Diagtool
       end
       diaglogger_info("[Collect] config file is stored in #{oslog}")
 
-      diaglogger_info("[Collect] Collecting process information...")
-      meminfo = c.collect_ps_eo()
-      diaglogger_info("[Collect] process informationis stored in #{meminfo}")
-
-      diaglogger_info("[Collect] Collecting OS memory information...")
-      meminfo = c.collect_meminfo()
-      diaglogger_info("[Collect] OS memory information is stored in #{meminfo}")
-
       diaglogger_info("[Collect] Collecting date/time information...")
       if system('which chronyc > /dev/null 2>&1')
-        ntp = c.collect_ntp(command="chrony")
+        ntp = c.collect_cmd_output(command="chronyc sources")
+        diaglogger_info("[Collect] date/time information is stored in #{ntp}")
       elsif system('which ntpq > /dev/null 2>&1')
-        ntp = c.collect_ntp(command="ntp")
+        ntp = c.collect_cmd_output(command="ntpq -p")
+        diaglogger_info("[Collect] date/time information is stored in #{ntp}")
       else
         diaglogger_warn("[Collect] chrony/ntp does not exist. skip collectig date/time information")
       end
-      diaglogger_info("[Collect] date/time information is stored in #{ntp}")
-			
-      diaglogger_info("[Collect] Collecting netstat information...")
-      if system('which netstat > /dev/null 2>&1')
-        netstat_n = c.collect_netstat_plan()
-        netstat_s = c.collect_netstat_s()
-        if @conf[:mask] == 'yes'
-          diaglogger_info("[Mask] Masking netstat file : #{netstat_n}...")
-          netstat_n = m.mask_tdlog(netstat_n, clean = true)
-        end
-        diaglogger_info("[Collect] netstat information is stored in #{netstat_n} and #{netstat_s}")		
-      else
-        diaglogger_warn("[Collect] netstat does not exist. skip collectig netstat")
-      end
       
+      ###
+      #  Correct OS information
+      ###
+      @cmd_list.each { |cmd|
+        diaglogger_info("[Collect] Collecting command output : command = #{cmd}")
+        if system(cmd + '> /dev/null 2>&1')
+          out = c.collect_cmd_output(cmd)
+          if @conf[:mask] == 'yes'
+            diaglogger_info("[Mask] Masking command output file : #{out}...")
+            out = m.mask_tdlog(out, clean = true)
+          end
+          diaglogger_info("[Collect] Collecting command output #{cmd.split[0]} stored in #{out}")
+        end
+      }
+			
+      ###
+      #  Correct information to be validated
+      ###
       diaglogger_info("[Collect] Collecting systctl information...")
-      sysctl = c.collect_sysctl()
+      sysctl = c.collect_cmd_output("sysctl -a")
       diaglogger_info("[Collect] sysctl information is stored in #{sysctl}")
 			
       diaglogger_info("[Valid] Validating systctl information...")
@@ -177,7 +173,7 @@ module Diagtool
       end
 
       diaglogger_info("[Collect] Collecting ulimit information...")
-      ulimit = c.collect_ulimit()
+      ulimit = c.collect_cmd_output(cmd="sh -c 'ulimit -n'")
       diaglogger_info("[Collect] ulimit information is stored in #{ulimit}")
 
       diaglogger_info("[Valid] Validating ulimit information...")
@@ -189,16 +185,23 @@ module Diagtool
       end
 
       if @conf[:mask] == 'yes'
-        diaglogger_info("[Mask] Masking td-agent config file : #{tdconf}...")
-        m.mask_tdlog(tdconf, clean = true)
-        tdlog.each do | file |
-          diaglogger_info("[Mask] Masking td-agent log file : #{file}...")
-          filename = file.split("/")[-1]
-          if filename.include?(".gz")
-            m.mask_tdlog_gz(file, clean = true)
-          elsif
-            m.mask_tdlog(file, clean = true)
-          end
+        tdconf.each { | file |
+          diaglogger_info("[Mask] Masking td-agent config file : #{file}...")
+          m.mask_tdlog(file, clean = true)
+        }
+      end
+
+      if @conf[:mask] == 'yes'
+        if tdlog != nil
+          tdlog.each { | file |
+            diaglogger_info("[Mask] Masking td-agent log file : #{file}...")
+            filename = file.split("/")[-1]
+            if filename.include?(".gz")
+              m.mask_tdlog_gz(file, clean = true)
+            elsif
+              m.mask_tdlog(file, clean = true)
+            end
+          }
         end
       end
       
@@ -206,15 +209,16 @@ module Diagtool
         diaglogger_info("[Mask] Export mask log file : #{@masklog}")
         m.export_masklog(@masklog)
       end
-      
+
       tar_file = c.compress_output()
       diaglogger_info("[Collect] Generate tar file #{tar_file}")
     end
 
     def parse_diagconf(params)
       options = {
-        :precheck => '', :basedir => '', :mask => '', :words => [], :wfile => '', :seed => '', :tdconf =>'', :tdlog => ''
+        :precheck => '', :basedir => '', :type =>'', :mask => '', :words => [], :wfile => '', :seed => '', :tdconf =>'', :tdlog => ''
       }
+      ### Parse precheck flag
       if params[:precheck]
         options[:precheck] = params[:precheck]
       else
@@ -231,6 +235,13 @@ module Diagtool
           raise "output directory '-o' must be specified"
         end
       end
+      ### Parse fluent type
+      if params[:type] == 'fluentd' || params[:type] == 'fluentbit'
+        options[:type] = params[:type]
+      else
+        raise "fluentd type '-t' must be specified (fluentd or fluentbit)"
+      end
+      ### Parse mask flag
       if params[:mask] == nil
         options[:mask] = 'no'
       else
@@ -240,7 +251,11 @@ module Diagtool
           raise "invalid arguments '#{params[:mask]}' : input of '-m|--mask' should be 'yes' or 'no'"
         end
       end
+     
+      ### Parse uder-defined keyword list which will be used in the mask function 
       options[:words] = params[:"word-list"] if params[:"word-list"] != nil
+
+      ### Parse uder-defined keyword file which will be used in the mask function 
       if params[:"word-file"] != nil
         f = params[:"word-file"]
         if File.exist?(f)
@@ -252,8 +267,11 @@ module Diagtool
         end
       end
       options[:words] = options[:words].uniq 
+
+      ### Parse hash seed which will be used in the mask function 
       options[:seed] = params[:"hash-seed"] if params[:"hash-seed"] != nil
-      
+
+      ### Parse the path of fluentd config file
       if params[:conf] != nil
         f = params[:conf]
         if File.exist?(f)
@@ -263,6 +281,7 @@ module Diagtool
         end
       end
 
+      ### Parse the path of fluentd log file
       if params[:log] != nil
         f = params[:log]
         if File.exist?(f)

data/lib/fluent/diagtool/validutils.rb

@@ -33,7 +33,8 @@ module Diagtool
         :net_ipv4_tcp_max_syn_backlog => "8096",
         :net_ipv4_tcp_slow_start_after_idle => "0",
         :net_ipv4_tcp_tw_reuse => "1",
-        :net_ipv4_ip_local_port_range => ["10240", "65535"]}
+        :net_ipv4_ip_local_port_range => ["10240", "65535"]
+      }
       @logger.debug("Initialize Validation Utils:")
       @logger.debug("    Default ulimit: #{@def_ulimit}")
       @logger.debug("    Default sysctl: #{@def_sysctl}")
@@ -57,7 +58,7 @@ module Diagtool
       v = Hash.new { |i,j| i[j] = Hash.new(&h.default_proc) }
       @logger.info("Loading sysctl file: #{sysctl_file}")
       File.readlines(sysctl_file).each{ |line|
-        if line.include?("net")
+        if line.include? "net"
           line_net = line.chomp.gsub(".","_").split("=")
           key = line_net[0].strip.to_sym
           if line_net[1].strip! =~ /\s/
@@ -66,17 +67,19 @@ module Diagtool
             value= line_net[1]
           end
           h[key] = value
-          if @def_sysctl[key] == value
-            @logger.info("#{key} => #{value} is correct")
-            v[key]['value'] = value
-            v[key]['recommend'] = @def_sysctl[key]	
-            v[key]['result'] = "correct"
-          else
-            @logger.warn("#{key} => #{value} is incorrect, should be #{@def_sysctl[key]}")
-            v[key]['value'] = value
-            v[key]['recommend'] = @def_sysctl[key]
-            v[key]['result'] = "incorrect"
-          end
+	  if @def_sysctl.key? key
+            if @def_sysctl[key] == value
+              @logger.info("#{key} => #{value} is correct")
+              v[key]['value'] = value
+              v[key]['recommend'] = @def_sysctl[key]	
+              v[key]['result'] = "correct"
+            else
+              @logger.warn("#{key} => #{value} is incorrect, should be #{@def_sysctl[key]}")
+              v[key]['value'] = value
+              v[key]['recommend'] = @def_sysctl[key]
+              v[key]['result'] = "incorrect"
+            end
+	  end
         end
       }
       if h == @sysctl

data/lib/fluent/diagtool/version.rb

@@ -1,5 +1,5 @@
 module Fluent
   module Diagtool
-    VERSION = "0.1.5"
+    VERSION = "1.0.0"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-diagtool
 version: !ruby/object:Gem::Version
-  version: 0.1.5
+  version: 1.0.0
 platform: ruby
 authors:
 - kubotat
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-05-28 00:00:00.000000000 Z
+date: 2020-10-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fileutils
@@ -44,7 +44,7 @@ description: Bringing productivity of trouble shooting to the next level  by aut
 email:
 - tkubota@ctc-america.com
 executables:
-- diagtool
+- fluent-diagtool
 extensions: []
 extra_rdoc_files: []
 files:
@@ -58,7 +58,8 @@ files:
 - bin/console
 - bin/setup
 - bin/word_list_sample
-- exe/diagtool
+- exclude_list01
+- exe/fluent-diagtool
 - fluent-diagtool.gemspec
 - lib/fluent/diagtool/collectutils.rb
 - lib/fluent/diagtool/diagutils.rb
@@ -84,8 +85,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.6.2
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: Diagnostic Tool for Fluentd