fluent-diagtool 0.1.1 → 0.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e43d24f82679a78fbb98b3a19231dd3cc418c57bec4b2aa7366f0bc935c5b3ec
-  data.tar.gz: a59aa1ce4b3751969ba4e164e7521c5aba4e6fd526c8a873cbd3f4b0aa059df7
+  metadata.gz: 459fd1aefd11e12f79ef9f12cd99e289412fc99c3711191508aa3b17af985a6d
+  data.tar.gz: 8ed32d7c995ec7a6afb9a618421eb49590f44db96ed10f4babb8440cb3fde271
 SHA512:
-  metadata.gz: 8aa091d9aa0de2bfa11947956e4587dd5c2a3f67a6c681ce31398a8aaf941f5450156ac6da1f1fb0a1ce6cf81252b2175355a8053446a1c906306a78b0bd3240
-  data.tar.gz: 96e441e30fdf836a731865d9992342c21436efd76d5ab4ec95254cdbdba3bb9363e68d35f594572dccfc44f67565fdca4014667acaf2632bd7760c09625a4f7f
+  metadata.gz: 97adbb75557f1c7c2cf0624e2c9f4aa8ac1631a91d1b30f269fa137223ee20855ceb5764d47cf32f3b01ec0b1e80d541b0ecb069c27d136e456809b935f91efe
+  data.tar.gz: e6bc2057366a4f864d7d081235f29bcc0ddbc118412695f52fc6516a642421e9d9770a5a1fd6c7fc2deb7f4cd90a1d3e8d90cb0bcec07e99fd8ef4dc41a90f8a

data/.gitignore

@@ -0,0 +1,13 @@
+~*
+#*
+*~
+[._]*.s[a-w][a-z]
+.DS_Store
+
+*.gem
+.bundle
+Gemfile.lock
+vendor
+.ruby-version
+
+test/tmp/

data/AUTHORS

@@ -1 +1 @@
-TOMONORI KUBOTA <tmnr.kubota _at_ gmail.com>
+TOMONORI KUBOTA <tkubota _at_ ctc-america.com>

data/Gemfile.lock

@@ -1,12 +1,16 @@
 PATH
   remote: .
   specs:
-    fluent-diagtool (0.1.0)
+    fluent-diagtool (0.1.5)
+      fileutils
+      json
 
 GEM
   remote: https://rubygems.org/
   specs:
     diff-lcs (1.3)
+    fileutils (1.4.1)
+    json (2.3.0)
     rake (12.3.3)
     rspec (3.9.0)
       rspec-core (~> 3.9.0)

data/README.md

@@ -1,7 +1,7 @@
 # Fluentd Diagnostic Tool
 
-The diagtool enable users to automate the date collection which is required for trouble shooting. The data collected by diagtool include the configuration and log files of the td-agent and diagnostic information of operating system such as network and memory status and stats. In some cases, configuration and log files contains the security sensitive information, such as IP addresses and Hostname. The diagtool also provides the functions to generate mask on IP addresses, Hostname(in FQDN style) and user defined keywords described in the collected data.<br> 
-The scope of data collection:<br>
+The diagtool enable users to automate the date collection which is required for trouble shooting. The data collected by diagtool include the configuration and log files of the td-agent and diagnostic information of operating system such as network and memory status and stats. In some cases, configuration and log files contains the security sensitive information, such as IP addresses and Hostname. The diagtool also provides the functions to generate mask on IP addresses, Hostname(in FQDN style) and user defined keywords described in the collected data.   
+The scope of data collection:  
 - TD Agent information
   - configuration files (*)
   - log files (*)
@@ -11,28 +11,85 @@ The scope of data collection:
   - OS log file
   - OS parameters
     - OS and kernel version
-    - time/date information
-    - maximum number of file descriptor(ulimit)
+    - time/date information(ntp -q/chronyc sources)
+    - maximum number of file descriptor(ulimit -n)
     - kernel network parameters(sysctl)
-  - network conectivity status/stats
-  - memory information
-<br>  
-(*) The diagtool automatically gather the path of td-agent configuration files and log files and use them during data collection. 
+  - snapshot of current process(ps)
+  - network conectivity status/stats(netstat -plan/netstat -s)
+  - memory information(/proc/meminfo)
+<br>   
 
 ## Prerequisite
+The diagtool provides support for td-agent based installation running on Linux OS. The td-agent is a stable distribution package of Fluentd.  
+The differences between Fluentd and td-agent are described in followed url:  
+https://www.fluentd.org/faqs
 
+## Diagtool Installation
+
+```
+# gem install fluent-diagtool
+Fetching: fileutils-1.0.2.gem (100%)
+Successfully installed fileutils-1.0.2
+Fetching: json-2.1.0.gem (100%)
+Building native extensions. This could take a while...
+Successfully installed json-2.1.0
+Fetching: fluent-diagtool-0.1.2.gem (100%)
+Successfully installed fluent-diagtool-0.1.2
+3 gems installed
+```
 
 ## Usage
 ```
-# ruby diagtool.rb --help
-Usage: diagtool.rb -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}
+# diagtool --help
+Usage: /usr/local/bin/diagtool -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}
+        --precheck                   Run Precheck (Optional)
     -o, --output DIR                 Output directory (Mandatory)
     -m, --mask yes|no                Enable mask function (Optional : Default=no)
     -w, --word-list word1,word2      Provide a list of user-defined words which will to be masked (Optional : Default=None)
-    -f, --word-file listfile         provide a file which describes a List of user-defined words (Optional : Default=None)
+    -f, --word-file list_file        provide a file which describes a List of user-defined words (Optional : Default=None)
     -s, --hash-seed seed             provide a word which will be used when generate the mask (Optional : Default=None)
+    -c, --conf config_file           provide a full path of td-agent configuration file (Optional : Default=None)
+    -l, --log log_file               provide a full path of td-agent log file (Optional : Default=None)
+```
+### Pre-check
+The diagtool automatically extract the path of td-agent configuration and log files from td-agent daemon and use them during data collection if the td-agent is managed as daemon. The precheck options provides the function to confirm if the diagtool could gather the td-agent information as expected. 
+The following command output shows the case when the diagtool successfully gather information from daemon.
+```
+# diagtool --precheck
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck] Check OS parameters...
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    operating system = CentOS Linux 8 (Core)
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    kernel version = Linux 4.18.0-147.el8.x86_64
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck] Check td-agent parameters...
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    td-agent conf path = /etc/td-agent/
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    td-agent conf file = td-agent.conf
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    td-agent log path = /var/log/td-agent/
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck]    td-agent log = td-agent.log
+2020-05-28 00:39:02 -0400: [Diagtool] [INFO] [Precheck] Precheck completed. You can run diagtool command without -c and -l options
 ```
-The list of user-defined words can be specified both -e option and -f option.
+In some cases, users do not manage td-agent as daemon but use own script to run td-agent with command line options. In that cases, users need to speccify the path of td-agent configuration and log files with -c and -l options respectively.  
+The following example shows the precheck results when the diagtool is not able to extract the path of td-agent configuration and log files.
+```
+# diagtool --precheck
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck] Check OS parameters...
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    operating system = CentOS Linux 8 (Core)
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    kernel version = Linux 4.18.0-147.5.1.el8_1.x86_64
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck] Check td-agent parameters...
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    td-agent conf path =
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    td-agent conf file =
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    td-agent log path =
+2020-05-28 05:45:14 +0000: [Diagtool] [INFO] [Precheck]    td-agent log =
+2020-05-28 05:45:14 +0000: [Diagtool] [WARN] [Precheck]    can not find td-agent conf path: please run diagtool command with -c /path/to/<td-agent conf file>
+2020-05-28 05:45:14 +0000: [Diagtool] [WARN] [Precheck]    can not find td-agent log path: please run diagtool command with -l /path/to/<td-agent log file>
+```
+
+### Run diagtool
+
+#### The "@include" directive in td-agent configuration file
+The "@include" directive is a function to reuse configuration defined in another configuration files. The diagtool read the td-agent configuration and collect the files described in "@include" directive as well. The details of "@include" directive are described in followed url:  
+https://docs.fluentd.org/configuration/config-file#6-re-use-your-config-the-include-directive
+
+#### User defined words to be masked 
+The user-defined words can be specified both -e option and -f option and the words are merged when both options are selected.
 The format of user-defined words list file specified in -f option should be followed format.
 ```
 # cat word_list_sample
@@ -43,7 +100,7 @@ NOTE: When user specified the keywork, only the exact match words will be masked
 
 #### Command sample:
 ```
-# ruby diagtool.rb -o /tmp/work1 -w passwd1,passwd2 -f word_list_sample -m yes
+# diagtool -o /tmp/work1 -w passwd1,passwd2 -f word_list_sample -m yes
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO] Parsing command options...
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO]    Option : Output directory = /tmp/work1
 2020-05-12 18:21:19 -0400: [Diagtool] [INFO]    Option : Mask = yes
@@ -99,9 +156,9 @@ NOTE: When user specified the keywork, only the exact match words will be masked
 2020-05-12 18:21:22 -0400: [Diagtool] [INFO] [Mask] Export mask log file : ./mask_20200512182119.json
 2020-05-12 18:21:22 -0400: [Diagtool] [INFO] [Collect] Generate tar file /tmp/work1/diagout-20200512182119.tar.gz
 ```
-## Mask Function
-When run diagtool with mask option, the log of mask is also created in 'mask_{timestamp}.json' file. Users are able to confirm how the mask was generated on each files. 
-<br>
+#### Mask Function
+When run diagtool with mask option, the log of mask is also created in 'mask_{timestamp}.json' file. Users are able to confirm how the mask was generated on each files.  
+The diagtool provides hash-seed option with '-s'. When hash-seed is specified, the mask will be generated with original word and hash-seed so that users could use unique mask value.
 #### Mask sample - IP address: IPv4_{md5hash}
 ```
     "Line112-8": {

data/exe/diagtool

@@ -26,13 +26,21 @@ include Diagtool
 params = {}
 OptionParser.new do |opt|
   opt.banner = "Usage: #{$0} -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}"
+  opt.on('--precheck', 'Run Precheck (Optional)')
   opt.on('-o','--output DIR', String, 'Output directory (Mandatory)')
   opt.on('-m','--mask yes|no', String, 'Enable mask function (Optional : Default=no)')
   opt.on('-w','--word-list word1,word2', Array, 'Provide a list of user-defined words which will to be masked (Optional : Default=None)')
-  opt.on('-f','--word-file listfile', String, 'provide a file which describes a List of user-defined words (Optional : Default=None)')
+  opt.on('-f','--word-file list_file', String, 'provide a file which describes a List of user-defined words (Optional : Default=None)')
   opt.on('-s','--hash-seed seed', String, 'provide a word which will be used when generate the mask (Optional : Default=None)')
+  opt.on('-c','--conf config_file', String, 'provide a full path of td-agent configuration file (Optional : Default=None)')
+  opt.on('-l','--log log_file', String, 'provide a full path of td-agent log file (Optional : Default=None)')
 end.parse!(into: params)
+
 diag = DiagUtils.new(params)
-diag.diagtool()
+if params[:precheck]
+  diag.run_precheck()
+else
+  diag.run_diagtool()
+end
 
 

data/fluent-diagtool.gemspec

@@ -12,14 +12,13 @@ Gem::Specification.new do |spec|
   spec.license       = "Apache-2.0"
   spec.required_ruby_version = Gem::Requirement.new(">= 2.3.0")
 
-
-  # Specify which files should be added to the gem when it is released.
-  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
   spec.files         = Dir.chdir(File.expand_path('..', __FILE__)) do
-    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|tool|sample|features)/}) }
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|tool|sample)/}) }
   end
   spec.bindir        = "exe"
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
+  spec.add_runtime_dependency("fileutils")
+  spec.add_runtime_dependency("json")
 end

data/lib/fluent/diagtool/collectutils.rb

@@ -24,17 +24,36 @@ module Diagtool
       @logger = Logger.new(STDOUT, level: log_level, formatter: proc {|severity, datetime, progname, msg|
         "#{datetime}: [Diagutils] [#{severity}] #{msg}\n"
       })
+      @precheck = conf[:precheck]
       @time_format = conf[:time]
-      @output_dir = conf[:output_dir]
+      @basedir = conf[:basedir]
       @workdir = conf[:workdir]
-			
-      @tdenv = get_tdenv()
-      @tdconf = @tdenv['FLUENT_CONF'].split('/')[-1]
-      @tdconf_path = @tdenv['FLUENT_CONF'].gsub(@tdconf,'')
-      @tdlog =  @tdenv['TD_AGENT_LOG_FILE'].split('/')[-1]
-      @tdlog_path = @tdenv['TD_AGENT_LOG_FILE'].gsub(@tdlog,'')
-
-      @osenv = get_osenv()
+      @outdir = conf[:outdir]			
+       
+      @tdenv = gen_tdenv()
+      if not conf[:tdconf].empty?
+	@tdconf = conf[:tdconf].split('/')[-1]
+        @tdconf_path = conf[:tdconf].gsub(@tdconf,'')
+      elsif
+	if not @tdenv['FLUENT_CONF'].empty?
+      	  @tdconf = @tdenv['FLUENT_CONF'].split('/')[-1]
+      	  @tdconf_path = @tdenv['FLUENT_CONF'].gsub(@tdconf,'')
+	else
+	  raise "The path of td-agent configuration file need to be specified."  if conf[:precheck] == false
+	end
+      end
+      if not conf[:tdlog].empty?
+        @tdlog = conf[:tdlog].split('/')[-1]
+        @tdlog_path = conf[:tdlog].gsub(@tdlog,'')
+      elsif
+        if not @tdenv['TD_AGENT_LOG_FILE'].empty?
+          @tdlog =  @tdenv['TD_AGENT_LOG_FILE'].split('/')[-1]
+          @tdlog_path = @tdenv['TD_AGENT_LOG_FILE'].gsub(@tdlog,'')
+        else
+          raise "The path of td-agent log file need to be specified." if conf[:precheck] == false
+	end
+      end 
+      @osenv = gen_osenv()
       @oslog_path = '/var/log/'
       @oslog = 'messages'
       @sysctl_path = '/etc/'
@@ -48,31 +67,66 @@ module Diagtool
       @logger.info("    td-agent log path = #{@tdlog_path}")
       @logger.info("    td-agent log = #{@tdlog}")
     end
-    def get_osenv()
+    
+    def gen_osenv()
       stdout, stderr, status = Open3.capture3('hostnamectl')
       os_dict = {}
       stdout.each_line { |l|
         s = l.split(":")
         os_dict[s[0].chomp.strip] = s[1].chomp.strip
       }
-      File.open(@workdir+'/os_env.output', 'w') do |f|
-        f.puts(stdout)
+      if @precheck == false  # SKip if precheck is true
+        File.open(@outdir+'/os_env.output', 'w') do |f|
+          f.puts(stdout)
+        end
       end
       return os_dict
     end
-    def get_tdenv()
+    
+    def gen_tdenv()
       stdout, stderr, status = Open3.capture3('systemctl cat td-agent')
       env_dict = {}
-        File.open(@workdir+'/td-agent_env.output', 'w') do |f|
-          f.puts(stdout)
-        end
+      if status.success?
+	if @precheck == false  # SKip if precheck is true
+          File.open(@outdir+'/td-agent_env.output', 'w') do |f|
+            f.puts(stdout)
+          end
+	end  
         stdout.split().each do | l |
-        if l.include?('Environment')
-          env_dict[l.split('=')[1]] = l.split('=')[2]
-        end
+          if l.include?('Environment')
+            env_dict[l.split('=')[1]] = l.split('=')[2]
+          end
+      	end
+      else
+        exe = 'fluentd'
+        stdout, stderr, status = Open3.capture3("ps aux | grep #{exe} | grep -v grep")
+        line = stdout.split(/\n/)
+	log_path = ''
+        conf_path = ''
+        line.each { |l|
+          cmd = l.split.drop(10)
+          i = 0
+          log_pos = 0
+          conf_pos = 0
+          if cmd[-1] != '--under-supervisor'
+            cmd.each { |c|
+              if c.include?("--log") || c.include?("-l")
+                log_pos = i + 1
+                log_path = cmd[log_pos]
+              elsif c.include?("--conf") || c.include?("-c")
+                conf_pos = i + 1
+                conf_path = cmd[conf_pos]
+              end
+              i+=1
+            }
+	  end
+	}
+        env_dict['FLUENT_CONF'] = conf_path
+        env_dict['TD_AGENT_LOG_FILE'] = log_path
       end
       return env_dict
     end
+    
     def export_env()
       env = {
         :os => @osenv['Operating System'],
@@ -84,60 +138,118 @@ module Diagtool
       }
       return env
     end
+    
     def collect_tdconf()
-      FileUtils.mkdir_p(@workdir+@tdconf_path)
-      FileUtils.cp(@tdconf_path+@tdconf, @workdir+@tdconf_path)
-      return @workdir+@tdconf_path+@tdconf
+      target_dir = @workdir+@tdconf_path
+      FileUtils.mkdir_p(target_dir)
+      FileUtils.cp(@tdconf_path+@tdconf, target_dir)
+      conf = @workdir+@tdconf_path+@tdconf
+      conf_list = []
+      conf_list.push target_dir + @tdconf
+      File.readlines(conf).each { |line|
+      if line.include? '@include'
+        f = line.split()[1]
+        if f.start_with?(/\//)  # /tmp/work1/b.conf
+          if f.include?('*')
+            Dir.glob(f).each { |ff|
+              conf_inc = target_dir + ff.gsub(/\//,'__')
+              FileUtils.cp(ff, conf_inc)
+              conf_list.push conf_inc
+             }
+	  else 
+	    conf_inc = target_dir+f.gsub(/\//,'__')
+            FileUtils.cp(f, conf_inc)
+            conf_list.push  conf_inc
+	  end
+        else
+	  f = f.gsub('./','') if f.include?('./')
+          if f.include?('*')
+            Dir.glob(@tdconf_path+f).each{ |ff|
+              conf_inc = target_dir + ff.gsub(@tdconf_path,'').gsub(/\//,'__')
+              FileUtils.cp(ff, conf_inc)
+              conf_list.push conf_inc
+            }
+	  else
+            conf_inc = target_dir+f.gsub(/\//,'__')
+            FileUtils.cp(@tdconf_path+f, conf_inc)
+            conf_list.push  conf_inc
+	  end
+        end
+      end
+     }
+     return conf_list
     end
+
     def collect_tdlog()
-      FileUtils.mkdir_p(@workdir+@tdlog_path)
-      FileUtils.cp_r(@tdlog_path, @workdir+@oslog_path)
-      return Dir.glob(@workdir+@tdlog_path+@tdlog+'*')
+      target_dir = @workdir+@tdlog_path
+      FileUtils.mkdir_p(target_dir)
+      Dir.glob(@tdlog_path+@tdlog+'*').each{ |f| 
+        FileUtils.cp(f, target_dir)
+      }
+      return Dir.glob(target_dir+@tdlog+'*')
     end
+    
     def collect_sysctl()
-      FileUtils.mkdir_p(@workdir+@sysctl_path)
-      FileUtils.cp(@sysctl_path+@sysctl, @workdir+@sysctl_path)
-      return @workdir+@sysctl_path+@sysctl
+      target_dir = @workdir+@sysctl_path
+      FileUtils.mkdir_p(target_dir)
+      FileUtils.cp(@sysctl_path+@sysctl, target_dir)
+      return target_dir+@sysctl
     end
+    
     def collect_oslog()
-      FileUtils.mkdir_p(@workdir+@oslog_path)
-      FileUtils.cp(@oslog_path+@oslog, @workdir+@oslog_path)
-      return @workdir+@oslog_path+@oslog
+      target_dir = @workdir+@oslog_path
+      FileUtils.mkdir_p(target_dir)
+      FileUtils.cp(@oslog_path+@oslog, target_dir)
+      return target_dir+@oslog
     end
+    
     def collect_ulimit()
-      output = @workdir+'/ulimit_n.output'
+      output = @outdir+'/ulimit_n.output'
       stdout, stderr, status = Open3.capture3("ulimit -n")
       File.open(output, 'w') do |f|
         f.puts(stdout)
       end
       return output
     end
+   
+    def collect_ps_eo()
+      output = @outdir+'/ps_eo.output'
+      stdout, stderr, status = Open3.capture3("ps -eo pid,ppid,stime,time,%mem,%cpu,cmd")
+      File.open(output, 'w') do |f|
+        f.puts(stdout)
+      end
+      return output
+    end
+
     def collect_meminfo()
-      output = @workdir+'/meminfo.output'
+      output = @outdir+'/meminfo.output'
       stdout, stderr, status = Open3.capture3("cat /proc/meminfo")
       File.open(output, 'w') do |f|
         f.puts(stdout)
       end
       return output
     end
-    def collect_netstat_n()
-      output = @workdir+'/netstat_n.output'
-      stdout, stderr, status = Open3.capture3("netstat -n")
+    
+    def collect_netstat_plan()
+      output = @outdir+'/netstat_plan.output'
+      stdout, stderr, status = Open3.capture3("netstat -plan")
       File.open(output, 'w') do |f|
         f.puts(stdout)
       end
       return output
     end
+    
     def collect_netstat_s()
-      output = @workdir+'/netstat_s.output'
+      output = @outdir+'/netstat_s.output'
       stdout, stderr, status = Open3.capture3("netstat -s")
       File.open(output, 'w') do |f|
         f.puts(stdout)
       end
       return output
     end
+    
     def collect_ntp(command)
-      output = @workdir+'/ntp_info.output'
+      output = @outdir+'/ntp_info.output'
       stdout_date, stderr_date, status_date = Open3.capture3("date")
       stdout_ntp, stderr_ntp, status_ntp = Open3.capture3("chronyc sources") if command == "chrony"
       stdout_ntp, stderr_ntp, status_ntp = Open3.capture3("ntpq -p") if command == "ntp"
@@ -147,19 +259,21 @@ module Diagtool
       end
       return output
     end
+    
     def collect_tdgems()
-      output = @workdir+'/tdgem_list.output'
+      output = @outdir+'/tdgem_list.output'
       stdout, stderr, status = Open3.capture3("td-agent-gem list | grep fluent")
       File.open(output, 'w') do |f|
         f.puts(stdout)
       end
       return output
     end
+    
     def compress_output()
-      Dir.chdir(@output_dir)
+      Dir.chdir(@basedir)
       tar_file = 'diagout-'+@time_format+'.tar.gz'
       stdout, stderr, status = Open3.capture3("tar cvfz #{tar_file} #{@time_format}")
-      return @output_dir + '/' + tar_file
+      return @basedir + '/' + tar_file
     end
   end
 end

data/lib/fluent/diagtool/diagutils.rb

@@ -27,10 +27,62 @@ module Diagtool
       time = Time.new
       @time_format = time.strftime("%Y%m%d%0k%M%0S")
       @conf = parse_diagconf(params)
+      #@conf[:time] = @time_format
+      #@conf[:workdir] = @conf[:basedir] + '/' + @time_format
+      #@conf[:outdir] = @conf[:workdir] + '/output'
+
+      #FileUtils.mkdir_p(@conf[:workdir])
+      #FileUtils.mkdir_p(@conf[:outdir])
+      
+      #diaglog = @conf[:workdir] + '/diagtool.output'
+      #@masklog = './mask_' + @time_format + '.json'
+      #@logger = Logger.new(STDOUT, formatter: proc {|severity, datetime, progname, msg|
+      #  "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
+      #})
+      #@logger_file = Logger.new(diaglog, formatter: proc {|severity, datetime, progname, msg|
+      #  "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
+      #})
+      #diaglogger_info("Parsing command options...")
+      #diaglogger_info("   Option : Output directory = #{@conf[:basedir]}")
+      #diaglogger_info("   Option : Mask = #{@conf[:mask]}")
+      #diaglogger_info("   Option : Word list = #{@conf[:words]}")
+      #diaglogger_info("   Option : Hash Seed = #{@conf[:seed]}")
+    end
+    
+    def run_precheck()
+      prechecklog = Logger.new(STDOUT, formatter: proc {|severity, datetime, progname, msg|
+        "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
+      })
+      loglevel = 'WARN'
+      c = CollectUtils.new(@conf, loglevel)
+      c_env = c.export_env()
+      prechecklog.info("[Precheck] Check OS parameters...")
+      prechecklog.info("[Precheck]    operating system = #{c_env[:os]}")
+      prechecklog.info("[Precheck]    kernel version = #{c_env[:kernel]}")
+      prechecklog.info("[Precheck] Check td-agent parameters...")
+      prechecklog.info("[Precheck]    td-agent conf path = #{c_env[:tdconf_path]}")
+      prechecklog.info("[Precheck]    td-agent conf file = #{c_env[:tdconf]}")
+      prechecklog.info("[Precheck]    td-agent log path = #{c_env[:tdlog_path]}")
+      prechecklog.info("[Precheck]    td-agent log = #{c_env[:tdlog]}")
+      if c_env[:tdconf_path] == nil || c_env[:tdconf] == nil
+	prechecklog.warn("[Precheck]    can not find td-agent conf path: please run diagtool command with -c /path/to/<td-agent conf file>")
+      end
+      if c_env[:tdlog_path] == nil || c_env[:tdlog] == nil
+        prechecklog.warn("[Precheck]    can not find td-agent log path: please run diagtool command with -l /path/to/<td-agent log file>")
+      end
+      if c_env[:tdconf_path] != nil && c_env[:tdconf] != nil && c_env[:tdlog_path] != nil && c_env[:tdlog] != nil
+	 prechecklog.info("[Precheck] Precheck completed. You can run diagtool command without -c and -l options")
+      end
+    end
+
+    def run_diagtool()
       @conf[:time] = @time_format
-      @conf[:workdir] = @conf[:output_dir] + '/' + @time_format
+      @conf[:workdir] = @conf[:basedir] + '/' + @time_format
+      @conf[:outdir] = @conf[:workdir] + '/output'
       FileUtils.mkdir_p(@conf[:workdir])
+      FileUtils.mkdir_p(@conf[:outdir])
       diaglog = @conf[:workdir] + '/diagtool.output'
+
       @masklog = './mask_' + @time_format + '.json'
       @logger = Logger.new(STDOUT, formatter: proc {|severity, datetime, progname, msg|
         "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
@@ -39,12 +91,11 @@ module Diagtool
         "#{datetime}: [Diagtool] [#{severity}] #{msg}\n"
       })
       diaglogger_info("Parsing command options...")
-      diaglogger_info("   Option : Output directory = #{@conf[:output_dir]}")
+      diaglogger_info("   Option : Output directory = #{@conf[:basedir]}")
       diaglogger_info("   Option : Mask = #{@conf[:mask]}")
       diaglogger_info("   Option : Word list = #{@conf[:words]}")
       diaglogger_info("   Option : Hash Seed = #{@conf[:seed]}")
-    end
-    def diagtool()
+
       loglevel = 'WARN'
       diaglogger_info("Initializing parameters...")
       c = CollectUtils.new(@conf, loglevel)
@@ -78,10 +129,14 @@ module Diagtool
         oslog = m.mask_tdlog(oslog, clean = true)
       end
       diaglogger_info("[Collect] config file is stored in #{oslog}")
-			
+
+      diaglogger_info("[Collect] Collecting process information...")
+      meminfo = c.collect_ps_eo()
+      diaglogger_info("[Collect] process informationis stored in #{meminfo}")
+
       diaglogger_info("[Collect] Collecting OS memory information...")
       meminfo = c.collect_meminfo()
-      diaglogger_info("[Collect] config file is stored in #{meminfo}")
+      diaglogger_info("[Collect] OS memory information is stored in #{meminfo}")
 
       diaglogger_info("[Collect] Collecting date/time information...")
       if system('which chronyc > /dev/null 2>&1')
@@ -95,7 +150,7 @@ module Diagtool
 			
       diaglogger_info("[Collect] Collecting netstat information...")
       if system('which netstat > /dev/null 2>&1')
-        netstat_n = c.collect_netstat_n()
+        netstat_n = c.collect_netstat_plan()
         netstat_s = c.collect_netstat_s()
         if @conf[:mask] == 'yes'
           diaglogger_info("[Mask] Masking netstat file : #{netstat_n}...")
@@ -134,9 +189,11 @@ module Diagtool
       end
 
       if @conf[:mask] == 'yes'
-        diaglogger_info("[Mask] Masking td-agent config file : #{tdconf}...")
-        m.mask_tdlog(tdconf, clean = true)
-        tdlog.each do | file |
+	tdconf.each { | file |
+	  diaglogger_info("[Mask] Masking td-agent config file : #{file}...")
+	  m.mask_tdlog(file, clean = true)
+	}
+        tdlog.each { | file |
           diaglogger_info("[Mask] Masking td-agent log file : #{file}...")
           filename = file.split("/")[-1]
           if filename.include?(".gz")
@@ -144,7 +201,7 @@ module Diagtool
           elsif
             m.mask_tdlog(file, clean = true)
           end
-        end
+	}
       end
       
       if @conf[:mask] == 'yes'
@@ -158,20 +215,23 @@ module Diagtool
 
     def parse_diagconf(params)
       options = {
-        :output_dir => '',
-        :mask => 'no',
-        :words => [],
-        :wfile => '',
-        :seed => ''
+        :precheck => '', :basedir => '', :mask => '', :words => [], :wfile => '', :seed => '', :tdconf =>'', :tdlog => ''
       }
-      if params[:output] != nil
-        if Dir.exist?(params[:output])
-          options[:output_dir] = params[:output]
+      if params[:precheck]
+        options[:precheck] = params[:precheck]
+      else
+        options[:precheck] = false
+      end
+      if options[:precheck] == false
+        if params[:output] != nil
+          if Dir.exist?(params[:output])
+            options[:basedir] = params[:output]
+          else
+            raise "output directory '#{basedir}' does not exist"
+          end
         else
-          raise "output directory '#{output_dir}' does not exist"
+          raise "output directory '-o' must be specified"
         end
-      else
-        raise "output directory '-o' must be specified"
       end
       if params[:mask] == nil
         options[:mask] = 'no'
@@ -195,20 +255,43 @@ module Diagtool
       end
       options[:words] = options[:words].uniq 
       options[:seed] = params[:"hash-seed"] if params[:"hash-seed"] != nil
+      
+      if params[:conf] != nil
+        f = params[:conf]
+        if File.exist?(f)
+	  options[:tdconf] = params[:conf]
+        else
+          raise "#{params[:conf]} : No such file or directory"
+        end
+      end
+
+      if params[:log] != nil
+        f = params[:log]
+        if File.exist?(f)
+          options[:tdlog] = params[:log]
+        else
+          raise "#{params[:log]} : No such file or directory"
+        end
+      end
+
       return options	
     end
+    
     def diaglogger_debug(str)
       @logger.debug(str)
       @logger_file.debug(str)
     end
+    
     def diaglogger_info(str)
       @logger.info(str)
       @logger_file.info(str)
     end
+    
     def diaglogger_warn(str)
       @logger.warn(str)
       @logger_file.warn(str)
     end
+    
     def diaglogger_error(str)
       @logger.error(str)
       @logger_file.error(str)

data/lib/fluent/diagtool/maskutils.rb

@@ -29,13 +29,10 @@ module Diagtool
       })
       @logger.debug("Initialize Maskutils: sanitized word = #{conf[:words]}")
       @hash_seed = conf[:seed]
-      @id = {
-        :fid =>'',
-        :lid =>'',
-        :cid =>''
-      }
+      @id = {}
       @masklog = Hash.new { |h,k| h[k] = Hash.new(&h.default_proc) }
     end
+    
     def mask_tdlog(input_file, clean)
       line_id = 0
       f = File.open(input_file+'.mask', 'w')
@@ -51,6 +48,7 @@ module Diagtool
       FileUtils.rm(input_file) if clean == true
       return input_file+'.mask'
     end
+    
     def mask_tdlog_gz(input_file, clean)
       line_id = 0
       f = File.open(input_file+'.mask', 'w')
@@ -68,6 +66,7 @@ module Diagtool
       FileUtils.rm(input_file) if clean == true
       return input_file+'.mask'
     end
+    
     def mask_tdlog_inspector(line)
       i = 0
       contents=[]
@@ -157,6 +156,7 @@ module Diagtool
       @logger.debug("Masked Line: #{line_masked}")
       return line_masked
     end
+    
     def mask_direct_pattern(str)
       is_mask = false
       if str.include?(">")
@@ -173,6 +173,7 @@ module Diagtool
       end
       return is_mask, str_mask
     end	
+    
     def mask_url_pattern(str)
       is_mask = false
       url = str.split('://')
@@ -215,6 +216,7 @@ module Diagtool
       str_mask << ":" if str.end_with?(':')
       return is_mask, str_mask
     end
+    
     def mask_equal_pattern(str)
       is_mask = false
       l = str.split('=') ## Mask host=<address:ip/hostname> or bind=<address: ip/hostname>
@@ -228,6 +230,7 @@ module Diagtool
       str_mask = l.join('=')
       return is_mask, str_mask
     end
+    
     def mask_colon_pattern(str)
       is_mask = false
       l = str.split(':')
@@ -242,6 +245,7 @@ module Diagtool
       str_mask << ":" if str.end_with?(':')
       return is_mask, str_mask
     end
+    
     def mask_slash_pattern(str)
       is_mask = false
       l = str.split('/')
@@ -256,14 +260,17 @@ module Diagtool
       str_mask << ":" if str.end_with?(':')
       return is_mask, str_mask
     end
+    
     def is_ipv4?(str)
       !!(str =~ /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/)
     end
+    
     def is_fqdn?(str)
       #!!(str =~ /^\b((?=[a-z0-9-]{1,63}\.)[a-z0-9]+(-[a-z0-9]+)*\.)+([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$/)
       !!(str =~ /^\b(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.){2,}([A-Za-z]|[A-Za-z][A-Za-z\-]*[A-Za-z]){2,}$/)
       #!!(str =~ /^\b(?=^.{1,254}$)(^(?:(?!\d+\.)[a-zA-Z0-9_\-]{1,63}\.?)+(?:[a-zA-Z]{2,})$)/)
     end
+    
     def is_words?(str)
       value = false
       @words.each do | l |
@@ -274,6 +281,7 @@ module Diagtool
       end
       return value
     end
+    
     def mask_ipv4_fqdn_words(str)
       str = str.to_s
       mtype = ''
@@ -299,11 +307,13 @@ module Diagtool
       end
       return is_mask, str, str_mask
     end
+    
     def put_masklog(str, str_mask)
       uid = "Line#{@id[:lid]}-#{@id[:cid]}"
       @masklog[@id[:fid]][uid]['original'] = str
       @masklog[@id[:fid]][uid]['mask'] = str_mask
     end
+    
     def export_masklog(output_file)
       masklog_json = JSON.pretty_generate(@masklog)
       File.open(output_file, 'w') do |f|

data/lib/fluent/diagtool/validutils.rb

@@ -38,6 +38,7 @@ module Diagtool
       @logger.debug("    Default ulimit: #{@def_ulimit}")
       @logger.debug("    Default sysctl: #{@def_sysctl}")
     end
+    
     def valid_ulimit(ulimit_file)
       @logger.info("Loading ulimit file: #{ulimit_file}")
       File.readlines(ulimit_file).each { |line|
@@ -50,6 +51,7 @@ module Diagtool
         end 
       }
     end
+    
     def valid_sysctl(sysctl_file)
       h = Hash.new()
       v = Hash.new { |i,j| i[j] = Hash.new(&h.default_proc) }

data/lib/fluent/diagtool/version.rb

@@ -1,5 +1,5 @@
 module Fluent
   module Diagtool
-    VERSION = "0.1.1"
+    VERSION = "0.1.6"
   end
 end

metadata

@@ -1,15 +1,43 @@
 --- !ruby/object:Gem::Specification
 name: fluent-diagtool
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.6
 platform: ruby
 authors:
 - kubotat
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-05-19 00:00:00.000000000 Z
-dependencies: []
+date: 2020-05-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fileutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Bringing productivity of trouble shooting to the next level  by automating
   collection of Fluentd configurations, settings and OS parameters as well as masking
   sensitive information in logs and configurations.
@@ -20,6 +48,7 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- ".gitignore"
 - AUTHORS
 - Gemfile
 - Gemfile.lock
@@ -27,7 +56,6 @@ files:
 - README.md
 - Rakefile
 - bin/console
-- bin/diagtool.rb
 - bin/setup
 - bin/word_list_sample
 - exe/diagtool

data/bin/diagtool.rb

@@ -1,37 +0,0 @@
-#
-# Fluentd
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#        http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-#
-
-#!/usr/bin/ruby
-require 'optparse'
-require '../lib/collectutils'
-require '../lib/maskutils'
-require '../lib/validutils'
-require '../lib/diagutils'
-include Diagtool
-
-params = {}
-OptionParser.new do |opt|
-  opt.banner = "Usage: #{$0} -o OUTPUT_DIR -m {yes | no} -w {word1,[word2...]} -f {listfile} -s {hash seed}"
-  opt.on('-o','--output DIR', String, 'Output directory (Mandatory)')
-  opt.on('-m','--mask yes|no', String, 'Enable mask function (Optional : Default=no)')
-  opt.on('-w','--word-list word1,word2', Array, 'Provide a list of user-defined words which will to be masked (Optional : Default=None)')
-  opt.on('-f','--word-file listfile', String, 'provide a file which describes a List of user-defined words (Optional : Default=None)')
-  opt.on('-s','--hash-seed seed', String, 'provide a word which will be used when generate the mask (Optional : Default=None)')
-end.parse!(into: params)
-diag = DiagUtils.new(params)
-diag.diagtool()
-
-