mihari 0.17.0 → 0.17.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a5030ba15be3b81b4a71d9f38f3a0305d023dcddb3121e0c700a0cb82f1dcd9
-  data.tar.gz: fc4f48e49a363550d165367e7aba0e3802535f3b13bcb32304d71a82eab256c7
+  metadata.gz: e28756adc591b441b67579ca8e4a0e7fcaadc1cf332a3c9e57e01438c19d6554
+  data.tar.gz: 0c9bf23eb782882354b6332878a41168293b4957724a27b2beab077f3caf3256
 SHA512:
-  metadata.gz: 79487f0a7350c6299d159a32d7bc9655ac76934c2f2d224b059de59f128c18c4b23725269687cc3bd777d111c7245100f0b735d7011be997fc1bfed293b9f7b1
-  data.tar.gz: 3071b839707180910cd1286041696b9f6b768abff4bb6e9a724ce553945118edb26784c70c848f0f5f888e1be7854c9e0b6d6a1645c8019d726926c10bda065f
+  metadata.gz: a91dc83d7a9988df7aff4b1ad90029c67223dfa717354a4c00fcba40e01b409ff77aba4c07969cc767dafa4d89bc6a5d9af807099665245ff0a578f3df4ae6cd
+  data.tar.gz: 34d6cc2546ce6b9e1a0b5918ec2a14dcf3f8f692222d950a7c27a7f28206308f78c0c40c66487108e5bcedfd70b25bbc9fa9f478862bf390aa041249ab3d31a9

data/.travis.yml

@@ -4,4 +4,4 @@ language: ruby
 cache: bundler
 rvm:
   - 2.6
-before_install: gem install bundler -v 2.0.1
+before_install: gem install bundler -v 2.1

data/README.md

@@ -6,22 +6,22 @@
 [![Coverage Status](https://coveralls.io/repos/github/ninoseki/mihari/badge.svg?branch=master)](https://coveralls.io/github/ninoseki/mihari?branch=master)
 [![CodeFactor](https://www.codefactor.io/repository/github/ninoseki/mihari/badge)](https://www.codefactor.io/repository/github/ninoseki/mihari)
 
-mihari(`見張り`) is a sidekick tool for [TheHive](https://github.com/TheHive-Project/TheHive) for monitoring malicious hosts (C2 / landing page / phishing, etc.) continuously.
+Mihari is a sidekick tool for [TheHive](https://github.com/TheHive-Project/TheHive) for monitoring malicious hosts (C2 / landing page / phishing, etc.) continuously.
 
 ## How it works
 
-- mihari makes a query against Shodan, Censys, VirusTotal, SecurityTrails, etc. and extracts artifacts from the results.
-- mihari checks whether TheHive contains the artifacts or not.
+- Mihari makes a query against Shodan, Censys, VirusTotal, SecurityTrails, etc. and extracts artifacts from the results.
+- Mihari checks whether TheHive contains the artifacts or not.
   - If it doesn't contain the artifacts:
-    - mihari creates an alert on TheHive.
-    - mihari sends a notification to Slack. (Optional)
-    - mihari creates an event on MISP. (Optional)
+    - Mihari creates an alert on TheHive.
+    - Mihari sends a notification to Slack. (Optional)
+    - Mihari creates an event on MISP. (Optional)
 
 ![img](https://github.com/ninoseki/mihari/raw/master/screenshots/eyecatch.png)
 
 Check this blog post for more details: [Continuous C2 hunting with Censys, Shodan, Onyphe and TheHive](https://hackmd.io/s/SkUaSrqoE).
 
-You can use mihari without TheHive. But note that mihari depends on TheHive to manage artifacts. It means mihari might make duplications when without TheHive.
+You can use mihari without TheHive but note that mihari depends on TheHive to manage artifacts. It means mihari might make duplications when without TheHive.
 
 ### Screenshots
 
@@ -51,7 +51,7 @@ docker pull ninoseki/mihari
 
 ## Basic usage
 
-mihari supports the following services by default.
+Mihari supports the following services by default.
 
 - [BinaryEdge](https://www.binaryedge.io/)
 - [Censys](http://censys.io)
@@ -96,11 +96,14 @@ Commands:
   mihari virustotal [IP|DOMAIN]               # VirusTotal resolutions lookup by an ip or domain
   mihari zoomeye [QUERY]                      # ZoomEye search by a query
 
+Options:
+  [--config=CONFIG]  # path to config file
+
 ```
 
 ### Cross searches
 
-mihari has cross search features. A cross search is a search across a number of services.
+Mihari has cross search features. A cross search is a search across a number of services.
 
 You can get aggregated results by using the following commands.
 
@@ -224,7 +227,7 @@ The input is a JSON data should have `title`, `description` and `artifacts` key.
 
 ## Configuration
 
-All configuration is done via ENV variables.
+Configuration can be done via environment variables or a YAML file.
 
 | Key                    | Desc.                          | Required or optional           |
 |------------------------|--------------------------------|--------------------------------|
@@ -249,6 +252,20 @@ All configuration is done via ENV variables.
 | ZOOMEYE_USERNAMME      | ZoomEye username               | Optional                       |
 | ZOOMEYE_PASSWORD       | ZoomEye password               | Optional                       |
 
+Instead of using environment variables, you can use a YAML file for configuration.
+
+```bash
+mihari virustotal 1.1.1.1 --config /path/to/yaml.yml
+```
+
+The YAML file should be a hash like below:
+
+```yaml
+thehive_api_endpoint: https://localhost
+thehive_api_key: foo
+virustotal_api_key: foo
+```
+
 You can check the configuration status via `status` command.
 
 ```bash
@@ -299,7 +316,7 @@ See `/examples` for more.
 
 ## Caching
 
-mihari caches execution results in `/tmp/mihari` and the default cache duration is 7 days. If you want to clear the cache, please clear `/tmp/mihari`.
+Mihari caches execution results in `/tmp/mihari` and the default cache duration is 7 days. If you want to clear the cache, please clear `/tmp/mihari`.
 
 ## Using it with Docker
 

data/lib/mihari.rb

@@ -24,6 +24,7 @@ require "mihari/errors"
 
 require "mihari/artifact"
 require "mihari/cache"
+require "mihari/config"
 require "mihari/type_checker"
 
 require "mihari/html"

data/lib/mihari/analyzers/circl.rb

@@ -43,8 +43,6 @@ module Mihari
         else
           raise InvalidInputError, "#{@query}(type: #{@type || 'unknown'}) is not supported."
         end
-      rescue ::PassiveCIRCL::Error => _e
-        nil
       end
 
       def passive_dns_lookup

data/lib/mihari/analyzers/dnpedia.rb

@@ -35,8 +35,6 @@ module Mihari
         rows.map do |row|
           [row.dig("name"), row.dig("zoneid")].join(".")
         end
-      rescue ::DNPedia::Error => _e
-        nil
       end
     end
   end

data/lib/mihari/analyzers/dnstwister.rb

@@ -55,8 +55,6 @@ module Mihari
         Parallel.map(domains) do |domain|
           resolvable?(domain) ? domain : nil
         end.compact
-      rescue ::DNSTwister::Error => _e
-        nil
       end
     end
   end

data/lib/mihari/analyzers/passivetotal.rb

@@ -54,8 +54,6 @@ module Mihari
         else
           raise InvalidInputError, "#{query}(type: #{type || 'unknown'}) is not supported." unless valid_type?
         end
-      rescue ::PassiveTotal::Error => _e
-        nil
       end
 
       def passive_dns_lookup

data/lib/mihari/analyzers/pulsedive.rb

@@ -51,8 +51,6 @@ module Mihari
         (properties.dig("dns") || []).map do |property|
           property.dig("value") if ["A", "PTR"].include?(property.dig("name"))
         end.compact
-      rescue ::Pulsedive::ResponseError => _e
-        nil
       end
     end
   end

data/lib/mihari/analyzers/securitytrails.rb

@@ -52,8 +52,6 @@ module Mihari
         else
           raise InvalidInputError, "#{query}(type: #{type || 'unknown'}) is not supported." unless valid_type?
         end
-      rescue ::SecurityTrails::Error => _e
-        nil
       end
 
       def domain_lookup

data/lib/mihari/analyzers/securitytrails_domain_feed.rb

@@ -53,8 +53,6 @@ module Mihari
         new_domains.select do |domain|
           regexp.match? domain
         end
-      rescue ::SecurityTrails::Error => _e
-        nil
       end
 
       def new_domains

data/lib/mihari/analyzers/shodan.rb

@@ -45,8 +45,6 @@ module Mihari
 
       def search_with_page(query, page: 1)
         api.host.search(query, page: page)
-      rescue ::Shodan::Error => _e
-        nil
       end
 
       def search

data/lib/mihari/analyzers/urlscan.rb

@@ -41,8 +41,6 @@ module Mihari
 
       def search
         api.search(query, size: 10_000)
-      rescue ::UrlScan::ResponseError => _e
-        nil
       end
 
       def valid_target_type?

data/lib/mihari/analyzers/virustotal.rb

@@ -50,16 +50,10 @@ module Mihari
         else
           raise InvalidInputError, "#{indicator}(type: #{type || 'unknown'}) is not supported." unless valid_type?
         end
-      rescue ::VirusTotal::Error => _e
-        nil
       end
 
       def domain_lookup
-        begin
-          res = api.domain.resolutions(indicator)
-        rescue ::VirusTotal::Error => _e
-          return nil
-        end
+        res = api.domain.resolutions(indicator)
 
         data = res.dig("data") || []
         data.map do |item|
@@ -68,11 +62,7 @@ module Mihari
       end
 
       def ip_lookup
-        begin
-          res = api.ip_address.resolutions(indicator)
-        rescue ::VirusTotal::Error => _e
-          return nil
-        end
+        res = api.ip_address.resolutions(indicator)
 
         data = res.dig("data") || []
         data.map do |item|

data/lib/mihari/cli.rb

@@ -5,6 +5,8 @@ require "json"
 
 module Mihari
   class CLI < Thor
+    class_option :config, type: :string, desc: "path to config file"
+
     desc "censys [QUERY]", "Censys IPv4 search by a query"
     method_option :title, type: :string, desc: "title"
     method_option :description, type: :string, desc: "description"
@@ -280,15 +282,29 @@ module Mihari
         %w(title description artifacts).all? { |key| json.key? key }
       end
 
+      def load_configuration
+        config = options["config"]
+        Config.load_from_yaml(config) if config
+      end
+
       def run_analyzer(analyzer_class, query:, options:)
+        load_configuration
+
         options = symbolize_hash_keys(options)
+        options = normalize_options(options)
 
         analyzer = analyzer_class.new(query, **options)
         analyzer.run
       end
 
       def symbolize_hash_keys(hash)
-        hash.map{ |k, v| [k.to_sym, v] }.to_h
+        hash.map { |k, v| [k.to_sym, v] }.to_h
+      end
+
+      def normalize_options(options)
+       # Delete :config because it is not intended to use for running an analyzer
+       options.delete(:config)
+       options
       end
 
       def refang(indicator)

data/lib/mihari/config.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+require "yaml"
+
+module Mihari
+  class Config
+    class << self
+      def load_from_yaml(path)
+        raise ArgumentError, "#{path} does not exist." unless File.exist?(path)
+
+        data = File.read(path)
+        begin
+          yaml = YAML.safe_load(data)
+        rescue TypeError => _e
+          return
+        end
+
+        yaml.each do |key, value|
+          ENV[key.upcase] = value
+        end
+      end
+    end
+  end
+end

data/lib/mihari/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Mihari
-  VERSION = "0.17.0"
+  VERSION = "0.17.1"
 end

data/mihari.gemspec

@@ -24,7 +24,7 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.add_development_dependency "bundler", "~> 2.0"
+  spec.add_development_dependency "bundler", "~> 2.1"
   spec.add_development_dependency "coveralls", "~> 0.8"
   spec.add_development_dependency "fakefs", "~> 0.20"
   spec.add_development_dependency "rake", "~> 13.0"
@@ -46,7 +46,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency "misp", "~> 0.1"
   spec.add_dependency "murmurhash3", "~> 0.1"
   spec.add_dependency "net-ping", "~> 2.0"
-  spec.add_dependency "onyphe", "~> 1.0"
+  spec.add_dependency "onyphe", "~> 1.1"
   spec.add_dependency "parallel", "~> 1.19"
   spec.add_dependency "passive_circl", "~> 0.1"
   spec.add_dependency "passivetotalx", "~> 0.1"
@@ -55,8 +55,8 @@ Gem::Specification.new do |spec|
   spec.add_dependency "securitytrails", "~> 1.0"
   spec.add_dependency "shodanx", "~> 0.2"
   spec.add_dependency "slack-notifier", "~> 2.3"
-  spec.add_dependency "thor", "~> 0.20"
-  spec.add_dependency "urlscan", "~> 0.4"
+  spec.add_dependency "thor", "~> 1.0"
+  spec.add_dependency "urlscan", "~> 0.5"
   spec.add_dependency "virustotalx", "~> 1.1"
   spec.add_dependency "zoomeye-rb", "~> 0.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mihari
 version: !ruby/object:Gem::Version
-  version: 0.17.0
+  version: 0.17.1
 platform: ruby
 authors:
 - Manabu Niseki
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-12-11 00:00:00.000000000 Z
+date: 2019-12-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '2.1'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '2.1'
 - !ruby/object:Gem::Dependency
   name: coveralls
   requirement: !ruby/object:Gem::Requirement
@@ -310,14 +310,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1.1'
 - !ruby/object:Gem::Dependency
   name: parallel
   requirement: !ruby/object:Gem::Requirement
@@ -436,28 +436,28 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.20'
+        version: '1.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.20'
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: urlscan
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.5'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.5'
 - !ruby/object:Gem::Dependency
   name: virustotalx
   requirement: !ruby/object:Gem::Requirement
@@ -534,6 +534,7 @@ files:
 - lib/mihari/artifact.rb
 - lib/mihari/cache.rb
 - lib/mihari/cli.rb
+- lib/mihari/config.rb
 - lib/mihari/configurable.rb
 - lib/mihari/emitters/base.rb
 - lib/mihari/emitters/misp.rb