mihari 0.8.2 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2be8fa691c300847f8986ad6f89f23360ae32c3c42865c9da439bf8ea966e698
-  data.tar.gz: d2ad505e1fb086d334c232cf4811effcff2109593cac150779eda415e672a4c9
+  metadata.gz: a1801d9d520fb6de9a4a27d17145b775b074f87e7e1ff88fcb7c50d5630d19e4
+  data.tar.gz: 886708647767e1d5d5a7aaaf7bfd9aca8d25bc714f7f2faf72e0c2c2c4b3073f
 SHA512:
-  metadata.gz: 032e012ae1ed2999bd626b7d3f4c7f6a06df58789ea3a885e3efb3e6b51c172d4868d6f1eee45af60a4ed430bb9967f6d27288f4a62a8cf051747c71811cb83d
-  data.tar.gz: 938f80eed835fa6a671e23f079c9f60b061b0006bec7281b8e0f8161dbebda1039fd2e066f8dd587df1bc2ab2e5ffe673737a24c04124cd5dc7066432b53bb54
+  metadata.gz: 727703b118506f0b310ed1422a757f669748057bd6b4bdb091c638d716209c5ff220088c1396c91924ca275c6fb959e2014f80cb5b9ecbdf72eee23025cf2a7d
+  data.tar.gz: 150c88bac7b129001b109b1966dc527eda5a7e734438507028e5e2a79f74d1884d6cf98d63a4142a8c45f37f501093a7214187aac88265d9a15a9e6f3d24d03e

data/README.md

@@ -9,14 +9,18 @@ mihari(`見張り`) is a sidekick tool for [TheHive](https://github.com/TheHive-
 
 ## How it works
 
-- mihari checks whether a TheHive instance contains given artifacts or not.
+- mihari makes a query against Shodan, Censys, VirusTotal, SecurityTrails, etc. and extracts artifacts from the query results.
+- mihari checks whether a TheHive instance contains the artifacts or not.
   - If it doesn't contain the artifacts:
     - mihari creates an alert with the artifacts on the TheHive instance.
     - mihari sends a notification to Slack. (Optional)
+    - mihari creates an event on MISP. (Optional)
 
 ![img](./screenshots/eyecatch.png)
 
-Check this blog post for more detail: [Continuous C2 hunting with Censys, Shodan, Onyphe and TheHive](https://hackmd.io/s/SkUaSrqoE)
+Check this blog post for more detail: [Continuous C2 hunting with Censys, Shodan, Onyphe and TheHive](https://hackmd.io/s/SkUaSrqoE).
+
+You can use mihari without TheHive. But note that mihari depends on TheHive to manage artifacts. It means mihari might make duplications when without TheHive.
 
 ### Screenshots
 
@@ -28,12 +32,22 @@ Check this blog post for more detail: [Continuous C2 hunting with Censys, Shodan
 
 ![img](./screenshots/slack.png)
 
+- MISP event example
+
+![img](./screenshots/misp.png)
+
 ## Installation
 
 ```bash
 gem install mihari
 ```
 
+Or you can use this tool with Docker.
+
+```bash
+docker pull ninoseki/mihari
+```
+
 ## Basic usage
 
 mihari supports Censys, Shodan, Onyphe, urlscan, SecurityTrails, crt.sh and VirusTotal by default.
@@ -56,6 +70,52 @@ Commands:
 
 ```
 
+### Example usages
+
+```bash
+# Censys lookup for PANDA C2
+$ mihari censys '("PANDA" AND "SMAdmin" AND "layui")' --title "PANDA C2"
+{
+  "title": "PANDA C2",
+  "description": "query = (\"PANDA\" AND \"SMAdmin\" AND \"layui\")",
+  "artifacts": [
+    "154.223.165.223",
+    "154.194.2.31",
+    "45.114.127.119",
+    "..."
+  ],
+  "tags": []
+}
+
+# VirusTotal passive DNS lookup for a FAKESPY host
+$ mihari virustotal "jppost-hi.top" --title "FAKESPY host passive DNS results"
+{
+  "title": "FAKESPY host passive DNS results",
+  "description": "indicator = jppost-hi.top",
+  "artifacts": [
+    "185.22.152.28",
+    "192.236.200.44",
+    "193.148.69.12",
+    "..."
+  ],
+  "tags": []
+}
+
+# SecurityTrails domain feed lookup for finding (possibly) Apple phishing websites
+mihari securitytrails_domain_feed "apple-" --type new
+{
+  "title": "SecurityTrails domain feed lookup",
+  "description": "Regexp = /apple-/",
+  "artifacts": [
+    "apple-sign.online",
+    "apple-log-in.com",
+    "apple-locator-id.info",
+    "..."
+  ],
+  "tags": []
+}
+```
+
 ### Import from JSON
 
 ```bash
@@ -88,6 +148,8 @@ All configuration is done via ENV variables.
 |------------------------|------------------------|--------------------------------|
 | THEHIVE_API_ENDPOINT   | TheHive URL            | Required                       |
 | THEHIVE_API_KEY        | TheHive API key        | Required                       |
+| MISP_API_ENDPOINT      | MISP URL               | Optional                       |
+| MISP_API_KEY           | MISP API key           | Optional                       |
 | SLACK_WEBHOOK_URL      | Slack Webhook URL      | Optional                       |
 | SLACK_CHANNEL          | Slack channel name     | Optional (default: `#general`) |
 | CENSYS_ID              | Censys API ID          | Optional                       |
@@ -97,7 +159,7 @@ All configuration is done via ENV variables.
 | SHODAN_API_KEY         | Shodan API key         | Optional                       |
 | VIRUSTOTAL_API_KEY     | VirusTotal API key     | Optional                       |
 
-## How to create a custom analyzer
+## How to create a custom script
 
 Create a class which extends `Mihari::Analyzers::Base` and implements the following methods.
 
@@ -143,6 +205,30 @@ See `/examples` for more.
 
 mihari caches execution results in `/tmp/mihari` and the default cache duration is 7 days. If you want to clear the cache, please clear `/tmp/mihari`.
 
+## Using it with Docker
+
+```bash
+$ docker run --rm ninoseki/mihari
+Commands:
+  mihari alerts                               # Show the alerts on TheHive
+  mihari censys [QUERY]                       # Censys IPv4 lookup by a given...
+  mihari crtsh [QUERY]                        # crt.sh lookup by a given query
+  mihari help [COMMAND]                       # Describe available commands o...
+  mihari import_from_json                     # Give a JSON input via STDIN
+  mihari onyphe [QUERY]                       # Onyphe datascan lookup by a g...
+  mihari securitytrails [IP|DOMAIN]           # SecurityTrails resolutions lo...
+  mihari securitytrails_domain_feed [REGEXP]  # SecurityTrails new domain fee...
+  mihari shodan [QUERY]                       # Shodan host lookup by a given...
+  mihari status                               # Show the current configuratio...
+  mihari urlscan [QUERY]                      # urlscan lookup by a given query
+  mihari virustotal [IP|DOMAIN]               # VirusTotal resolutions lookup...
+
+# Note that you should pass configurations via environment variables
+$ docker run --rm ninoseki/mihari -e THEHIVE_API_ENDPOINT="http://THEHIVE_URL" -e THEHIVE_API_KEY="API KEY" mihari
+# or
+$ docker run --rm ninoseki/mihari --env-file ~/.mihari.env mihari
+```
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/docker/Dockerfile

@@ -0,0 +1,11 @@
+FROM ruby:2.6
+
+RUN cd /tmp/ \
+  && git clone https://github.com/ninoseki/mihari.git \
+  && cd mihari \
+  && gem build mihari.gemspec -o mihari.gem \
+  && gem install mihari.gem
+
+ENTRYPOINT ["mihari"]
+
+CMD ["--help"]

data/lib/mihari.rb

@@ -42,6 +42,7 @@ require "mihari/notifiers/slack"
 require "mihari/notifiers/exception_notifier"
 
 require "mihari/emitters/base"
+require "mihari/emitters/misp"
 require "mihari/emitters/slack"
 require "mihari/emitters/stdout"
 require "mihari/emitters/the_hive"

data/lib/mihari/analyzers/base.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "parallel"
+
 module Mihari
   module Analyzers
     class Base
@@ -29,7 +31,9 @@ module Mihari
       end
 
       def run
-        Mihari.emitters.each do |emitter_class|
+        set_unique_artifacts
+
+        Parallel.each(Mihari.emitters) do |emitter_class|
           emitter = emitter_class.new
           next unless emitter.valid?
 
@@ -64,6 +68,12 @@ module Mihari
 
         @unique_artifacts ||= @the_hive.artifact.find_non_existing_artifacts(uncached_artifacts)
       end
+
+      private
+
+      def set_unique_artifacts
+        unique_artifacts
+      end
     end
   end
 end

data/lib/mihari/emitters/misp.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+require "misp"
+require "net/ping"
+
+module Mihari
+  module Emitters
+    class MISP < Base
+      # @return [true, false]
+      def valid?
+        api_endpoint? && api_key? && ping?
+      end
+
+      def emit(title:, description:, artifacts:, tags: [])
+        event = ::MISP::Event.new(info: title)
+
+        artifacts.each do |artifact|
+          event.attributes << build_attribute(artifact)
+        end
+
+        tags.each do |tag|
+          event.add_tag name: tag
+        end
+
+        event.create
+      end
+
+      private
+
+      def build_attribute(artifact)
+        ::MISP::Attribute.new(value: artifact.data, type: to_misp_type(type: artifact.data_type, value: artifact.data))
+      end
+
+      def hash_type(value)
+        case value.length
+        when 32
+          "md5"
+        when 40
+          "sha1"
+        when 64
+          "sha256"
+        when 128
+          "sha512"
+        end
+      end
+
+      def to_misp_type(type:, value:)
+        type = type.to_sym
+        table = {
+          ip: "ip-dst",
+          mail: "email-dst",
+          url: "url",
+          domain: "domain",
+        }
+        return table[type] if table.key?(type)
+
+        hash_type value
+      end
+
+      def create_attribute(artifact)
+        artifact.data_type
+      end
+
+      def api_endpoint?
+        api_endpoint = ::MISP.configuration.api_endpoint
+        !api_endpoint.nil? && !api_endpoint.empty?
+      end
+
+      def api_key?
+        api_key = ::MISP.configuration.api_key
+        !api_key.nil? && !api_key.empty?
+      end
+
+      def ping?
+        base_url = ::MISP.configuration.api_endpoint
+        base_url = base_url.end_with?("/") ? base_url[0..-2] : base_url
+        url = "#{base_url}/users/login"
+
+        http = Net::Ping::HTTP.new(url)
+        http.ping?
+      end
+    end
+  end
+end

data/lib/mihari/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Mihari
-  VERSION = "0.8.2"
+  VERSION = "0.9.0"
 end

data/mihari.gemspec

@@ -40,8 +40,10 @@ Gem::Specification.new do |spec|
   spec.add_dependency "hachi", "~> 0.2"
   spec.add_dependency "lightly", "~> 0.3"
   spec.add_dependency "mem", "~> 0.1"
+  spec.add_dependency "misp", "~> 0.1"
   spec.add_dependency "net-ping", "~> 2.0"
   spec.add_dependency "onyphe", "~> 0.2"
+  spec.add_dependency "parallel", "~> 1.17"
   spec.add_dependency "public_suffix", "~> 4.0"
   spec.add_dependency "securitytrails", "~> 0.2"
   spec.add_dependency "shodanx", "~> 0.2"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mihari
 version: !ruby/object:Gem::Version
-  version: 0.8.2
+  version: 0.9.0
 platform: ruby
 authors:
 - Manabu Niseki
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-09-19 00:00:00.000000000 Z
+date: 2019-09-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -220,6 +220,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: misp
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
 - !ruby/object:Gem::Dependency
   name: net-ping
   requirement: !ruby/object:Gem::Requirement
@@ -248,6 +262,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.2'
+- !ruby/object:Gem::Dependency
+  name: parallel
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.17'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.17'
 - !ruby/object:Gem::Dependency
   name: public_suffix
   requirement: !ruby/object:Gem::Requirement
@@ -363,6 +391,7 @@ files:
 - Rakefile
 - bin/console
 - bin/setup
+- docker/Dockerfile
 - examples/ipinfo_hosted_domains.rb
 - examples/vt_passive_dns.rb
 - exe/mihari
@@ -382,6 +411,7 @@ files:
 - lib/mihari/cache.rb
 - lib/mihari/cli.rb
 - lib/mihari/emitters/base.rb
+- lib/mihari/emitters/misp.rb
 - lib/mihari/emitters/slack.rb
 - lib/mihari/emitters/stdout.rb
 - lib/mihari/emitters/the_hive.rb
@@ -399,6 +429,7 @@ files:
 - mihari.gemspec
 - screenshots/alert.png
 - screenshots/eyecatch.png
+- screenshots/misp.png
 - screenshots/slack.png
 homepage: https://github.com/ninoseki/mihari
 licenses: