mihari 5.4.3 → 5.4.5

data/docs/emitters/database.md

@@ -0,0 +1,22 @@
+# Database
+
+This emitter stores data in a database. This emitter uses SQLite3 by default but you can change to use MySQL or PostgreSQL. The database is a primary database of Mihari. Each data generated by Mihari is stored in the database. You can view the data via the built-in web app.
+
+Mihari loads a database URL via environment variable `DATABASE_URL`. Defaults to `sqlite3:///mihari.db"` (SQLite3).
+
+If you want to use MySQL or PostgreSQL, please set a database URL for that.
+
+- MySQL: `mysql2://username:password@host:3306/database` (+ `gem install mysql2`)
+- PostgreSQL: `postgres://username:password@host:5432/database` (+ `gem install pg`)
+
+```yaml
+emitter: database
+```
+
+!!! note
+
+    You have to initialize the database by `mihari db migrate`.
+
+## ER Diagram
+
+![](https://imgur.com/krhoSgh.png)

data/docs/emitters/hive.md

@@ -0,0 +1,18 @@
+# TheHive
+
+- [https://thehive-project.org/](https://thehive-project.org/)
+
+This emitter creates an alert on TheHive. TheHive v4 & v5 are supported.
+
+```yaml
+emitter: the_hive
+url: ...
+api_key: ...
+api_version: ...
+```
+
+| Name        | Type   | Required? | Default                    | Desc.               |
+| ----------- | ------ | --------- | -------------------------- | ------------------- |
+| url         | String | No        | ENV[”THEHIVE_URL”]         | TheHive API URL     |
+| api_key     | String | No        | ENV[”THEHIVE_API_KEY”]     | TheHive API key     |
+| api_version | String | No        | ENV[”THEHIVE_API_VERSION”] | TheHive API version |

data/docs/emitters/index.md

@@ -0,0 +1,7 @@
+# Emitters
+
+- [Database](database.md)
+- [TheHive](hive.md)
+- [MISP](misp.md)
+- [Slack](slack.md)
+- [Webhook](webhook.md)

data/docs/emitters/misp.md

@@ -0,0 +1,16 @@
+# MISP
+
+- [https://www.misp-project.org/](https://www.misp-project.org/)
+
+This emitter creates an event on MISP based on an alert.
+
+```yaml
+emitter: misp
+url: ...
+api_key: ...
+```
+
+| Name    | Type   | Required? | Default             | Desc.        |
+| ------- | ------ | --------- | ------------------- | ------------ |
+| url     | String | No        | ENV[”MISP_URL”]     | MISP API URL |
+| api_key | String | No        | ENV[”MISP_API_KEY”] | MISP API key |

data/docs/emitters/slack.md

@@ -0,0 +1,16 @@
+# Slack
+
+- [https://slack.com/](https://slack.com/intl/ja-jp/)
+
+This emitter post a message to Slack via incoming webhook.
+
+```yaml
+emitter: slack
+webhook_url: ...
+channel: ...
+```
+
+| Name        | Type   | Required? | Default                         | Desc.             |
+| ----------- | ------ | --------- | ------------------------------- | ----------------- |
+| webhook_url | String | No        | ENV[SLACK_WEBHOOK_URL]          | Slack webhook URL |
+| channel     | String | No        | ENV[SLACK_CHANNEL] / `#general` | Slack channel     |

data/docs/emitters/webhook.md

@@ -0,0 +1,63 @@
+# Webhook
+
+This emitter creates an HTTP request payload based on the specified conditions.
+
+```yaml
+emitter: webhook
+url: ...
+method: ...
+headers: ...
+template: ...
+```
+
+| Name     | Type   | Required? | Default | Desc.                                                |
+| -------- | ------ | --------- | ------- | ---------------------------------------------------- |
+| url      | String | Yes       |         | URL                                                  |
+| method   | String | No        | POST    | HTTP request method (GET or POST)                    |
+| headers  | Hash   | No        |         | HTTP request headers                                 |
+| template | String | No        |         | ERB template to customize the payload in JSON format |
+
+You can customize the payload by using **template**.
+
+A template is an ERB template. It should generate a valid JSON.
+
+- [https://github.com/ruby/erb](https://github.com/ruby/erb)
+
+You can use the following variables to build the JSON.
+
+| Name        | Type                    | Default | Desc.        |
+| ----------- | ----------------------- | ------- | ------------ |
+| title       | String                  |         |              |
+| description | String                  |         |              |
+| source      | String                  |         | ID of a rule |
+| tags        | Array<String>           | []      |              |
+| artifacts   | Array<Mihari::Artifact> |         |              |
+
+## Example
+
+**ThreatFox**
+
+```yaml
+- emitter: webhook
+  url: https://threatfox-api.abuse.ch/api/v1/
+  headers:
+    api-key: YOUR_API_KEY
+  template: threatfox.erb
+```
+
+```ruby
+{
+	"query": "submit_ioc",
+	"threat_type": "payload_delivery",
+	"ioc_type": "ip:port",
+	"malware": "foobar",
+	"confidence_level": 100,
+	"anonymous": 0,
+	"iocs": [
+		<% @artifacts.select { |artifact| artifact.data_type == "ip" }.each_with_index do |artifact, idx| %>
+			"<%= artifact.data %>:80"
+			<%= ',' if idx < (@artifacts.length - 1) %>
+		<% end %>
+	]
+}
+```

data/docs/enrichers/google_public_dns.md

@@ -0,0 +1,19 @@
+---
+tags:
+  - DNS record
+---
+
+# Google Public DNS
+
+- [https://developers.google.com/speed/public-dns](https://developers.google.com/speed/public-dns)
+
+This enricher uses Google Public DNS to enrich an URL and domain artifact.
+
+```yaml
+enricher: google_public_dns
+```
+
+## Supported Artifacts
+
+- URL
+- Domain

data/docs/enrichers/index.md

@@ -0,0 +1,6 @@
+# Enrichers
+
+- [Google Public DNS](google_public_dns.md)
+- [IPInfo](ipinfo.md)
+- [Shodan](shodan.md)
+- [Whois](whois.md)

data/docs/enrichers/ipinfo.md

@@ -0,0 +1,19 @@
+---
+tags:
+  - Autonomous system
+  - Geolocation
+---
+
+# ipinfo.io
+
+- [https://ipinfo.io/](https://ipinfo.io/)
+
+This enricher uses ipinfo.io API to enrich an IP artifact.
+
+```yaml
+enricher: ipinfo
+```
+
+## Supported Artifacts
+
+- IP address

data/docs/enrichers/shodan.md

@@ -0,0 +1,22 @@
+---
+tags:
+  - Port
+  - CPE
+  - DNS record
+---
+
+# Shodan
+
+- [https://www.shodan.io/](https://www.shodan.io/dashboard)
+
+This enricher uses Shodan InternetDB API to enrich an artifact.
+
+[https://internetdb.shodan.io/](https://internetdb.shodan.io/)
+
+```yaml
+enricher: shodan
+```
+
+## Supported Artifacts
+
+- IP address

data/docs/enrichers/whois.md

@@ -0,0 +1,17 @@
+---
+tags:
+  - Whois
+---
+
+# Whois
+
+This enricher uses “whois” command to enrich an artifact.
+
+```yaml
+enricher: whois
+```
+
+## Supported Artifacts
+
+- URL
+- Domain

data/docs/github_actions.md

@@ -0,0 +1,43 @@
+# GitHub Actions
+
+GitHub Actions is a good way to run Mihari searches continuously.
+
+The following is an example of a GitHub Actions workflow to run Mihari.
+
+```yaml
+name: Mihari searches
+
+on:
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install dependencies
+        run: sudo apt-get -yqq install sqlite3 libsqlite3-dev
+      - name: Set up Ruby 3.2
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "3.2"
+          bundler-cache: true
+      - name: Run Mihari
+        run: |
+          mihari search /path/to/rule.yml
+```
+
+!!! tip
+
+    You need to install `libpq-dev` for PostgreSQL, `libmysqlclient-dev` for MySQL.
+
+This example assumes that you have `Gemfile` in your repository.
+
+```ruby
+source "https://rubygems.org"
+
+gem "pg"     # if you use PostgresSQL
+gem "mysql2" # if you use MySQL
+
+gem "mihari"
+```

data/docs/index.md

@@ -0,0 +1,13 @@
+# Mihari
+
+A query aggregator for OSINT based threat hunting.
+
+Mihari can aggregate multiple searches across multiple services in a single rule & persist findings in a database.
+
+- [Requirements](./requirements.md)
+- [Installation](./installation.md)
+- [How to Write a Rule](./rule.md)
+- [Usage](./usage.md)
+- [Configuration](./configuration.md)
+- [GitHub Actions](./github_actions.md)
+- [Alternatives](./alternatives.md)

data/docs/installation.md

@@ -0,0 +1,31 @@
+# Installation
+
+## Ruby Gem
+
+Mihari is packaged as a Ruby Gem.
+
+```bash
+gem install mihari
+```
+
+Mihari uses SQLite3 as a primary database by default. Thus a gem for SQLite (`sqlite3`) is installed by default.
+
+If you want to use MySQL or PostgreSQL instead of SQLite3, please install a gem for that by yourself.
+
+**MySQL**
+
+```bash
+gem install mysql2
+```
+
+**PostgreSQL**
+
+```bash
+gem install pg
+```
+
+# Docker
+
+You can built the Docker image by yourself.
+
+`Dockerfile` is available at [https://github.com/ninoseki/mihari/tree/master/docker](https://github.com/ninoseki/mihari/tree/master/docker).

data/docs/requirements.md

@@ -0,0 +1,20 @@
+# Requirements
+
+- Runtime:
+  - Ruby 2.7+ / 3.0+ (tested with 2.7, 3.0, 3.1 and 3.2)
+- Database:
+  - SQLite3, PostgreSQL and MySQL
+- Others:
+  - MISP
+  - TheHive
+
+| Name       | Supported versions      |
+| ---------- | ----------------------- |
+| Ruby       | v2.7, v3.0, v3.1 & v3.2 |
+| PostgreSQL | v15                     |
+| SQLite     | v3                      |
+| MySQL      | v8                      |
+| MISP       | v2.4                    |
+| TheHive    | v3 & v4                 |
+
+You need to have a database to persistent the data. See [Database](./emitters/database.md) for details.

data/docs/rule.md

@@ -0,0 +1,165 @@
+# How to Write a Rule
+
+Mihari has [Sigma](https://github.com/SigmaHQ/sigma) like format to describe a set of search queries to express a rule.
+
+Mihari has three main components to compose a rule.
+
+![](https://imgur.com/BBT99BG.png)
+
+- Analyzers/Queries: a list of queries (analyzers) that builds a list of artifacts
+- Enrichers: a list of enrichers that enriches a list of artifacts
+- Emitters: a list of emitters that emits a list of artifacts as an alert
+
+An artifact has five types:
+
+- IP address (`ip`)
+- Domain (`domain`)
+- URL (`url`)
+- Mail (`mail`)
+- Hash (`hash`)
+
+An alert can have multiple artifacts bundled by a rule.
+
+Let's break down the following example:
+
+```yaml
+id: c7f6968e-dbe1-4612-b0bb-8407a4fe05df
+title: Example
+description: Mihari rule example
+created_on: "2023-01-01"
+updated_on: "2023-01-02"
+author: ninoseki
+references:
+  - https://github.com/ninoseki/mihari
+related:
+  - 6254bb74-5e5d-42ad-bc1e-231da0293b0f
+tags:
+  - foo
+  - bar
+queries:
+  - analyzer: shodan
+    query: ip:1.1.1.1
+  - analyzer: censys
+    query: ip:8.8.8.8
+enrichers:
+  - enricher: whois
+  - enricher: ipinfo
+  - enricher: shodan
+  - enricher: google_public_dns
+emitters:
+  - emitter: database
+  - emitter: misp
+  - emitter: slack
+  - emitter: the_hive
+data_types:
+  - hash
+  - ip
+  - domain
+  - url
+  - mail
+falsepositives: []
+```
+
+## ID
+
+`id` is an unique ID of a rule. UUID v4 is recommended.
+
+## Title
+
+`title` is a title of a rule.
+
+## Description
+
+`description` is a short description of a rule.
+
+## Created/Updated On
+
+`created_on` is a date of a rule creation. Optional.
+Also a rule can have `updated_on` that is a date of a rule modification. Optional.
+
+## Tags
+
+`tags` is a list of tags of a rule.
+
+## Author
+
+`author` is an author of a rule. Optional.
+
+## References
+
+`references` is a list of a references of a rule. Optional.
+
+## Related
+
+`related` is a list of related rule IDs. Optional.
+
+## Queries
+
+`queries` is a list of queries/analyzers.
+See [Analyzers](./analyzers/index.md) to know details of each analyzer.
+
+## Enrichers
+
+`enrichers` is a list of enrichers.
+See [Enrichers](./enrichers/index.md) to know details of each enricher.
+
+Defaults to:
+
+- `google_public_dns`
+- `ipinfo`
+- `shodan`
+- `whois`
+
+## Emitters
+
+`emitters` is a list of emitters.
+See [Emitters](./emitters/index.md) to know details of each emitter.
+
+Defaults to:
+
+- `database`
+- `misp`
+- `slack`
+- `the_hive`
+
+## Data Types
+
+`data_types` is a list of data (artifact) types to allow by a rule. Types not defined in here will be automatically rejected.
+
+Defaults to:
+
+- `ip`
+- `domain`
+- `url`
+- `mail`
+- `hash`
+
+## False positives
+
+`falsepositives` is a list of false positive values. A string or regexp can be used in here.
+
+## Artifact TTL
+
+`artifact_ttl` (alias: `artifact_lifetime`) is an integer value of artifact TTL (Time-To-Live) in seconds.
+
+Mihari rejects a same artifact in a same rule in general.
+
+But you may want to get a same artifact after a certain period of time. `artifact_ttl` is for that. If a rule finds a same artifact after `artifact_ttl` seconds have been passed, that artifact will be included in an alert.
+
+## How to Run a Rule
+
+Once you finish writing a rule, you can run the rule by `mihari` CLI.
+
+!!! note
+
+    You have to initialize the database by `mihari db migrate` if you haven't already done.
+
+```bash
+mihari search /path/to/rule.yml
+```
+
+The command outputs an alert to the standard output. Also you can confirm it with a built-in web app.
+
+```bash
+mihari web
+```

data/docs/tags.md

@@ -0,0 +1,3 @@
+# Tags
+
+[TAGS]

data/docs/usage.md

@@ -0,0 +1,100 @@
+# Usage
+
+```bash
+$ mihari
+Commands:
+  mihari --version, -v        # Print the version
+  mihari alert                # Sub commands for alert
+  mihari db                   # Sub commands for DB
+  mihari help [COMMAND]       # Describe available commands or one specific command
+  mihari rule                 # Sub commands for rule
+  mihari search [PATH_OR_ID]  # Search by a rule
+  mihari web                  # Launch the web app
+```
+
+## `mihari db`
+
+This sub command is for initializing/migrating database.
+
+```bash
+mihari db migrate
+```
+
+See [Database](./emitters/database.md) for detailed database configuration.
+
+## `mihari rule`
+
+This sub command is for validating/initializing a rule.
+
+```bash
+mihari rule init /path/to/rule.yml
+mihari rule validate /path/to/rule.yml
+```
+
+## `mihari search`
+
+This is a command for running a rule.
+
+```bash
+mihari search /path/to/rule.yml
+```
+
+Mihari asks whether really you want to update a rule if there is a diff by default.
+
+```bash
+$ mihari search /path/to/rule.yml
+There is a diff in the rule (6254bb74-5e5d-42ad-bc1e-231da0293b0f). Are you sure you want to overwrite the rule? (y/n)
+```
+
+It can be suppressed by providing `-f`.
+
+```bash
+mihari search -f /path/to/rule.yml
+```
+
+## `mihari add`
+
+You may want to add an alert manually. You can do that by this command.
+
+```bash
+mihari alert /path/to/alert.yml
+```
+
+## `mihari web`
+
+This command is for launching the built-in web app.
+
+```bash
+mihari web
+```
+
+It stars the app with `localhost:9292`. You can configure it by providing options.
+
+```bash
+$ mihari help web
+Usage:
+  mihari web
+
+Options:
+  [--port=N]                                         # Hostname to listen on
+                                                     # Default: 9292
+  [--host=HOST]                                      # Port to listen on
+                                                     # Default: localhost
+  [--threads=THREADS]                                # min:max threads to use
+                                                     # Default: 0:5
+  [--verbose], [--no-verbose]                        # Report each request
+                                                     # Default: true
+  [--worker-timeout=N]                               # Worker timeout value (in seconds)
+                                                     # Default: 60
+  [--hide-config-values], [--no-hide-config-values]  # Whether to hide config values or not
+                                                     # Default: true
+  [--open], [--no-open]                              # Whether to open the app in browser or not
+                                                     # Default: true
+  [--rack-env=RACK_ENV]                              # Rack environment
+                                                     # Default: production
+```
+
+!!! tip
+
+    The built-in web app offers API to interact with Mihari.
+    The API docs are available on `/redoc-static.html`