mihari 5.4.4 → 5.4.6

data/docs/analyzers/virustotal_intelligence.md

@@ -0,0 +1,29 @@
+---
+tags:
+  - IP address
+  - Domain
+  - URL
+  - Hash
+---
+
+# VirusTotal Intelligence
+
+- [https://www.virustotal.com](https://www.virustotal.com/gui/home/search)
+
+This analyzer uses VirusTotal Intelligence API. Pagination is supported.
+
+```yaml
+analyzer: virustotal_intelligence
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”VIRUSTOTAL_API_KEY"]`.

data/docs/analyzers/zoomeye.md

@@ -0,0 +1,33 @@
+# ZoomEye
+
+- [https://zoomeye.org/](https://zoomeye.org/)
+
+This analyzer uses ZoomEye API v3. Pagination is supported.
+
+An API endpoint to use is changed based on a `type` option.
+
+| Type | API endpoint   | Artifact type |
+| ---- | -------------- | ------------- |
+| web  | `/web/search`  | IP address    |
+| host | `/host/search` | IP address    |
+
+```yaml
+analyzer: zoomeye
+query: ...
+type: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### Type
+
+`type` determines a search type. `web` or `host`.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”ZOOMEYE_API_KEY"]`.

data/docs/configuration.md

@@ -0,0 +1,35 @@
+# Configuration
+
+Configuration can be done via environment variables.
+
+| Environmental Variable | Description                     | Default                |
+| ---------------------- | ------------------------------- | ---------------------- |
+| DATABASE_URL           | Database URL                    | `sqlite3:///mihari.db` |
+| BINARYEDGE_API_KEY     | BinaryEdge API key              |                        |
+| CENSYS_ID              | Censys API ID                   |                        |
+| CENSYS_SECRET          | Censys secret                   |                        |
+| CIRCL_PASSIVE_PASSWORD | CIRCL passive DNS/SSL password  |                        |
+| CIRCL_PASSIVE_USERNAME | CIRCL passive DNS/SSL username, |                        |
+| IPINFO_API_KEY         | IPInfo API key (token)          |                        |
+| MISP_URL               | MISP URL                        |                        |
+| MISP_API_KEY           | MISP API key                    |                        |
+| ONYPHE_API_KEY         | Onyphe API key                  |                        |
+| OTX_API_KEY            | OTX API key                     |                        |
+| PASSIVETOTAL_API_KEY   | PassiveTotal API key            |                        |
+| PASSIVETOTAL_USERNAME  | PassiveTotal username           |                        |
+| PULSEDIVE_API_KEY      | Pulsedive API key               |                        |
+| SECURITYTRAILS_API_KEY | SecurityTrails API key          |                        |
+| SHODAN_API_KEY         | Shodan API key                  |                        |
+| SLACK_CHANNEL          | Slack channel name              | `#general`             |
+| SLACK_WEBHOOK_URL      | Slack Webhook URL               |                        |
+| THEHIVE_URL            | TheHive URL,                    |                        |
+| THEHIVE_API_KEY        | TheHive API key,                |                        |
+| URLSCAN_API_KEY        | urlscan.io API key,             |                        |
+| VIRUSTOTAL_API_KEY     | VirusTotal API key              |                        |
+| ZOOMEYE_API_KEY        | ZoomEye API key                 |                        |
+| SENTRY_DSN             | Sentry DSN                      |                        |
+| RETRY_INTERVAL         | Retry interval                  | 5                      |
+| RETRY_TIMES            | Retry times                     | 3                      |
+| PAGINATION_LIMIT       | Pagination limit                | 100                    |
+
+Or you can set values through `.env` file. Values in `.env` file will be automatically loaded.

data/docs/emitters/database.md

@@ -0,0 +1,22 @@
+# Database
+
+This emitter stores data in a database. This emitter uses SQLite3 by default but you can change to use MySQL or PostgreSQL. The database is a primary database of Mihari. Each data generated by Mihari is stored in the database. You can view the data via the built-in web app.
+
+Mihari loads a database URL via environment variable `DATABASE_URL`. Defaults to `sqlite3:///mihari.db"` (SQLite3).
+
+If you want to use MySQL or PostgreSQL, please set a database URL for that.
+
+- MySQL: `mysql2://username:password@host:3306/database` (+ `gem install mysql2`)
+- PostgreSQL: `postgres://username:password@host:5432/database` (+ `gem install pg`)
+
+```yaml
+emitter: database
+```
+
+!!! note
+
+    You have to initialize the database by `mihari db migrate`.
+
+## ER Diagram
+
+![](https://imgur.com/krhoSgh.png)

data/docs/emitters/hive.md

@@ -0,0 +1,26 @@
+# TheHive
+
+- [https://thehive-project.org/](https://thehive-project.org/)
+
+This emitter creates an alert on TheHive. TheHive v4 & v5 are supported.
+
+```yaml
+emitter: the_hive
+url: ...
+api_key: ...
+api_version: ...
+```
+
+## Components
+
+### URL
+
+`url` is a TheHive URL. Optional. Defaults to `ENV[”THEHIVE_URL”]`.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”THEHIVE_API_KEY”]`.
+
+### API Version
+
+`api_version` is a version of The Hive API. Optional. Defaults to `ENV[”THEHIVE_API_VERSION”]`.

data/docs/emitters/index.md

@@ -0,0 +1,7 @@
+# Emitters
+
+- [Database](database.md)
+- [TheHive](hive.md)
+- [MISP](misp.md)
+- [Slack](slack.md)
+- [Webhook](webhook.md)

data/docs/emitters/misp.md

@@ -0,0 +1,21 @@
+# MISP
+
+- [https://www.misp-project.org/](https://www.misp-project.org/)
+
+This emitter creates an event on MISP based on an alert.
+
+```yaml
+emitter: misp
+url: ...
+api_key: ...
+```
+
+## Components
+
+### URL
+
+`url` is a MISP URL. Optional. Defaults to `ENV[MISP_URL]`.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”MISP_API_KEY”]`.

data/docs/emitters/slack.md

@@ -0,0 +1,26 @@
+# Slack
+
+- [https://slack.com/](https://slack.com/intl/ja-jp/)
+
+This emitter post a message to Slack via incoming webhook.
+
+```yaml
+emitter: slack
+webhook_url: ...
+channel: ...
+```
+
+| Name        | Type   | Required? | Default                         | Desc.             |
+| ----------- | ------ | --------- | ------------------------------- | ----------------- |
+| webhook_url | String | No        | ENV[SLACK_WEBHOOK_URL]          | Slack webhook URL |
+| channel     | String | No        | ENV[SLACK_CHANNEL] / `#general` | Slack channel     |
+
+## Components
+
+### Webhook URL
+
+`url` is a Slack's incoming webhook URL. Optional. Defaults to `ENV[SLACK_WEBHOOK_URL]`.
+
+### API Key
+
+`channel` is a Slack channel to sent a message. Optional. Defaults to `ENV[SLACK_CHANNEL]` or `#general`.

data/docs/emitters/webhook.md

@@ -0,0 +1,63 @@
+# Webhook
+
+This emitter creates an HTTP request payload based on the specified conditions.
+
+```yaml
+emitter: webhook
+url: ...
+method: ...
+headers: ...
+template: ...
+```
+
+## Components
+
+### URL
+
+`url` is a webhook URL.
+
+### Method
+
+`method` is an HTTP method. Optional. Defaults to `POST`.
+
+### Headers
+
+`headers` (hash) is HTTP headers. Optional.
+
+### Template
+
+`template` is an [ERB](https://github.com/ruby/erb) template to customize the payload to sent. A template should generate a valid JSON.
+
+You can use the following parameters inside an ERB template.
+
+- `rule`: a rule
+- `artifacts`: a list of artifacts
+
+## Examples
+
+### ThreatFox
+
+```yaml
+- emitter: webhook
+  url: https://threatfox-api.abuse.ch/api/v1/
+  headers:
+    api-key: YOUR_API_KEY
+  template: threatfox.erb
+```
+
+```ruby
+{
+	"query": "submit_ioc",
+	"threat_type": "payload_delivery",
+	"ioc_type": "ip:port",
+	"malware": "foobar",
+	"confidence_level": 100,
+	"anonymous": 0,
+	"iocs": [
+		<% @artifacts.select { |artifact| artifact.data_type == "ip" }.each_with_index do |artifact, idx| %>
+			"<%= artifact.data %>:80"
+			<%= ',' if idx < (@artifacts.length - 1) %>
+		<% end %>
+	]
+}
+```

data/docs/enrichers/google_public_dns.md

@@ -0,0 +1,19 @@
+---
+tags:
+  - DNS record
+---
+
+# Google Public DNS
+
+- [https://developers.google.com/speed/public-dns](https://developers.google.com/speed/public-dns)
+
+This enricher uses Google Public DNS to enrich an URL and domain artifact.
+
+```yaml
+enricher: google_public_dns
+```
+
+## Supported Artifacts
+
+- URL
+- Domain

data/docs/enrichers/index.md

@@ -0,0 +1,6 @@
+# Enrichers
+
+- [Google Public DNS](google_public_dns.md)
+- [IPInfo](ipinfo.md)
+- [Shodan](shodan.md)
+- [Whois](whois.md)

data/docs/enrichers/ipinfo.md

@@ -0,0 +1,19 @@
+---
+tags:
+  - Autonomous system
+  - Geolocation
+---
+
+# ipinfo.io
+
+- [https://ipinfo.io/](https://ipinfo.io/)
+
+This enricher uses ipinfo.io API to enrich an IP artifact.
+
+```yaml
+enricher: ipinfo
+```
+
+## Supported Artifacts
+
+- IP address

data/docs/enrichers/shodan.md

@@ -0,0 +1,22 @@
+---
+tags:
+  - Port
+  - CPE
+  - DNS record
+---
+
+# Shodan
+
+- [https://www.shodan.io/](https://www.shodan.io/dashboard)
+
+This enricher uses Shodan InternetDB API to enrich an artifact.
+
+[https://internetdb.shodan.io/](https://internetdb.shodan.io/)
+
+```yaml
+enricher: shodan
+```
+
+## Supported Artifacts
+
+- IP address

data/docs/enrichers/whois.md

@@ -0,0 +1,17 @@
+---
+tags:
+  - Whois
+---
+
+# Whois
+
+This enricher uses “whois” command to enrich an artifact.
+
+```yaml
+enricher: whois
+```
+
+## Supported Artifacts
+
+- URL
+- Domain

data/docs/github_actions.md

@@ -0,0 +1,43 @@
+# GitHub Actions
+
+GitHub Actions is a good way to run Mihari searches continuously.
+
+The following is an example of a GitHub Actions workflow to run Mihari.
+
+```yaml
+name: Mihari searches
+
+on:
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install dependencies
+        run: sudo apt-get -yqq install sqlite3 libsqlite3-dev
+      - name: Set up Ruby 3.2
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "3.2"
+          bundler-cache: true
+      - name: Run Mihari
+        run: |
+          mihari search /path/to/rule.yml
+```
+
+!!! tip
+
+    You need to install `libpq-dev` for PostgreSQL, `libmysqlclient-dev` for MySQL.
+
+This example assumes that you have `Gemfile` in your repository.
+
+```ruby
+source "https://rubygems.org"
+
+gem "pg"     # if you use PostgresSQL
+gem "mysql2" # if you use MySQL
+
+gem "mihari"
+```

data/docs/index.md

@@ -0,0 +1,13 @@
+# Mihari
+
+A query aggregator for OSINT based threat hunting.
+
+Mihari can aggregate multiple searches across multiple services in a single rule & persist findings in a database.
+
+- [Requirements](./requirements.md)
+- [Installation](./installation.md)
+- [How to Write a Rule](./rule.md)
+- [Usage](./usage.md)
+- [Configuration](./configuration.md)
+- [GitHub Actions](./github_actions.md)
+- [Alternatives](./alternatives.md)

data/docs/installation.md

@@ -0,0 +1,31 @@
+# Installation
+
+## Ruby Gem
+
+Mihari is packaged as a Ruby Gem.
+
+```bash
+gem install mihari
+```
+
+Mihari uses SQLite3 as a primary database by default. Thus a gem for SQLite (`sqlite3`) is installed by default.
+
+If you want to use MySQL or PostgreSQL instead of SQLite3, please install a gem for that by yourself.
+
+**MySQL**
+
+```bash
+gem install mysql2
+```
+
+**PostgreSQL**
+
+```bash
+gem install pg
+```
+
+# Docker
+
+You can built the Docker image by yourself.
+
+`Dockerfile` is available at [https://github.com/ninoseki/mihari/tree/master/docker](https://github.com/ninoseki/mihari/tree/master/docker).

data/docs/requirements.md

@@ -0,0 +1,20 @@
+# Requirements
+
+- Runtime:
+  - Ruby 2.7+ / 3.0+ (tested with 2.7, 3.0, 3.1 and 3.2)
+- Database:
+  - SQLite3, PostgreSQL and MySQL
+- Others:
+  - MISP
+  - TheHive
+
+| Name       | Supported versions      |
+| ---------- | ----------------------- |
+| Ruby       | v2.7, v3.0, v3.1 & v3.2 |
+| PostgreSQL | v15                     |
+| SQLite     | v3                      |
+| MySQL      | v8                      |
+| MISP       | v2.4                    |
+| TheHive    | v3 & v4                 |
+
+You need to have a database to persistent the data. See [Database](./emitters/database.md) for details.

data/docs/rule.md

@@ -0,0 +1,171 @@
+# How to Write a Rule
+
+Mihari has [Sigma](https://github.com/SigmaHQ/sigma) like format to describe a set of search queries to express a rule.
+
+Mihari has three main components to compose a rule.
+
+![](https://imgur.com/BBT99BG.png)
+
+- Analyzers/Queries: a list of queries (analyzers) that builds a list of artifacts
+- Enrichers: a list of enrichers that enriches a list of artifacts
+- Emitters: a list of emitters that emits a list of artifacts as an alert
+
+An artifact has five types:
+
+- IP address (`ip`)
+- Domain (`domain`)
+- URL (`url`)
+- Mail (`mail`)
+- Hash (`hash`)
+
+An alert can have multiple artifacts bundled by a rule.
+
+!!! note
+
+    A rule is assumed to be executed multiple times continuously. An alert generated by a rule will only have new findings at that time.
+
+Let's break down the following example:
+
+```yaml
+id: c7f6968e-dbe1-4612-b0bb-8407a4fe05df
+title: Example
+description: Mihari rule example
+created_on: "2023-01-01"
+updated_on: "2023-01-02"
+author: ninoseki
+references:
+  - https://github.com/ninoseki/mihari
+related:
+  - 6254bb74-5e5d-42ad-bc1e-231da0293b0f
+tags:
+  - foo
+  - bar
+queries:
+  - analyzer: shodan
+    query: ip:1.1.1.1
+  - analyzer: censys
+    query: ip:8.8.8.8
+enrichers:
+  - enricher: whois
+  - enricher: ipinfo
+  - enricher: shodan
+  - enricher: google_public_dns
+emitters:
+  - emitter: database
+  - emitter: misp
+  - emitter: slack
+  - emitter: the_hive
+data_types:
+  - hash
+  - ip
+  - domain
+  - url
+  - mail
+falsepositives: []
+```
+
+## Components
+
+### ID
+
+`id` is an unique ID of a rule. UUID v4 is recommended.
+
+### Title
+
+`title` is a title of a rule.
+
+### Description
+
+`description` is a short description of a rule.
+
+### Created/Updated On
+
+`created_on` is a date of a rule creation. Optional.
+Also a rule can have `updated_on` that is a date of a rule modification. Optional.
+
+### Tags
+
+`tags` is a list of tags of a rule.
+
+### Author
+
+`author` is an author of a rule. Optional.
+
+### References
+
+`references` is a list of a references of a rule. Optional.
+
+### Related
+
+`related` is a list of related rule IDs. Optional.
+
+### Queries
+
+`queries` is a list of queries/analyzers.
+See [Analyzers](./analyzers/index.md) to know details of each analyzer.
+
+### Enrichers
+
+`enrichers` is a list of enrichers.
+See [Enrichers](./enrichers/index.md) to know details of each enricher.
+
+Defaults to:
+
+- `google_public_dns`
+- `ipinfo`
+- `shodan`
+- `whois`
+
+### Emitters
+
+`emitters` is a list of emitters.
+See [Emitters](./emitters/index.md) to know details of each emitter.
+
+Defaults to:
+
+- `database`
+- `misp`
+- `slack`
+- `the_hive`
+
+### Data Types
+
+`data_types` is a list of data (artifact) types to allow by a rule. Types not defined in here will be automatically rejected.
+
+Defaults to:
+
+- `ip`
+- `domain`
+- `url`
+- `mail`
+- `hash`
+
+### False positives
+
+`falsepositives` is a list of false positive values. A string or regexp can be used in here.
+
+### Artifact TTL
+
+`artifact_ttl` (alias: `artifact_lifetime`) is an integer value of artifact TTL (Time-To-Live) in seconds.
+
+Mihari rejects a same artifact in a same rule in general.
+
+But you may want to get a same artifact after a certain period of time. `artifact_ttl` is for that. If a rule finds a same artifact after `artifact_ttl` seconds have been passed, that artifact will be included in an alert.
+
+## How to Run a Rule
+
+Once you finish writing a rule, you can run the rule by `mihari` CLI.
+
+!!! note
+
+    You have to initialize the database by `mihari db migrate` if you haven't already done.
+
+```bash
+mihari search /path/to/rule.yml
+```
+
+The command outputs an alert to the standard output. Also you can confirm it with a built-in web app.
+
+```bash
+mihari web
+```

data/docs/tags.md

@@ -0,0 +1,3 @@
+# Tags
+
+[TAGS]