mihari 5.7.0 → 5.7.2

data/docs/analyzers/index.md

@@ -1,104 +0,0 @@
-# Analyzers
-
-- [BinaryEdge](binaryedge.md)
-- [Censys](censys.md)
-- [Circle Passive DNS/SSL](circl.md)
-- [crt.sh](crtsh.md)
-- [dnstwister](dnstwister.md)
-- [Feed](feed.md)
-- [Fofa](fofa.md)
-- [GreyNoise](greynoise.md)
-- [HunterHow](hunterhow.md)
-- [Onyphe](onyphe.md)
-- [OTX](otx.md)
-- [PassiveTotal](passivetotal.md)
-- [PulseDive](pulsedive.md)
-- [SecurityTrails](securitytrails.md)
-- [Shodan](shodan.md)
-- [urlscan.io](urlscan.md)
-- [VirusTotal](virustotal.md)
-- [VirusTotal Intelligence](virustotal_intelligence.md)
-
-## Options
-
-All the analyzers can have optional `options`.
-
-```yaml
-analyzer: ...
-query: ...
-options:
-  retry_times: ...
-  retry_interval: ...
-  retry_exponential_backoff: ...
-  timeout: ...
-  ignore_error: ...
-```
-
-Also the following analyzers can have pagination options.
-
-- [Shodan](./shodan.md)
-- [BinaryEdge](./binaryedge.md)
-- [Censys](./censys.md)
-- [ZoomEye](./zoomeye.md)
-- [urlscan.io](./urlscan.md)
-- [VirusTotal Intelligence](./virustotal_intelligence.md)
-- [HunterHow](./hunterhow.md)
-
-```yaml
-options:
-  pagination_interval: ...
-  pagination_limit: ...
-```
-
-### Retry Times
-
-`retry_times` (`integer`) is a number of times of retry when something goes wrong. Optional. Defaults to 3.
-
-### Retry Interval
-
-`retry_interval` (`integer`) is an interval in seconds between retries. Optional. Defaults to 5.
-
-### Retry Exponential Backoff
-
-`retry_exponential_backoff` (`bool`) controls whether to do exponential backoff. Optional. Defaults to `true`.
-
-### Timeout
-
-`timeout` (`integer`) is an HTTP timeout in seconds. Optional.
-
-### Ignore Error
-
-`ignore_error` (`bool`) controls whether to ignore an error or not. Optional. Defaults to `false`.
-
-Mihari uses fail-fast approach. For example, if Shodan returns an error, the Censys query next is not triggered because Mihari raises an error before it.
-
-```yaml
-queries:
-  - analyzer: shodan
-    query: ip:1.1.1.1
-  - analyzer: censys
-    query: ip:8.8.8.8
-```
-
-You can set `ignore_error` option to make it fault tolerance.
-
-```yaml
-queries:
-  - analyzer: shodan
-    query: ip:1.1.1.1
-    options:
-      ignore_error: true
-  - analyzer: censys
-    query: ip:8.8.8.8
-```
-
-### Pagination Interval
-
-`pagination_interval` (`integer`) is an interval in seconds between pagination. Optional. Defaults to 0.
-
-### Pagination Limit
-
-`pagination_limit` (`integer`) is an limit for pagination. Optional. Defaults to 100.
-
-In the worst case, if something wrong with Mihari or a service, Mihari can drain API quota by doing pagination forever.
-`pagination_limit` is a safety valve for that. A number of pagination is limited as `pagination_limit` times.

data/docs/analyzers/onyphe.md

@@ -1,26 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# ONYPHE
-
-- [https://www.onyphe.io/](https://www.onyphe.io/)
-
-This analyzer uses ONYPHE API v2 (`/api/v2/simple/datascan`) to search.
-
-```yaml
-analyzer: onyphe
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”ONYPHE_API_KEY”"]`.

data/docs/analyzers/otx.md

@@ -1,28 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
----
-
-# OTX
-
-- [https://otx.alienvault.com/](https://otx.alienvault.com/dashboard/new)
-
-This analyzer uses [OTX API v1](https://otx.alienvault.com/api) (`/api/v1/indicators/`) API to search.
-
-```yaml
-analyzer: otx
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a passive DNS search query. Domain or IP address.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”OTX_API_KEY”"]`.

data/docs/analyzers/passivetotal.md

@@ -1,52 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
-  - Passive SSL
-  - Reverse Whois
----
-
-# PassiveTotal
-
-- [https://community.riskiq.com/](https://community.riskiq.com/home)
-
-This analyzer uses [PassvieTotal API](https://api.passivetotal.org/index.html).
-
-An API endpoint to use is changed based on a type of a query.
-
-| Query                                   | API endpoint                  | Artifact   |
-| --------------------------------------- | ----------------------------- | ---------- |
-| IP address                              | `/v2/dns/passive`             | Domain     |
-| Domain                                  | `/v2/dns/passive`             | IP address |
-| Mail                                    | `/v2/whois/search`            | Domain     |
-| Hash (SSL certificate SHA1 fingerprint) | `/v2/ssl-certificate/history` | IP address |
-
-```yaml
-analyzer: passivetotal
-query: ...
-username: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `passivetotal` and `pt`.
-
-### Query
-
-`query` (`string`) is a passive DNS/SSL or reverse whois search query. Domain, IP address, mail or SHA1 certificate fingerprint.
-
-- Passive DNS: Domain, IP Address
-- Passive SSL: SHA1 certificate fingerprint
-- Reverse whois: mail
-
-### Username
-
-`username` (`string`) is a username. Optional. Defaults to `ENV[”PASSIVETOTAL_USERNAME"]`.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”PASSIVETOTAL_API_KEY"]`.

data/docs/analyzers/pulsedive.md

@@ -1,28 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
----
-
-# Pulsedive
-
-- [https://pulsedive.com/](https://pulsedive.com/)
-
-This analyzer uses [Pulsedive API](https://pulsedive.com/api/) (`/api/info.php`) to search.
-
-```yaml
-analyzer: pulsedive
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a passive DNS search query. Domain or IP address.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”PULSEDIVE_API_KEY"]`.

data/docs/analyzers/securitytrails.md

@@ -1,41 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
-  - Reverse Whois
----
-
-# SecurityTrails
-
-- [https://securitytrails.com/](https://securitytrails.com/)
-
-This analyzer uses [SecurityTrails API](https://docs.securitytrails.com/docs).
-
-An API endpoint to use is changed based on a type of a query.
-
-| Query type | API endpoint       | Artifact   |
-| ---------- | ------------------ | ---------- |
-| IP address | `/v1/domains/list` | Domain     |
-| Domain     | `/v1/history/`     | IP address |
-| Mail       | `/v1/domains/list` | Domain     |
-
-```yaml
-analyzer: securitytrails
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `securitytrails` and `st`.
-
-### Query
-
-`query` (`string`) is a passive DNS search/reverse whois query. Domain, IP address or mail.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”SECURITYTRAILS_API_KEY"]`.

data/docs/analyzers/shodan.md

@@ -1,26 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# Shodan
-
-- [https://shodan.io/](https://shodan.io/)
-
-This analyzer uses [Shodan REST AP](https://developer.shodan.io/api) (`/shodan/host/search`) API to search. Pagination is supported.
-
-```yaml
-analyzer: shodan
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”SHODAN_API_KEY"]`.

data/docs/analyzers/urlscan.md

@@ -1,28 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Artifact:URL
----
-
-# urlscan.io
-
-- [https://urlscan.io/](https://urlscan.io/)
-
-This analyzer uses [urlscan.io](http://urlscan.io) API (`/api/v1/search`) to search. Pagination is supported.
-
-```yaml
-analyzer: urlscan
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”URLSCAN_API_KEY"]`.

data/docs/analyzers/virustotal.md

@@ -1,43 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
----
-
-# VirusTotal
-
-- [https://www.virustotal.com](https://www.virustotal.com/gui/home/search)
-
-This analyzer uses VirusTotal API v3.
-
-An API endpoint to use is changed based on a type of a query.
-
-::: top
-
-    Note that this analyzer only checks passive DNS data of a given query (domain or IP address).
-
-| Query      | API endpoint            | Artifact   |
-| ---------- | ----------------------- | ---------- |
-| IP address | `/api/v3/ip_addresses/` | Domain     |
-| Domain     | `/api/v3/domains/`      | IP address |
-
-```yaml
-analyzer: virustotal
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `virustoal` and `vt`.
-
-### Query
-
-`query` (`string`) is a passive DNS search query. Domain or IP address.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”VIRUSTOTAL_API_KEY"]`.

data/docs/analyzers/virustotal_intelligence.md

@@ -1,33 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Artifact:URL
-  - Artifact:Hash
----
-
-# VirusTotal Intelligence
-
-- [https://www.virustotal.com](https://www.virustotal.com/gui/home/search)
-
-This analyzer uses VirusTotal Intelligence API. Pagination is supported.
-
-```yaml
-analyzer: virustotal_intelligence
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `virustotal_intelligence` and ``.
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”VIRUSTOTAL_API_KEY"]`.

data/docs/analyzers/zoomeye.md

@@ -1,38 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# ZoomEye
-
-- [https://zoomeye.org/](https://zoomeye.org/)
-
-This analyzer uses ZoomEye API v3. Pagination is supported.
-
-An API endpoint to use is changed based on a `type` option.
-
-| Type | API endpoint   | Artifact type |
-| ---- | -------------- | ------------- |
-| web  | `/web/search`  | IP address    |
-| host | `/host/search` | IP address    |
-
-```yaml
-analyzer: zoomeye
-query: ...
-type: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### Type
-
-`type` (`string`) determines a search type. `web` or `host`.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”ZOOMEYE_API_KEY"]`.

data/docs/configuration.md

@@ -1,35 +0,0 @@
-# Configuration
-
-Configuration can be done via environment variables.
-
-| Environmental Variable | Description                     | Default                |
-| ---------------------- | ------------------------------- | ---------------------- |
-| DATABASE_URL           | Database URL                    | `sqlite3:///mihari.db` |
-| BINARYEDGE_API_KEY     | BinaryEdge API key              |                        |
-| CENSYS_ID              | Censys API ID                   |                        |
-| CENSYS_SECRET          | Censys secret                   |                        |
-| CIRCL_PASSIVE_PASSWORD | CIRCL passive DNS/SSL password  |                        |
-| CIRCL_PASSIVE_USERNAME | CIRCL passive DNS/SSL username, |                        |
-| IPINFO_API_KEY         | IPInfo API key (token)          |                        |
-| MISP_URL               | MISP URL                        |                        |
-| MISP_API_KEY           | MISP API key                    |                        |
-| ONYPHE_API_KEY         | Onyphe API key                  |                        |
-| OTX_API_KEY            | OTX API key                     |                        |
-| PASSIVETOTAL_API_KEY   | PassiveTotal API key            |                        |
-| PASSIVETOTAL_USERNAME  | PassiveTotal username           |                        |
-| PULSEDIVE_API_KEY      | Pulsedive API key               |                        |
-| SECURITYTRAILS_API_KEY | SecurityTrails API key          |                        |
-| SHODAN_API_KEY         | Shodan API key                  |                        |
-| SLACK_CHANNEL          | Slack channel name              | `#general`             |
-| SLACK_WEBHOOK_URL      | Slack Webhook URL               |                        |
-| THEHIVE_URL            | TheHive URL,                    |                        |
-| THEHIVE_API_KEY        | TheHive API key,                |                        |
-| URLSCAN_API_KEY        | urlscan.io API key,             |                        |
-| VIRUSTOTAL_API_KEY     | VirusTotal API key              |                        |
-| ZOOMEYE_API_KEY        | ZoomEye API key                 |                        |
-| SENTRY_DSN             | Sentry DSN                      |                        |
-| RETRY_INTERVAL         | Retry interval                  | 5                      |
-| RETRY_TIMES            | Retry times                     | 3                      |
-| PAGINATION_LIMIT       | Pagination limit                | 100                    |
-
-Or you can set values through `.env` file. Values in `.env` file will be automatically loaded.

data/docs/emitters/database.md

@@ -1,22 +0,0 @@
-# Database
-
-This emitter stores data in a database. This emitter uses SQLite3 by default but you can change to use MySQL or PostgreSQL. The database is a primary database of Mihari. Each data generated by Mihari is stored in the database. You can view the data via the built-in web app.
-
-Mihari loads a database URL via environment variable `DATABASE_URL`. Defaults to `sqlite3:///mihari.db"` (SQLite3).
-
-If you want to use MySQL or PostgreSQL, please set a database URL for that.
-
-- MySQL: `mysql2://username:password@host:3306/database` (+ `gem install mysql2`)
-- PostgreSQL: `postgres://username:password@host:5432/database` (+ `gem install pg`)
-
-```yaml
-emitter: database
-```
-
-!!! note
-
-    You have to initialize the database by `mihari db migrate`.
-
-## ER Diagram
-
-![](https://imgur.com/krhoSgh.png)

data/docs/emitters/hive.md

@@ -1,26 +0,0 @@
-# TheHive
-
-- [https://thehive-project.org/](https://thehive-project.org/)
-
-This emitter creates an alert on TheHive. TheHive v4 & v5 are supported.
-
-```yaml
-emitter: thehive
-url: ...
-api_key: ...
-api_version: ...
-```
-
-## Components
-
-### URL
-
-`url` (`string`) is a TheHive URL. Optional. Defaults to `ENV[”THEHIVE_URL”]`.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”THEHIVE_API_KEY”]`.
-
-### API Version
-
-`api_version` (`string`) is a version of The Hive API. Optional. `v4` or `v5`. Defaults to `ENV[”THEHIVE_API_VERSION”]`.

data/docs/emitters/index.md

@@ -1,36 +0,0 @@
-# Emitters
-
-- [Database](database.md)
-- [TheHive](hive.md)
-- [MISP](misp.md)
-- [Slack](slack.md)
-- [Webhook](webhook.md)
-
-## Options
-
-All the emitters can have optional `options`.
-
-```yaml
-emitter: ...
-options:
-  timeout: ...
-  retry_times: ...
-  retry_interval: ...
-  retry_exponential_backoff: ...
-```
-
-### Timeout
-
-`timeout` (`integer`) is an HTTP timeout in seconds. Optional.
-
-### Retry Times
-
-`retry_times` (`integer`) is a number of times of retry when something goes wrong. Optional. Defaults to 3.
-
-### Retry Interval
-
-`retry_interval` (`integer`) is an interval in seconds between retries. Optional. Defaults to 5.
-
-### Retry Exponential Backoff
-
-`retry_exponential_backoff` (`bool`) controls whether to do exponential backoff. Optional. Defaults to `true`.

data/docs/emitters/misp.md

@@ -1,21 +0,0 @@
-# MISP
-
-- [https://www.misp-project.org/](https://www.misp-project.org/)
-
-This emitter creates an event on MISP based on an alert. MISP v2 is supported.
-
-```yaml
-emitter: misp
-url: ...
-api_key: ...
-```
-
-## Components
-
-### URL
-
-`url` (`string`) is a MISP URL. Optional. Defaults to `ENV[MISP_URL]`.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”MISP_API_KEY”]`.

data/docs/emitters/slack.md

@@ -1,21 +0,0 @@
-# Slack
-
-- [https://slack.com/](https://slack.com/intl/ja-jp/)
-
-This emitter post a message to Slack via incoming webhook.
-
-```yaml
-emitter: slack
-webhook_url: ...
-channel: ...
-```
-
-## Components
-
-### Webhook URL
-
-`url` (`string`) is a Slack's incoming webhook URL. Optional. Defaults to `ENV[SLACK_WEBHOOK_URL]`.
-
-### API Key
-
-`channel` (`string`) is a Slack channel to sent a message. Optional. Defaults to `ENV[SLACK_CHANNEL]` or `#general`.

data/docs/emitters/webhook.md

@@ -1,63 +0,0 @@
-# Webhook
-
-This emitter creates an HTTP request payload based on the specified conditions.
-
-```yaml
-emitter: webhook
-url: ...
-method: ...
-headers: ...
-template: ...
-```
-
-## Components
-
-### URL
-
-`url` (`string`) is a webhook URL.
-
-### Method
-
-`method` (`string`)is an HTTP method. Optional. Defaults to `POST`.
-
-### Headers
-
-`headers` (`hash`) are HTTP headers. Optional.
-
-### Template
-
-`template` (`string`) is an [ERB](https://github.com/ruby/erb) template to customize the payload to sent. A template should generate a valid JSON.
-
-You can use the following parameters inside an ERB template.
-
-- `rule`: a rule
-- `artifacts`: a list of artifacts
-
-## Examples
-
-### ThreatFox
-
-```yaml
-- emitter: webhook
-  url: https://threatfox-api.abuse.ch/api/v1/
-  headers:
-    api-key: YOUR_API_KEY
-  template: threatfox.erb
-```
-
-```ruby
-{
-	"query": "submit_ioc",
-	"threat_type": "payload_delivery",
-	"ioc_type": "ip:port",
-	"malware": "foobar",
-	"confidence_level": 100,
-	"anonymous": 0,
-	"iocs": [
-		<% @artifacts.select { |artifact| artifact.data_type == "ip" }.each_with_index do |artifact, idx| %>
-			"<%= artifact.data %>:80"
-			<%= ',' if idx < (@artifacts.length - 1) %>
-		<% end %>
-	]
-}
-```