mihari 5.6.2 → 5.7.1

data/docs/analyzers/feed.md

@@ -1,73 +0,0 @@
-# Feed
-
-This analyzer can ingest a feed (JSON or CSV) by specifying conditions.
-
-Note that you should write a selector to get proper IoCs from a feed. A selector is based on [jr](https://github.com/yuya-takeyama/jr).
-
-```yaml
-analyzer: feed
-query: ...
-selector: ...
-method: ...
-headers: ...
-params: ...
-data: ...
-json: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a URL of a feed.
-
-!!! note
-
-    I know this is a strange naming. It's just for keeping the convention with other analyzers.
-
-### Method
-
-`method` (`string`) is an HTTP method. Defaults to `GET`.
-
-### Selector
-
-`selector` (`string`) is a `jr` selector.
-
-### Headers
-
-`headers` (`hash`) is an HTTP headers. Optional.
-
-### Params
-
-`params` (`hash`) is an HTTP query params. Optional.
-
-### Data
-
-`data` (`hash`) is an HTTP form data. Optional.
-
-### JSON
-
-`json` (`hash`) is an JSON body. Optional.
-
-## Examples
-
-### ThreatFox
-
-```yaml
-analyzer: feed
-query: "https://threatfox-api.abuse.ch/api/v1/"
-method: POST
-json:
-  query: get_iocs
-  days: 1
-headers:
-selector: "map(&:data).unwrap.map(&:ioc).map { |v| v.start_with?('http://', 'https://') ? v :  v.split(':').first }"
-```
-
-### URLhaus
-
-```yaml
-analyzer: feed
-query: "https://urlhaus.abuse.ch/feeds/country/JP/"
-selector: "map { |v| v[1] }"
-```

data/docs/analyzers/greynoise.md

@@ -1,26 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# GreyNoise
-
-- [https://www.greynoise.io/](https://www.greynoise.io/)
-
-This analyzer uses GreyNoise API (`/v2/experimental/gnql`) to search. Pagination is supported.
-
-```yaml
-analyzer: greynoise
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a GNQL search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”GREYNOISE_API_KEY"]`.

data/docs/analyzers/hunterhow.md

@@ -1,33 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# Hunter How
-
-- [https://hunter.how/](https://hunter.how/)
-
-This analyzer uses Hunter How API (`https://api.hunter.how/search`) to search. Pagination is supported.
-
-```yaml
-analyzer: hunterhow
-query: ...
-api_key: ...
-start_time: ...
-end_time: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### Start/End Time
-
-- `start_time` (`date`): Only show results after the given date.
-- `end_time` (`date`): Only show results after the given date.
-
-### API key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”HUNTERHOW_API_KEY"]`.

data/docs/analyzers/index.md

@@ -1,103 +0,0 @@
-# Analyzers
-
-- [BinaryEdge](binaryedge.md)
-- [Censys](censys.md)
-- [Circle Passive DNS/SSL](circl.md)
-- [crt.sh](crtsh.md)
-- [dnstwister](dnstwister.md)
-- [Feed](feed.md)
-- [GreyNoise](greynoise.md)
-- [HunterHow](hunterhow.md)
-- [Onyphe](onyphe.md)
-- [OTX](otx.md)
-- [PassiveTotal](passivetotal.md)
-- [PulseDive](pulsedive.md)
-- [SecurityTrails](securitytrails.md)
-- [Shodan](shodan.md)
-- [urlscan.io](urlscan.md)
-- [VirusTotal](virustotal.md)
-- [VirusTotal Intelligence](virustotal_intelligence.md)
-
-## Options
-
-All the analyzers can have optional `options`.
-
-```yaml
-analyzer: ...
-query: ...
-options:
-  retry_times: ...
-  retry_interval: ...
-  retry_exponential_backoff: ...
-  timeout: ...
-  ignore_error: ...
-```
-
-Also the following analyzers can have pagination options.
-
-- [Shodan](./shodan.md)
-- [BinaryEdge](./binaryedge.md)
-- [Censys](./censys.md)
-- [ZoomEye](./zoomeye.md)
-- [urlscan.io](./urlscan.md)
-- [VirusTotal Intelligence](./virustotal_intelligence.md)
-- [HunterHow](./hunterhow.md)
-
-```yaml
-options:
-  pagination_interval: ...
-  pagination_limit: ...
-```
-
-### Retry Times
-
-`retry_times` (`integer`) is a number of times of retry when something goes wrong. Optional. Defaults to 3.
-
-### Retry Interval
-
-`retry_interval` (`integer`) is an interval in seconds between retries. Optional. Defaults to 5.
-
-### Retry Exponential Backoff
-
-`retry_exponential_backoff` (`bool`) controls whether to do exponential backoff. Optional. Defaults to `true`.
-
-### Timeout
-
-`timeout` (`integer`) is an HTTP timeout in seconds. Optional.
-
-### Ignore Error
-
-`ignore_error` (`bool`) controls whether to ignore an error or not. Optional. Defaults to `false`.
-
-Mihari uses fail-fast approach. For example, if Shodan returns an error, the Censys query next is not triggered because Mihari raises an error before it.
-
-```yaml
-queries:
-  - analyzer: shodan
-    query: ip:1.1.1.1
-  - analyzer: censys
-    query: ip:8.8.8.8
-```
-
-You can set `ignore_error` option to make it fault tolerance.
-
-```yaml
-queries:
-  - analyzer: shodan
-    query: ip:1.1.1.1
-    options:
-      ignore_error: true
-  - analyzer: censys
-    query: ip:8.8.8.8
-```
-
-### Pagination Interval
-
-`pagination_interval` (`integer`) is an interval in seconds between pagination. Optional. Defaults to 0.
-
-### Pagination Limit
-
-`pagination_limit` (`integer`) is an limit for pagination. Optional. Defaults to 100.
-
-In the worst case, if something wrong with Mihari or a service, Mihari can drain API quota by doing pagination forever.
-`pagination_limit` is a safety valve for that. A number of pagination is limited as `pagination_limit` times.

data/docs/analyzers/onyphe.md

@@ -1,26 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# ONYPHE
-
-- [https://www.onyphe.io/](https://www.onyphe.io/)
-
-This analyzer uses ONYPHE API v2 (`/api/v2/simple/datascan`) to search.
-
-```yaml
-analyzer: onyphe
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”ONYPHE_API_KEY”"]`.

data/docs/analyzers/otx.md

@@ -1,28 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
----
-
-# OTX
-
-- [https://otx.alienvault.com/](https://otx.alienvault.com/dashboard/new)
-
-This analyzer uses [OTX API v1](https://otx.alienvault.com/api) (`/api/v1/indicators/`) API to search.
-
-```yaml
-analyzer: otx
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a passive DNS search query. Domain or IP address.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”OTX_API_KEY”"]`.

data/docs/analyzers/passivetotal.md

@@ -1,52 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
-  - Passive SSL
-  - Reverse Whois
----
-
-# PassiveTotal
-
-- [https://community.riskiq.com/](https://community.riskiq.com/home)
-
-This analyzer uses [PassvieTotal API](https://api.passivetotal.org/index.html).
-
-An API endpoint to use is changed based on a type of a query.
-
-| Query                                   | API endpoint                  | Artifact   |
-| --------------------------------------- | ----------------------------- | ---------- |
-| IP address                              | `/v2/dns/passive`             | Domain     |
-| Domain                                  | `/v2/dns/passive`             | IP address |
-| Mail                                    | `/v2/whois/search`            | Domain     |
-| Hash (SSL certificate SHA1 fingerprint) | `/v2/ssl-certificate/history` | IP address |
-
-```yaml
-analyzer: passivetotal
-query: ...
-username: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `passivetotal` and `pt`.
-
-### Query
-
-`query` (`string`) is a passive DNS/SSL or reverse whois search query. Domain, IP address, mail or SHA1 certificate fingerprint.
-
-- Passive DNS: Domain, IP Address
-- Passive SSL: SHA1 certificate fingerprint
-- Reverse whois: mail
-
-### Username
-
-`username` (`string`) is a username. Optional. Defaults to `ENV[”PASSIVETOTAL_USERNAME"]`.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”PASSIVETOTAL_API_KEY"]`.

data/docs/analyzers/pulsedive.md

@@ -1,28 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
----
-
-# Pulsedive
-
-- [https://pulsedive.com/](https://pulsedive.com/)
-
-This analyzer uses [Pulsedive API](https://pulsedive.com/api/) (`/api/info.php`) to search.
-
-```yaml
-analyzer: pulsedive
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a passive DNS search query. Domain or IP address.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”PULSEDIVE_API_KEY"]`.

data/docs/analyzers/securitytrails.md

@@ -1,41 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
-  - Reverse Whois
----
-
-# SecurityTrails
-
-- [https://securitytrails.com/](https://securitytrails.com/)
-
-This analyzer uses [SecurityTrails API](https://docs.securitytrails.com/docs).
-
-An API endpoint to use is changed based on a type of a query.
-
-| Query type | API endpoint       | Artifact   |
-| ---------- | ------------------ | ---------- |
-| IP address | `/v1/domains/list` | Domain     |
-| Domain     | `/v1/history/`     | IP address |
-| Mail       | `/v1/domains/list` | Domain     |
-
-```yaml
-analyzer: securitytrails
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `securitytrails` and `st`.
-
-### Query
-
-`query` (`string`) is a passive DNS search/reverse whois query. Domain, IP address or mail.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”SECURITYTRAILS_API_KEY"]`.

data/docs/analyzers/shodan.md

@@ -1,26 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# Shodan
-
-- [https://shodan.io/](https://shodan.io/)
-
-This analyzer uses [Shodan REST AP](https://developer.shodan.io/api) (`/shodan/host/search`) API to search. Pagination is supported.
-
-```yaml
-analyzer: shodan
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”SHODAN_API_KEY"]`.

data/docs/analyzers/urlscan.md

@@ -1,28 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Artifact:URL
----
-
-# urlscan.io
-
-- [https://urlscan.io/](https://urlscan.io/)
-
-This analyzer uses [urlscan.io](http://urlscan.io) API (`/api/v1/search`) to search. Pagination is supported.
-
-```yaml
-analyzer: urlscan
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”URLSCAN_API_KEY"]`.

data/docs/analyzers/virustotal.md

@@ -1,43 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Passive DNS
----
-
-# VirusTotal
-
-- [https://www.virustotal.com](https://www.virustotal.com/gui/home/search)
-
-This analyzer uses VirusTotal API v3.
-
-An API endpoint to use is changed based on a type of a query.
-
-::: top
-
-    Note that this analyzer only checks passive DNS data of a given query (domain or IP address).
-
-| Query      | API endpoint            | Artifact   |
-| ---------- | ----------------------- | ---------- |
-| IP address | `/api/v3/ip_addresses/` | Domain     |
-| Domain     | `/api/v3/domains/`      | IP address |
-
-```yaml
-analyzer: virustotal
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `virustoal` and `vt`.
-
-### Query
-
-`query` (`string`) is a passive DNS search query. Domain or IP address.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”VIRUSTOTAL_API_KEY"]`.

data/docs/analyzers/virustotal_intelligence.md

@@ -1,33 +0,0 @@
----
-tags:
-  - Artifact:IP
-  - Artifact:Domain
-  - Artifact:URL
-  - Artifact:Hash
----
-
-# VirusTotal Intelligence
-
-- [https://www.virustotal.com](https://www.virustotal.com/gui/home/search)
-
-This analyzer uses VirusTotal Intelligence API. Pagination is supported.
-
-```yaml
-analyzer: virustotal_intelligence
-query: ...
-api_key: ...
-```
-
-## Components
-
-### Analyzer
-
-`analyzer` (`string`) should be either of `virustotal_intelligence` and ``.
-
-### Query
-
-`query` (`string`) is a search query.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”VIRUSTOTAL_API_KEY"]`.

data/docs/analyzers/zoomeye.md

@@ -1,38 +0,0 @@
----
-tags:
-  - Artifact:IP
----
-
-# ZoomEye
-
-- [https://zoomeye.org/](https://zoomeye.org/)
-
-This analyzer uses ZoomEye API v3. Pagination is supported.
-
-An API endpoint to use is changed based on a `type` option.
-
-| Type | API endpoint   | Artifact type |
-| ---- | -------------- | ------------- |
-| web  | `/web/search`  | IP address    |
-| host | `/host/search` | IP address    |
-
-```yaml
-analyzer: zoomeye
-query: ...
-type: ...
-api_key: ...
-```
-
-## Components
-
-### Query
-
-`query` (`string`) is a search query.
-
-### Type
-
-`type` (`string`) determines a search type. `web` or `host`.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”ZOOMEYE_API_KEY"]`.

data/docs/configuration.md

@@ -1,35 +0,0 @@
-# Configuration
-
-Configuration can be done via environment variables.
-
-| Environmental Variable | Description                     | Default                |
-| ---------------------- | ------------------------------- | ---------------------- |
-| DATABASE_URL           | Database URL                    | `sqlite3:///mihari.db` |
-| BINARYEDGE_API_KEY     | BinaryEdge API key              |                        |
-| CENSYS_ID              | Censys API ID                   |                        |
-| CENSYS_SECRET          | Censys secret                   |                        |
-| CIRCL_PASSIVE_PASSWORD | CIRCL passive DNS/SSL password  |                        |
-| CIRCL_PASSIVE_USERNAME | CIRCL passive DNS/SSL username, |                        |
-| IPINFO_API_KEY         | IPInfo API key (token)          |                        |
-| MISP_URL               | MISP URL                        |                        |
-| MISP_API_KEY           | MISP API key                    |                        |
-| ONYPHE_API_KEY         | Onyphe API key                  |                        |
-| OTX_API_KEY            | OTX API key                     |                        |
-| PASSIVETOTAL_API_KEY   | PassiveTotal API key            |                        |
-| PASSIVETOTAL_USERNAME  | PassiveTotal username           |                        |
-| PULSEDIVE_API_KEY      | Pulsedive API key               |                        |
-| SECURITYTRAILS_API_KEY | SecurityTrails API key          |                        |
-| SHODAN_API_KEY         | Shodan API key                  |                        |
-| SLACK_CHANNEL          | Slack channel name              | `#general`             |
-| SLACK_WEBHOOK_URL      | Slack Webhook URL               |                        |
-| THEHIVE_URL            | TheHive URL,                    |                        |
-| THEHIVE_API_KEY        | TheHive API key,                |                        |
-| URLSCAN_API_KEY        | urlscan.io API key,             |                        |
-| VIRUSTOTAL_API_KEY     | VirusTotal API key              |                        |
-| ZOOMEYE_API_KEY        | ZoomEye API key                 |                        |
-| SENTRY_DSN             | Sentry DSN                      |                        |
-| RETRY_INTERVAL         | Retry interval                  | 5                      |
-| RETRY_TIMES            | Retry times                     | 3                      |
-| PAGINATION_LIMIT       | Pagination limit                | 100                    |
-
-Or you can set values through `.env` file. Values in `.env` file will be automatically loaded.

data/docs/emitters/database.md

@@ -1,22 +0,0 @@
-# Database
-
-This emitter stores data in a database. This emitter uses SQLite3 by default but you can change to use MySQL or PostgreSQL. The database is a primary database of Mihari. Each data generated by Mihari is stored in the database. You can view the data via the built-in web app.
-
-Mihari loads a database URL via environment variable `DATABASE_URL`. Defaults to `sqlite3:///mihari.db"` (SQLite3).
-
-If you want to use MySQL or PostgreSQL, please set a database URL for that.
-
-- MySQL: `mysql2://username:password@host:3306/database` (+ `gem install mysql2`)
-- PostgreSQL: `postgres://username:password@host:5432/database` (+ `gem install pg`)
-
-```yaml
-emitter: database
-```
-
-!!! note
-
-    You have to initialize the database by `mihari db migrate`.
-
-## ER Diagram
-
-![](https://imgur.com/krhoSgh.png)

data/docs/emitters/hive.md

@@ -1,26 +0,0 @@
-# TheHive
-
-- [https://thehive-project.org/](https://thehive-project.org/)
-
-This emitter creates an alert on TheHive. TheHive v4 & v5 are supported.
-
-```yaml
-emitter: thehive
-url: ...
-api_key: ...
-api_version: ...
-```
-
-## Components
-
-### URL
-
-`url` (`string`) is a TheHive URL. Optional. Defaults to `ENV[”THEHIVE_URL”]`.
-
-### API Key
-
-`api_key` (`string`) is an API key. Optional. Defaults to `ENV[”THEHIVE_API_KEY”]`.
-
-### API Version
-
-`api_version` (`string`) is a version of The Hive API. Optional. `v4` or `v5`. Defaults to `ENV[”THEHIVE_API_VERSION”]`.