mihari 5.4.4 → 5.4.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f1630070f66c1405dede2ff3d57403f219e7b407293ee7e3dc2b01c765ff148
-  data.tar.gz: 040a0d28d0eae3b8f0ec81736c8119f667d85e8b473cf1e5022a5e0486e46bf4
+  metadata.gz: 685244b2cf09a001eacff1c1e0fa1e4fecb44e8affee30dd2bd7914e65cba594
+  data.tar.gz: 427285d6992f44011dee5b0038c79da4b0b0958062869b1547d36190d1b27656
 SHA512:
-  metadata.gz: 7a3bb9a719b30d527170567ccea897ead5f253009a2fe4c028b5ca942bd170d5447d9ef3f403e8a80217f1a84371213a62dfad1dcb2a79738b7fc6c9d7427f08
-  data.tar.gz: 8681c17478418bd04025745d2b0470183b8ebda9b8ef9f561ef1d76ab834091f3d27d0ca04323e597d065b3af95ea3a53b1bc288a79690f8fd93e185e1683c5b
+  metadata.gz: 06e99eed502d4df71a79104a7dc1d29bed954866dd3523971f1883316dec2eb74ffdff1f78df64e711ffb65f92e4474e7b45b4057e5e8e84295e4a2677b87dd4
+  data.tar.gz: 434ede07d0f8c50626bc975f5c0278013ae7bd989e90c7ccbf8d8f031c93e1313fa9d8fffc17d21f1aaf6893b41a027087ef78f3d523461b857038aee0a8d1b4

data/README.md

@@ -13,25 +13,9 @@
 
 ---
 
-Mihari is a tool for OSINT based threat hunting.
+A query aggregator for OSINT based threat hunting.
 
-## How it works
-
-![img](https://github.com/ninoseki/mihari/raw/master/images/overview.jpg)
-
-- Mihari makes a query against Shodan, Censys, VirusTotal, SecurityTrails, etc. and extracts artifacts (IP addresses, domains, URLs or hashes).
-- Mihari checks whether the database (SQLite3, PostgreSQL or MySQL) contains the artifacts or not.
-  - If it doesn't contain the artifacts:
-    - Mihari saves artifacts in the database.
-    - Mihari creates an alert on TheHive.
-    - Mihari sends a notification to Slack.
-    - Mihari creates an event on MISP.
-
-Also, you can check the alerts on a built-in web app.
-
-![img](https://github.com/ninoseki/mihari/raw/master/images/web_alerts.png)
-
-## Supported services
+Mihari can aggregate multiple searches across multiple services in a single rule & persist findings in a database.
 
 Mihari supports the following services by default.
 
@@ -52,13 +36,7 @@ Mihari supports the following services by default.
 - [VirusTotal](http://virustotal.com) & [VirusTotal Intelligence](https://www.virustotal.com/gui/intelligence-overview)
 - [ZoomEye](https://zoomeye.org)
 
-## Docs
-
-- [Mihari Knowledge Base](https://www.notion.so/Mihari-Knowledge-Base-266994ff61204428ba6cfcebe40b0bd1)
-
-## Presentations
-
-- [Adversary Infrastructure Tracking with Mihari](https://ninoseki.github.io/presentations/Adversary%20Infrastructure%20Tracking%20with%20Mihari.pdf)
+See [documentation](https://ninoseki.github.io/mihari/) for more details.
 
 ## License
 

data/docs/alternatives.md

@@ -0,0 +1,5 @@
+# Alternatives
+
+- [InQuest/ThreatIngestor](https://github.com/InQuest/ThreatIngestor) - Extract and aggregate threat intelligence.
+- [thalesgroup-cert/Watcher](https://github.com/thalesgroup-cert/Watcher) - Watcher - Open Source Cybersecurity Threat Hunting Platform. Developed with Django & React JS.
+- [projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) - Quickly discover exposed hosts on the internet using multiple search engines.

data/docs/analyzers/binaryedge.md

@@ -0,0 +1,26 @@
+---
+tags:
+  - IP address
+---
+
+# BinaryEdge
+
+- [https://www.binaryedge.io/](https://www.binaryedge.io/)
+
+This analyzer uses [BinaryEdge API V2](https://docs.binaryedge.io/api-v2/) (`/v2/query/search`) to search. Pagination is supported.
+
+```yaml
+analyzer: binaryedge
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”BINARYEDGE_API_KEY"]`.

data/docs/analyzers/censys.md

@@ -0,0 +1,31 @@
+---
+tags:
+  - IP address
+---
+
+# Censys
+
+- [https://censys.io/](https://censys.io/)
+
+This analyzer uses [Censys Search 2.0 REST API](https://search.censys.io/api) to search. Pagination is supported.
+
+```yaml
+analyzer: censys
+query: ...
+id: ...
+secret: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### ID
+
+`id` is a Cencys ID. Optional. Defaults to `ENV[”CENSYS_ID”]`.
+
+### Secret
+
+`secret` is a Cencys secret. Optional. Defaults to `ENV[”CENSYS_SECRET”]`.

data/docs/analyzers/circl.md

@@ -0,0 +1,37 @@
+---
+tags:
+  - IP address
+  - Passive DNS
+  - Passive SSL
+---
+
+# CIRCL Passive DNS/SSL
+
+- [https://www.circl.lu/services/passive-dns/](https://www.circl.lu/services/passive-dns/)
+- [https://www.circl.lu/services/passive-ssl/](https://www.circl.lu/services/passive-ssl/)
+
+This analyzer uses CIRCL passive DNS API or passive SSL API:
+
+- Use passive DNS API if a query(input) is a domain
+- Use passive SSL API if a query(input) is a SHA1 certificate fingerprint
+
+```yaml
+analyzer: circl
+query: ...
+password: ...
+username: ...
+```
+
+## Components
+
+### Query
+
+`query` is a domain or SHA1 certificate fingerprint.
+
+### Username
+
+`username` is a username. Optional. Defaults to `ENV[”CIRCL_PASSIVE_USERNAME”]`.
+
+### Password
+
+`password` is a password. Optional. Defaults to `ENV[”CIRCL_PASSIVE_PASSWORD”]`.

data/docs/analyzers/crtsh.md

@@ -0,0 +1,26 @@
+---
+tags:
+  - Domain
+---
+
+# crt.sh
+
+- [https://crt.sh/](https://crt.sh/)
+
+This analyzer uses [crt.sh](http://crt.sh)'s (unofficial?) REST API.
+
+```yaml
+analyzer: crtsh
+query: ...
+exclude_expired: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### Exclude Expired
+
+`exclude_expired` (boolean) determines whether to exclude expired domains or not. Optional. Defaults to `true`.

data/docs/analyzers/dnstwister.md

@@ -0,0 +1,25 @@
+---
+tags:
+  - Domain
+---
+
+# dnstwister
+
+- [https://dnstwister.report/](https://dnstwister.report/)
+
+This analyzer uses [dnstwister API](https://dnstwister.report/api/) to search.
+
+```yaml
+analyzer: dnstwister
+query: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+!!! tip
+
+    There is no need to input a domain in hexadecimal format. This analyzer automatically converts a domain (in string format) into a hexadecimal value.

data/docs/analyzers/feed.md

@@ -0,0 +1,73 @@
+# Feed
+
+This analyzer can ingest a feed (JSON or CSV) by specifying conditions.
+
+Note that you should write a selector to get proper IoCs from a feed. A selector is based on [jr](https://github.com/yuya-takeyama/jr).
+
+```yaml
+analyzer: feed
+query: ...
+selector: ...
+method: ...
+headers: ...
+params: ...
+data: ...
+json: ...
+```
+
+## Components
+
+### Query
+
+`query` is a URL of a feed.
+
+!!! note
+
+    I know this is a strange naming. It's just for keeping the convention with other analyzers.
+
+### Method
+
+`method` is an HTTP method. Defaults to `GET`.
+
+### Selector
+
+`selector` is a `jr` selector.
+
+### Headers
+
+`headers` (hash) is an HTTP headers. Optional.
+
+### Params
+
+`params` (hash) is an HTTP query params. Optional.
+
+### Data
+
+`data` (hash) is an HTTP form data. Optional.
+
+### JSON
+
+`json` (hash) is an JSON body. Optional.
+
+## Examples
+
+### ThreatFox
+
+```yaml
+analyzer: feed
+query: "https://threatfox-api.abuse.ch/api/v1/"
+method: POST
+json:
+  query: get_iocs
+  days: 1
+headers:
+selector: "map(&:data).unwrap.map(&:ioc).map { |v| v.start_with?('http://', 'https://') ? v :  v.split(':').first }"
+```
+
+### URLhaus
+
+```yaml
+analyzer: feed
+query: "https://urlhaus.abuse.ch/feeds/country/JP/"
+selector: "map { |v| v[1] }"
+```

data/docs/analyzers/greynoise.md

@@ -0,0 +1,26 @@
+---
+tags:
+  - IP address
+---
+
+# GreyNoise
+
+- [https://www.greynoise.io/](https://www.greynoise.io/)
+
+This analyzer uses GreyNoise API (`/v2/experimental/gnql`) to search. Pagination is supported.
+
+```yaml
+analyzer: greynoise
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a GNQL search query.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”GREYNOISE_API_KEY"]`.

data/docs/analyzers/hunterhow.md

@@ -0,0 +1,33 @@
+---
+tags:
+  - IP address
+---
+
+# Hunter How
+
+- [https://hunter.how/](https://hunter.how/)
+
+This analyzer uses Hunter How API (`https://api.hunter.how/search`) to search. Pagination is supported.
+
+```yaml
+analyzer: hunterhow
+query: ...
+api_key: ...
+start_time: ...
+end_time: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### Start/End Time
+
+- `start_time` (date): Only show results after the given date.
+- `end_time` (date): Only show results after the given date.
+
+### API key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”HUNTERHOW_API_KEY"]`.

data/docs/analyzers/index.md

@@ -0,0 +1,79 @@
+# Analyzers
+
+- [BinaryEdge](binaryedge.md)
+- [Censys](censys.md)
+- [Circle Passive DNS/SSL](circl.md)
+- [crt.sh](crtsh.md)
+- [dnstwister](dnstwister.md)
+- [Feed](feed.md)
+- [GreyNoise](greynoise.md)
+- [HunterHow](hunterhow.md)
+- [Onyphe](onyphe.md)
+- [OTX](otx.md)
+- [PassiveTotal](passivetotal.md)
+- [PulseDive](pulsedive.md)
+- [SecurityTrails](securitytrails.md)
+- [Shodan](shodan.md)
+- [urlscan.io](urlscan.md)
+- [VirusTotal](virustotal.md)
+- [VirusTotal Intelligence](virustotal_intelligence.md)
+
+## Options
+
+All the analyzers can have optional `options`.
+
+```yaml
+analyzer: ...
+query: ...
+options:
+  interval: ...
+  pagination_limit: ...
+  retry_times: ...
+  retry_interval: ...
+  ignore_error: ...
+```
+
+### Interval
+
+`interval` is an interval in seconds between pagination. (If an analyzer does pagination). Optional.
+
+### Pagination Limit
+
+`pagination_limit` is an limit for pagination. Defaults to 100.
+
+In the worst case, if something wrong with Mihari or a service, Mihari can drain API quota by doing pagination forever.
+`pagination_limit` is a safety valve for that. A number of pagination is limited as `pagination_limit` times.
+
+### Retry Times
+
+`retry_times` is a number of times of retry when something goes wrong. Defaults to 3.
+
+### Retry Interval
+
+`retry_interval` is an interval in seconds between retries. Defaults to 5.
+
+### Ignore Error
+
+`ignore_error` controls whether to ignore an error or not. Defaults to `false`.
+
+Mihari uses fail-fast approach. For example, if Shodan returns an error, the Censys query next is not triggered because Mihari raises an error before it.
+
+```yaml
+queries:
+  - analyzer: shodan
+    query: ip:1.1.1.1
+  - analyzer: censys
+    query: ip:8.8.8.8
+```
+
+You can set `ignore_error` option to make it fault tolerance.
+
+```yaml
+queries:
+  - analyzer: shodan
+    query: ip:1.1.1.1
+    options:
+      ignore_error: true
+  - analyzer: censys
+    query: ip:8.8.8.8
+```

data/docs/analyzers/onyphe.md

@@ -0,0 +1,26 @@
+---
+tags:
+  - IP address
+---
+
+# ONYPHE
+
+- [https://www.onyphe.io/](https://www.onyphe.io/)
+
+This analyzer uses ONYPHE API v2 (`/api/v2/simple/datascan`) to search.
+
+```yaml
+analyzer: onyphe
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”ONYPHE_API_KEY”"]`.

data/docs/analyzers/otx.md

@@ -0,0 +1,28 @@
+---
+tags:
+  - IP address
+  - Domain
+  - Passive DNS
+---
+
+# OTX
+
+- [https://otx.alienvault.com/](https://otx.alienvault.com/dashboard/new)
+
+This analyzer uses [OTX API v1](https://otx.alienvault.com/api) (`/api/v1/indicators/`) API to search.
+
+```yaml
+analyzer: otx
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a passive DNS search query. Domain or IP address.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”OTX_API_KEY”"]`.

data/docs/analyzers/passivetotal.md

@@ -0,0 +1,48 @@
+---
+tags:
+  - IP address
+  - Domain
+  - Passive DNS
+  - Passive SSL
+  - Reverse Whois
+---
+
+# PassiveTotal
+
+- [https://community.riskiq.com/](https://community.riskiq.com/home)
+
+This analyzer uses [PassvieTotal API](https://api.passivetotal.org/index.html).
+
+An API endpoint to use is changed based on a type of a query.
+
+| Query                                   | API endpoint                  | Artifact   |
+| --------------------------------------- | ----------------------------- | ---------- |
+| IP address                              | `/v2/dns/passive`             | Domain     |
+| Domain                                  | `/v2/dns/passive`             | IP address |
+| Mail                                    | `/v2/whois/search`            | Domain     |
+| Hash (SSL certificate SHA1 fingerprint) | `/v2/ssl-certificate/history` | IP address |
+
+```yaml
+analyzer: passivetotal
+query: ...
+username: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a passive DNS/SSL or reverse whois search query. Domain, IP address, mail or SHA1 certificate fingerprint.
+
+- Passive DNS: Domain, IP Address
+- Passive SSL: SHA1 certificate fingerprint
+- Reverse whois: mail
+
+### Username
+
+`username` is a username. Optional. Defaults to `ENV[”PASSIVETOTAL_USERNAME"]`.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”PASSIVETOTAL_API_KEY"]`.

data/docs/analyzers/pulsedive.md

@@ -0,0 +1,28 @@
+---
+tags:
+  - IP address
+  - Domain
+  - Passive DNS
+---
+
+# Pulsedive
+
+- [https://pulsedive.com/](https://pulsedive.com/)
+
+This analyzer uses [Pulsedive API](https://pulsedive.com/api/) (`/api/info.php`) to search.
+
+```yaml
+analyzer: pulsedive
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a passive DNS search query. Domain or IP address.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”PULSEDIVE_API_KEY"]`.

data/docs/analyzers/securitytrails.md

@@ -0,0 +1,37 @@
+---
+tags:
+  - IP address
+  - Domain
+  - Passive DNS
+  - Reverse Whois
+---
+
+# SecurityTrails
+
+- [https://securitytrails.com/](https://securitytrails.com/)
+
+This analyzer uses [SecurityTrails API](https://docs.securitytrails.com/docs).
+
+An API endpoint to use is changed based on a type of a query.
+
+| Query type | API endpoint       | Artifact   |
+| ---------- | ------------------ | ---------- |
+| IP address | `/v1/domains/list` | Domain     |
+| Domain     | `/v1/history/`     | IP address |
+| Mail       | `/v1/domains/list` | Domain     |
+
+```yaml
+analyzer: securitytrails
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a passive DNS search/reverse whois query. Domain, IP address or mail.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”SECURITYTRAILS_API_KEY"]`.

data/docs/analyzers/shodan.md

@@ -0,0 +1,26 @@
+---
+tags:
+  - IP address
+---
+
+# Shodan
+
+- [https://shodan.io/](https://shodan.io/)
+
+This analyzer uses [Shodan REST AP](https://developer.shodan.io/api) (`/shodan/host/search`) API to search. Pagination is supported.
+
+```yaml
+analyzer: shodan
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”SHODAN_API_KEY"]`.

data/docs/analyzers/urlscan.md

@@ -0,0 +1,28 @@
+---
+tags:
+  - IP address
+  - Domain
+  - URL
+---
+
+# urlscan.io
+
+- [https://urlscan.io/](https://urlscan.io/)
+
+This analyzer uses [urlscan.io](http://urlscan.io) API (`/api/v1/search`) to search. Pagination is supported.
+
+```yaml
+analyzer: urlscan
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a search query.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”URLSCAN_API_KEY"]`.

data/docs/analyzers/virustotal.md

@@ -0,0 +1,39 @@
+---
+tags:
+  - IP address
+  - Domain
+  - Passive DNS
+---
+
+# VirusTotal
+
+- [https://www.virustotal.com](https://www.virustotal.com/gui/home/search)
+
+This analyzer uses VirusTotal API v3.
+
+An API endpoint to use is changed based on a type of a query.
+
+::: top
+
+    Note that this analyzer only checks passive DNS data of a given query (domain or IP address).
+
+| Query      | API endpoint            | Artifact   |
+| ---------- | ----------------------- | ---------- |
+| IP address | `/api/v3/ip_addresses/` | Domain     |
+| Domain     | `/api/v3/domains/`      | IP address |
+
+```yaml
+analyzer: virustotal
+query: ...
+api_key: ...
+```
+
+## Components
+
+### Query
+
+`query` is a passive DNS search query. Domain or IP address.
+
+### API Key
+
+`api_key` is an API key. Optional. Defaults to `ENV[”VIRUSTOTAL_API_KEY"]`.