parse_ioc 0.4.0__tar.gz → 0.6.0__tar.gz

parse_ioc-0.6.0/PKG-INFO

@@ -0,0 +1,244 @@
+Metadata-Version: 2.4
+Name: parse_ioc
+Version: 0.6.0
+Summary: Parse a list of indicators into a dictionary or JSON structure for programmatic use.
+Project-URL: Homepage, https://github.com/bonifield/parse_ioc
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# ParseIOC
+
+Parse a list of indicators into a dictionary, JSON structure, or Sqlite database for programmatic use.
+
+Handles:
+- emails and their domains
+- URL domains, and username+password and ports if present
+- ipv4 and ipv6 addresses and networks
+- md5, sha1, sha256, and sha512 hashes
+
+...with the goal of **mapping this dictionary to fields in a SIEM** or similar database.
+
+This enables programatic use of an IOC file to quickly query specific values in a SIEM, such as `source.ip` or `file.hash.sha256`.
+
+This tool will support more formats (imphash, ssdeep, ja3+) "eventually".
+
+**Otherwise-unidentifiable strings are categorized as "file".**
+
+IOCs are categorized as one of these types:
+- `domain`
+- `email`
+- `file_path_linux`
+- `file_path_windows`
+- `ipv4`
+- `ipv4_network`
+- `ipv6`
+- `ipv6_network`
+- `md5`
+- `sha1`
+- `sha256`
+- `sha512`
+- extra: `credentials`
+- extra: `port`
+
+## Setup
+
+```
+uv add parse_ioc
+# or
+python3 -m pip install parse_ioc
+```
+
+## Usage
+
+### Imports
+
+```
+from parse_ioc import ParseIOC, parse_multi, map_fields, to_sqlite
+```
+- `ParseIOC`: class that parses each indicator
+- `parse_multi`: function that wraps `ParseIOC` to create one large dictionary, or a list of dictionaries, containing the parsed indicators
+- `map_fields`: function that maps parsed indicators by type to fields used in a SIEM, specified a TOML config
+- `to_sqlite`: writes parsed indicators to a simple Sqlite3 database
+
+### Categorize a single indicator using `ParseIOC()`
+```
+>>> i = ParseIOC("https://192.168.20.20/bad")
+>>> i.to_dict
+{'ioc': '192.168.20.20', 'ioc_type': 'ipv4'}
+>>> i.to_json
+'{"ioc": "192.168.20.20", "ioc_type": "ipv4"}'
+```
+
+"extra" details are added when appropriate, such as credentials in a URL, ports, or domains in email addresses
+```
+>>> e = ParseIOC("alice@email.local")
+>>> print(json.dumps(e.to_dict, indent=4))
+{
+    "ioc": "alice@email.local",
+    "ioc_type": "email",
+    "extra": [
+        {
+            "ioc": "email.local",
+            "ioc_type": "domain"
+            ...
+```
+
+### Categorize a **list** or **file** of indicators using `parse_multi()`
+
+using `mode="combined"` produces one massive dictionary of categorized indicators
+```
+>>> p = parse_multi(ioc_list, mode="combined")
+# or
+>>> p = parse_multi("ioc_examples.txt", mode="combined")
+>>> import json
+>>> print(json.dumps(p, indent=4))
+{
+    "email": [
+        "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+        "alice@email.local",
+        ...
+```
+
+using `mode="single"` produces a **list of dictionaries**, similar to calling `ParseIOC(ioc)` by itself in a loop and putting the results in a list
+```
+>>> p = parse_multi("tests/ioc_examples.txt", mode="single")
+>>> print(json.dumps(p, indent=4))
+[
+    {
+        "ioc": "website.local",
+        "ioc_type": "domain"
+    },
+    {
+        "ioc": "anotherwebsite.local",
+        "ioc_type": "domain",
+        "extra": [
+            {
+                "ioc": 9443,
+                "ioc_type": "port"
+            }
+        ]
+    },
+    {
+        "ioc": "bob@email.local",
+        "ioc_type": "email",
+        "extra": [
+            {
+                "ioc": "email.local",
+                "ioc_type": "domain"
+            }
+```
+
+### Map a **list** or **file** of indicators to SIEM fields using `map_fields()`
+
+- use the `map_ecs.toml` file as a reference or template
+
+example shows parsing a file into [Elastic Common Schema (ECS)](https://www.elastic.co/docs/reference/ecs/ecs-field-reference) fields
+```
+>>> m = map_fields("tests/ioc_examples.txt", "map_ecs.toml")
+>>> print(json.dumps(m, indent=4))
+{
+    "email.from.address": [
+        "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+        "alice@email.local",
+        "bob@emailsubdomain.bad.local.xn--p1ai",
+        "bob@email.local",
+        ...
+```
+
+specify `keep_types=True` in `map_fields()` to nest the SIEM values under the original type categorizations
+```
+>>> m = map_fields("tests/ioc_examples.txt", "map_ecs.toml", keep_types=True)
+>>> print(json.dumps(m, indent=4))
+{
+    "email": {
+        "email.from.address": [
+            "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+            "alice@email.local",
+             ...
+```
+
+### Create Sqlite database of parsed indicators using `to_sqlite()`
+- table name: `indicators`
+- columns: `ioc` (`TEXT`, primary key) and `type` (`TEXT`)
+- uses `parse_multi(input_object, mode="combined")`
+- `extra` data has its own row, not linked to its parent data
+	- ex. a port won't have any reference to the original URL it came from
+```
+>>> to_sqlite("tests/ioc_examples.txt", "iocs.db")
+exported 41 indicators to iocs.db
+```
+
+example output using `sqlite3`
+```
+$ sqlite3 iocs.db "SELECT ioc,type FROM indicators;"
+xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai|email
+alice@email.local|email
+bob@emailsubdomain.bad.local.xn--p1ai|email
+bob@email.local|email
+192.168.1.1|ipv4
+...
+9443|port
+email.local|domain
+...
+```
+
+## Gotchas
+- Domain names must have a period (`.`) otherwise the string will be considered a file
+- This regex in the domain checker, determines if the parsed domain name drops further down to the `file` checker or not:
+```
+file_extensions = re.compile(r"\.(?:exe|dll|msi|bat|cmd|elf|scr|cpl|ps1|vbs|pdf|docx|xlsx|pptx|doc|xls|ppt|rtf|csv|txt|log|xml|zip|rar|7z|tar|gz|iso|img|dmg|cab|png|jpg|jpeg|gif|ico|bmp|svg|mp3|mp4|wav|avi|mov|js|php|css|html|htm|sql|conf|ini|yaml|yml|json)$", re.IGNORECASE)
+```
+
+## Expected output from running `tests/parse_ioc.py` using example data
+```
+$ uv run tests/parseonly.py -f tests/ioc_examples.txt -h
+usage: parseonly.py [-h] -f FILENAME [-m] [-j] [-k]
+
+parse indicators
+
+options:
+  -h, --help           show this help message and exit
+  -f, --file FILENAME  input filename
+  -m, --map            show mapped format
+  -j, --json           display indented JSON
+  -k, --keep-types     keep field types in output dict
+```
+
+```
+$ uv run tests/parseonly.py -f tests/ioc_examples.txt -mjk
+{
+    "email": {
+        "email.from.address": [
+            "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+            "bob@emailsubdomain.bad.local",
+            "alice@email.local",
+            "bob@emailsubdomain.bad.local.xn--p1ai",
+            "bob@email.local"
+        ],
+        "email.sender.address": [
+            "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+            "bob@emailsubdomain.bad.local",
+    ...
+    "ipv4": {
+        "source.ip": [
+            "192.168.20.20",
+            "8.8.8.8",
+            "1.2.3.4",
+            "192.168.1.1"
+        ],
+    ...
+    "ipv4_network": {
+        "source.ip": [
+            "192.168.1.0/24",
+            "8.8.8.0/24"
+    ...
+```
+
+## TODO
+
+- add optional type conversions to match TheHive5 default observable types
+	- change the list of categorizations in README to a markdown table, showing the converted types
+- need to streamline and optimize code
+- remove the filepath regex statements created by AI

parse_ioc-0.6.0/README.md

@@ -0,0 +1,234 @@
+# ParseIOC
+
+Parse a list of indicators into a dictionary, JSON structure, or Sqlite database for programmatic use.
+
+Handles:
+- emails and their domains
+- URL domains, and username+password and ports if present
+- ipv4 and ipv6 addresses and networks
+- md5, sha1, sha256, and sha512 hashes
+
+...with the goal of **mapping this dictionary to fields in a SIEM** or similar database.
+
+This enables programatic use of an IOC file to quickly query specific values in a SIEM, such as `source.ip` or `file.hash.sha256`.
+
+This tool will support more formats (imphash, ssdeep, ja3+) "eventually".
+
+**Otherwise-unidentifiable strings are categorized as "file".**
+
+IOCs are categorized as one of these types:
+- `domain`
+- `email`
+- `file_path_linux`
+- `file_path_windows`
+- `ipv4`
+- `ipv4_network`
+- `ipv6`
+- `ipv6_network`
+- `md5`
+- `sha1`
+- `sha256`
+- `sha512`
+- extra: `credentials`
+- extra: `port`
+
+## Setup
+
+```
+uv add parse_ioc
+# or
+python3 -m pip install parse_ioc
+```
+
+## Usage
+
+### Imports
+
+```
+from parse_ioc import ParseIOC, parse_multi, map_fields, to_sqlite
+```
+- `ParseIOC`: class that parses each indicator
+- `parse_multi`: function that wraps `ParseIOC` to create one large dictionary, or a list of dictionaries, containing the parsed indicators
+- `map_fields`: function that maps parsed indicators by type to fields used in a SIEM, specified a TOML config
+- `to_sqlite`: writes parsed indicators to a simple Sqlite3 database
+
+### Categorize a single indicator using `ParseIOC()`
+```
+>>> i = ParseIOC("https://192.168.20.20/bad")
+>>> i.to_dict
+{'ioc': '192.168.20.20', 'ioc_type': 'ipv4'}
+>>> i.to_json
+'{"ioc": "192.168.20.20", "ioc_type": "ipv4"}'
+```
+
+"extra" details are added when appropriate, such as credentials in a URL, ports, or domains in email addresses
+```
+>>> e = ParseIOC("alice@email.local")
+>>> print(json.dumps(e.to_dict, indent=4))
+{
+    "ioc": "alice@email.local",
+    "ioc_type": "email",
+    "extra": [
+        {
+            "ioc": "email.local",
+            "ioc_type": "domain"
+            ...
+```
+
+### Categorize a **list** or **file** of indicators using `parse_multi()`
+
+using `mode="combined"` produces one massive dictionary of categorized indicators
+```
+>>> p = parse_multi(ioc_list, mode="combined")
+# or
+>>> p = parse_multi("ioc_examples.txt", mode="combined")
+>>> import json
+>>> print(json.dumps(p, indent=4))
+{
+    "email": [
+        "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+        "alice@email.local",
+        ...
+```
+
+using `mode="single"` produces a **list of dictionaries**, similar to calling `ParseIOC(ioc)` by itself in a loop and putting the results in a list
+```
+>>> p = parse_multi("tests/ioc_examples.txt", mode="single")
+>>> print(json.dumps(p, indent=4))
+[
+    {
+        "ioc": "website.local",
+        "ioc_type": "domain"
+    },
+    {
+        "ioc": "anotherwebsite.local",
+        "ioc_type": "domain",
+        "extra": [
+            {
+                "ioc": 9443,
+                "ioc_type": "port"
+            }
+        ]
+    },
+    {
+        "ioc": "bob@email.local",
+        "ioc_type": "email",
+        "extra": [
+            {
+                "ioc": "email.local",
+                "ioc_type": "domain"
+            }
+```
+
+### Map a **list** or **file** of indicators to SIEM fields using `map_fields()`
+
+- use the `map_ecs.toml` file as a reference or template
+
+example shows parsing a file into [Elastic Common Schema (ECS)](https://www.elastic.co/docs/reference/ecs/ecs-field-reference) fields
+```
+>>> m = map_fields("tests/ioc_examples.txt", "map_ecs.toml")
+>>> print(json.dumps(m, indent=4))
+{
+    "email.from.address": [
+        "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+        "alice@email.local",
+        "bob@emailsubdomain.bad.local.xn--p1ai",
+        "bob@email.local",
+        ...
+```
+
+specify `keep_types=True` in `map_fields()` to nest the SIEM values under the original type categorizations
+```
+>>> m = map_fields("tests/ioc_examples.txt", "map_ecs.toml", keep_types=True)
+>>> print(json.dumps(m, indent=4))
+{
+    "email": {
+        "email.from.address": [
+            "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+            "alice@email.local",
+             ...
+```
+
+### Create Sqlite database of parsed indicators using `to_sqlite()`
+- table name: `indicators`
+- columns: `ioc` (`TEXT`, primary key) and `type` (`TEXT`)
+- uses `parse_multi(input_object, mode="combined")`
+- `extra` data has its own row, not linked to its parent data
+	- ex. a port won't have any reference to the original URL it came from
+```
+>>> to_sqlite("tests/ioc_examples.txt", "iocs.db")
+exported 41 indicators to iocs.db
+```
+
+example output using `sqlite3`
+```
+$ sqlite3 iocs.db "SELECT ioc,type FROM indicators;"
+xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai|email
+alice@email.local|email
+bob@emailsubdomain.bad.local.xn--p1ai|email
+bob@email.local|email
+192.168.1.1|ipv4
+...
+9443|port
+email.local|domain
+...
+```
+
+## Gotchas
+- Domain names must have a period (`.`) otherwise the string will be considered a file
+- This regex in the domain checker, determines if the parsed domain name drops further down to the `file` checker or not:
+```
+file_extensions = re.compile(r"\.(?:exe|dll|msi|bat|cmd|elf|scr|cpl|ps1|vbs|pdf|docx|xlsx|pptx|doc|xls|ppt|rtf|csv|txt|log|xml|zip|rar|7z|tar|gz|iso|img|dmg|cab|png|jpg|jpeg|gif|ico|bmp|svg|mp3|mp4|wav|avi|mov|js|php|css|html|htm|sql|conf|ini|yaml|yml|json)$", re.IGNORECASE)
+```
+
+## Expected output from running `tests/parse_ioc.py` using example data
+```
+$ uv run tests/parseonly.py -f tests/ioc_examples.txt -h
+usage: parseonly.py [-h] -f FILENAME [-m] [-j] [-k]
+
+parse indicators
+
+options:
+  -h, --help           show this help message and exit
+  -f, --file FILENAME  input filename
+  -m, --map            show mapped format
+  -j, --json           display indented JSON
+  -k, --keep-types     keep field types in output dict
+```
+
+```
+$ uv run tests/parseonly.py -f tests/ioc_examples.txt -mjk
+{
+    "email": {
+        "email.from.address": [
+            "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+            "bob@emailsubdomain.bad.local",
+            "alice@email.local",
+            "bob@emailsubdomain.bad.local.xn--p1ai",
+            "bob@email.local"
+        ],
+        "email.sender.address": [
+            "xn--bob--o4d3g5c@xn--c1a3a5a.xn--p1ai",
+            "bob@emailsubdomain.bad.local",
+    ...
+    "ipv4": {
+        "source.ip": [
+            "192.168.20.20",
+            "8.8.8.8",
+            "1.2.3.4",
+            "192.168.1.1"
+        ],
+    ...
+    "ipv4_network": {
+        "source.ip": [
+            "192.168.1.0/24",
+            "8.8.8.0/24"
+    ...
+```
+
+## TODO
+
+- add optional type conversions to match TheHive5 default observable types
+	- change the list of categorizations in README to a markdown table, showing the converted types
+- need to streamline and optimize code
+- remove the filepath regex statements created by AI