txt2stix 0.0.4__tar.gz

txt2stix-0.0.4/.env.example

@@ -0,0 +1,16 @@
+## AI Settings
+INPUT_TOKEN_LIMIT=
+OPENAI_API_KEY=
+ANTHROPIC_API_KEY=
+GOOGLE_API_KEY=
+OPENROUTER_API_KEY=
+DEEPSEEK_API_KEY=
+TEMPERATURE=
+## BIN LIST
+BIN_LIST_API_KEY=
+## CTIBUTLER
+CTIBUTLER_BASE_URL=
+CTIBUTLER_API_KEY=
+## VULMATCH
+VULMATCH_BASE_URL=
+VULMATCH_API_KEY=

txt2stix-0.0.4/.env.markdown

@@ -0,0 +1,52 @@
+# Environment file info
+
+If you're running in production, you should set these securely.
+
+However, if you just want to experiment, set the following values
+
+## AI Settings
+
+* `INPUT_TOKEN_LIMIT`: `15000`
+	* (REQUIRED IF USING AI MODES) Ensure the input/output token count meets requirements and is supported by the model selected. Will not allow files with more than tokens specified to be processed
+* `TEMPERATURE`: `0.0` 
+	* The temperature value ranges from 0 to 2, with lower values indicating greater determinism and higher values indicating more randomness in responses.
+
+**A small note on selecting a provider**
+
+Below are the models you can use.
+
+We strongly recommend using OpenAI because of it's use of structured outputs. Whilst the other models should work, they can often ignore prompts for our expected response structured leading to failures.
+
+* `OPENAI_API_KEY`: YOUR_API_KEY
+	* (REQUIRED IF USING OPENAI MODELS DIRECTLY IN AI MODES) get it from: https://platform.openai.com/api-keys
+* `OPENROUTER_API_KEY`=
+	* (REQUIRED IF USING MODELS PROVIDED BY OPENROUTER IN AI MODES) get it from: https://openrouter.ai/settings/keys
+* `DEEPSEEK_API_KEY`=
+	* (REQUIRED IF USING DEEPSEEK MODELS DIRECTLY IN AI MODES) get it from: https://platform.deepseek.com/api-key
+* `ANTHROPIC_API_KEY`: YOUR_API_KEY
+	* (REQUIRED IF USING ANTHROPIC MODELS DIRECTLY IN AI MODES) get it from" https://console.anthropic.com/settings/keys
+* `GOOGLE_API_KEY`:
+	* (REQUIRED IF USING GOOGLE GEMINI MODELS DIRECTLY IN AI MODES) get it from the Google Cloud Platform (making sure the Gemini API is enabled for the project)
+
+## BIN List
+
+* `BIN_LIST_API_KEY`: BLANK
+	*  for enriching credit card extractions needed for extracting credit card information. You get an API key here https://rapidapi.com/trade-expanding-llc-trade-expanding-llc-default/api/bin-ip-checker
+
+## CTIBUTLER
+
+txt2stix requires [ctibutler](https://github.com/muchdogesec/ctibutler) to lookup ATT&CK, CAPEC, CWE, ATLAS, and locations in blogs
+
+* `CTIBUTLER_BASE_URL`: `'http://api.ctibutler.com'` (recommended)
+	* If you are running CTI Butler locally, be sure to set `'http://host.docker.internal:8006/api/'` in the `.env` file otherwise you will run into networking errors.
+* `CTIBUTLER_API_KEY`:
+	* If using `'http://api.ctibutler.com'`, [get your API key here](http://app.ctibutler.com). Can be left blank if running locally.
+
+## VULMATCH FOR CVE AND CPE LOOKUPS
+
+txt2stix requires [vulmatch](https://github.com/muchdogesec/vulmatch) to lookup CVEs and CPEs in blogs
+
+* `VULMATCH_BASE_URL`: `'http://api.vulmatch.com'` (recommended)
+	* If you are running Vulmatch locally, be sure to set `'http://host.docker.internal:8005/api/'` in the `.env` file otherwise you will run into networking errors.
+* `VULMATCH_API_KEY`:
+	* If using `'http://api.vulmatch.com'`, [get your API key here](http://app.vulmatch.com). Can be left blank if running locally.

txt2stix-0.0.4/.github/workflows/create-release.yml

@@ -0,0 +1,71 @@
+name: Create Release
+run-name: Creating release
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - 'main'
+
+jobs:
+  create-release:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      id-token: write
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install build dependencies
+        run: python3 -m pip install build hatchling
+
+      - name: Extract version from pyproject.toml
+        id: get-version
+        run: |
+          VERSION=$(python -m hatchling version)
+          echo "version=$VERSION" >> $GITHUB_OUTPUT
+
+      - name: Build package
+        run: python3 -m build
+
+      - name: Determine release tag
+        id: release-tag
+        run: |
+          if [[ "${{ github.ref_name }}" == "main" ]]; then
+            echo "tag=${{ steps.get-version.outputs.version }}" >> $GITHUB_OUTPUT
+          else
+            echo "tag=${{ steps.get-version.outputs.version }}-beta-$(date +"%Y-%m-%d-%H-%M-%S")" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Create GitHub Release
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          TAG="${{ steps.release-tag.outputs.tag }}"
+          gh release create "$TAG" --repo "${{ github.repository }}" --notes ""
+          gh release upload "$TAG" dist/** --repo "${{ github.repository }}"
+
+      - name: Clean up old beta releases (keep latest 10)
+        if: github.ref_name != 'main'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh release list --limit 100 --repo "${{ github.repository }}" --json tagName,createdAt \
+            | jq -r '.[] | select(.tagName | test("-beta-")) | [.tagName, .createdAt] | @tsv' \
+            | sort -k2 -r \
+            | tail -n +11 \
+            | cut -f1 \
+            | while read old_tag; do
+                echo "Deleting old beta release: $old_tag"
+                gh release delete "$old_tag" --repo "${{ github.repository }}" --cleanup-tag --yes
+              done
+
+      - name: Publish package distributions to PyPI
+        if: github.ref_name == 'main'
+        uses: pypa/gh-action-pypi-publish@release/v1

txt2stix-0.0.4/.github/workflows/run-tests.yml

@@ -0,0 +1,65 @@
+name: Run Tests
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+permissions:
+  contents: read
+
+jobs:
+  run-tests:
+    runs-on: ubuntu-latest
+    environment: txt2stix_tests
+    
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+    
+          
+      - name: Set .env testing purpose
+        run: |
+            echo > .env
+            echo "CTIBUTLER_BASE_URL=${{ secrets.CTIBUTLER_BASE_URL }}" >> .env
+            echo "CTIBUTLER_API_KEY=${{ secrets.CTIBUTLER_API_KEY }}" >> .env
+            echo "VULMATCH_BASE_URL=${{ secrets.VULMATCH_BASE_URL }}" >> .env
+            echo "VULMATCH_API_KEY=${{ secrets.VULMATCH_API_KEY }}" >> .env
+            echo "TEST_AI_MODEL=${{ secrets.TEST_AI_MODEL }}" >> .env
+            echo "INPUT_TOKEN_LIMIT=1000"  >> .env
+
+            echo "OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY}}" >> .env
+            echo "ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY}}" >> .env
+            echo "GOOGLE_API_KEY=${{ secrets.GOOGLE_API_KEY}}" >> .env
+            echo "OPENROUTER_API_KEY=${{ secrets.OPENROUTER_API_KEY}}" >> .env
+
+            echo "BIN_LIST_API_KEY=${{ secrets.BIN_LIST_API_KEY }}" >> .env
+
+      - name: unit tests
+        id: unit-test
+        run: |
+          set -a; 
+          source .env;
+          set +a;
+          pip install -e .[tests]
+
+          pytest --cov --cov-branch --cov-report=xml --junitxml=junit.xml -o junit_family=legacy
+
+      - name: Upload coverage reports to Codecov
+        if: ${{ !cancelled() }}
+        uses: codecov/codecov-action@v5
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+
+      - name: Upload test results to Codecov
+        if: ${{ !cancelled() }}
+        uses: codecov/test-results-action@v1
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}

txt2stix-0.0.4/.gitignore

@@ -0,0 +1,11 @@
+
+.DS_Store
+.env
+__pycache__/
+
+# venv in docs
+txt2stix-venv/
+
+# output dirs
+output/
+logs/

txt2stix-0.0.4/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2020 DOGESEC (https://www.dogesec.com/)
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

txt2stix-0.0.4/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.4
+Name: txt2stix
+Version: 0.0.4
+Summary: txt2stix is a Python script that is designed to identify and extract IoCs and TTPs from text files, identify the relationships between them, convert them to STIX 2.1 objects, and output as a STIX 2.1 bundle.
+Project-URL: Homepage, https://github.com/muchdogesec/txt2stix
+Project-URL: Issues, https://github.com/muchdogesec/txt2stix/issues
+Author-email: DOGESEC <support@dogesec.com>
+License-File: LICENSE
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Requires-Dist: base58>=2.1.1
+Requires-Dist: beautifulsoup4>=4.12.3
+Requires-Dist: llama-index-core>=0.12.42
+Requires-Dist: llama-index-llms-anthropic>=0.7.2
+Requires-Dist: llama-index-llms-deepseek>=0.1.2
+Requires-Dist: llama-index-llms-gemini>=0.5.0
+Requires-Dist: llama-index-llms-openai-like>=0.4.0
+Requires-Dist: llama-index-llms-openai>=0.4.5
+Requires-Dist: llama-index-llms-openrouter>=0.3.2
+Requires-Dist: mistune>=3.0.2
+Requires-Dist: pathvalidate>=3.2.0
+Requires-Dist: phonenumbers>=8.13.39
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: requests>=2.32.4
+Requires-Dist: schwifty>=2024.6.1
+Requires-Dist: stix2extensions
+Requires-Dist: tld>=0.13
+Requires-Dist: tldextract>=5.1.2
+Requires-Dist: validators>=0.28.3
+Provides-Extra: tests
+Requires-Dist: pytest; extra == 'tests'
+Requires-Dist: pytest-cov; extra == 'tests'
+Requires-Dist: pytest-subtests; extra == 'tests'
+Requires-Dist: requests; extra == 'tests'
+Description-Content-Type: text/markdown
+
+# txt2stix
+
+[![codecov](https://codecov.io/gh/muchdogesec/txt2stix/graph/badge.svg?token=KTHAIYBH1I)](https://codecov.io/gh/muchdogesec/txt2stix)
+
+## Before you begin...
+
+We have build two products on-top of txt2stix that provide more user-friendly experience:
+
+* [Stixify: Extract machine readable cyber threat intelligence from unstructured data](https://github.com/muchdogesec/stixify)
+* [Obstracts: Turn any blog into structured threat intelligence](https://github.com/muchdogesec/obstracts)
+
+## Overview
+
+![txt2stix](docs/txt2stix.png)
+
+txt2stix is a Python script that is designed to identify and extract IoCs and TTPs from text files, identify the relationships between them, convert them to STIX 2.1 objects, and output as a STIX 2.1 bundle.
+
+The general design goal of txt2stix was to keep it flexible, but simple, so that new extractions could be added or modified over time.
+
+In short txt2stix;
+
+1. takes a txt file input
+2. extracts observables for enabled extractions (ai, pattern, or lookup)
+3. converts extracted observables to STIX 2.1 objects
+4. generates the relationships between extracted observables (ai, standard)
+5. converts extracted relationships to STIX 2.1 SRO objects
+6. outputs a STIX 2.1 bundle
+
+## tl;dr
+
+[![txt2stix](https://img.youtube.com/vi/TWVGCou9oGk/0.jpg)](https://www.youtube.com/watch?v=TWVGCou9oGk)
+
+[Watch the demo](https://www.youtube.com/watch?v=TWVGCou9oGk).
+
+## Usage
+
+### Setup
+
+Install the required dependencies using:
+
+```shell
+# clone the latest code
+git clone https://github.com/muchdogesec/txt2stix
+cd txt2stix
+# create a venv
+python3 -m venv txt2stix-venv
+source txt2stix-venv/bin/activate
+# install requirements
+pip3 install .
+```
+
+### Set variables
+
+txt2stix has various settings that are defined in an `.env` file.
+
+To create a template for the file:
+
+```shell
+cp .env.example .env
+```
+
+To see more information about how to set the variables, and what they do, read the `.env.markdown` file.
+
+### Usage
+
+```shell
+python3 txt2stix.py \
+	--relationship_mode MODE \
+	--input_file FILE.txt \
+	...
+```
+
+The following arguments are available:
+
+#### Input settings
+
+* `--input_file` (REQUIRED): the file to be converted. Must be `.txt`
+
+#### STIX Report generation settings
+
+
+* `--name` (REQUIRED): name of file, max 72 chars. Will be used in the STIX Report Object created.
+* `--report_id` (OPTIONAL): Sometimes it is required to control the id of the `report` object generated. You can therefore pass a valid UUIDv4 in this field to be assigned to the report. e.g. passing `2611965-930e-43db-8b95-30a1e119d7e2` would create a STIX object id `report--2611965-930e-43db-8b95-30a1e119d7e2`. If this argument is not passed, the UUID will be randomly generated.
+* `--tlp_level` (OPTIONAL): Options are `clear`, `green`, `amber`, `amber_strict`, `red`. Default if not passed, is `clear`.
+* `--confidence` (OPTIONAL): value between 0-100. Default if not passed is null.
+* `--labels` (OPTIONAL): comma seperated list of labels. Case-insensitive (will all be converted to lower-case). Allowed `a-z`, `0-9`. e.g.`label1,label2` would create 2 labels.
+* `--created` (OPTIONAL): by default all object `created` times will take the time the script was run. If you want to explicitly set these times you can do so using this flag. Pass the value in the format `YYYY-MM-DDTHH:MM:SS.sssZ` e.g. `2020-01-01T00:00:00.000Z`
+* `--use_identity` (OPTIONAL): can pass a full STIX 2.1 identity object (make sure to properly escape). Will be validated by the STIX2 library.
+* `--external_refs` (OPTIONAL): txt2stix will automatically populate the `external_references` of the report object it creates for the input. You can use this value to add additional objects to `external_references`. Note, you can only add `source_name` and `external_id` values currently. Pass as `source_name=external_id`. e.g. `--external_refs txt2stix=demo1 source=id` would create the following objects under the `external_references` property: `{"source_name":"txt2stix","external_id":"demo1"},{"source_name":"source","external_id":"id"}`
+
+#### Output settings
+
+How the extractions are performed
+
+* `--use_extractions` (REQUIRED): if you only want to use certain extraction types, you can pass their slug found in either `includes/ai/config.yaml`, `includes/lookup/config.yaml` `includes/pattern/config.yaml` (e.g. `pattern_ipv4_address_only`). Default if not passed, no extractions applied. You can also pass a catch all wildcard `*` which will match all extraction paths (e.g. `'pattern_*'` would run all extractions starting with `pattern_` -- make sure to use quotes when using a wildcard)
+	* Important: if using any AI extractions (`ai_*`), you must set an AI API key in your `.env` file
+	* Important: if you are using any MITRE ATT&CK, CAPEC, CWE, ATLAS or Location extractions you must set `CTIBUTLER` or NVD CPE or CVE extractions you must set `VULMATCH` settings in your `.env` file
+* `--relationship_mode` (REQUIRED): either.
+	* `ai`: AI provider must be enabled. extractions performed by either regex or AI for extractions user selected. Rich relationships created from AI provider from extractions.
+	* `standard`: extractions performed by either regex or AI (AI provider must be enabled) for extractions user selected. Basic relationships created from extractions back to master Report object generated.
+* `--ignore_extraction_boundary` (OPTIONAL, default `false`, not compatible with AI extractions): in some cases the same string will create multiple extractions depending on extractions set (e.g. `https://www.google.com/file.txt` could create a url, url with file, domain, subdomain, and file). The default behaviour is for txt2stix to take the longest extraction and ignore everything else (e.g. only extract url with file, and ignore url, file, domain, subdomain, and file). If you want to override this behaviour and get all extractions in the output, set this flag to `true`.
+* `--ignore_image_refs` (default `true`): images references in documents don't usually need extracting. e.g. `<img src="https://example.com/image.png" alt="something">` you would not want domain or file extractions extracting `example.com` and `image.png`. Hence these are ignored by default (they are removed from text sent to extraction). Note, only the `img src` is ignored, all other values e.g. `alt` are considered. If you want extractions to consider this data, set it to `false`
+* `--ignore_link_refs` (default `true`): link references in documents don't usually need extracting e.g. `<a href="https://example.com/link.html" title="something">Bad Actor</a>` you would only want `Bad actor` to be considered for extraction. Hence these part of the link are ignored by default (they are removed from text sent to extraction). Note, only the `a href` is ignored, all other values e.g. `title` are considered. Setting this to `false` will also include everything inside the link tag (e.g. `example.com` would extract as a domain)
+
+#### AI settings
+
+If any AI extractions, or AI relationship mode is set, you must set the following accordingly
+
+* `--ai_settings_extractions`:
+	* defines the `provider:model` to be used for extractions. You can supply more than one provider. Seperate with a space (e.g. `openrouter:openai/gpt-4o` `openrouter:deepseek/deepseek-chat`) If more than one provider passed, txt2stix will take extractions from all models, de-dupelicate them, and them package them in the output. Currently supports:
+		* Provider (env var required `OPENROUTER_API_KEY`): `openrouter:`, providers/models `openai/gpt-4o`, `deepseek/deepseek-chat` ([More here](https://openrouter.ai/models))
+		* Provider (env var required `OPENAI_API_KEY`): `openai:`, models e.g.: `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `gpt-4` ([More here](https://platform.openai.com/docs/models))
+		* Provider (env var required `ANTHROPIC_API_KEY`): `anthropic:`, models e.g.: `claude-3-5-sonnet-latest`, `claude-3-5-haiku-latest`, `claude-3-opus-latest` ([More here](https://docs.anthropic.com/en/docs/about-claude/models))
+		* Provider (env var required `GOOGLE_API_KEY`): `gemini:models/`, models: `gemini-1.5-pro-latest`, `gemini-1.5-flash-latest` ([More here](https://ai.google.dev/gemini-api/docs/models/gemini))
+		* Provider (env var required `DEEPSEEK_API_KEY`): `deepseek:`, models `deepseek-chat` ([More here](https://api-docs.deepseek.com/quick_start/pricing))
+	* See `tests/manual-tests/cases-ai-extraction-type.md` for some examples
+* `--ai_settings_relationships`:
+	* similar to `ai_settings_extractions` but defines the model used to generate relationships. Only one model can be provided. Passed in same format as `ai_settings_extractions`
+	* See `tests/manual-tests/cases-ai-relationships.md` for some examples
+* `--ai_content_check_provider`: Passing this flag will get the AI to try and classify the text in the input to 1) determine if it is talking about threat intelligence, and 2) what type of threat intelligence it is talking about. For context, we use this to filter out non-threat intel posts in Obstracts and Stixify. You pass `provider:model` with this flag to determine the AI model you wish to use to perform the check.
+* `--ai_create_attack_flow`: passing this flag will also prompt the AI model (the same entered for `--ai_settings_relationships`) to generate an [Attack Flow](https://center-for-threat-informed-defense.github.io/attack-flow/) for the MITRE ATT&CK extractions to define the logical order in which they are being described. You must pass `--ai_settings_relationships` for this to work.
+
+## Adding new extractions
+
+It is very likely you'll want to extend txt2stix to include new extractions to;
+
+* Add a new lookup extraction: add your lookup to `includes/lookups` as a `.txt` file. Lookups should be a list of items seperated by new lines to be searched for in documents. Once this is added, update `includes/extractions/lookup/config.yaml` with a new record pointing to your lookup. You can now use this lookup time at script run-time.
+* Add a new AI extraction: Edit `includes/extractions/ai/config.yaml` with a new record for your extraction. You can craft the prompt used in the config to control how the LLM performs the extraction.
+
+Currently it is not possible to easily add any other types of extractions (without modifying the logic at a code level).
+
+## Detailed documentation
+
+If you would like to understand how txt2stix works in more detail, please refer to the documentation in `/docs/README.md`.
+
+This documentation is paticularly helpful to read for those of you wanting to add your own custom extractions.
+
+## Useful supporting tools
+
+* [A Quick Start Guide to txt2stix](https://www.dogesec.com/blog/txt2stix_quickstart_guide/)
+* [An example of how to use txt2stix with Attack Flows](https://www.dogesec.com/blog/understading_structure_attack_flows/)
+* [STIX2 Python Library](https://pypi.org/project/stix2/): APIs for serializing and de-serializing STIX2 JSON content
+* [STIX 2 Pattern Validator](https://pypi.org/project/stix2-patterns/): a tool for checking the syntax of the Cyber Threat Intelligence (CTI) STIX Pattern expressions
+* [STIX Viewer](https://github.com/traut/stixview): Quickly load bundles produced from your report
+
+## Support
+
+[Minimal support provided via the DOGESEC community](https://community.dogesec.com/).
+
+## License
+
+[Apache 2.0](/LICENSE).