shadowaitools 1.0.0__tar.gz

shadowaitools-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alpha Quantum
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

shadowaitools-1.0.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include README.md
+include LICENSE

shadowaitools-1.0.0/PKG-INFO

@@ -0,0 +1,336 @@
+Metadata-Version: 2.1
+Name: shadowaitools
+Version: 1.0.0
+Summary: Shadow AI detection from the logs you already have: parse a DNS, proxy or firewall export locally, look up each unique domain against the AI Tools Blocklist, and get an inventory of the AI tools in use with category, AI type and vendor training verdicts.
+Home-page: https://www.shadowaitools.com
+Author: Alpha Quantum
+Author-email: info@alpha-quantum.com
+License: MIT
+Project-URL: Homepage, https://www.shadowaitools.com
+Project-URL: Documentation, https://www.shadowaitools.com
+Project-URL: Source, https://github.com/explainableaixai/shadowaitools
+Project-URL: Tracker, https://www.shadowaitools.com/contact.php
+Project-URL: AI Tools Blocklist, https://www.aitoolsblocklist.com
+Project-URL: AI Agent Allow List, https://www.aiagentallowlist.com
+Project-URL: Shadow AI Tools, https://www.shadowaitools.com
+Keywords: shadow ai,shadow ai detection,shadow ai tools,shadow ai discovery,ai tools inventory,dns log analysis,proxy log analysis,firewall log,ai governance,ai acceptable use,data loss prevention,ai blocklist,generative ai usage,ai risk management
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Security
+Classifier: Topic :: System :: Networking :: Monitoring
+Classifier: Topic :: System :: Systems Administration
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.20.0
+
+# shadowaitools
+
+`shadowaitools` turns a DNS, proxy or firewall export into an inventory of the AI tools in use on your network. It runs on your own machine: the export is parsed locally, every hostname is reduced to its registrable domain, and each unique domain is looked up once against the [AI blocklist for web filtering](https://www.aitoolsblocklist.com) at aitoolsblocklist.com. What comes back is a list of tools with category, AI type, hit counts, the users or devices that reached them, and the vendor's position on training with your data, dated.
+
+It is the command line and Python counterpart of the hosted service at shadowaitools.com, which lets you [find the AI tools employees use](https://www.shadowaitools.com) from the same export in a browser and produces the per-user breakdown, the sanctioned versus unsanctioned split and a PDF evidence pack.
+
+Only `requests` is required. Python 3.7 and newer.
+
+---
+
+## Installation
+
+```bash
+pip install shadowaitools
+```
+
+This installs the `shadowaitools` command and the `shadowaitools` package. The API key is an AI Tools Blocklist key from the account area at aitoolsblocklist.com; pass it as `--key` or export it once:
+
+```bash
+export SHADOWAITOOLS_API_KEY=your_key
+```
+
+## Quick start
+
+```bash
+shadowaitools scan nextdns-export.csv --csv inventory.csv --json inventory.json
+```
+
+```
+Shadow AI inventory (csv export, 400 lines, 45 unique domains, 45 lookups)
+AI tools found: 34   users involved: 10   train on your data by default: 16   terms silent: 15
+
+domain                      hits   ai type     category                    trains on data   sanctioned  users
+--------------------------  -----  ----------  --------------------------  ---------------  ----------  ----------------------
+character.ai                10     ai_native   Text & Language             yes              no          laptop-marketing-02 +2
+openai.com                  10     ai_native   Code & Development          opt_out_default  yes         laptop-eng-07 +4
+otter.ai                    7      ai_native   Audio, Voice & Music        yes              no          laptop-sales-09 +2
+midjourney.com              5      ai_native   Image & Visual              unstated         no          laptop-exec-01 +1
+elevenlabs.io               4      ai_native   Text & Language             opt_out_default  no          laptop-eng-07 +1
+fireflies.ai                3      ai_native   Audio, Voice & Music        no               no          desktop-support-03 +1
+```
+
+In Python:
+
+```python
+from shadowaitools import scan, to_csv
+
+inventory = scan(
+    "zscaler-web.csv",
+    api_key="your_key",
+    sanctioned=["openai.com", "github.com"],
+    cache_file="lookups.json",
+)
+
+print(inventory["summary"])
+# {'lines': 400, 'records': 400, 'unique_domains': 45, 'lookups': 45, 'ai_tools_found': 34,
+#  'sanctioned': 2, 'unsanctioned': 32, 'users_involved': 10, 'training_default_yes': 16,
+#  'training_no': 3, 'unstated': 15, 'quota_remaining': 9999940}
+
+for tool in inventory["tools"]:
+    if not tool["sanctioned"] and tool["ai_type"] == "ai_native":
+        print(tool["domain"], tool["hits"], [u["name"] for u in tool["users"]], tool["trains_on_data"])
+
+with open("inventory.csv", "w") as fh:
+    fh.write(to_csv(inventory))
+```
+
+## How a scan works
+
+1. **Parse.** The first non-empty line decides the format. A header row with a known column name means CSV (comma, tab, semicolon or pipe). `hostname=` or `dstname=` tokens mean key=value syslog. `query[A]` means dnsmasq. The Squid `access.log` layout and Windows DNS Server debug packets have their own detectors. Anything else is read line by line for the first hostname and the first private IP address.
+2. **Reduce.** Each hostname becomes a registrable domain (`chat.openai.com` to `openai.com`, `news.bbc.co.uk` to `bbc.co.uk`). Names in reserved zones (`.local`, `.internal`, `.lan`, `.corp`, `.home`, `.arpa`, `.test`, `.example`) are dropped before anything is sent.
+3. **Look up.** Each unique domain is sent once to `GET https://www.aitoolsblocklist.com/api/check?domain=<domain>` with the key in the `X-API-Key` header. With `cache_file`, domains seen on a previous run are answered from the cache.
+4. **Assemble.** Domains with `blocked: true` become tools. Hits, hosts and users are attached from the parse, the sanctioned flag from your list, and the category, AI type and training fields from the lookup.
+
+A first run on a 400-line export with 45 distinct domains makes 45 lookups. A month of resolver logs with two million lines usually collapses to a few thousand registrable domains, because most traffic goes to a small set of hosts, and the second run through the same cache pays only for domains that are new that day. The `domains` command prints the exact count before any lookup is made.
+
+The user or device column is optional. When the export carries `Identities`, `user`, `Source User`, `device_name`, `client_ip`, `src` or a similar column, every tool lists who reached it; when it does not, the inventory still has the tools and the hit counts.
+
+## Accepted exports
+
+| Source | Format | Hostname column or field | User column or field |
+|---|---|---|---|
+| Cisco Umbrella activity export | csv | `Domain` | `Identities`, `Internal IP` |
+| Cloudflare Gateway DNS log | csv | `QueryName` | `DeviceName`, `SourceIP`, `Email` |
+| DNSFilter query log | csv | `domain` | `client`, `device` |
+| NextDNS log export | csv | `domain` | `client_ip`, `device_name` |
+| Palo Alto URL filtering log | csv | `URL` | `Source User`, `Source address` |
+| Zscaler web log | csv | `url` | `user`, `cip` |
+| Fortinet FortiGate web filter syslog | key-value | `hostname=` | `user=`, `srcip=` |
+| SonicWall syslog | key-value | `dstname=` | `usr=`, `src=` |
+| Pi-hole and dnsmasq | dnsmasq | `query[A] name` | `from address` |
+| Squid | squid | `CONNECT host:443` | client IP, authenticated user |
+| Windows DNS Server debug log | windows-dns | encoded question name | client address |
+| Plain hostname or URL list | generic | the line | none |
+| Any other text log | generic | first hostname on the line | first private IPv4 on the line |
+
+Anything with a header row and a hostname column works, whatever produced it. The `formats` command prints this list from the installed version.
+
+## API
+
+### `scan(source, api_key=None, **options)`
+
+`source` is a file path, a string of contents, `bytes` or an open file. Options:
+
+| Option | Default | Meaning |
+|---|---|---|
+| `api_key` | env `SHADOWAITOOLS_API_KEY` or `ATB_API_KEY` | AI Tools Blocklist key |
+| `concurrency` | 1 | parallel lookups |
+| `pause` | 0.0 | seconds to wait after each lookup |
+| `max_lines` | none | stop parsing after this many lines |
+| `cache_file` | none | JSON file of previous lookups, reused and extended |
+| `sanctioned` | `[]` | domains to flag as sanctioned |
+| `on_progress` | none | callback `(done, total)` |
+| `timeout`, `max_retries`, `base_url` | 30, 2, production | passed to the client |
+
+Returns a dict with `generated`, `format`, `summary`, `by_category` and `tools`.
+
+### Tool fields
+
+| Field | Values |
+|---|---|
+| `domain`, `hosts`, `hits`, `users` | from the export |
+| `sanctioned` | `True` when the domain is in your list |
+| `primary_category` | one of 18 functional categories |
+| `categories` | `[{"category", "subcategory"}]`, a tool can sit in several |
+| `ai_type` | `ai_native` or `ai_enabled` |
+| `trains_on_data` | `yes`, `no`, `opt_out_default`, `unstated` |
+| `opt_out_available`, `enterprise_no_training`, `api_no_training` | same value set |
+| `terms_checked` | ISO date the vendor terms were last read |
+
+### Other functions
+
+| Function | Purpose |
+|---|---|
+| `parse_log(source, max_lines=None)` | `{"format", "lines", "records": [{"host", "user"}]}` with no network call |
+| `extract_domains(records)` | `[{"domain", "hosts", "hits", "users"}]` sorted by hits |
+| `registrable_domain(host)` | reduce a hostname |
+| `to_csv(inventory)` | one CSV row per tool |
+| `to_table(inventory)` | fixed-width text table |
+| `Client(api_key).lookup(domain)` | the raw lookup |
+
+### Exceptions
+
+`AuthenticationError` (401), `QuotaError` (403, inactive account or monthly quota used up), `RateLimitError` (429 after two retries) and the base `ShadowAIToolsError` (anything else, including 503 after retries). Each carries `.status` and `.body`.
+
+## Command line
+
+```
+shadowaitools scan <file> [--key KEY] [--json FILE] [--csv FILE] [--cache FILE]
+                          [--sanctioned a.com,b.com] [--max-lines N] [--concurrency N]
+                          [--pause SECONDS] [--quiet]
+shadowaitools domains <file>     unique registrable domains and hit counts, no lookups
+shadowaitools formats            the accepted export formats
+```
+
+`domains` is the dry run: it prints the format that was detected and the number of lookups a scan would need.
+
+## Worked examples
+
+### Weekly report for a security team
+
+```python
+# weekly_shadow_ai.py
+import datetime
+import json
+from shadowaitools import scan, to_csv
+
+week = datetime.date.today().isocalendar()[1]
+inv = scan(
+    f"/exports/umbrella-week-{week}.csv",
+    cache_file="/var/lib/shadowaitools/lookups.json",
+    sanctioned=open("/etc/shadowaitools/approved.txt").read().split(),
+    pause=0.05,
+)
+
+with open(f"/reports/shadow-ai-week-{week}.csv", "w") as fh:
+    fh.write(to_csv(inv))
+
+risky = [t for t in inv["tools"] if not t["sanctioned"] and t["trains_on_data"] in ("yes", "opt_out_default")]
+print(f"week {week}: {inv['summary']['ai_tools_found']} tools, {len(risky)} unsanctioned tools that train on input")
+for t in sorted(risky, key=lambda t: -t["hits"])[:10]:
+    print(f"  {t['domain']:28} {t['hits']:5} hits  {len(t['users']):3} users  {t['primary_category']}")
+```
+
+### Comparing two weeks
+
+```python
+from shadowaitools import scan
+
+before = scan("proxy-week-36.log", cache_file="lookups.json")
+after = scan("proxy-week-37.log", cache_file="lookups.json")
+
+seen_before = {t["domain"] for t in before["tools"]}
+new_tools = [t for t in after["tools"] if t["domain"] not in seen_before]
+print("new AI tools this week:", [t["domain"] for t in new_tools])
+```
+
+Because both scans share the cache, the second one only pays for domains that did not appear in the first.
+
+### Streaming a large export in chunks
+
+Exports from a busy resolver run to millions of lines. `parse_log` accepts a string, so a file can be read in blocks and grouped before a single lookup is made.
+
+```python
+from shadowaitools import Client, extract_domains, parse_log, registrable_domain
+
+records = []
+with open("dns-month.log", encoding="utf-8", errors="replace") as fh:
+    block = []
+    for line in fh:
+        block.append(line)
+        if len(block) == 200_000:
+            records.extend(parse_log("".join(block))["records"])
+            block = []
+    if block:
+        records.extend(parse_log("".join(block))["records"])
+
+groups = extract_domains(records)
+print(len(records), "records,", len(groups), "unique domains")
+
+client = Client("your_key")
+tools = []
+for g in groups:
+    r = client.lookup(g["domain"])
+    if r.get("blocked"):
+        tools.append((g["domain"], g["hits"], r.get("primary_category"), r.get("trains_on_data")))
+```
+
+### Only the lookup
+
+```python
+from shadowaitools import Client
+
+c = Client("your_key")
+print(c.lookup("chat.openai.com"))
+# {'domain': 'openai.com', 'blocked': True, 'primary_category': 'Code & Development', 'ai_type': 'ai_native',
+#  'categories': [...], 'trains_on_data': 'opt_out_default', 'opt_out_available': 'yes', ...}
+print(c.lookup("example.com")["blocked"])
+# False
+```
+
+## Why the logs are the right starting point
+
+Every governance framework begins with an inventory. The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) puts "Map" before "Measure" and "Manage": an organisation has to know which AI systems are in use before any control can be applied. The [ENISA](https://www.enisa.europa.eu/) work on AI cybersecurity makes the same point for European organisations, and the [CISA](https://www.cisa.gov/) guidance on secure AI deployment assumes that operators can enumerate the AI services their people reach. On the data-protection side, the [ICO's guidance on AI and data protection](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/) treats the flow of personal data into third-party AI services as a processing activity that has to be documented.
+
+The inventory you need already exists in the DNS filter, proxy or firewall. It is complete in a way a survey can never be, it costs nothing to export, and it names the hostnames rather than the products people remember. The missing step is classification, which is what the lookup adds: is this domain an AI tool, what kind, and what does the vendor do with the input. Alongside the tool inventory, the same organisation usually wants the opposite control for its own agents, which is where an allow list for AI agents comes in: the [AI agent allow list](https://www.aiagentallowlist.com) tells a browsing agent which pages on a site it may open and which it must not, so the two datasets cover both directions of AI traffic.
+
+## Hosted audit
+
+The package produces the inventory. The hosted [shadow AI inventory](https://www.shadowaitools.com) at shadowaitools.com produces the report: upload the same export, get every tool with category and risk level, the per-user breakdown, dated training verdicts, the sanctioned split against your approved list, sector policy verdicts from the AI Policy Profiles, a CSV and a PDF evidence pack. The free preview names a fifth of the tools found and comes with a preview PDF; full reports are one-time purchases, and the subscription plans on aitoolsblocklist.com include one to ten audits a month.
+
+## Related packages
+
+- [`aiblocklist`](https://pypi.org/project/aiblocklist/) and [`aitoolsblocklist`](https://pypi.org/project/aitoolsblocklist/): Python clients for the lookup and feed APIs of the [AI domain blocklist](https://www.aitoolsblocklist.com), 20,000+ AI tool domains in 18 categories, refreshed daily, with EDL, PAC, hosts and DNS feeds.
+- [`aiagentallowlist`](https://pypi.org/project/aiagentallowlist/): per-URL allow, deny and flag verdicts for browsing agents across 40 million+ domains, up to 28 verified page types each.
+- [`websiteclassificationapi`](https://pypi.org/project/websiteclassificationapi/): the [website categorization API](https://www.websitecategorizationapi.com), 700+ IAB content categories for any URL.
+- [`cipawebfiltering`](https://pypi.org/project/cipawebfiltering/) and [`phishingdetectionapi`](https://pypi.org/project/phishingdetectionapi/): the [CIPA web filtering](https://www.cipawebfiltering.com) client for schools and the [phishing detection API](https://www.phishingdetectionapi.com) with 390,000+ DNS-verified active phishing domains.
+- The [web filtering database](https://www.webfilteringdatabase.com), 120M+ domains in 59 categories, for the same firewalls and resolvers.
+- Node.js: [`shadowaitools` on npm](https://www.npmjs.com/package/shadowaitools), same formats, same inventory, same cache file.
+- Source: [github.com/explainableaixai/shadowaitools](https://github.com/explainableaixai/shadowaitools) and [gitlab.com/url-classifications/shadowaitools](https://gitlab.com/url-classifications/shadowaitools).
+
+## Frequently asked questions
+
+**How do I detect shadow AI on my network?**
+Export a week of DNS, proxy or firewall logs, run `shadowaitools scan export.csv`, and read the inventory. Every hostname is matched against 20,000+ known AI tool domains, so the result is the AI traffic that actually crossed your network. For a report with the per-user breakdown and a PDF, upload the same file at [shadowaitools.com](https://www.shadowaitools.com).
+
+**Is anything installed on endpoints or inspected in transit?**
+No. The tool reads a log export that your DNS filter, proxy or firewall already produces. There is no agent, no TLS inspection and no change to the network.
+
+**Does the export leave my machine?**
+No. It is parsed locally. Only unique registrable domains are looked up, one request each, and reserved internal zones are never sent.
+
+**Which exports are supported?**
+Cisco Umbrella, Zscaler, Palo Alto, Fortinet, Cloudflare Gateway, DNSFilter, NextDNS, Pi-hole, SonicWall, Squid, Windows DNS Server debug logs, any CSV with a header row, key=value syslog lines and plain hostname lists. See the table above.
+
+**What is the difference between `ai_native` and `ai_enabled`?**
+`ai_native` is a service whose product is the AI: a chatbot, a code assistant, an image generator. `ai_enabled` is an ordinary product that has added AI features, such as an office suite with a built-in assistant. Both appear in the inventory because both can receive pasted content; the flag lets you treat them differently.
+
+**How current is the database behind the lookup?**
+It is rebuilt daily, with about 300,000 new domains checked every day against a 120-million-domain corpus, so new tools are caught close to launch. The training verdicts come from a review of 13,000+ vendor terms and carry the date they were checked.
+
+**Can I limit how many lookups a scan makes?**
+Yes. `shadowaitools domains file` shows the count first; `--max-lines` caps parsing; `--cache` makes repeated runs free for domains already seen; `--pause` slows the run down.
+
+**Who is behind shadowaitools?**
+Alpha Quantum, which also builds the [AI tools blocklist](https://www.aitoolsblocklist.com), the [AI agent allow list](https://www.aiagentallowlist.com), the [website categorization API](https://www.websitecategorizationapi.com) and the [web filtering database](https://www.webfilteringdatabase.com).
+
+## Links
+
+- Hosted shadow AI audit: [https://www.shadowaitools.com](https://www.shadowaitools.com)
+- The lookup database: [https://www.aitoolsblocklist.com](https://www.aitoolsblocklist.com)
+- AI agent allow list: [https://www.aiagentallowlist.com](https://www.aiagentallowlist.com)
+- NIST AI Risk Management Framework: [https://www.nist.gov/itl/ai-risk-management-framework](https://www.nist.gov/itl/ai-risk-management-framework)
+- ENISA: [https://www.enisa.europa.eu/](https://www.enisa.europa.eu/)
+- CISA: [https://www.cisa.gov/](https://www.cisa.gov/)
+- ICO, AI and data protection: [https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/)
+
+## License
+
+MIT