aiblocklist 1.0.0__tar.gz

aiblocklist-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alpha Quantum
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aiblocklist-1.0.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include README.md
+include LICENSE

aiblocklist-1.0.0/PKG-INFO

@@ -0,0 +1,401 @@
+Metadata-Version: 2.1
+Name: aiblocklist
+Version: 1.0.0
+Summary: AI blocklist client: classify any domain as an AI tool (20,000+ domains, 18 categories, daily refresh, vendor training-on-your-data verdicts) and download the feed database for DNS sinkholes, proxies, firewalls and SOAR playbooks.
+Home-page: https://www.aitoolsblocklist.com
+Author: Alpha Quantum
+Author-email: info@alpha-quantum.com
+License: MIT
+Project-URL: Homepage, https://www.aitoolsblocklist.com
+Project-URL: Documentation, https://www.aitoolsblocklist.com/ai-blocklist-api.php
+Project-URL: Source, https://github.com/explainableaixai/aiblocklist
+Project-URL: Tracker, https://www.aitoolsblocklist.com/contact.php
+Project-URL: AI Tools Blocklist, https://www.aitoolsblocklist.com
+Project-URL: AI Agent Allow List, https://www.aiagentallowlist.com
+Project-URL: Shadow AI Tools, https://www.shadowaitools.com
+Keywords: ai blocklist,ai tools blocklist,block ai tools,shadow ai,dns sinkhole,external dynamic list,web filtering,secure web gateway,dlp,acceptable use policy,generative ai,ai governance,domain classification,training on your data
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: Proxy Servers
+Classifier: Topic :: Security
+Classifier: Topic :: System :: Networking :: Firewalls
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.20.0
+
+# aiblocklist
+
+Python client for the AI Tools Blocklist API, a database of [classified AI tool domains](https://www.aitoolsblocklist.com) built for the teams that run proxies, DNS resolvers, DLP pipelines and security automation. One call classifies a domain; one download gives you the whole list for local matching.
+
+What each record carries:
+
+- 20,000+ AI-tool domains in 18 functional categories with subcategories, multi-label, rebuilt daily
+- an AI type: `ai_native` for tools that are the AI, `ai_enabled` for products with an AI feature inside
+- the vendor's data-use position: trains on input, opt-out available, enterprise tier exempt, API tier exempt, and the date the terms were checked
+- for feed and database plans, the full CSV plus hosted EDL, PAC, hosts and DNS feeds
+
+Only `requests` is required. Python 3.7 and newer.
+
+---
+
+## Installation
+
+```bash
+pip install aiblocklist
+```
+
+## Quick start
+
+```python
+from aiblocklist import AIBlocklistClient
+
+client = AIBlocklistClient("YOUR_API_KEY")
+
+r = client.check("chatgpt.com")
+r.blocked               # True
+r.primary_category      # "Text & Language"
+r.ai_type               # "ai_native"
+r.category_names        # ["Text & Language"]
+r.subcategory_names     # ["General assistants & chatbots"]
+r.trains_on_data        # "opt_out_default"
+r.terms_checked         # "2026-09-17"
+r.quota_remaining       # lookups left in the current 30-day cycle
+
+client.check("example.com").blocked   # False
+```
+
+`check()` returns a `Lookup`, a `dict` subclass with the raw JSON and convenience properties, so `r["blocked"]` and `r.blocked` are the same value. The key comes from the account area at aitoolsblocklist.com and is sent as `X-API-Key`. Public methods (`stats()`, `categories()`, `clause()`) work without a key.
+
+From the command line:
+
+```bash
+export ATB_API_KEY=YOUR_API_KEY
+python -m aiblocklist chatgpt.com midjourney.com example.com
+python -m aiblocklist --stats
+```
+
+## Methods
+
+| Method | Endpoint | Key | Returns |
+|---|---|---|---|
+| `check(domain)` | `GET /api/check?domain=` | yes | `Lookup` |
+| `is_blocked(domain)` | same | yes | `bool` |
+| `check_many(domains, pause=0)` | same, sequential, deduplicated | yes | `list[Lookup]` |
+| `data_use(domain)` | same | yes | dict of the five data-use fields |
+| `feeds.status()` | `GET /api/database/?action=status` | feed or database plan | plan and file state |
+| `feeds.database_info()` | `GET /api/database/?action=database_info` | feed or database plan | file name, timestamp, size |
+| `feeds.download_database(path)` | `GET /api/database/?action=download_database` | feed or database plan | `{"file", "bytes"}` |
+| `feeds.download_categories(path)` | `GET /api/database/?action=download_categories` | feed or database plan | `{"file", "bytes"}` |
+| `stats()` | `GET /api/stats.php` | no | totals and 18 categories with counts |
+| `categories()` | same | no | the category list |
+| `clause(domain, field)` | `GET /api/data-use-clause.php` | no | `{"clause", "url"}` or `None` |
+
+Pass a bare domain or any URL; `clean_domain()` strips scheme, path, port, credentials and a leading `www.`. Subdomains resolve to the registrable domain on the server.
+
+## The response, field by field
+
+```json
+{
+  "domain": "chatgpt.com",
+  "blocked": true,
+  "primary_category": "Text & Language",
+  "ai_type": "ai_native",
+  "categories": [{"category": "Text & Language", "subcategory": "General assistants & chatbots"}],
+  "trains_on_data": "opt_out_default",
+  "opt_out_available": "yes",
+  "enterprise_no_training": "yes",
+  "api_no_training": "yes",
+  "terms_checked": "2026-09-17",
+  "quota_remaining": 9999986
+}
+```
+
+| Field | Values |
+|---|---|
+| `blocked` | `true` when the domain is a known AI tool, otherwise `false` with an empty `categories` list |
+| `primary_category` | one of the 18 categories |
+| `ai_type` | `ai_native` or `ai_enabled` |
+| `categories` | every category and subcategory the tool belongs to |
+| `trains_on_data` | `yes`, `no`, `opt_out_default`, `unstated` |
+| `opt_out_available` | `yes`, `no`, `unstated` |
+| `enterprise_no_training` | `yes`, `no`, `unstated` |
+| `api_no_training` | `yes`, `no`, `unstated` |
+| `terms_checked` | ISO date the vendor terms were last read |
+| `quota_remaining` | lookups left on the plan |
+
+Both found and not-found are HTTP 200. Errors are raised as exceptions (see below).
+
+## Worked examples
+
+### 1. FastAPI middleware for an internal egress service
+
+An internal HTTP egress service (or any FastAPI app that proxies outbound requests) checks the destination host once, keeps the verdict for a day, and enforces a category policy. Assistants that train on input by default are allowed only with a header that the DLP layer downstream can key on.
+
+```python
+import time
+from fastapi import FastAPI, Request
+from fastapi.responses import JSONResponse
+from aiblocklist import AIBlocklistClient, AIBlocklistError
+
+app = FastAPI()
+client = AIBlocklistClient("YOUR_API_KEY")
+cache = {}                          # host -> (verdict, expires)
+TTL = 86400
+BLOCK = {"Image & Visual", "Audio & Voice", "Companions & Social"}
+
+
+def verdict_for(host: str) -> str:
+    now = time.time()
+    hit = cache.get(host)
+    if hit and hit[1] > now:
+        return hit[0]
+    try:
+        r = client.check(host)
+    except AIBlocklistError:
+        return "allow"              # fail open on API trouble, log it
+    verdict = "allow"
+    if r.blocked:
+        if BLOCK & set(r.category_names):
+            verdict = "block"
+        elif r.trains_on_data in ("yes", "opt_out_default"):
+            verdict = "warn"
+    cache[host] = (verdict, now + TTL)
+    return verdict
+
+
+@app.middleware("http")
+async def ai_policy(request: Request, call_next):
+    host = request.headers.get("x-target-host", "")
+    v = verdict_for(host) if host else "allow"
+    if v == "block":
+        return JSONResponse({"error": "AI tool blocked by policy", "host": host}, status_code=403)
+    response = await call_next(request)
+    if v == "warn":
+        response.headers["X-AI-Policy"] = "trains-on-input"
+    return response
+```
+
+The same shape works as a Flask `before_request` hook. With a one-day cache, a gateway that sees a few thousand distinct hosts uses a few thousand lookups a month.
+
+### 2. Classifying a proxy export with pandas
+
+Take the unique hosts from a proxy or DNS export, classify them, and produce a table of AI tools with their categories and training terms. `check_many()` deduplicates, so the cost is one lookup per unique host.
+
+```python
+import pandas as pd
+from aiblocklist import AIBlocklistClient
+
+client = AIBlocklistClient("YOUR_API_KEY")
+
+log = pd.read_csv("proxy_export.csv")               # columns: timestamp, user, host, bytes
+hosts = log["host"].dropna().unique().tolist()
+
+results = client.check_many(hosts, pause=0.05)
+found = pd.DataFrame([
+    {
+        "domain": r.domain,
+        "primary_category": r.primary_category,
+        "subcategories": "; ".join(r.subcategory_names),
+        "ai_type": r.ai_type,
+        "trains_on_data": r.trains_on_data,
+        "enterprise_no_training": r["enterprise_no_training"],
+        "terms_checked": r.terms_checked,
+    }
+    for r in results if r.blocked
+])
+
+# join back to users so the report says who reached what
+usage = log.merge(found, left_on="host", right_on="domain")
+summary = usage.groupby(["domain", "primary_category", "trains_on_data"])["user"].nunique()
+print(summary.sort_values(ascending=False).head(25))
+found.to_csv("ai_tools_found.csv", index=False)
+```
+
+For exports with hundreds of thousands of lines, the hosted [shadow AI inventory](https://www.shadowaitools.com) service does this with per-user breakdowns, sanctioned lists and a PDF evidence pack, without writing the join yourself.
+
+### 3. A nightly feed refresh for a DNS resolver
+
+Feed and database plans download the full CSV. This cron job fetches it, keeps only the categories the organisation blocks, and writes an Unbound local-zone file. Swap the output format for dnsmasq, a hosts file or a firewall EDL as needed.
+
+```python
+#!/usr/bin/env python3
+import csv, os, subprocess
+from aiblocklist import AIBlocklistClient, PlanError
+
+client = AIBlocklistClient(os.environ["ATB_API_KEY"])
+BLOCK = {"Image & Visual", "Audio & Voice", "Companions & Social", "Agents & Automation"}
+CSV_PATH = "/var/lib/aiblocklist/ai_tools_full.csv"
+ZONE_TMP = "/etc/unbound/unbound.conf.d/ai-block.conf.tmp"
+ZONE = "/etc/unbound/unbound.conf.d/ai-block.conf"
+
+try:
+    info = client.feeds.database_info()
+except PlanError as exc:
+    raise SystemExit("this key is lookup-only: %s" % exc.body.get("plan"))
+
+print("database %s updated %s (%s)" % (info["database_file"], info["last_updated"], info["file_size_human"]))
+client.feeds.download_database(CSV_PATH)
+
+n = 0
+with open(CSV_PATH, newline="", encoding="utf-8") as src, open(ZONE_TMP, "w") as out:
+    out.write("server:\n")
+    for row in csv.DictReader(src):
+        if row.get("primary_category") in BLOCK:
+            out.write('  local-zone: "%s." always_nxdomain\n' % row["domain"])
+            n += 1
+os.replace(ZONE_TMP, ZONE)
+subprocess.run(["unbound-control", "reload"], check=False)
+print("%d domains sinkholed" % n)
+```
+
+```
+30 5 * * * /usr/bin/python3 /opt/aiblocklist/refresh_zone.py >> /var/log/aiblocklist.log 2>&1
+```
+
+### 4. Citing the vendor's own terms
+
+`clause()` returns the verbatim sentence behind a public data-use field with the URL it came from, which is what an approval ticket or a policy exception needs.
+
+```python
+c = client.clause("chatgpt.com", "trains_consumer_default")
+if c:
+    print(c["clause"])
+    print(c["url"])
+```
+
+Fields: `trains_consumer_default`, `optout_available`, `enterprise_no_training`, `api_no_training`.
+
+## Error handling
+
+```python
+from aiblocklist import (
+    AIBlocklistError, AuthenticationError, QuotaError, PlanError,
+    BadRequestError, NotFoundError, ServiceUnavailableError,
+)
+
+try:
+    r = client.check("notion.so")
+except AuthenticationError:      # 401: no key or unknown key
+    ...
+except QuotaError as exc:        # 403: inactive account or quota exhausted
+    print(exc.status, exc.body)
+except ServiceUnavailableError:  # 503 after retries
+    ...
+except AIBlocklistError:         # anything else, including network failures
+    ...
+```
+
+| Exception | HTTP | Meaning |
+|---|---|---|
+| `BadRequestError` | 400 | empty domain, or an unknown clause field |
+| `AuthenticationError` | 401 | no key, or a key that matches no account |
+| `QuotaError` | 403 | inactive account or monthly quota exhausted |
+| `PlanError` | 403 | database endpoints on a lookup-only plan; `body["plan"]` names it |
+| `NotFoundError` | 404 | database file not available for the account |
+| `ServiceUnavailableError` | 503 | lookup service busy; retried `max_retries` times first |
+
+The client is a context manager, so the HTTP session closes cleanly:
+
+```python
+with AIBlocklistClient("YOUR_API_KEY") as client:
+    print(client.stats()["total_tools"])
+```
+
+## Configuration
+
+```python
+client = AIBlocklistClient(
+    api_key="YOUR_API_KEY",
+    base_url="https://www.aitoolsblocklist.com",   # default
+    timeout=30,                                     # seconds; downloads use at least 300
+    max_retries=2,                                  # on 503 and network errors
+)
+```
+
+Keep the key out of source: read it from an environment variable or a secrets manager and pass it to the constructor. The command line entry reads `ATB_API_KEY` for that reason. Lookups are metered per call, so cache verdicts in your own process or datastore for a day; the list is rebuilt daily and a shorter cache buys nothing. Downloads are streamed to disk in 64 KB chunks, so a full database file never sits in memory.
+
+## Why a classified AI tool list belongs in the security stack
+
+The risk in AI tool usage is not the tool, it is the input. A contract pasted into a summariser, a customer list dropped into a spreadsheet assistant, a repository fed to a code explainer: each is an outbound request to a hostname, and the hostname is the only thing every control point on the network can see without an agent or TLS inspection. Classifying that hostname is the step that turns a policy into an enforceable rule.
+
+### Data loss prevention needs the destination, not just the content
+
+[Data loss prevention](https://en.wikipedia.org/wiki/Data_loss_prevention_software) systems inspect what leaves. They cannot decide whether leaving is acceptable without knowing where it is going and what the recipient does with it. The [AI tool classification database](https://www.aitoolsblocklist.com) supplies both: the functional category of the destination and the vendor's own position on training, opt-out and enterprise exemptions. A DLP rule that says "block source code to AI assistants that train on input" is expressible only with that data.
+
+### Frameworks ask for an inventory first
+
+The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) starts its Map function with knowing which AI systems are in use and which third parties receive data. The [ENISA](https://www.enisa.europa.eu/) guidance on AI cybersecurity takes the same order: identify, then govern. Under the [EU AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj), deployers carry obligations that depend on knowing which systems staff use. None of this is possible from a survey; it is possible from network evidence joined to a classified list.
+
+### Prompt-borne exposure is a recognised risk class
+
+The [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) lists sensitive information disclosure among its leading risks. For an organisation that does not build LLM applications but whose staff use hundreds of them, the mitigation is at the network edge: know which destinations are AI tools, permit the ones whose terms are acceptable, and block or warn on the rest. Categories and subcategories make that policy fine-grained; the daily rebuild keeps it current as tools launch.
+
+## Neighbouring products
+
+This package governs what people reach. What an organisation's own AI agents may open on the web is the reverse question, answered by the [AI agent allow list](https://www.aiagentallowlist.com): verified page-type URLs for 40 million+ domains, up to 28 types per domain, and a per-URL allow or deny verdict so a browsing agent can read documentation and pricing while staying off login, checkout and upload pages. Its client is `aiagentallowlist`. Together the two give a [per-URL policy for browsing agents](https://www.aiagentallowlist.com) and a per-domain policy for staff.
+
+Before either policy is written, most teams want to [find the AI tools employees use](https://www.shadowaitools.com) today. That service takes a DNS, proxy or firewall export, matches it against this same list, and returns the tools found with categories, risk flags, training verdicts, a per-user breakdown and a dated CSV and PDF.
+
+## Frequently asked questions
+
+**What is the AI Tools Blocklist?**
+A daily-refreshed database of 20,000+ AI-tool domains classified into 18 functional categories with subcategories, with the vendor's data-use terms attached to each record. It is delivered as a lookup API, as downloadable CSV and JSON, and as hosted feeds in EDL, PAC, hosts and DNS formats from [aitoolsblocklist.com](https://www.aitoolsblocklist.com).
+
+**How do I check whether a domain is an AI tool in Python?**
+`pip install aiblocklist`, then `AIBlocklistClient("YOUR_API_KEY").check("domain.com").blocked`. The same `Lookup` object carries the category, subcategories, AI type and the five data-use fields.
+
+**How do I find out whether an AI vendor trains on my data?**
+`client.data_use("domain.com")` returns `trains_on_data`, `opt_out_available`, `enterprise_no_training`, `api_no_training` and `terms_checked`. `client.clause("domain.com", "trains_consumer_default")` returns the verbatim clause with its URL.
+
+**Can I download the whole list?**
+Feed and database plans can: `client.feeds.download_database(path)` streams the CSV, `download_categories(path)` the category tree. The Lookup API plan is lookup-only and receives a `PlanError` on those calls.
+
+**Is there a bulk endpoint?**
+No. `check_many()` runs sequential lookups, deduplicated, with an optional pause. For large lists, download the database once and match locally.
+
+**How current is the list?**
+Rebuilt daily. `client.stats()` shows the live totals; `client.feeds.database_info()` shows the timestamp of your file.
+
+**Does the list work with schools and CIPA filtering?**
+Yes. Districts use the categories to allow approved tutors while blocking essay generators, deepfake tools and companion chatbots, alongside the general filtering database at [cipawebfiltering.com](https://www.cipawebfiltering.com).
+
+**Who builds it?**
+Alpha Quantum, also behind the [website categorization API](https://www.websitecategorizationapi.com), the [web filtering database](https://www.webfilteringdatabase.com), the [AI agent allow list](https://www.aiagentallowlist.com) and the [shadow AI inventory](https://www.shadowaitools.com) service.
+
+## Related packages
+
+- [`aiblocklist` on npm](https://www.npmjs.com/package/aiblocklist), the Node.js version of this client
+- [`aitoolsblocklist`](https://pypi.org/project/aitoolsblocklist/) on PyPI and [on npm](https://www.npmjs.com/package/aitoolsblocklist), the original AI Tools Blocklist client
+- [`shadowaitools`](https://pypi.org/project/shadowaitools/) on PyPI and [on npm](https://www.npmjs.com/package/shadowaitools), local log scanner for shadow AI tools
+- [`aiagentallowlist`](https://pypi.org/project/aiagentallowlist/) on PyPI and [on npm](https://www.npmjs.com/package/aiagentallowlist), client for the AI agent allow list
+- [`phishingdetectionapi`](https://pypi.org/project/phishingdetectionapi/) on PyPI and [on npm](https://www.npmjs.com/package/phishingdetectionapi), from [phishingdetectionapi.com](https://www.phishingdetectionapi.com)
+- [`webfilteringdatabase`](https://www.npmjs.com/package/webfilteringdatabase) on npm, from [webfilteringdatabase.com](https://www.webfilteringdatabase.com)
+- [`websiteclassificationapi`](https://pypi.org/project/websiteclassificationapi/) on PyPI and [`websitecategorization`](https://www.npmjs.com/package/websitecategorization) on npm, from [websitecategorizationapi.com](https://www.websitecategorizationapi.com)
+- [`cipawebfiltering`](https://pypi.org/project/cipawebfiltering/) on PyPI and [on npm](https://www.npmjs.com/package/cipawebfiltering)
+- [PII detection API](https://www.piidetectionapi.com) for prompts and DLP logs
+
+Source: [github.com/explainableaixai/aiblocklist](https://github.com/explainableaixai/aiblocklist), mirror at [gitlab.com/url-classifications/aiblocklist](https://gitlab.com/url-classifications/aiblocklist).
+
+## Links
+
+- Product, plans and feeds: [https://www.aitoolsblocklist.com](https://www.aitoolsblocklist.com)
+- NIST AI Risk Management Framework: [https://www.nist.gov/itl/ai-risk-management-framework](https://www.nist.gov/itl/ai-risk-management-framework)
+- OWASP Top 10 for LLM Applications: [https://owasp.org/www-project-top-10-for-large-language-model-applications/](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
+- ENISA: [https://www.enisa.europa.eu/](https://www.enisa.europa.eu/)
+- EU AI Act, Regulation (EU) 2024/1689: [https://eur-lex.europa.eu/eli/reg/2024/1689/oj](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)
+- Data loss prevention software: [https://en.wikipedia.org/wiki/Data_loss_prevention_software](https://en.wikipedia.org/wiki/Data_loss_prevention_software)
+
+## License
+
+MIT