rastrolog 0.1.0__tar.gz

rastrolog-0.1.0/.gitignore

@@ -0,0 +1,39 @@
+# Private planning notes (never publish)
+AI Traffic Lead Magnet*.md
+docs/private/
+
+# Claude Code local overrides
+.claude/settings.local.json
+
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+dist/
+build/
+*.egg-info/
+
+# JS
+node_modules/
+.astro/
+.pnpm-store/
+coverage/
+playwright-report/
+test-results/
+*.tsbuildinfo
+
+# Generated
+js/core/src/*.gen.ts
+
+# OS / editors
+.DS_Store
+.idea/
+.vscode/
+
+# Build-time copy (canonical file is the repo-root signals.json)
+python/src/rastrolog/signals.json

rastrolog-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Carlos Saldaña Matar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rastrolog-0.1.0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.5
+Name: rastrolog
+Version: 0.1.0
+Summary: See which AI crawlers read your site and which AI chat products send you visitors.
+Project-URL: Homepage, https://github.com/csmatar/rastrolog
+Project-URL: Issues, https://github.com/csmatar/rastrolog/issues
+Project-URL: Changelog, https://github.com/csmatar/rastrolog/blob/main/CHANGELOG.md
+Author: Carlos Saldaña Matar
+License-Expression: MIT
+License-File: LICENSE
+Keywords: access-log,ai,analytics,claudebot,crawler,gptbot,perplexitybot,referrer,robots.txt
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Framework :: Django
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: Log Analysis
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: rich>=13.9
+Requires-Dist: typer>=0.15
+Provides-Extra: django
+Requires-Dist: django>=4.2; extra == 'django'
+Provides-Extra: fastapi
+Requires-Dist: fastapi>=0.115; extra == 'fastapi'
+Description-Content-Type: text/markdown
+
+# rastrolog
+
+**See which AI crawlers read your site and which AI chat products send you visitors.**
+
+`rastrolog` reads your access logs and prints two tables:
+
+1. **AI crawlers** by user agent (GPTBot, ClaudeBot, PerplexityBot, …): what they fetched, and whether each one is there to train, to fetch a page for a user, or to build a search index.
+2. **AI referrals**: people who clicked a link inside ChatGPT, Claude, Perplexity, Gemini or Copilot, and the pages they landed on.
+
+It classifies where a visit came from, never who the visitor is. There is no hosting, no storage and no telemetry.
+
+## Install
+
+```bash
+pip install rastrolog        # or: uv tool install rastrolog
+```
+
+## Use
+
+```bash
+rastrolog parse access.log                     # nginx, Apache combined, CloudFront, ALB; .gz is fine
+rastrolog parse access.log.gz --since 7d --json
+rastrolog parse access.log --by page --top 20  # which pages AI tools touch
+rastrolog check "https://chatgpt.com/"         # classify one referrer or user agent
+```
+
+`--json` writes the same report as JSON on stdout, for piping into anything else.
+
+## In your app
+
+FastAPI / Starlette:
+
+```python
+from rastrolog.middleware.asgi import AITrafficMiddleware
+
+app.add_middleware(
+    AITrafficMiddleware, on_match=lambda match, scope: counter.labels(match.id).inc()
+)
+# in a handler: request.state.ai_traffic -> rastrolog.Match | None
+```
+
+Django:
+
+```python
+MIDDLEWARE = [..., "rastrolog.middleware.django.AITrafficMiddleware"]
+RASTROLOG = {"on_match": "myproject.metrics.count_ai_traffic", "log": True}
+# in a view: request.ai_traffic -> rastrolog.Match | None
+```
+
+Library:
+
+```python
+from rastrolog import classify_user_agent, classify_referrer
+
+classify_user_agent("Mozilla/5.0 ... GPTBot/1.3 ...")
+# Match(kind='crawler', id='openai-gptbot', vendor='openai', token='GPTBot', purpose='training', ...)
+classify_referrer("https://www.google.com/")  # None
+```
+
+## What it detects
+
+The lists live in [`signals.json`](https://github.com/csmatar/rastrolog/blob/main/signals.json). Crawler tokens cite the vendor's own documentation, except Bytespider (ByteDance publishes none, so that entry is flagged `vendor_documented: false` and sourced from a reputable third party instead). Referrer hosts cite either vendor docs or a published AI-source list from an analytics vendor (Matomo, Plausible). robots.txt-only tokens (`Google-Extended`, `Applebot-Extended`) never appear in a user agent string; they're read by the robots.txt checker, not by log classification. Every entry has a real fixture in [`conformance/`](https://github.com/csmatar/rastrolog/blob/main/conformance/).
+
+Known gaps, stated plainly:
+
+- **Google AI Overviews and AI Mode** send visitors with a plain `google.com` referrer, which can't be told apart from ordinary search. rastrolog doesn't guess.
+- **AWS ALB logs** have no Referer field, so ALB logs show crawlers only.
+- Some AI apps strip the referrer entirely. Those visits look like direct traffic.
+
+## Get notified when the list changes
+
+New AI crawlers appear every few months, and your robots.txt goes out of date the day they do. Until the email signup page launches, watch this repo's releases (**Watch → Custom → Releases**). Every release that changes `signals.json` says what was added.

rastrolog-0.1.0/README.md

@@ -0,0 +1,72 @@
+# rastrolog
+
+**See which AI crawlers read your site and which AI chat products send you visitors.**
+
+`rastrolog` reads your access logs and prints two tables:
+
+1. **AI crawlers** by user agent (GPTBot, ClaudeBot, PerplexityBot, …): what they fetched, and whether each one is there to train, to fetch a page for a user, or to build a search index.
+2. **AI referrals**: people who clicked a link inside ChatGPT, Claude, Perplexity, Gemini or Copilot, and the pages they landed on.
+
+It classifies where a visit came from, never who the visitor is. There is no hosting, no storage and no telemetry.
+
+## Install
+
+```bash
+pip install rastrolog        # or: uv tool install rastrolog
+```
+
+## Use
+
+```bash
+rastrolog parse access.log                     # nginx, Apache combined, CloudFront, ALB; .gz is fine
+rastrolog parse access.log.gz --since 7d --json
+rastrolog parse access.log --by page --top 20  # which pages AI tools touch
+rastrolog check "https://chatgpt.com/"         # classify one referrer or user agent
+```
+
+`--json` writes the same report as JSON on stdout, for piping into anything else.
+
+## In your app
+
+FastAPI / Starlette:
+
+```python
+from rastrolog.middleware.asgi import AITrafficMiddleware
+
+app.add_middleware(
+    AITrafficMiddleware, on_match=lambda match, scope: counter.labels(match.id).inc()
+)
+# in a handler: request.state.ai_traffic -> rastrolog.Match | None
+```
+
+Django:
+
+```python
+MIDDLEWARE = [..., "rastrolog.middleware.django.AITrafficMiddleware"]
+RASTROLOG = {"on_match": "myproject.metrics.count_ai_traffic", "log": True}
+# in a view: request.ai_traffic -> rastrolog.Match | None
+```
+
+Library:
+
+```python
+from rastrolog import classify_user_agent, classify_referrer
+
+classify_user_agent("Mozilla/5.0 ... GPTBot/1.3 ...")
+# Match(kind='crawler', id='openai-gptbot', vendor='openai', token='GPTBot', purpose='training', ...)
+classify_referrer("https://www.google.com/")  # None
+```
+
+## What it detects
+
+The lists live in [`signals.json`](https://github.com/csmatar/rastrolog/blob/main/signals.json). Crawler tokens cite the vendor's own documentation, except Bytespider (ByteDance publishes none, so that entry is flagged `vendor_documented: false` and sourced from a reputable third party instead). Referrer hosts cite either vendor docs or a published AI-source list from an analytics vendor (Matomo, Plausible). robots.txt-only tokens (`Google-Extended`, `Applebot-Extended`) never appear in a user agent string; they're read by the robots.txt checker, not by log classification. Every entry has a real fixture in [`conformance/`](https://github.com/csmatar/rastrolog/blob/main/conformance/).
+
+Known gaps, stated plainly:
+
+- **Google AI Overviews and AI Mode** send visitors with a plain `google.com` referrer, which can't be told apart from ordinary search. rastrolog doesn't guess.
+- **AWS ALB logs** have no Referer field, so ALB logs show crawlers only.
+- Some AI apps strip the referrer entirely. Those visits look like direct traffic.
+
+## Get notified when the list changes
+
+New AI crawlers appear every few months, and your robots.txt goes out of date the day they do. Until the email signup page launches, watch this repo's releases (**Watch → Custom → Releases**). Every release that changes `signals.json` says what was added.

rastrolog-0.1.0/hatch_build.py

@@ -0,0 +1,26 @@
+"""Bundle the repo-root signals.json into the wheel.
+
+Building from a git checkout, the canonical file is ../signals.json.
+Building from an sdist, the sdist force-include already placed it at
+src/rastrolog/signals.json, so there is nothing to add.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from hatchling.builders.hooks.plugin.interface import BuildHookInterface
+
+
+class SignalsBuildHook(BuildHookInterface):
+    def initialize(self, version: str, build_data: dict[str, Any]) -> None:
+        if version == "editable":
+            return  # dev installs read the repo-root file directly (see rastrolog/signals.py)
+        root_signals = Path(self.root).parent / "signals.json"
+        if root_signals.is_file():
+            build_data["force_include"][str(root_signals)] = "rastrolog/signals.json"
+            return
+        if not (Path(self.root) / "src" / "rastrolog" / "signals.json").is_file():
+            msg = "signals.json not found at the repo root or in src/rastrolog/"
+            raise FileNotFoundError(msg)

rastrolog-0.1.0/pyproject.toml

@@ -0,0 +1,99 @@
+[project]
+name = "rastrolog"
+version = "0.1.0"
+description = "See which AI crawlers read your site and which AI chat products send you visitors."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Carlos Saldaña Matar" }]
+requires-python = ">=3.10"
+keywords = ["ai", "crawler", "gptbot", "claudebot", "perplexitybot", "robots.txt", "referrer", "access-log", "analytics"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Framework :: Django",
+    "Framework :: FastAPI",
+    "Intended Audience :: Developers",
+    "Intended Audience :: System Administrators",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Internet :: Log Analysis",
+    "Topic :: Internet :: WWW/HTTP",
+    "Typing :: Typed",
+]
+dependencies = ["rich>=13.9", "typer>=0.15"]
+
+[project.optional-dependencies]
+fastapi = ["fastapi>=0.115"]
+django = ["django>=4.2"]
+
+[project.urls]
+Homepage = "https://github.com/csmatar/rastrolog"
+Issues = "https://github.com/csmatar/rastrolog/issues"
+Changelog = "https://github.com/csmatar/rastrolog/blob/main/CHANGELOG.md"
+
+[project.scripts]
+rastrolog = "rastrolog.cli:app"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0",
+    "pytest-cov>=7.0",
+    "mypy>=2.0",
+    "ruff>=0.16",
+    "jsonschema>=4.26",
+    "types-jsonschema",
+    "starlette>=0.46",
+    "django>=5.2,<6; python_version < '3.12'",
+    "django>=6.0; python_version >= '3.12'",
+    "pre-commit>=4.0",
+]
+
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.sdist.force-include]
+"../signals.json" = "src/rastrolog/signals.json"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/rastrolog"]
+
+[tool.hatch.build.targets.wheel.hooks.custom]
+path = "hatch_build.py"
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP", "B", "SIM", "RUF", "PT", "C4", "PTH"]
+ignore = ["E501"]  # ruff format owns line length; long literals in fixtures are fine
+allowed-confusables = ["–", "·", "✗"]  # deliberate typography in CLI output
+
+[tool.mypy]
+strict = true
+python_version = "3.10"
+files = ["src"]
+
+[[tool.mypy.overrides]]
+module = ["django.*", "asgiref.*"]
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = ["-ra", "--strict-markers", "-m", "not perf"]
+markers = ["perf: slow performance smoke test, run with -m perf"]
+
+[tool.coverage.run]
+source = ["rastrolog"]
+branch = true
+
+[tool.coverage.report]
+exclude_also = ["if TYPE_CHECKING:", "raise NotImplementedError"]

rastrolog-0.1.0/scripts/make_fixture_logs.py

@@ -0,0 +1,135 @@
+"""Write conformance/logs/{nginx,apache,cloudfront,alb}.log from one event table.
+
+The four files describe the same 14 requests plus one malformed line, so every
+format must produce the same report (ALB has no Referer field, so no referrals).
+Run after changing EVENTS, then regenerate the golden files with write_golden.py.
+"""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Callable
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from urllib.parse import quote
+
+from rastrolog.formats import DEFAULT_CLOUDFRONT_FIELDS
+
+ROOT = Path(__file__).resolve().parents[2]
+LOGS = ROOT / "conformance" / "logs"
+UA_CASES = json.loads((ROOT / "conformance" / "user_agents.json").read_text(encoding="utf-8"))
+MONTHS = ("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
+MALFORMED = "this line is not a valid access log entry"
+MALFORMED_AFTER = 7  # insert after the 7th event
+QUOTED_UA = 'Mozilla/5.0 (X11; Linux x86_64) "Quoted" Browser/1.0'
+
+
+def ua(key: str) -> str:
+    for case in UA_CASES:
+        if case.get("label") == key or (case["expect"] and case["expect"]["id"] == key):
+            return str(case["ua"])
+    raise KeyError(key)
+
+
+GPTBOT, CLAUDEBOT = ua("openai-gptbot"), ua("anthropic-claudebot")
+CHATGPT_USER, PERPLEXITYBOT = ua("openai-chatgpt-user"), ua("perplexity-perplexitybot")
+GOOGLEBOT, BROWSER = ua("google-googlebot"), ua("browser-chrome")
+
+EVENTS: list[tuple[str, str, str, str]] = [
+    ("2026-09-20T08:00:00", "/docs", GPTBOT, ""),
+    ("2026-09-21T08:00:00", "/pricing", GPTBOT, ""),
+    ("2026-09-22T08:00:00", "/docs", GPTBOT, ""),
+    ("2026-09-22T09:00:00", "/blog/ai-traffic", CLAUDEBOT, ""),
+    ("2026-09-23T09:00:00", "/docs", CLAUDEBOT, ""),
+    ("2026-09-23T10:00:00", "/pricing", CHATGPT_USER, ""),
+    ("2026-09-24T10:00:00", "/robots.txt", PERPLEXITYBOT, ""),
+    ("2026-09-24T11:00:00", "/docs", GOOGLEBOT, ""),
+    ("2026-09-25T12:00:00", "/pricing?utm_source=chatgpt.com", BROWSER, "https://chatgpt.com/"),
+    ("2026-09-25T13:00:00", "/pricing", BROWSER, "https://chatgpt.com/"),
+    ("2026-09-26T12:00:00", "/docs", BROWSER, "https://www.perplexity.ai/search?q=ai+traffic+logs"),
+    ("2026-09-26T13:00:00", "/blog/ai-traffic", BROWSER, "https://claude.ai/"),
+    ("2026-09-27T12:00:00", "/", BROWSER, "https://www.google.com/"),
+    ("2026-09-27T13:00:00", "/", QUOTED_UA, ""),
+]
+
+
+def _ts(text: str) -> datetime:
+    return datetime.fromisoformat(text).replace(tzinfo=timezone.utc)
+
+
+def _clf(ts: datetime, offset_hours: int) -> str:
+    local = ts.astimezone(timezone(timedelta(hours=offset_hours)))
+    sign = "+" if offset_hours >= 0 else "-"
+    return (
+        f"{local.day:02d}/{MONTHS[local.month - 1]}/{local.year}:"
+        f"{local:%H:%M:%S} {sign}{abs(offset_hours):02d}00"
+    )
+
+
+def _combined(offset_hours: int, quote_escape: str) -> Callable[[datetime, str, str, str], str]:
+    def line(ts: datetime, target: str, agent: str, referrer: str) -> str:
+        return (
+            f'203.0.113.7 - - [{_clf(ts, offset_hours)}] "GET {target} HTTP/1.1" 200 5120 '
+            f'"{referrer or "-"}" "{agent.replace(chr(34), quote_escape)}"'
+        )
+
+    return line
+
+
+def _cloudfront(ts: datetime, target: str, agent: str, referrer: str) -> str:
+    path, _, query = target.partition("?")
+    values = {
+        "date": ts.strftime("%Y-%m-%d"),
+        "time": ts.strftime("%H:%M:%S"),
+        "x-edge-location": "LAX50-C1",
+        "sc-bytes": "5120",
+        "c-ip": "203.0.113.7",
+        "cs-method": "GET",
+        "cs(Host)": "d111111abcdef8.cloudfront.net",
+        "cs-uri-stem": path,
+        "sc-status": "200",
+        "cs(Referer)": quote(referrer, safe=":/?=&+") if referrer else "-",
+        "cs(User-Agent)": quote(agent, safe="/;:()+,.=@-_"),
+        "cs-uri-query": query or "-",
+    }
+    return "\t".join(values.get(name, "-") for name in DEFAULT_CLOUDFRONT_FIELDS)
+
+
+def _alb(ts: datetime, target: str, agent: str, referrer: str) -> str:
+    iso = ts.strftime("%Y-%m-%dT%H:%M:%S.000000Z")
+    return (
+        f"https {iso} app/my-lb/50dc6c495c0c9188 203.0.113.7:4321 10.0.0.1:80 "
+        f'0.000 0.001 0.000 200 200 120 5120 "GET https://www.example.com:443{target} HTTP/1.1" '
+        f'"{agent.replace(chr(34), chr(39))}" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 '
+        "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/web/73e2d6bc24d8a067 "
+        '"Root=1-67891233-abcdef012345678912345678" "www.example.com" '
+        '"arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012" '
+        f'0 {iso} "forward" "-" "-" "10.0.0.1:80" "200" "-" "-"'
+    )
+
+
+WRITERS: dict[str, tuple[list[str], Callable[[datetime, str, str, str], str]]] = {
+    "nginx": ([], _combined(0, "\\x22")),
+    "apache": ([], _combined(-5, '\\"')),
+    "cloudfront": (
+        ["#Version: 1.0", "#Fields: " + " ".join(DEFAULT_CLOUDFRONT_FIELDS)],
+        _cloudfront,
+    ),
+    "alb": ([], _alb),
+}
+
+
+def main() -> None:
+    LOGS.mkdir(parents=True, exist_ok=True)
+    for name, (header, write) in WRITERS.items():
+        lines = list(header)
+        for index, (when, target, agent, referrer) in enumerate(EVENTS, start=1):
+            lines.append(write(_ts(when), target, agent, referrer))
+            if index == MALFORMED_AFTER:
+                lines.append(MALFORMED)
+        (LOGS / f"{name}.log").write_text("\n".join(lines) + "\n", encoding="utf-8")
+        print(f"wrote conformance/logs/{name}.log ({len(lines)} lines)")
+
+
+if __name__ == "__main__":
+    main()

rastrolog-0.1.0/scripts/write_golden.py

@@ -0,0 +1,33 @@
+"""Regenerate conformance/logs/*.expected.json from the Python implementation.
+
+Run it after an intentional behaviour change and review the diff before
+committing. The golden files are also the contract for the TypeScript parser
+in js/core (Epic 2/3), so an unreviewed change here breaks parity.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+from rastrolog.parse import ParseStats, iter_records
+from rastrolog.report import Aggregator
+
+LOGS = Path(__file__).resolve().parents[2] / "conformance" / "logs"
+
+
+def main() -> int:
+    for log in sorted(LOGS.glob("*.log")):
+        stats = ParseStats()
+        aggregator = Aggregator()
+        aggregator.add_all(iter_records(log, stats=stats))
+        report = aggregator.result(top=10, skipped=stats.skipped)
+        target = log.with_suffix(".expected.json")
+        target.write_text(json.dumps(report.to_dict(), indent=2) + "\n", encoding="utf-8")
+        print(f"wrote {target.relative_to(LOGS.parents[1])}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

rastrolog-0.1.0/src/rastrolog/__init__.py

@@ -0,0 +1,15 @@
+"""Classify AI crawler and AI referral traffic."""
+
+from importlib.metadata import version as _version
+
+from rastrolog.classify import Match, classify_referrer, classify_request, classify_user_agent
+
+__version__ = _version("rastrolog")
+
+__all__ = [
+    "Match",
+    "__version__",
+    "classify_referrer",
+    "classify_request",
+    "classify_user_agent",
+]