analytics-platform 0.3.7__tar.gz → 0.3.9__tar.gz

{analytics_platform-0.3.7/analytics_platform.egg-info → analytics_platform-0.3.9}/PKG-INFO

@@ -1,9 +1,13 @@
 Metadata-Version: 2.4
 Name: analytics-platform
-Version: 0.3.7
+Version: 0.3.9
 Summary: Firewall, error monitoring and server analytics for your Python server, run from your analytics dashboard.
 Author: steviec87
 License-Expression: MIT
+Project-URL: Homepage, https://sitegoose.com
+Project-URL: Source, https://github.com/StevieC87/Sitegoose-Packages/tree/main/platform-python
+Project-URL: Issues, https://github.com/StevieC87/Sitegoose-Packages/issues
+Project-URL: Security, https://sitegoose.com/security
 Keywords: firewall,waf,rate-limit,error-monitoring,analytics,server-monitoring,django,flask,fastapi,celery,rq,dramatiq,wsgi,asgi
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
@@ -35,7 +39,42 @@ dependencies; Python 3.8 or later.
 pip install analytics-platform
 ```
 
-## Django
+Put your keys in `.env` (site settings has all three, ready to copy), or in
+your host's environment:
+
+```bash
+ANALYTICS_SITE_ID=YOUR_SITE_ID
+ANALYTICS_SITE_SECRET=YOUR_SITE_SECRET
+ANALYTICS_ENDPOINT=https://YOUR_COLLECTOR_URL
+```
+
+The package reads only the `ANALYTICS_*` lines of the `.env` in the folder the
+server starts in (Django: beside `manage.py`), and never writes them into
+`os.environ`, so your own `.env` loading is unaffected. Behind Cloudflare, add
+`ANALYTICS_TRUST_PROXY=cloudflare`; behind Nginx or a load balancer,
+`ANALYTICS_TRUST_PROXY=1`. Several services on one site? Give each
+`ANALYTICS_SERVER_NAME=api` (or `worker`, and so on).
+
+## Flask and FastAPI: one line
+
+```python
+from analytics_platform.start import platform
+```
+
+Put it near the top of the module that makes your app. On the app's first
+request, the package puts itself in front of it: Flask through
+`init_flask(app)`, and FastAPI or any Starlette app through its middleware, as
+the outermost of yours. It works whether the import is above or below
+`app = Flask(...)`. An app that already has the platform by hand gets nothing
+added a second time, and a FastAPI app mounted inside another isn't counted
+twice.
+
+Use the same import wherever you call `platform.track()` and the rest: it's
+one platform for the whole process. It's made on first use, not on import, so
+under Gunicorn or uWSGI with `--preload` its background threads start in each
+worker, not in the master.
+
+## Django: one line in settings
 
 ```python
 # settings.py
@@ -43,36 +82,31 @@ MIDDLEWARE = [
     "analytics_platform.django.PlatformMiddleware",  # first
     # ...
 ]
+```
 
-ANALYTICS_PLATFORM = {
-    "site_id": "YOUR_SITE_ID",
-    "secret": os.environ["ANALYTICS_SITE_SECRET"],  # site settings; keep it on the server
-    "endpoint": "https://YOUR_COLLECTOR_URL",
-    # "trust_proxy": 1,             # behind Nginx or a load balancer
-    # "trust_proxy": "cloudflare",  # every request through Cloudflare
-}
+That's all with the keys in `.env` or the environment. Options in code go in
+`ANALYTICS_PLATFORM`, and win over the environment:
+
+```python
+ANALYTICS_PLATFORM = {"trust_proxy": 1, "errors": {"capture_logging": True}}
 ```
 
 In your code: `from analytics_platform.django import get_platform`. The
 package tells the collector it is a Django site (`analytics-platform-django/…`),
 so attack patterns written for Django reach it.
 
-## FastAPI / Starlette
+## Setting it up in code instead
+
+For options in code with Flask, FastAPI or another app (use this or the line
+above, not both). Any key left out is read from `ANALYTICS_*`:
 
 ```python
 from analytics_platform import create_platform, PlatformMiddleware
 
-platform = create_platform(site_id="YOUR_SITE_ID", secret=os.environ["ANALYTICS_SITE_SECRET"],
-                           endpoint="https://YOUR_COLLECTOR_URL")
-app.add_middleware(PlatformMiddleware, platform=platform)
-```
-
-## Flask
+platform = create_platform(errors={"capture_logging": True})
 
-```python
-platform = create_platform(site_id="YOUR_SITE_ID", secret=os.environ["ANALYTICS_SITE_SECRET"],
-                           endpoint="https://YOUR_COLLECTOR_URL")
-platform.init_flask(app)
+app.add_middleware(PlatformMiddleware, platform=platform)   # FastAPI / Starlette
+platform.init_flask(app)                                    # Flask
 ```
 
 Other WSGI apps: `app = platform.wsgi(app)`; other ASGI apps: `app = platform.asgi(app)`.

analytics_platform-0.3.7/PKG-INFO → analytics_platform-0.3.9/README.md

@@ -1,29 +1,3 @@
-Metadata-Version: 2.4
-Name: analytics-platform
-Version: 0.3.7
-Summary: Firewall, error monitoring and server analytics for your Python server, run from your analytics dashboard.
-Author: steviec87
-License-Expression: MIT
-Keywords: firewall,waf,rate-limit,error-monitoring,analytics,server-monitoring,django,flask,fastapi,celery,rq,dramatiq,wsgi,asgi
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: System Administrators
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Framework :: Django
-Classifier: Framework :: Flask
-Classifier: Framework :: FastAPI
-Classifier: Topic :: Internet :: WWW/HTTP
-Classifier: Topic :: Internet :: Log Analysis
-Classifier: Topic :: Security
-Classifier: Topic :: System :: Monitoring
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Dynamic: license-file
-
 # analytics-platform (Python)
 
 One package for your Python server, run from your analytics dashboard: a
@@ -35,7 +9,42 @@ dependencies; Python 3.8 or later.
 pip install analytics-platform
 ```
 
-## Django
+Put your keys in `.env` (site settings has all three, ready to copy), or in
+your host's environment:
+
+```bash
+ANALYTICS_SITE_ID=YOUR_SITE_ID
+ANALYTICS_SITE_SECRET=YOUR_SITE_SECRET
+ANALYTICS_ENDPOINT=https://YOUR_COLLECTOR_URL
+```
+
+The package reads only the `ANALYTICS_*` lines of the `.env` in the folder the
+server starts in (Django: beside `manage.py`), and never writes them into
+`os.environ`, so your own `.env` loading is unaffected. Behind Cloudflare, add
+`ANALYTICS_TRUST_PROXY=cloudflare`; behind Nginx or a load balancer,
+`ANALYTICS_TRUST_PROXY=1`. Several services on one site? Give each
+`ANALYTICS_SERVER_NAME=api` (or `worker`, and so on).
+
+## Flask and FastAPI: one line
+
+```python
+from analytics_platform.start import platform
+```
+
+Put it near the top of the module that makes your app. On the app's first
+request, the package puts itself in front of it: Flask through
+`init_flask(app)`, and FastAPI or any Starlette app through its middleware, as
+the outermost of yours. It works whether the import is above or below
+`app = Flask(...)`. An app that already has the platform by hand gets nothing
+added a second time, and a FastAPI app mounted inside another isn't counted
+twice.
+
+Use the same import wherever you call `platform.track()` and the rest: it's
+one platform for the whole process. It's made on first use, not on import, so
+under Gunicorn or uWSGI with `--preload` its background threads start in each
+worker, not in the master.
+
+## Django: one line in settings
 
 ```python
 # settings.py
@@ -43,36 +52,31 @@ MIDDLEWARE = [
     "analytics_platform.django.PlatformMiddleware",  # first
     # ...
 ]
+```
 
-ANALYTICS_PLATFORM = {
-    "site_id": "YOUR_SITE_ID",
-    "secret": os.environ["ANALYTICS_SITE_SECRET"],  # site settings; keep it on the server
-    "endpoint": "https://YOUR_COLLECTOR_URL",
-    # "trust_proxy": 1,             # behind Nginx or a load balancer
-    # "trust_proxy": "cloudflare",  # every request through Cloudflare
-}
+That's all with the keys in `.env` or the environment. Options in code go in
+`ANALYTICS_PLATFORM`, and win over the environment:
+
+```python
+ANALYTICS_PLATFORM = {"trust_proxy": 1, "errors": {"capture_logging": True}}
 ```
 
 In your code: `from analytics_platform.django import get_platform`. The
 package tells the collector it is a Django site (`analytics-platform-django/…`),
 so attack patterns written for Django reach it.
 
-## FastAPI / Starlette
+## Setting it up in code instead
+
+For options in code with Flask, FastAPI or another app (use this or the line
+above, not both). Any key left out is read from `ANALYTICS_*`:
 
 ```python
 from analytics_platform import create_platform, PlatformMiddleware
 
-platform = create_platform(site_id="YOUR_SITE_ID", secret=os.environ["ANALYTICS_SITE_SECRET"],
-                           endpoint="https://YOUR_COLLECTOR_URL")
-app.add_middleware(PlatformMiddleware, platform=platform)
-```
+platform = create_platform(errors={"capture_logging": True})
 
-## Flask
-
-```python
-platform = create_platform(site_id="YOUR_SITE_ID", secret=os.environ["ANALYTICS_SITE_SECRET"],
-                           endpoint="https://YOUR_COLLECTOR_URL")
-platform.init_flask(app)
+app.add_middleware(PlatformMiddleware, platform=platform)   # FastAPI / Starlette
+platform.init_flask(app)                                    # Flask
 ```
 
 Other WSGI apps: `app = platform.wsgi(app)`; other ASGI apps: `app = platform.asgi(app)`.

{analytics_platform-0.3.7 → analytics_platform-0.3.9}/analytics_platform/__init__.py

@@ -20,6 +20,7 @@ import http
 import time
 import weakref
 
+from . import _env
 from .core import Core, EXIT_DEADLINE, SESSION_HEADER, VERSION, begin_exit, header, request_target, within
 from .errors import add_breadcrumb, current_request, current_trail
 from .queries import current_queries
@@ -77,7 +78,8 @@ MODULES = ("proxy", "firewall", "errors", "analytics", "performance", "crawlers"
 
 class Platform:
     """site_id, secret (site settings; keep it on the server), endpoint (the
-    collector's URL); trust_proxy (how to find the visitor's address: None
+    collector's URL), each read from ANALYTICS_SITE_ID, ANALYTICS_SITE_SECRET and
+    ANALYTICS_ENDPOINT when left out; trust_proxy (ANALYTICS_TRUST_PROXY) (how to find the visitor's address: None
     uses the connection's, a number of proxies in X-Forwarded-For,
     "cloudflare"); debug (print send problems); per module True / False / {options}.
     client: the name this package goes by to the collector ("python", or
@@ -86,14 +88,20 @@ class Platform:
     keep Server speed and leave database queries, outgoing calls or job
     queues out of it."""
 
-    def __init__(self, site_id, secret, endpoint, trust_proxy=None, debug=False, firewall=True, errors=True, analytics=True,
+    def __init__(self, site_id=None, secret=None, endpoint=None, trust_proxy=None, debug=False, firewall=True, errors=True, analytics=True,
                  performance=True, crawlers=True, forms=True, headers=True, audit=True, flags=True, maintenance=True,
                  inventory=True,
                  posture=True, resources=True, proxy=True, server_name=None, client="python", queries=True,
                  calls=True, queues=True):
+        # Each of these left out is read from ANALYTICS_* in the environment (_env.py).
+        given = _env.from_environment({"site_id": site_id, "secret": secret, "endpoint": endpoint,
+                                       "trust_proxy": trust_proxy, "server_name": server_name})
+        site_id, secret, endpoint = given["site_id"], given["secret"], given["endpoint"]
+        trust_proxy, server_name = given["trust_proxy"], given["server_name"]
         self.core = Core(site_id, secret, endpoint, debug, server_name=server_name, client=client)
         if not self.core.valid:
-            print("[analytics-platform] site_id, secret and endpoint are required; nothing is sent and every request passes")
+            print("[analytics-platform] site_id, secret and endpoint are required (or ANALYTICS_SITE_ID, ANALYTICS_SITE_SECRET "
+                  "and ANALYTICS_ENDPOINT in the environment); nothing is sent and every request passes")
         settings = {"firewall": firewall, "errors": errors, "analytics": analytics, "performance": performance,
                     "crawlers": crawlers, "forms": forms,
                     "headers": headers, "audit": audit, "flags": flags, "maintenance": maintenance, "inventory": inventory,
@@ -577,6 +585,12 @@ class Platform:
 
     def init_flask(self, app):
         """Flask: the firewall in front of the app, and its request errors."""
+        # Once per app, by hand or by analytics_platform.start.
+        extensions = getattr(app, "extensions", None)
+        if isinstance(extensions, dict):
+            if "analytics_platform" in extensions:
+                return app
+            extensions["analytics_platform"] = self
         app.wsgi_app = self.wsgi(app.wsgi_app, timing=False)
         from flask import g, got_request_exception, request
 

analytics_platform-0.3.9/analytics_platform/_env.py

@@ -0,0 +1,71 @@
+"""The options read from ANALYTICS_* when they aren't passed, as the Node and
+Go packages do: ANALYTICS_SITE_ID, ANALYTICS_SITE_SECRET, ANALYTICS_ENDPOINT,
+and optionally ANALYTICS_TRUST_PROXY ("cloudflare" or a number of proxies)
+and ANALYTICS_SERVER_NAME. And the ANALYTICS_* lines of a .env file, for
+analytics_platform.start and Django: read, never written to os.environ, so
+whatever loads the app's own .env files still sets them as it would."""
+import os
+import re
+
+_LINE = re.compile(r"^\s*(?:export\s+)?(ANALYTICS_[A-Z0-9_]+)\s*=\s*(.*)$")
+
+
+def options_from(env):
+    """{site_id, secret, endpoint, trust_proxy, server_name} found in env."""
+    def read(name):
+        value = env.get(name)
+        return value.strip() if isinstance(value, str) and value.strip() else None
+
+    found = {}
+    for option, name in (("site_id", "ANALYTICS_SITE_ID"), ("secret", "ANALYTICS_SITE_SECRET"),
+                         ("endpoint", "ANALYTICS_ENDPOINT"), ("server_name", "ANALYTICS_SERVER_NAME")):
+        if read(name):
+            found[option] = read(name)
+    if "endpoint" not in found and read("ANALYTICS_FIREWALL_ENDPOINT"):
+        found["endpoint"] = read("ANALYTICS_FIREWALL_ENDPOINT")
+    trust = read("ANALYTICS_TRUST_PROXY")
+    if trust:
+        if trust.lower() == "cloudflare":
+            found["trust_proxy"] = "cloudflare"
+        elif trust.lower() == "true":
+            found["trust_proxy"] = 1
+        elif trust.isdigit() and int(trust) > 0:
+            found["trust_proxy"] = int(trust)
+    return found
+
+
+def read_dotenv(path):
+    """The ANALYTICS_* lines of a .env file as {name: value}; {} without one.
+    Not a dotenv: no expansion, and every other line is ignored."""
+    try:
+        with open(path, encoding="utf-8") as handle:
+            text = handle.read()
+    except (OSError, UnicodeDecodeError):
+        return {}
+    found = {}
+    for line in text.splitlines():
+        match = _LINE.match(line)
+        if not match:
+            continue
+        value = match.group(2).strip()
+        quote = value[:1]
+        if quote in ("'", '"') and value.find(quote, 1) > 0:
+            value = value[1:value.find(quote, 1)]
+        else:
+            value = re.sub(r"\s+#.*$", "", value)
+        found[match.group(1)] = value
+    return found
+
+
+def from_environment(options, dotenv=None):
+    """options with what's missing filled from os.environ, then (when given)
+    from a .env file's ANALYTICS_* lines."""
+    filled = dict(options)
+    sources = [options_from(os.environ)]
+    if dotenv:
+        sources.append(options_from(read_dotenv(dotenv)))
+    for source in sources:
+        for key, value in source.items():
+            if filled.get(key) is None:
+                filled[key] = value
+    return filled

{analytics_platform-0.3.7 → analytics_platform-0.3.9}/analytics_platform/core.py

@@ -23,7 +23,7 @@ import urllib.error
 import urllib.parse
 import urllib.request
 
-VERSION = "0.3.7"
+VERSION = "0.3.9"
 TIMEOUT = 5
 MIN_SYNC = 30
 RETRY = 60
@@ -285,19 +285,24 @@ CLIENTS = ("python", "django")
 
 
 
-# A dashboard header is added only when it is well formed: a token name that
-# isn't one of the headers that frame the response (a Content-Length or
-# Transfer-Encoding of ours would desync it, a Set-Cookie or Content-Type
-# would change what it is), and a value of visible ASCII, spaces and tabs,
-# so no line break, control character or byte a server rejects gets through.
-FRAMING = frozenset(("content-length", "transfer-encoding", "connection", "set-cookie", "content-type"))
-_HEADER_NAME = re.compile(r"[A-Za-z][A-Za-z0-9-]{0,63}")
+# A dashboard header is added only when it is one of the nine the dashboard
+# can produce (expresscollect/Analytics/platform.js, headersOf), in any case,
+# with a value of visible ASCII, spaces and tabs, so no line break, control
+# character or byte a server rejects gets through. A list of what's allowed,
+# not of what isn't: whoever got into our servers still couldn't make this
+# app grant another site its users' data (Access-Control-Allow-*), send every
+# page elsewhere (Refresh), log everyone out (Clear-Site-Data) or change what
+# a response is. Shared cases: sdk/protocols/header-cases.json.
+SECURITY_HEADERS = frozenset((
+    "strict-transport-security", "x-content-type-options", "x-frame-options", "referrer-policy",
+    "permissions-policy", "cross-origin-opener-policy", "content-security-policy",
+    "content-security-policy-report-only", "reporting-endpoints"))
 _HEADER_VALUE = re.compile(r"[\x20-\x7e\t]*")
 
 
 def safe_header(pair):
     return (isinstance(pair, (list, tuple)) and len(pair) == 2 and isinstance(pair[0], str)
-            and _HEADER_NAME.fullmatch(pair[0]) is not None and pair[0].lower() not in FRAMING
+            and pair[0].lower() in SECURITY_HEADERS
             and isinstance(pair[1], str) and len(pair[1]) <= 4000 and _HEADER_VALUE.fullmatch(pair[1]) is not None)
 
 class Core:

{analytics_platform-0.3.7 → analytics_platform-0.3.9}/analytics_platform/django.py

@@ -1,11 +1,12 @@
-"""Django: add the middleware first in MIDDLEWARE and the options to settings.
+"""Django: add the middleware first in MIDDLEWARE. That's all when the keys
+are in the environment or in the project's .env (beside manage.py):
+ANALYTICS_SITE_ID, ANALYTICS_SITE_SECRET and ANALYTICS_ENDPOINT.
 
     MIDDLEWARE = ["analytics_platform.django.PlatformMiddleware", ...]
-    ANALYTICS_PLATFORM = {
-        "site_id": "...",
-        "secret": os.environ["ANALYTICS_SITE_SECRET"],
-        "endpoint": "https://...",
-    }
+
+Options in code go in settings, and win over the environment:
+
+    ANALYTICS_PLATFORM = {"trust_proxy": 1, "errors": {"capture_logging": True}}
 
 In your code, get_platform() is the same platform: get_platform().track(...).
 The visitor's address is REMOTE_ADDR unless you set trust_proxy. The package
@@ -13,11 +14,13 @@ says it is Django (analytics-platform-django/<version>), so request rules
 aimed at Django sites reach it. Every connection's queries are timed for
 Server speed, under the request that ran them ("queries": False leaves them out).
 """
+import os
 import time
 
 from django.conf import settings
 from django.http import HttpResponse
 
+from . import _env
 from . import SESSION_HEADER, Platform, _inflight_end, _inflight_start, _queries_end, _queries_start
 from .errors import current_request, current_trail
 
@@ -27,8 +30,13 @@ _platform = None
 def get_platform():
     global _platform
     if _platform is None:
+        # Settings first, then ANALYTICS_* in the environment, then the
+        # ANALYTICS_* lines of the project's .env (beside manage.py), which is
+        # read, never loaded into the environment.
         options = dict(getattr(settings, "ANALYTICS_PLATFORM", {}))
         options.setdefault("client", "django")
+        base = getattr(settings, "BASE_DIR", None) or os.getcwd()
+        options = _env.from_environment(options, os.path.join(str(base), ".env"))
         _platform = Platform(**options)
         _connect_login_failed()
         _watch_queries(_platform)

{analytics_platform-0.3.7 → analytics_platform-0.3.9}/analytics_platform/proxy.py

@@ -49,6 +49,21 @@ PASS_HEADERS = ("content-type", "user-agent", "accept-language", "referer", "ori
                 "sec-fetch-site", "sec-ch-ua", "sec-ch-ua-mobile", "sec-ch-ua-platform")
 SAFE_PATH = re.compile(r"[A-Za-z0-9/_.-]{1,120}\Z")
 FILE_PATH = re.compile(r"v(\d+(?:\.\d+\.\d+)?)/([a-z]+\.js)\Z")
+# What a collector answer may be served as from the site's own domain. Its
+# type comes from the collector, so anything but these three becomes plain
+# text, and nosniff stops a browser guessing: a server that is no longer ours
+# still can't answer <prefix>/api/... with a page that runs as the customer's
+# own site. Shared cases: sdk/protocols/proxy-answer-cases.json.
+ANSWER_TYPES = frozenset(("application/json", "text/plain", "image/webp"))
+_VISIBLE = re.compile(r"[\x20-\x7e]*")
+
+
+def answer_type(value):
+    value = value if isinstance(value, str) else ""
+    kind = value.split(";", 1)[0].strip().lower()
+    return value if kind in ANSWER_TYPES and _VISIBLE.fullmatch(value) else "text/plain; charset=utf-8"
+
+
 NOT_FOUND = (404, [("Content-Type", "text/plain; charset=utf-8")], b"Not found")
 
 
@@ -236,8 +251,8 @@ class Proxy:
 
     @staticmethod
     def _answer_headers(headers):
-        out = []
-        for name in ("Content-Type", "Cache-Control", "Retry-After"):
+        out = [("Content-Type", answer_type(headers.get("Content-Type"))), ("X-Content-Type-Options", "nosniff")]
+        for name in ("Cache-Control", "Retry-After"):
             value = headers.get(name)
             if value:
                 out.append((name, value))

analytics_platform-0.3.9/analytics_platform/start.py

@@ -0,0 +1,118 @@
+"""The one-line setup. Near the top of the module that makes your app:
+
+    from analytics_platform.start import platform
+
+It reads ANALYTICS_SITE_ID, ANALYTICS_SITE_SECRET and ANALYTICS_ENDPOINT from
+the environment, or else from the ANALYTICS_* lines of the .env in the folder
+the server starts in (read, never written to os.environ).
+
+Flask and FastAPI (any Starlette app): that's all. On an app's first request
+it puts itself in front, as platform.init_flask(app) and
+app.add_middleware(PlatformMiddleware, platform=platform) do by hand, so it
+doesn't matter whether this import comes before or after the app is made. An
+app that already has either by hand gets nothing added a second time, and a
+Starlette app mounted inside another isn't counted twice.
+
+Django: add "analytics_platform.django.PlatformMiddleware" first in
+MIDDLEWARE instead, and use get_platform() from there.
+
+The platform is made on first use (the first request, or the first
+platform.track()), not on import: under Gunicorn or uWSGI with the app loaded
+before the workers fork, its background threads then start in each worker,
+not in the master. Use the same import wherever you call platform.track() and
+the rest: it's one platform for the process.
+"""
+import os
+import threading
+
+from . import Platform, PlatformMiddleware, _env
+
+_lock = threading.Lock()
+_made = None
+
+
+def _platform():
+    global _made
+    if _made is None:
+        with _lock:
+            if _made is None:
+                _made = Platform(**_env.from_environment({}, os.path.join(os.getcwd(), ".env")))
+    return _made
+
+
+class _Platform:
+    """Stands for the platform until its first use, then hands every
+    attribute to it."""
+
+    def __getattr__(self, name):
+        return getattr(_platform(), name)
+
+    def __repr__(self):
+        return "<analytics_platform.start platform%s>" % ("" if _made is not None else " (not made yet)")
+
+
+platform = _Platform()
+
+_MARK = "_analytics_platform_started"
+
+
+def _hook_flask():
+    try:
+        from flask import Flask
+    except Exception:
+        return
+    if getattr(Flask.__call__, _MARK, False):
+        return
+    original = Flask.__call__
+    hook_lock = threading.Lock()
+
+    # The first request's own thread hooks the app in before Flask marks it
+    # as started (after that it refuses new before_request functions); other
+    # threads wait for it, so none gets past without it.
+    def __call__(self, environ, start_response):
+        if not self.__dict__.get(_MARK):
+            with hook_lock:
+                if not self.__dict__.get(_MARK):
+                    try:
+                        _platform().init_flask(self)
+                    except Exception as error:
+                        print("[analytics-platform] couldn't add itself to this Flask app; call platform.init_flask(app) by hand: %s" % error)
+                    self.__dict__[_MARK] = True
+        return original(self, environ, start_response)
+
+    setattr(__call__, _MARK, True)
+    Flask.__call__ = __call__
+
+
+def _hook_starlette():
+    try:
+        from starlette.applications import Starlette
+        from starlette.middleware import Middleware
+    except Exception:
+        return
+    if getattr(Starlette.__call__, _MARK, False):
+        return
+    original = Starlette.__call__
+
+    # Before the app builds its middleware stack, on its first call. Only the
+    # outermost app: a mounted one is called with scope["app"] already set by
+    # the app it's mounted in. Outermost of your own middleware, as
+    # add_middleware() called last puts it.
+    async def __call__(self, scope, receive, send):
+        if not self.__dict__.get(_MARK):
+            self.__dict__[_MARK] = True
+            outermost = "app" not in scope
+            by_hand = any(getattr(m, "cls", None) is PlatformMiddleware for m in getattr(self, "user_middleware", []))
+            if outermost and not by_hand and getattr(self, "middleware_stack", None) is None:
+                self.user_middleware.insert(0, Middleware(PlatformMiddleware, platform=platform))
+        await original(self, scope, receive, send)
+
+    setattr(__call__, _MARK, True)
+    Starlette.__call__ = __call__
+
+
+# Each only when its framework is installed.
+_hook_flask()
+_hook_starlette()
+
+__all__ = ["platform"]

analytics_platform-0.3.7/README.md → analytics_platform-0.3.9/analytics_platform.egg-info/PKG-INFO

@@ -1,3 +1,33 @@
+Metadata-Version: 2.4
+Name: analytics-platform
+Version: 0.3.9
+Summary: Firewall, error monitoring and server analytics for your Python server, run from your analytics dashboard.
+Author: steviec87
+License-Expression: MIT
+Project-URL: Homepage, https://sitegoose.com
+Project-URL: Source, https://github.com/StevieC87/Sitegoose-Packages/tree/main/platform-python
+Project-URL: Issues, https://github.com/StevieC87/Sitegoose-Packages/issues
+Project-URL: Security, https://sitegoose.com/security
+Keywords: firewall,waf,rate-limit,error-monitoring,analytics,server-monitoring,django,flask,fastapi,celery,rq,dramatiq,wsgi,asgi
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Framework :: Django
+Classifier: Framework :: Flask
+Classifier: Framework :: FastAPI
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Internet :: Log Analysis
+Classifier: Topic :: Security
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
 # analytics-platform (Python)
 
 One package for your Python server, run from your analytics dashboard: a
@@ -9,7 +39,42 @@ dependencies; Python 3.8 or later.
 pip install analytics-platform
 ```
 
-## Django
+Put your keys in `.env` (site settings has all three, ready to copy), or in
+your host's environment:
+
+```bash
+ANALYTICS_SITE_ID=YOUR_SITE_ID
+ANALYTICS_SITE_SECRET=YOUR_SITE_SECRET
+ANALYTICS_ENDPOINT=https://YOUR_COLLECTOR_URL
+```
+
+The package reads only the `ANALYTICS_*` lines of the `.env` in the folder the
+server starts in (Django: beside `manage.py`), and never writes them into
+`os.environ`, so your own `.env` loading is unaffected. Behind Cloudflare, add
+`ANALYTICS_TRUST_PROXY=cloudflare`; behind Nginx or a load balancer,
+`ANALYTICS_TRUST_PROXY=1`. Several services on one site? Give each
+`ANALYTICS_SERVER_NAME=api` (or `worker`, and so on).
+
+## Flask and FastAPI: one line
+
+```python
+from analytics_platform.start import platform
+```
+
+Put it near the top of the module that makes your app. On the app's first
+request, the package puts itself in front of it: Flask through
+`init_flask(app)`, and FastAPI or any Starlette app through its middleware, as
+the outermost of yours. It works whether the import is above or below
+`app = Flask(...)`. An app that already has the platform by hand gets nothing
+added a second time, and a FastAPI app mounted inside another isn't counted
+twice.
+
+Use the same import wherever you call `platform.track()` and the rest: it's
+one platform for the whole process. It's made on first use, not on import, so
+under Gunicorn or uWSGI with `--preload` its background threads start in each
+worker, not in the master.
+
+## Django: one line in settings
 
 ```python
 # settings.py
@@ -17,36 +82,31 @@ MIDDLEWARE = [
     "analytics_platform.django.PlatformMiddleware",  # first
     # ...
 ]
+```
 
-ANALYTICS_PLATFORM = {
-    "site_id": "YOUR_SITE_ID",
-    "secret": os.environ["ANALYTICS_SITE_SECRET"],  # site settings; keep it on the server
-    "endpoint": "https://YOUR_COLLECTOR_URL",
-    # "trust_proxy": 1,             # behind Nginx or a load balancer
-    # "trust_proxy": "cloudflare",  # every request through Cloudflare
-}
+That's all with the keys in `.env` or the environment. Options in code go in
+`ANALYTICS_PLATFORM`, and win over the environment:
+
+```python
+ANALYTICS_PLATFORM = {"trust_proxy": 1, "errors": {"capture_logging": True}}
 ```
 
 In your code: `from analytics_platform.django import get_platform`. The
 package tells the collector it is a Django site (`analytics-platform-django/…`),
 so attack patterns written for Django reach it.
 
-## FastAPI / Starlette
+## Setting it up in code instead
+
+For options in code with Flask, FastAPI or another app (use this or the line
+above, not both). Any key left out is read from `ANALYTICS_*`:
 
 ```python
 from analytics_platform import create_platform, PlatformMiddleware
 
-platform = create_platform(site_id="YOUR_SITE_ID", secret=os.environ["ANALYTICS_SITE_SECRET"],
-                           endpoint="https://YOUR_COLLECTOR_URL")
-app.add_middleware(PlatformMiddleware, platform=platform)
-```
+platform = create_platform(errors={"capture_logging": True})
 
-## Flask
-
-```python
-platform = create_platform(site_id="YOUR_SITE_ID", secret=os.environ["ANALYTICS_SITE_SECRET"],
-                           endpoint="https://YOUR_COLLECTOR_URL")
-platform.init_flask(app)
+app.add_middleware(PlatformMiddleware, platform=platform)   # FastAPI / Starlette
+platform.init_flask(app)                                    # Flask
 ```
 
 Other WSGI apps: `app = platform.wsgi(app)`; other ASGI apps: `app = platform.asgi(app)`.

{analytics_platform-0.3.7 → analytics_platform-0.3.9}/analytics_platform.egg-info/SOURCES.txt

@@ -2,6 +2,7 @@ LICENSE
 README.md
 pyproject.toml
 analytics_platform/__init__.py
+analytics_platform/_env.py
 analytics_platform/analytics.py
 analytics_platform/audit.py
 analytics_platform/calls.py
@@ -25,6 +26,7 @@ analytics_platform/queues.py
 analytics_platform/request_rules.py
 analytics_platform/resources.py
 analytics_platform/rq.py
+analytics_platform/start.py
 analytics_platform.egg-info/PKG-INFO
 analytics_platform.egg-info/SOURCES.txt
 analytics_platform.egg-info/dependency_links.txt
@@ -37,12 +39,15 @@ tests/test_crawlers.py
 tests/test_errors.py
 tests/test_firewall.py
 tests/test_hardening.py
+tests/test_headers.py
 tests/test_limits.py
 tests/test_page_limits.py
 tests/test_proxy.py
+tests/test_proxy_answer.py
 tests/test_queries.py
 tests/test_redis.py
 tests/test_request_rules.py
 tests/test_resources.py
 tests/test_rq_dramatiq.py
+tests/test_start.py
 tests/test_static_late.py

{analytics_platform-0.3.7 → analytics_platform-0.3.9}/pyproject.toml

@@ -2,9 +2,15 @@
 requires = ["setuptools>=77"]
 build-backend = "setuptools.build_meta"
 
+[project.urls]
+Homepage = "https://sitegoose.com"
+Source = "https://github.com/StevieC87/Sitegoose-Packages/tree/main/platform-python"
+Issues = "https://github.com/StevieC87/Sitegoose-Packages/issues"
+Security = "https://sitegoose.com/security"
+
 [project]
 name = "analytics-platform"
-version = "0.3.7"
+version = "0.3.9"
 description = "Firewall, error monitoring and server analytics for your Python server, run from your analytics dashboard."
 readme = "README.md"
 requires-python = ">=3.8"

analytics_platform-0.3.9/tests/test_headers.py

@@ -0,0 +1,28 @@
+"""The security headers taken from the platform config, against the cases every
+client shares: sdk/protocols/header-cases.json. Only the nine the dashboard can
+produce get through, so all the packages add the same ones."""
+import json
+import os
+import unittest
+
+from analytics_platform.core import safe_header
+
+CASES = os.path.join(os.path.dirname(__file__), "..", "..", "protocols", "header-cases.json")
+
+
+class HeaderCases(unittest.TestCase):
+    def test_shared_cases(self):
+        with open(CASES, encoding="utf-8") as f:
+            cases = json.load(f)["cases"]
+        for case in cases:
+            value = case["value"] * case.get("repeat", 1)
+            with self.subTest(name=case["name"], value=case["value"][:40], repeat=case.get("repeat")):
+                self.assertEqual(safe_header([case["name"], value]), case["ok"])
+
+    def test_only_pairs_of_strings(self):
+        for pair in (None, "X-Frame-Options", ["X-Frame-Options"], ["X-Frame-Options", 1], ["X-Frame-Options", "DENY", "x"]):
+            self.assertFalse(safe_header(pair))
+
+
+if __name__ == "__main__":
+    unittest.main()

analytics_platform-0.3.9/tests/test_proxy_answer.py

@@ -0,0 +1,27 @@
+"""The type a collector answer is served with from the site's own domain,
+against the cases every client shares: sdk/protocols/proxy-answer-cases.json."""
+import json
+import os
+import unittest
+
+from analytics_platform.proxy import Proxy, answer_type
+
+CASES = os.path.join(os.path.dirname(__file__), "..", "..", "protocols", "proxy-answer-cases.json")
+
+
+class ProxyAnswerCases(unittest.TestCase):
+    def test_shared_cases(self):
+        with open(CASES, encoding="utf-8") as f:
+            cases = json.load(f)["cases"]
+        for case in cases:
+            with self.subTest(type=case["type"]):
+                self.assertEqual(answer_type(case["type"]), case["served"])
+
+    def test_nosniff_and_only_cache_and_retry(self):
+        upstream = {"Content-Type": "text/html", "Cache-Control": "no-store", "Set-Cookie": "a=b", "Refresh": "0;url=x"}
+        self.assertEqual(Proxy._answer_headers(upstream), [
+            ("Content-Type", "text/plain; charset=utf-8"), ("X-Content-Type-Options", "nosniff"), ("Cache-Control", "no-store")])
+
+
+if __name__ == "__main__":
+    unittest.main()

analytics_platform-0.3.9/tests/test_start.py

@@ -0,0 +1,156 @@
+"""The one-line setup: keys from ANALYTICS_* or a .env file (never written to
+os.environ), and analytics_platform.start putting itself in front of Flask
+and FastAPI/Starlette apps on their first request, and Django's settings
+block becoming optional. The framework tests skip where the framework isn't
+installed; run them where Flask, FastAPI (with httpx) and Django are."""
+import os
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from analytics_platform import Platform, PlatformMiddleware, _env  # noqa: E402
+
+SITE = "a" * 24
+KEYS = {"ANALYTICS_SITE_ID": SITE, "ANALYTICS_SITE_SECRET": "s3cret", "ANALYTICS_ENDPOINT": "http://127.0.0.1:9"}
+OFF = {name: False for name in ("firewall", "errors", "analytics", "performance", "crawlers", "forms", "headers", "audit",
+                                "flags", "maintenance", "inventory", "posture", "resources", "proxy")}
+
+
+def installed(name):
+    try:
+        __import__(name)
+        return True
+    except Exception:
+        return False
+
+
+class EnvTest(unittest.TestCase):
+    def test_options_from_the_environment(self):
+        found = _env.options_from({**KEYS, "ANALYTICS_TRUST_PROXY": "2", "ANALYTICS_SERVER_NAME": "api"})
+        self.assertEqual(found, {"site_id": SITE, "secret": "s3cret", "endpoint": "http://127.0.0.1:9",
+                                 "server_name": "api", "trust_proxy": 2})
+        self.assertEqual(_env.options_from({"ANALYTICS_TRUST_PROXY": "Cloudflare"})["trust_proxy"], "cloudflare")
+        self.assertEqual(_env.options_from({"ANALYTICS_FIREWALL_ENDPOINT": "https://x"})["endpoint"], "https://x")
+        self.assertEqual(_env.options_from({"ANALYTICS_TRUST_PROXY": "nonsense", "ANALYTICS_SITE_ID": "  "}), {})
+
+    def test_dotenv_lines_are_read_and_never_loaded(self):
+        with tempfile.TemporaryDirectory() as folder:
+            path = os.path.join(folder, ".env")
+            with open(path, "w") as handle:
+                handle.write('DATABASE_URL=x\nANALYTICS_SITE_ID="%s"\nexport ANALYTICS_SITE_SECRET=from-file # note\n' % SITE)
+            with mock.patch.dict(os.environ, {"ANALYTICS_ENDPOINT": "http://127.0.0.1:9"}, clear=True):
+                filled = _env.from_environment({"secret": None}, path)
+                self.assertNotIn("ANALYTICS_SITE_SECRET", os.environ)
+                self.assertNotIn("DATABASE_URL", os.environ)
+        self.assertEqual((filled["site_id"], filled["secret"], filled["endpoint"]), (SITE, "from-file", "http://127.0.0.1:9"))
+
+    def test_the_environment_wins_over_the_file_and_given_options_over_both(self):
+        with tempfile.TemporaryDirectory() as folder:
+            path = os.path.join(folder, ".env")
+            with open(path, "w") as handle:
+                handle.write("ANALYTICS_SITE_SECRET=file\nANALYTICS_SERVER_NAME=file\n")
+            with mock.patch.dict(os.environ, {"ANALYTICS_SITE_SECRET": "env"}, clear=True):
+                filled = _env.from_environment({"server_name": "given"}, path)
+        self.assertEqual((filled["secret"], filled["server_name"]), ("env", "given"))
+
+    def test_platform_with_no_keys_reads_them_from_the_environment(self):
+        with mock.patch.dict(os.environ, KEYS, clear=True):
+            platform = Platform(**OFF)
+        self.assertTrue(platform.core.valid)
+        with mock.patch.dict(os.environ, {}, clear=True), mock.patch("builtins.print") as printed:
+            self.assertFalse(Platform(**OFF).core.valid)
+        self.assertIn("ANALYTICS_SITE_SECRET", printed.call_args[0][0])
+
+
+def started():
+    """analytics_platform.start, with a fresh platform made from KEYS."""
+    from analytics_platform import start
+    start._made = None
+    return start
+
+
+@unittest.skipUnless(installed("flask"), "Flask isn't installed")
+class FlaskTest(unittest.TestCase):
+    def test_hooks_itself_in_on_the_first_request_wherever_the_import_is(self):
+        from flask import Flask
+        app = Flask("before")  # made before the import
+        app.get("/")(lambda: "ok")
+        with mock.patch.dict(os.environ, KEYS):
+            start = started()
+            response = app.test_client().get("/")
+        self.assertEqual(response.status_code, 200)
+        self.assertIs(app.extensions["analytics_platform"], start._made)
+        wrapped = app.wsgi_app
+        app.test_client().get("/")
+        self.assertIs(app.wsgi_app, wrapped, "wrapped once, not on every request")
+
+    def test_an_app_set_up_by_hand_is_left_alone(self):
+        from flask import Flask
+        with mock.patch.dict(os.environ, KEYS):
+            started()
+            own = Platform(**OFF)
+            app = Flask("by_hand")
+            app.get("/")(lambda: "ok")
+            own.init_flask(app)
+            wrapped = app.wsgi_app
+            self.assertEqual(app.test_client().get("/").status_code, 200)
+        self.assertIs(app.extensions["analytics_platform"], own)
+        self.assertIs(app.wsgi_app, wrapped)
+
+
+@unittest.skipUnless(installed("fastapi") and installed("httpx"), "FastAPI or httpx isn't installed")
+class FastAPITest(unittest.TestCase):
+    def test_outermost_of_your_middleware_and_not_in_a_mounted_app(self):
+        from fastapi import FastAPI
+        from fastapi.testclient import TestClient
+        from starlette.middleware.gzip import GZipMiddleware
+        with mock.patch.dict(os.environ, KEYS):
+            started()
+            app = FastAPI()
+            app.add_middleware(GZipMiddleware)
+            sub = FastAPI()
+            sub.get("/x")(lambda: {"ok": True})
+            app.mount("/sub", sub)
+            app.get("/")(lambda: {"ok": True})
+            client = TestClient(app)
+            self.assertEqual(client.get("/").status_code, 200)
+            self.assertEqual(client.get("/sub/x").status_code, 200)
+        self.assertEqual([m.cls for m in app.user_middleware], [PlatformMiddleware, GZipMiddleware])
+        self.assertEqual([m.cls for m in sub.user_middleware], [], "a mounted app isn't counted twice")
+
+    def test_an_app_with_it_by_hand_gets_no_second_one(self):
+        from fastapi import FastAPI
+        from fastapi.testclient import TestClient
+        with mock.patch.dict(os.environ, KEYS):
+            started()
+            app = FastAPI()
+            app.add_middleware(PlatformMiddleware, platform=Platform(**OFF))
+            app.get("/")(lambda: {"ok": True})
+            self.assertEqual(TestClient(app).get("/").status_code, 200)
+        self.assertEqual([m.cls for m in app.user_middleware], [PlatformMiddleware])
+
+
+@unittest.skipUnless(installed("django"), "Django isn't installed")
+class DjangoTest(unittest.TestCase):
+    def test_no_settings_block_the_keys_from_the_projects_dotenv(self):
+        from django.conf import settings
+        with tempfile.TemporaryDirectory() as folder:
+            with open(os.path.join(folder, ".env"), "w") as handle:
+                handle.write("\n".join("%s=%s" % item for item in KEYS.items()))
+            if not settings.configured:
+                settings.configure(BASE_DIR=folder)
+            else:
+                settings.BASE_DIR = folder
+            from analytics_platform import django as integration
+            integration._platform = None
+            with mock.patch.dict(os.environ, {}, clear=True):
+                platform = integration.get_platform()
+                self.assertNotIn("ANALYTICS_SITE_SECRET", os.environ)
+        self.assertTrue(platform.core.valid)
+
+
+if __name__ == "__main__":
+    unittest.main()