visionary-sdk 0.4.0__tar.gz

visionary_sdk-0.4.0/.gitignore

@@ -0,0 +1,7 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+# Copied from the repository root when packing.
+/LICENSE

visionary_sdk-0.4.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

visionary_sdk-0.4.0/PKG-INFO

@@ -0,0 +1,444 @@
+Metadata-Version: 2.5
+Name: visionary-sdk
+Version: 0.4.0
+Summary: Visionary collector: logs API requests, database and HTTP spans, errors and system metrics into your database.
+Project-URL: Homepage, https://github.com/styropyr0/Visionary
+Project-URL: Documentation, https://github.com/styropyr0/Visionary/tree/main/sdks/python#readme
+Project-URL: Source, https://github.com/styropyr0/Visionary/tree/main/sdks/python
+Project-URL: Issues, https://github.com/styropyr0/Visionary/issues
+Author: styropyr0
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: analytics,api,asgi,django,fastapi,flask,metrics,monitoring
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Django
+Classifier: Framework :: FastAPI
+Classifier: Framework :: Flask
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: System :: Monitoring
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Provides-Extra: all
+Requires-Dist: psutil>=5.9; extra == 'all'
+Requires-Dist: psycopg>=3.1; extra == 'all'
+Requires-Dist: pymysql>=1.0; extra == 'all'
+Provides-Extra: mysql
+Requires-Dist: pymysql>=1.0; extra == 'mysql'
+Provides-Extra: postgres
+Requires-Dist: psycopg>=3.1; extra == 'postgres'
+Provides-Extra: system
+Requires-Dist: psutil>=5.9; extra == 'system'
+Description-Content-Type: text/markdown
+
+# visionary-sdk (Python)
+
+The Python collector for [Visionary](../../Readme.md). It runs inside your web
+service and writes:
+
+- one row per HTTP request into `api_logs` (method, route pattern, status,
+  timing, user, redacted headers and bodies, error codes, and how many
+  database and outbound HTTP calls it made),
+- the database and HTTP calls of slow or failed requests into `spans`,
+- unhandled and reported exceptions into `errors`,
+- one `releases` row per deploy (when the release is known), and
+- a periodic host / process / runtime sample into `system_metrics`.
+
+The Visionary server reads these tables for its dashboards. All slow work
+happens on background threads, so requests are never blocked or slowed. If
+the database is down, the SDK only writes log lines.
+
+Every Visionary SDK behaves the same way. The contract is
+[`sdks/SPEC.md`](../SPEC.md).
+
+## Install
+
+```sh
+pip install visionary-sdk                 # core: stdlib only (D1, SQLite)
+pip install "visionary-sdk[system]"       # + psutil: host CPU/memory/network, process CPU/RSS/threads/fds
+pip install "visionary-sdk[postgres]"     # + psycopg 3
+pip install "visionary-sdk[mysql]"        # + PyMySQL
+```
+
+Requires Python 3.9+. Without the `system` extra, Linux hosts read every host
+and process column from `/proc` instead. On macOS and Windows most of them are
+NULL; disk usage, thread count, GC and HTTP throughput are still recorded.
+
+## Quickstart
+
+First configure a database (see [Choosing the database](#choosing-the-database)). For example:
+
+```sh
+export VISIONARY_DATABASE_URL=postgres://user:pass@localhost:5432/app
+```
+
+### Django
+
+```python
+# settings.py
+MIDDLEWARE = [
+    "visionary.integrations.django.VisionaryMiddleware",  # first, so it sees the final response
+    # ...
+]
+```
+
+The middleware calls `visionary.init()` the first time it loads, so nothing
+else is needed. It works under WSGI and ASGI, with sync and async views.
+
+- **Route:** `request.resolver_match.route` with a leading `/`, e.g. `/users/<int:id>/`.
+- **User:** if you don't call `set_user`, an authenticated `request.user` is
+  used (`pk` as the id, `get_username()` as the handle).
+
+### FastAPI / Starlette (any ASGI app)
+
+```python
+from fastapi import FastAPI
+from visionary.integrations.asgi import VisionaryASGIMiddleware
+
+app = FastAPI()
+app.add_middleware(VisionaryASGIMiddleware)
+# or, for any ASGI app: app = VisionaryASGIMiddleware(app)
+```
+
+The route is the matched route's path (`/users/{user_id}`). Streaming
+responses keep streaming. Only the first `VISIONARY_LOG_MAX_BODY_BYTES` of a
+body are copied.
+
+### Flask
+
+```python
+from flask import Flask
+from visionary.integrations.flask import Visionary
+
+app = Flask(__name__)
+Visionary(app)            # or: ext = Visionary(); ext.init_app(app)
+```
+
+The route is `request.url_rule.rule` (`/users/<int:user_id>`).
+
+### Anything else
+
+```python
+import visionary
+
+agent = visionary.init()   # reads the environment, starts, flushes at exit
+```
+
+`init()` never raises. If no database is configured, it logs one warning and
+returns a no-op agent, and the integrations pass requests straight through.
+
+For explicit control (tests, several agents, custom config):
+
+```python
+cfg = visionary.from_env()          # or visionary.Config(...)
+cfg.path_prefix = "/api"
+agent = visionary.Agent(cfg).start()
+app.add_middleware(VisionaryASGIMiddleware, agent=agent)
+...
+agent.close(timeout=10)             # drains the queue and flushes samples
+```
+
+## Choosing the database
+
+The SDK checks these in order:
+
+1. **Your own connection.** Use this when the app already has a driver:
+
+   ```python
+   import psycopg, visionary
+
+   cfg = visionary.from_env()
+   cfg.connection_factory = lambda: psycopg.connect(DSN)
+   cfg.dialect = "postgres"            # "postgres", "mysql" or "sqlite"
+   visionary.init(cfg)
+   ```
+
+   The factory must return a new DB-API connection. Visionary keeps that
+   connection for its own use:
+   - PostgreSQL and MySQL: one shared connection.
+   - SQLite: one connection per worker thread.
+
+   After an error it reconnects through the factory, and it closes the
+   connection at shutdown. Each batch runs as one transaction. If you hand it
+   an autocommit connection, batches are not atomic.
+
+2. **`VISIONARY_DATABASE_URL`**, chosen by scheme:
+
+   | URL | Backend |
+   |---|---|
+   | `postgres://user:pass@host:5432/db` (or `postgresql://`) | PostgreSQL via psycopg 3 |
+   | `mysql://user:pass@host:3306/db` | MySQL 8+ / MariaDB via PyMySQL |
+   | `sqlite:///absolute/path.db`, `sqlite:///C:/data/x.db`, `sqlite://relative.db` | SQLite via `sqlite3` |
+   | `d1://` | Cloudflare D1, credentials from `D1_*` |
+
+3. **Cloudflare D1**, when `D1_ACCOUNT_ID`, `D1_DATABASE_ID` and
+   `D1_API_TOKEN` are set. `D1_API_BASE` overrides the API URL.
+
+Tables and indexes are created at startup from the schema files shipped in
+the package (`visionary/schema/<dialect>/`). This happens on the background
+threads, so a slow database never delays boot. If the database is down at
+startup, creation is retried before each write.
+
+## Configuration
+
+Every setting has an environment variable, and `visionary.Config` has the
+same settings as attributes. The full list and semantics are in
+[SPEC.md, "Environment"](../SPEC.md#environment).
+
+| Variable | Default | Notes |
+|---|---|---|
+| `VISIONARY_DATABASE_URL` | | see above |
+| `D1_ACCOUNT_ID`, `D1_DATABASE_ID`, `D1_API_TOKEN`, `D1_API_BASE` | | D1 only |
+| `VISIONARY_INSTANCE` | hostname | labels `system_metrics` rows |
+| `VISIONARY_LOG_REQUESTS` | `true` | request logging on/off |
+| `VISIONARY_LOG_PATH_PREFIX` | empty | only log paths starting with this |
+| `VISIONARY_LOG_BODIES` | `true` | capture request/response bodies |
+| `VISIONARY_LOG_MAX_BODY_BYTES` | `8192` | per body |
+| `VISIONARY_LOG_RETENTION_DAYS` | `30` | `0` keeps forever |
+| `VISIONARY_LOG_REDACT_HEADERS`, `_REDACT_FIELDS`, `_REDACT_REQUEST_FIELDS` | empty | comma lists added to the defaults |
+| `VISIONARY_LOG_ERROR_CODE_FIELD`, `_ERROR_KEY_FIELD` | `error.code`, `error.key` | dotted JSON paths |
+| `VISIONARY_SYSTEM_METRICS` | `true` | sampler on/off |
+| `VISIONARY_SYSTEM_INTERVAL` | `15s` | `15`, `500ms`, `1m` |
+| `VISIONARY_SYSTEM_RETENTION_DAYS` | `30` | |
+| `VISIONARY_SERVICE` | empty | service name on every row |
+| `VISIONARY_RELEASE` | platform commit variable | release on every row; see [Telemetry](#telemetry-spans-errors-releases) |
+| `VISIONARY_SPANS` | `true` | spans and call totals on/off |
+| `VISIONARY_SPAN_THRESHOLD_MS` | `100` | store spans of requests at least this slow (or 5xx) |
+| `VISIONARY_SPAN_MAX` | `100` | span rows per request; later calls still count in the totals |
+| `VISIONARY_ERRORS` | `true` | error capture on/off |
+| `VISIONARY_INSTRUMENT_HTTP` | `false` | Python only: call `instrument_http()` at start |
+
+Booleans accept `true/false/1/0/yes/no`.
+
+## Telemetry (spans, errors, releases)
+
+### Service and release
+
+`VISIONARY_SERVICE` names the service, so several services can share one
+database. The release (`version`) is `VISIONARY_RELEASE`, else the first of
+`RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `VERCEL_GIT_COMMIT_SHA`,
+`HEROKU_SLUG_COMMIT`, `SOURCE_VERSION`, `GIT_COMMIT`, `GITHUB_SHA`,
+`K_REVISION`; commit hashes are cut to 12 characters. Both go on every
+`api_logs`, `system_metrics` and `errors` row (`spans`: service only). When
+the release is known, one `releases` row (`sdk = python`, `runtime` such as
+`CPython 3.13.1`) is written at start; the dashboards mark deploys with it.
+
+### Spans
+
+Every call the SDK can see is counted into `db_count`, `db_ms`, `http_count`
+and `http_ms` on the request's `api_logs` row (NULL for a kind of call that is
+not instrumented). When the request took at least `VISIONARY_SPAN_THRESHOLD_MS`
+or answered 5xx, its first `VISIONARY_SPAN_MAX` calls are also stored as
+`spans` rows, in start order. Spans outside a request are stored when they
+take at least the threshold themselves, with `request_id` NULL.
+
+The request is found through a contextvar, so spans from sync views, thread
+pools that copy the context (FastAPI sync endpoints, `asgiref`,
+`contextvars.copy_context().run`) and asyncio tasks all land on the right
+request.
+
+| Source | How |
+|---|---|
+| Django ORM / `connection.cursor()` | automatic: the middleware adds a wrapper to `execute_wrappers` of every database alias |
+| SQLAlchemy (sync or async) | `visionary.instrument_sqlalchemy(engine)` |
+| psycopg 3 (sync/async), psycopg2, PyMySQL, sqlite3 | `conn = visionary.instrument_connection(conn)` (or `instrument_cursor(cur)`) |
+| `requests`, `httpx` (sync and async), `urllib3` | `visionary.instrument_http()` once at startup, or `VISIONARY_INSTRUMENT_HTTP=true` |
+| anything else | `with visionary.span("cache", "GET session"):` (also `async with`) |
+
+```python
+engine = visionary.instrument_sqlalchemy(create_engine(DSN))
+
+with visionary.span("queue", "publish order.created", target="rabbitmq") as s:
+    channel.basic_publish(...)
+    s.set_row_count(1)       # optional; s.set_error(...) marks it failed
+```
+
+Database spans are named after the first keyword and main table
+(`SELECT users`, `INSERT bookings`, `BEGIN`), with the target
+`system/database` (`postgresql/app`, `sqlite/app.db`). The statement is stored
+sanitized: string and number literals become `?`, `IN (?, ?, ...)` becomes
+`IN (?)`, whitespace is collapsed. Parameter values are never stored. HTTP
+spans are named `METHOD host`; the detail is the URL without credentials,
+query or fragment. An HTTP span fails on a transport error or a 5xx answer.
+A call through `requests` makes one span, not a second one for urllib3.
+
+The SDK's own writes are never recorded as spans.
+
+### Errors
+
+The Django, ASGI and Flask integrations record every exception the
+framework turns into a 500 (`handled = 0`) and leave the exception and the
+framework's response unchanged. 4xx exceptions (`Http404`, `HTTPException`,
+`abort(404)`) are not errors. To record one you handled:
+
+```python
+try:
+    charge(card)
+except PaymentError:
+    visionary.capture_exception()      # or capture_exception(exc)
+```
+
+Inside a request the row carries its `request_id`, method, route, path and
+user. `type` is the class name, qualified with its module unless it is a
+builtin (`ValueError`, `psycopg.errors.UndefinedTable`); `stack` is
+`traceback.format_exception` (the last 16 KB).
+
+`fingerprint` groups the same error: the first 16 hex characters of
+`sha1(type + "\n" + frames)`, where `frames` are the 5 innermost frames of
+your own code (not the standard library, `site-packages` or this SDK), each
+written `function@path` with the path relative to its `sys.path` entry
+(`shop/views.py`), joined with `\n`. If no frame is your code, the 5
+innermost frames are used; with no frames at all, the message with digit
+runs replaced by `0`. At most 20 rows per fingerprint per minute are kept.
+
+Retention: `spans` and `errors` follow `VISIONARY_LOG_RETENTION_DAYS`;
+`releases` are kept.
+
+## Users
+
+```python
+import visionary
+
+visionary.set_user(user.id, handle=user.email, role="admin")
+```
+
+Call `set_user` from auth middleware or a view once the user is known. Only
+`id` is required. It is contextvar-based, so it works from sync views, thread
+pools (FastAPI sync endpoints) and asyncio tasks. Outside a request handled
+by an integration, it does nothing.
+
+`visionary.set_route("/custom/{pattern}")` works the same way for the route,
+for routers the integration can't see.
+
+## Hooks
+
+```python
+cfg = visionary.from_env()
+
+def lookup_user(user_id):
+    """Fill in handle/role for ids logged without them. Runs on the writer
+    thread, once per distinct user per batch; keep it cheap (cache)."""
+    u = USER_CACHE.get(user_id)
+    return (u.email, u.role) if u else None
+
+def db_stats():
+    """Your pool's stats for the db_* columns. Counters are cumulative;
+    Visionary stores per-sample deltas."""
+    s = pool.get_stats()                     # e.g. psycopg_pool
+    return {
+        "total": s["pool_size"], "idle": s["pool_available"],
+        "in_use": s["pool_size"] - s["pool_available"], "max": s["pool_max"],
+        "acquires": s.get("requests_num", 0), "empty_acquires": s.get("requests_queued", 0),
+        "acquire_wait_ms": s.get("requests_wait_ms", 0),
+    }
+
+cfg.lookup_user = lookup_user
+cfg.db_stats = db_stats
+visionary.init(cfg)
+```
+
+If `db_stats()` leaves out a key, that column is stored as NULL.
+
+With Django 5.1+ PostgreSQL pooling (`"OPTIONS": {"pool": ...}`) you don't need
+the hook: when `db_stats` is unset, the Django middleware sums the stats of
+Django's pools the same way. Without a pool the `db_*` columns stay NULL.
+
+## What is captured
+
+Each request (`api_logs`) records:
+
+- **Request id:** the incoming `X-Request-ID` if it is at most 128 characters,
+  otherwise a random 32-hex id. It is echoed on the response.
+- **Timing:** start time (`2026-10-17T17:34:00.123Z`) and duration in ms.
+- **Request line:** method, path, route pattern and query string.
+- **Status:** the final status (500 when the handler raised).
+- **Client:** IP (first `X-Forwarded-For` hop, otherwise the peer address) and user agent.
+- **User:** id, handle and role.
+- **Headers:** all request headers as JSON.
+- **Bodies:** content types, bodies up to the cap with `*_truncated` flags,
+  and full body sizes.
+- **Errors:** for `status >= 400`, the error code and key from the JSON response body.
+
+Bodies are not captured for `multipart/*`, `image/*`, `video/*`, `audio/*`,
+`application/octet-stream`, `application/zip` and `application/pdf`.
+
+Streaming responses are counted, and their first bytes are copied without
+buffering. File responses (`FileResponse`, Flask `send_file`) keep their
+sendfile path, so only their `Content-Length` is recorded.
+
+Each sample (`system_metrics`) records:
+
+- **Host:** CPU, load average (Unix only), memory, disk usage of the working
+  directory, and network rates.
+- **Process:** CPU, RSS, threads and open files.
+- **Runtime:** live Python threads (in the `goroutines` column), the
+  `tracemalloc` heap when tracing is on, and GC collections and pause time
+  (via `gc.callbacks`).
+- **Database pool:** whatever your `db_stats` hook returns, or Django's
+  PostgreSQL pools when it isn't set.
+- **HTTP:** requests, 5xx responses, requests per second and in-flight
+  requests seen by the integrations. These are counted even when request
+  logging is off.
+- **Node (node-exporter style):**
+  - CPU: count, user / system / iowait / steal share.
+  - Memory: available and page-cache memory, swap.
+  - Disk: total size, inode use, read/write KB/s and IOPS summed over whole
+    disks (partitions skipped), average I/O wait (await), and busy % of the
+    busiest disk.
+  - Network: packets/s, errors and drops.
+  - TCP: connections established and retransmits.
+  - Processes on the host, running and blocked.
+  - Context switches/s.
+  - This process: read/write KB/s and its open-file limit.
+
+Sources that don't exist on the platform are NULL:
+
+- **Linux only:** iowait, steal, disk busy, the TCP columns,
+  `procs_running` / `procs_blocked`.
+- **Not on Windows:** page cache, inodes and the open-file limit.
+- **Windows resolution:** disk await comes from whole-second counters there,
+  so it is coarse.
+
+On Linux the host, process and node columns are read from `/proc` when
+psutil isn't installed, so all of them are filled either way.
+
+Older `api_logs` and `system_metrics` tables are upgraded in place at
+startup: missing columns are added with `ALTER TABLE ... ADD COLUMN`, and
+existing rows keep NULL in them.
+
+## Redaction
+
+Redaction runs on the writer thread before anything is stored, and all
+matching is case-insensitive.
+
+- **Headers:** `Authorization`, `Proxy-Authorization`, `Cookie`, `Set-Cookie`,
+  `X-Api-Key` and `X-Auth-Token` (plus `VISIONARY_LOG_REDACT_HEADERS`) become
+  `[REDACTED]`.
+- **JSON keys:** `password`, `passwd`, `secret`, `client_secret`, `token`,
+  `access_token`, `refresh_token`, `id_token`, `api_key`, `apikey`, `otp`,
+  `pin`, `card_number`, `cvv` and `private_key` (plus
+  `VISIONARY_LOG_REDACT_FIELDS`) are redacted in request and response bodies.
+  `VISIONARY_LOG_REDACT_REQUEST_FIELDS` adds keys redacted in request bodies
+  only.
+- **Truncated bodies:** matching uses a regex, so bodies cut at the cap are
+  still scrubbed.
+- **Form bodies:** `application/x-www-form-urlencoded` request bodies are
+  scrubbed the same way (`password=[REDACTED]`).
+
+## Operational notes
+
+- **Request queue:** holds 5000 entries. When it is full, new entries are
+  dropped and the count is logged.
+- **Request writes:** batches of up to 50 are written every 3 s. A failed
+  batch is retried up to 3 times (1 s, then 2 s backoff), then dropped with an
+  error log.
+- **System writes:** samples are written 4 at a time. On failure they are kept
+  (at most an hour's worth) and retried with the next write.
+- **Retention:** runs 60 s after start, then daily, deleting in chunks of 5000.
+- **Telemetry writes:** spans, errors and the release row share the request
+  queue and batches (one transaction, or one D1 `batch` call, per flush).
+- **Logging:** all messages go to the `visionary` logger.
+- **Forking servers:** with servers like `gunicorn --preload`, the background
+  threads are restarted in each worker after fork.