mooring 0.3.2__tar.gz → 0.3.4__tar.gz

{mooring-0.3.2 → mooring-0.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mooring
-Version: 0.3.2
+Version: 0.3.4
 Summary: Git-free marimo notebook sharing via GitHub
 Requires-Python: >=3.12
 Requires-Dist: keyring
@@ -14,6 +14,8 @@ Requires-Dist: truststore
 Requires-Dist: uvicorn
 Provides-Extra: copilot
 Requires-Dist: github-copilot-sdk>=1.0.1; extra == 'copilot'
+Provides-Extra: pii
+Requires-Dist: gliner>=0.2.13; extra == 'pii'
 Description-Content-Type: text/markdown
 
 # ⚓ mooring

{mooring-0.3.2 → mooring-0.3.4}/docs/admins/ai-privacy.md

@@ -39,11 +39,15 @@ sees whatever your team wrote into those files.
 
 ## The four structural guarantees
 
-1. **Single choke point.** The context handed to the model is assembled in one
-   place (`ai/chat.py:build_system_context`) — from the schema text and the
-   notebook source, plus (only when team context is enabled) the team instructions
-   and the value-minimised data-dictionary slice. Nothing reaches the model except
-   through this one function.
+1. **Single choke point for the system context.** The context handed to the model
+   is assembled in one place (`ai/chat.py:build_system_context`) — from the schema
+   text and the notebook source, plus (only when team context is enabled) the team
+   instructions and the value-minimised data-dictionary slice. Two further egresses
+   exist by design and are value-free by construction: your **chat turns**, and the
+   agent's **tool reads** (it can re-read the notebook source via
+   `mooring_read_notebook_source`, and fetch dataset schemas). The opt-in
+   [structured-PII scan](#structured-pii-pre-flight-scan-opt-in-best-effort) runs at
+   all of these, not only `build_system_context`.
 2. **Value-free tools only.** The agent is given mooring's own tools (`ai/tools.py`):
    list datasets, get a schema, read the notebook source, and *propose* a cell —
    each value-free by construction. When a data dictionary is configured, three more
@@ -81,9 +85,11 @@ data often lives **outside** it — a network share, a warehouse export, a datab
 connection, a path built at runtime — and the schema most useful for writing code
 is frequently a *derived* frame (a join/filter result) that no file holds. To help
 there, mooring can read the schema of the dataframes **already loaded in your
-running kernel**. It is **on by default** and value-free, but — like team context —
-its safety comes from *how it is built*, not from physical impossibility, so it is
-documented here in full. Turn it off with `[ai] live_schema = false`.
+running kernel**. It is **on by default**, refreshed on every chat turn (so a frame
+you load after opening the chat is picked up without reopening), and value-free — but,
+like team context, its safety comes from *how it is built*, not from physical
+impossibility, so it is documented here in full. Turn it off with
+`[ai] live_schema = false`.
 
 How it stays value-blind (`ai/introspect.py`):
 
@@ -100,6 +106,11 @@ How it stays value-blind (`ai/introspect.py`):
 - **Fail-closed on the way back.** The reader (`_parse_frames`) accepts only the
   `{name, columns: [[str, str]], n_rows: int}` shape and drops everything else, so a
   value can't ride back on a key mooring doesn't read.
+- **The per-turn refresh adds no new value channel.** The schema is captured at
+  chat-open *and* re-probed on each turn through the **same** frozen probe and
+  fail-closed reader; a turn re-states the schema only when the kernel changed, and an
+  unchanged kernel is not re-sent. The refresh re-states already-value-free schema —
+  it opens no path a value could take that the open-time capture did not.
 
 Honest caveat: unlike `schema.py` (which physically only ever reads a file header),
 this probe runs in a namespace that *contains* values. Its value-blindness is the
@@ -145,13 +156,139 @@ Run `mooring ai dictionary check` to see exactly how your files parse — which
 was detected, how many tables/columns were kept, which keys were dropped, and any
 secret-scan findings — *before* enabling the feature or sharing the files.
 
+## Structured-PII pre-flight scan (opt-in, best-effort)
+
+The guarantees above stop the *data* from reaching the model. They cannot stop a
+human from **typing a real value** into a cell or the chat —
+`df.filter(pl.col("pan") == "4012 8888 8888 1881")`, or "why does account
+4012888888881881 fail?". As a thin extra floor, mooring can scan text on its way
+out for **well-formed structured identifiers** and warn before it leaves. It is
+**off by default** (`[ai.pii] enabled = false`) and, like team context, its safety
+is best-effort, not structural.
+
+**What it catches** (precision over recall): checksum-validated **payment cards**
+(Luhn), **IBANs** (mod-97), and **NHS numbers** (mod-11), plus shape-anchored
+**emails** and **UK NINOs**. **What it does not catch, by design:** addresses,
+account narratives, **UK sort codes**, **bank account numbers**, US SSNs, phone
+numbers, dates of birth, IP addresses, or any value split across two messages.
+Person **names** are out of reach of the structured scan too, but can be caught by
+the optional local-NER pass below. **A clean scan is not a value-free guarantee** —
+it is a safety net for the obvious, well-formed cases, and it complements (never
+replaces) the structural value-blindness above.
+
+It runs at every egress, and every finding is value-free — a line number and a
+*kind* (`payment card`, `email address`, …), never the matched value:
+
+- **Your chat prompt.** With `block_prompt = true` (the default once the feature is
+  on), a prompt that looks like it contains a card/IBAN/NHS/email/NINO is **held**;
+  you see which kinds tripped it and must click **"Send anyway"** — nothing reaches
+  the model until you confirm. (Set `block_prompt = false` for a warn-only advisory.)
+- **The notebook source and its schema.** On opening the copilot you get a one-time,
+  value-free banner if the notebook or a dataset schema looks like it contains PII.
+  The source is never rewritten (that would break your code). But a schema **column
+  name** that is itself a value — the result of a pivot/transpose on a PII key, e.g.
+  `df.pivot(on="customer_pan")` — is **withheld** from the schema the model sees.
+- **Team context.** An `instructions.md` carrying a checksum-validated card/IBAN/NHS
+  (or a secret) is withheld entirely; a stray email/NINO drops just that line; a
+  data-dictionary description that trips the scan is dropped.
+
+Run **`mooring ai pii check`** to scan your `context/` files and notebook sources
+**offline** (no Copilot, no network) before enabling the feature — it prints
+`path:line  kind` for each finding and never echoes a value. Put `# mooring: pii-ok`
+on a line to retire a reviewed false positive.
+
+Configure it under `[ai.pii]`: `enabled` (master switch), `block_prompt`
+(hold-and-confirm vs. a warn-only advisory on the chat prompt), and
+`scan_notebook_source` (the source/schema banner).
+
+## Name detection (opt-in, local NER)
+
+A person's name — `where name == "Jane Smith"` — has no checksum or fixed shape, so
+the structured scan above cannot see it. The optional **name pass** (`ai/ner.py`)
+closes that gap with a **local** zero-shot NER model ([GLiNER](https://github.com/urchade/GLiNER)),
+shipped as the `mooring[pii]` extra so the lean install and the frozen `.pyz` stay
+free of the heavy ML stack (torch + transformers). It is **off by default**, only
+acts when `[ai.pii] enabled` is also true, and is **best-effort** (NER both misses
+and false-positives — a clean scan is not proof of no names).
+
+Its privacy properties match the structured scan:
+
+- **Local only.** The model runs on the analyst's machine; the text is never sent
+  anywhere to be scanned. The single network touch is a **one-time model download**
+  from Hugging Face on first use — pre-fetch it on a managed/offline network with
+  **`mooring ai pii model`**.
+- **Value-free findings.** GLiNER returns the matched name; mooring reads **only**
+  the label and character offset, maps it to a line number, and **drops the text**.
+  A finding is `(line, "person name")` — never the name — so it logs and streams
+  over SSE as safely as the structured kinds. Pinned by `tests/test_ner.py`.
+- **No pickle, pinned.** The default model (`gliner-community/gliner_small-v2.5`) is
+  loaded as its **safetensors** `bf16` variant — `mooring ai pii model` fetches *only*
+  the safetensors file, never the repo's `pytorch_model.bin`, so nothing is unpickled.
+  It is **pinned to a specific commit** (`name_model_revision`) for reproducibility and
+  so a security review is against a fixed artifact.
+- **Same egress + UI.** A flagged chat prompt is held with the same "Send anyway"
+  confirm; `mooring ai pii check` runs the name pass too (when the model is already
+  cached) for the offline lint. At the chat prompt, a configured-but-uninstalled extra
+  **fails loud** (a `scan_error` advisory) rather than silently doing nothing; while the
+  model is still downloading the name pass is skipped (the message is still structurally
+  scanned) and the chat shows a "preparing model" status.
+
+Configure under `[ai.pii]`: `detect_names` (on/off), `name_model` / `name_model_revision`
+/ `name_model_variant` (which model, pinned commit, and safetensors variant —
+`name_model_variant = ""` loads a repo's default weights file for a model that has no
+variant safetensors), `name_labels` (entity labels to flag), and `name_threshold`
+(confidence cut-off; raise for fewer, safer hits). GLiNER is zero-shot, so `name_labels`
+is not limited to people — add `"organization"` to also flag **business names** (surfaced
+as an `organization` finding); other entity types (e.g. `"address"`) work the same way.
+Capitalised non-person terms make organisation detection more false-positive-prone, so it
+stays out of the default. Install and enable:
+
+```toml
+[ai.pii]
+enabled = true
+detect_names = true
+```
+```
+pip install mooring[pii]      # or: uvx mooring[pii]  (combine extras: mooring[copilot,pii])
+mooring ai pii model          # pre-download the model (recommended)
+```
+
+## Deploying name detection in an institutional / offline environment
+
+The model download is the only part of mooring that reaches a non-GitHub host
+(Hugging Face). In a locked-down environment, plan for:
+
+- **Firewall allow-list.** Outbound HTTPS is needed to `huggingface.co` **and** the
+  file backends — the LFS CDN and the newer **Xet** hosts (`cas-bridge.xethub.hf.co`,
+  `*.xethub.hf.co`). Allow-listing only `huggingface.co` passes the metadata fetch and
+  then fails on the actual download.
+- **TLS / SSL-intercepting proxy.** mooring enables the **OS trust store** globally
+  (`truststore`), so Hugging Face traffic honours your proxy's root CA automatically,
+  the same way GitHub traffic does — no separate CA bundle needed in the normal case.
+  `REQUESTS_CA_BUNDLE` / `SSL_CERT_FILE` still take precedence if you set them.
+- **Proxy / rate limits.** `HTTPS_PROXY` / `NO_PROXY` are honoured; set an `HF_TOKEN`
+  to lift the anonymous-download rate limit (faster, fewer throttles).
+- **Air-gapped (no egress).** Either point at an internal mirror with
+  `HF_ENDPOINT=https://<your-hf-proxy>` (e.g. Artifactory/Nexus), **or** provision the
+  cache out-of-band: run `mooring ai pii model` on a connected machine, copy
+  `~/.cache/huggingface` (or set a shared `HF_HOME`) to the target machines, and set
+  `HF_HUB_OFFLINE=1`. Relocate the cache with `HF_HOME` if the user profile is small
+  or roaming.
+- **Model governance.** The weights are a third-party artifact. The pinned default is
+  safetensors (no code-execution-on-load risk that a pickle `pytorch_model.bin` carries),
+  loaded locally; review the pinned `name_model` + `name_model_revision` through your
+  model-risk process, and re-pin a new revision only after review.
+
 ## The one thing to watch
 
 Anything **you type into a cell or the chat** is, by definition, visible to the
 assistant. If you hard-code a real value into a cell —
 `df.filter(pl.col("ssn") == "123-45-6789")` — that literal is part of the source
 the assistant can read. The chat reminds you of this; **never paste real values**.
-A future release will add an automatic PII/redaction guard on outbound text.
+The opt-in [structured-PII scan](#structured-pii-pre-flight-scan-opt-in-best-effort)
+above catches *well-formed* cards/IBANs/NHS numbers/emails/NINOs as a safety net,
+but it cannot catch a name, a sort code, an account number, or a value typed into
+prose — so the rule stands regardless.
 
 ## Verifying it yourself
 
@@ -177,3 +314,8 @@ The copilot needs the optional extra (`pip install mooring[copilot]` /
 `uvx mooring[copilot]`), a GitHub Copilot licence (`mooring ai login`), and your
 organisation's Copilot **CLI/agent policy** enabled. See
 [Configuration](configuration.md) for the `[ai]` settings.
+
+Optional **name detection** (the structured-PII guard's NER pass) needs the
+separate `mooring[pii]` extra (`pip install mooring[pii]`); without it the guard
+still does its stdlib structured-PII scan. See
+[Name detection](#name-detection-opt-in-local-ner).

{mooring-0.3.2 → mooring-0.3.4}/docs/admins/configuration.md

@@ -189,6 +189,32 @@ a fallback when no OS credential store is available; normally it's in the
 credential store instead. Run `selftest` to print the exact paths on a given
 machine.
 
+## Editing the user config from the command line
+
+Rather than hand-editing the file, `mooring config` reads and writes it by
+**dotted key** (`section.subsection.key`). It touches only the user file and
+leaves every other setting in place:
+
+```
+mooring config set ai.pii.enabled true          # bool
+mooring config set ai.pii.name_threshold 0.6     # number
+mooring config set ai.pii.name_labels person name organization   # several tokens = a list
+mooring config get ai.pii.enabled                # print the effective value
+mooring config unset ai.pii.enabled              # remove the key (revert to the default)
+mooring config list                              # print the whole effective config
+mooring config path                              # print the config.toml location
+```
+
+Value typing is automatic: `true`/`false` become booleans, `5`/`0.6` become
+numbers, **several tokens become a string list**, and anything else (a path or
+model id like `urchade/gliner_multi_pii-v1`) stays a string. To force a string
+that looks numeric, quote it as a TOML literal, e.g. `set some.key '"123"'`.
+
+`get` and `list` show the **effective** value (packaged default merged with your
+file); they do **not** reflect a one-run [environment-variable](#environment-variables)
+override. This works for any key, including the `[ai]` / `[ai.pii]` settings
+documented in [AI privacy](ai-privacy.md).
+
 ## The runtime setup form
 
 If a build ships **without** `client_id` / `owner` / `repo`, the hub shows a

{mooring-0.3.2 → mooring-0.3.4}/docs/users/ai-copilot.md

@@ -41,6 +41,13 @@ Keep both tabs side by side: chat on one, the marimo notebook on the other.
     Anything you type into a cell or the chat is visible to the assistant. Refer
     to columns by name — don't paste actual data values.
 
+    If your admin has turned on the **structured-PII scan**, a message that looks
+    like it contains a payment card, IBAN, NHS number, email, or UK NINO is **held**
+    and you'll see a **"Send anyway"** prompt — nothing is sent until you confirm.
+    It's a best-effort safety net: it won't catch names, sort codes, or account
+    numbers, so the rule still stands. See
+    [the privacy page](../admins/ai-privacy.md#structured-pii-pre-flight-scan-opt-in-best-effort).
+
 ## Team context (optional)
 
 If your admin enables it (`[ai] context = true`), the copilot also reads a
@@ -72,4 +79,8 @@ your files parse and to catch anything sensitive *before* you share them.
   choice is remembered. (Some models — and "Auto" — have no effort setting.)
 - The assistant can read your notebook's current code and ask for a dataset's
   schema on its own; you don't need to paste either in.
+- It also sees the schema of dataframes already loaded in the running notebook —
+  and this is **refreshed every time you send a message**, so if you load a new
+  dataframe mid-chat, just ask your next question; there's no need to reopen the
+  chat. (It sees column names and types only, never your data values.)
 - It writes Polars (`pl`) by default, matching mooring's bundled notebook stack.

{mooring-0.3.2 → mooring-0.3.4}/docs/users/cli.md

@@ -36,6 +36,8 @@ python mooring.pyz deps remove polars
 python mooring.pyz deps list
 python mooring.pyz deps lock
 python mooring.pyz build-requirements [-o FILE]
+python mooring.pyz config set <key> <value...> | config get <key>
+python mooring.pyz config unset <key> | config list | config path
 python mooring.pyz selftest
 python mooring.pyz version
 ```
@@ -178,6 +180,24 @@ afterwards to remove it from the team repo for everyone; a notebook you never
 shared just disappears. You are asked to confirm first — pass `-y`/`--yes` to
 skip the prompt (required when running non-interactively, e.g. from a script).
 
+### `config`
+
+Read and edit your user `config.toml` by **dotted key**, without hand-editing the
+file (every other setting is preserved):
+
+```
+python mooring.pyz config set ai.pii.enabled true
+python mooring.pyz config set ai.pii.name_labels person name organization
+python mooring.pyz config get ai.pii.enabled        # effective value
+python mooring.pyz config unset ai.pii.enabled      # revert to the default
+python mooring.pyz config list                      # whole effective config
+python mooring.pyz config path                      # config.toml location
+```
+
+`true`/`false` become booleans and numbers are parsed; several tokens become a
+list; anything else stays a string. See
+[Configuration → editing from the command line](../admins/configuration.md#editing-the-user-config-from-the-command-line).
+
 ### `selftest`
 
 Verify the bundled environment and print a diagnostic snapshot of the machine:

{mooring-0.3.2 → mooring-0.3.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mooring"
-version = "0.3.2"
+version = "0.3.4"
 description = "Git-free marimo notebook sharing via GitHub"
 readme = "README.md"
 requires-python = ">=3.12"
@@ -28,6 +28,13 @@ dependencies = [
 copilot = [
     "github-copilot-sdk>=1.0.1",
 ]
+# Optional NER name detection for the outbound-PII guard (Phase 2). GLiNER pulls
+# torch + transformers (hundreds of MB), so it ships as an extra — without it the
+# guard still does its stdlib structured-PII scan. Install with
+# `pip install mooring[pii]`. Pre-fetch the model with `mooring ai pii model`.
+pii = [
+    "gliner>=0.2.13",
+]
 
 [project.scripts]
 mooring = "mooring.cli:main"

{mooring-0.3.2 → mooring-0.3.4}/src/mooring/__init__.py

@@ -1,3 +1,3 @@
 """Mooring: git-free marimo notebook sharing via GitHub."""
 
-__version__ = "0.3.2"
+__version__ = "0.3.4"

{mooring-0.3.2 → mooring-0.3.4}/src/mooring/ai/base.py

@@ -65,6 +65,12 @@ class AIProvider(Protocol):
         model: str | None = None,
         reasoning_effort: str | None = None,
         dictionary=None,
+        pii_enabled: bool = False,
+        pii_block: bool = True,
+        pii_names: bool = False,
+        pii_name_labels: tuple[str, ...] | None = None,
+        pii_name_threshold: float = 0.7,
+        pii_name_model: str | None = None,
     ):
         """Open a long-lived, streaming chat session (a ``ChatBroadcaster``).