ariadne-data 0.2.0__tar.gz

ariadne_data-0.2.0/PKG-INFO

@@ -0,0 +1,363 @@
+Metadata-Version: 2.4
+Name: ariadne-data
+Version: 0.2.0
+Summary: Governed metrics for autonomous agents: OKF context layer + OSI semantic layer behind one MCP facade
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: duckdb>=1.0
+Requires-Dist: sqlglot>=25.0
+Requires-Dist: openai>=1.40
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: mcp
+Requires-Dist: mcp<2,>=1.2; extra == "mcp"
+Provides-Extra: documents
+Requires-Dist: liteparse>=2.4.0; extra == "documents"
+Requires-Dist: markitdown[docx]>=0.1.6; extra == "documents"
+Requires-Dist: firecrawl-anydoc==0.1.3; extra == "documents"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: mcp<2,>=1.2; extra == "dev"
+Requires-Dist: liteparse>=2.4.0; extra == "dev"
+Requires-Dist: markitdown[docx]>=0.1.6; extra == "dev"
+Requires-Dist: firecrawl-anydoc==0.1.3; extra == "dev"
+
+# Ariadne
+
+**Governed metrics for AI agents.** Ariadne sits between your data and your AI agent
+and makes sure the numbers coming back are the numbers your business agreed on.
+
+When an agent writes SQL directly against a database (text-to-SQL), it guesses: which
+table, which join, which filter, which definition of "revenue". Two runs of the same
+question can return different answers, and nothing tells the agent that cancelled orders
+must be excluded or that the currency changed in March. Ariadne replaces the guessing
+with a governed catalog:
+
+- A **semantic layer** holds the one canonical definition of each metric, written in
+  the open [OSI](https://ossie.apache.org/) YAML format. Ariadne compiles these to SQL
+  itself and executes them read-only — no agent ever hand-writes a query for a governed
+  number.
+- A **context layer** holds the business knowledge around those metrics — policies,
+  table caveats, provenance — as plain version-controlled Markdown. Every governed
+  answer carries its caveats with it, so the agent reports "revenue was X *— excluding
+  cancelled orders, per policy*" instead of a bare number.
+
+Agents query Ariadne through the local `ariadne` command line — no server to run,
+no extra setup. Ariadne also speaks [MCP](https://modelcontextprotocol.io/), the
+standard tool protocol, as an optional extra (`ariadne-data[mcp]`) for remote or
+host-integrated tool registration.
+
+## What you get
+
+### For querying agents
+
+- **Local commands** covering the full question lifecycle (see below) — governed answers
+  first, with a safe path for anything not yet modeled, plus saved answer examples as
+  labeled method references. No server required.
+- **Caveats attached to every number.** Governed answers include the full text of every
+  business policy the metric cites. Ungoverned answers are clearly labeled
+  `governed: false`.
+- **Fan-out protection.** Ad-hoc queries are checked for grain-preserving joins *before*
+  execution, so one-to-many joins can't silently inflate totals — a classic way
+  text-to-SQL agents produce wrong-but-confident numbers.
+
+### For data owners
+
+- **AI-assisted authoring, human gate.** One command profiles your databases and drafts
+  the entire metric catalog and context pages into a `staging/` area. Nothing is ever
+  served until a human reviews and promotes it.
+- **Bring your own agent.** In harness-led authoring, your agent's *own* model drafts
+  the catalog from measured profiles and converted documents (`ariadne-authoring`) —
+  no server-side LLM. Source access, admission and Promotion stay human decisions.
+- **Everything is files.** Metrics are YAML, business knowledge is Markdown. Review in
+  pull requests, diff definitions, roll back with git. No server database to migrate.
+- **Self-improving catalog.** When an agent answers an unmodeled question, its working
+  definition is automatically staged as a proposal — good ad-hoc answers become
+  governed metrics through the same human review.
+- **Document ingestion.** Feed existing data dictionaries, PDFs, or Office documents in;
+  get typed, provenance-cited proposals out, staged for review.
+
+### For everyone
+
+- **Lightweight.** A Python package plus embedded DuckDB. No API gateway, cache cluster,
+  or refresh workers to operate.
+- **Permissive runtime.** Only MIT/Apache-licensed dependencies — no AGPL, no proprietary
+  engine.
+
+## How it works
+
+```text
+ your data ──▶ profile ──▶ author (AI) ──▶ staging/ ──▶ human review ──▶ served catalog
+  (many sources)        drafts everything     nothing serves              metrics + policies
+                                                   from here                       │
+                                                                                   ▼
+                                    AI agent runs `ariadne` (local) or connects over MCP:
+                                               Ariadne compiles the metric
+                                               to SQL, checks it, runs it read-only,
+                                               returns the number + its business caveats
+```
+
+1. **Register your sources** in one `ariadne.yaml` file.
+2. **Profile & author**: `ariadne-author` inspects the actual data and combines it with
+   your seed notes (the rules statistics can't see) to draft the catalog into `staging/`.
+   Or let your own agent's model draft through `ariadne-authoring` Git proposals (see
+   "Harness-led authoring" below) — either way, a human reviews, and only a human
+   promotes or admits.
+3. **Review & promote (legacy staged project)**: a human reads the staging area, then
+   `ariadne-promote` merges approved artifacts into the served catalog and marks them
+   verified. **Git-managed project**: a human admits the exact reviewed, validated
+   candidate revision with `ariadne-admit --confirm` (see
+   [docs/how-to/git-authoring.md](docs/how-to/git-authoring.md)).
+4. **Query**: your agent runs `ariadne` locally against the promoted catalog (`--served`)
+   or the pinned admitted snapshot (`--project DIR`) — or connects over MCP with the
+   optional extra.
+
+## Quickstart
+
+Requires Python ≥ 3.12 and [uv](https://docs.astral.sh/uv/). Authoring uses an LLM —
+set `OPENAI_API_KEY` (plus `OPENAI_BASE_URL` / `OPENAI_MODEL` if not using OpenAI
+directly) in a `.env` file.
+
+```bash
+uv venv --python 3.12 .venv
+uv pip install --python .venv/bin/python -e .
+source .venv/bin/activate
+```
+
+**See it end-to-end** on a bundled example (seeds a demo shop, authors a catalog with a
+live LLM, pauses for your review, then queries a governed metric locally):
+
+```bash
+examples/retail-demo/demo.sh
+```
+
+**Point it at your own data:**
+
+```bash
+ariadne-init my-project && cd my-project   # scaffold registry, seed notes (local-first)
+# edit ariadne.yaml (your sources) and data/seed_notes.md (your business rules)
+ariadne-author --sources ariadne.yaml --notes data/seed_notes.md
+# review staging/, then:
+ariadne-promote
+ariadne list-metrics --sources ariadne.yaml      # query locally — no server needed
+ariadne get-metric gross_margin --sources ariadne.yaml
+```
+
+Or run the same project Git-managed — isolated revision-checked proposals, exact-
+revision human admission, pinned snapshots (the full legacy → new mapping lives in
+[docs/how-to/git-authoring.md](docs/how-to/git-authoring.md)):
+
+```bash
+ariadne-init --git my-project && cd my-project && git init
+# declare sources, then (agent or you):
+ariadne-authoring start --project .
+ariadne-authoring propose-sources --proposal ID --expected-revision REV --file sources.yaml
+#    HUMAN: ariadne-authoring approve-sources --project . --proposal ID \
+#             --revision REV --source-digest DIGEST
+ariadne-authoring edit --proposal ID --expected-revision REV --file edit.json
+ariadne-authoring validate --proposal ID --expected-revision REV
+ariadne-authoring review --proposal ID
+#    HUMAN: ariadne-admit --project . --proposal ID --revision REV \
+#             --reviewed-digest DIGEST --confirm
+ariadne list-metrics --project .   # queries the pinned admitted snapshot
+```
+
+Already have a legacy staged project? The human import moves its served artifacts
+into an unadmitted Git proposal without touching pending staging work — never
+admission:
+
+```bash
+ariadne-init --git . && git init
+ariadne-authoring import-legacy --project . --from-root .   # HUMAN
+```
+
+Imported legacy table pages bind to their approved catalog assets with the
+human `ariadne-authoring enrich-legacy` (ambiguous same-named tables refuse
+rather than guess), and the explicit `refresh-schema` generates their physical
+schema facts. Context knowledge — physical schemas, shared terminology,
+identifier mappings and bounded mapping evidence with automatic hypothesis
+capture — has its own guide:
+[docs/how-to/context-knowledge.md](docs/how-to/context-knowledge.md).
+Aliases, observations and measurements never authorize joins or replace the
+compile gate and live audits.
+
+Installing the packaged skill (`ariadne-skill install --scope user`) teaches your agent
+the whole workflow — query governed metrics first, respect the staging boundary, report
+caveats with every number.
+
+### Harness-led authoring — bring your own agent
+
+`ariadne-author` above drafts using an LLM configured on the host (`.env`). If your
+agent harness has its own model, it authors instead through the local
+`ariadne-authoring` commands — base install only, no OpenAI-compatible endpoint, no
+extra credential. In a Git-managed project (scaffolded with `ariadne-init --git`, the
+flow above) the agent drafts through isolated proposals; `ariadne-author --git` runs
+the same drafting loop straight into an unadmitted proposal. The legacy staged route
+below keeps working for non-Git projects:
+
+```bash
+ariadne-init my-project && cd my-project
+# 1. agent: propose source registry entries (inert data — nothing is opened yet)
+ariadne-authoring propose-sources --file sources-proposal.yaml
+# 2. HUMAN operator (never the agent): review the digest, install into ariadne.yaml
+#    ariadne-authoring approve-sources --digest <digest>
+# 3. agent: read the measured profile (null until step 2) and converted documents
+#    from data/docs/ (allowed roots live in ariadne.yaml's documents: section)
+ariadne-authoring inputs --document policy-notes.md
+# 4. agent's own model drafts Seed Notes + OSI + OKF, submits, repairs on errors
+#    (3 rounds max; each repair needs the current digest as --base-digest)
+ariadne-authoring submit --file submission.json   # cites its document references
+# 5. agent inspects and reports the exact digest, then STOPS
+ariadne-authoring review
+# 6. HUMAN (never the agent): ariadne-promote on the reviewed digest
+```
+
+Documents are pre-staged by the operator in `data/docs/` (or another configured
+root); PDF/Office/HTML conversion uses the optional `.[documents]` extra, and
+inputs are bounded (converted text arrives in parts). Metrics without an
+independent expected-value oracle are staged with `attestation: pending` —
+`governed: true` means Promoted, not independently value-attested.
+
+**Optional: stdio MCP toolsets.** With the `mcp` extra installed (see below),
+the same local stdio server exposes the role toolsets over one core — pick the
+registration at launch: `python -m ariadne.serve --project DIR` (combined,
+the default), `--toolset analyst` (the query tools `list_metrics`,
+`search_context`, `get_metric`, `list_datasets`, `define_metric`,
+`query_metrics`, `query_detail`, `query_records`, `resolve_term`,
+`save_example`, `search_examples`), or `--toolset authoring` (the Git proposal contract:
+`authoring_start_proposal`, `authoring_read_artifact`,
+`authoring_edit_proposal`, `authoring_propose_sources`, `authoring_inputs`,
+`authoring_upload_document` for bounded ≤5 MiB uploads,
+`authoring_refresh_schema`, `authoring_validate_proposal`,
+`authoring_inspect_mapping`, `authoring_resolve_term`,
+`authoring_review_proposal`) with identical
+contracts to the `ariadne-authoring` commands. Stdio is local and needs no
+token; the server runs for one project root. Source approval, admission,
+reconciliation, the legacy import/enrichment and Promotion are never exposed
+as tools.
+
+**Opt-in HTTP (issue #18).** `ariadne-serve --project DIR --http` serves the
+same toolsets at `/mcp`, on loopback (`127.0.0.1`) by default. Every request,
+including health/readiness checks, requires `Authorization: Bearer <token>`
+from the server environment `ARIADNE_HTTP_TOKEN`. Missing/wrong tokens fail
+before dispatch; the token never belongs in `ariadne.yaml`, responses or logs.
+Stdio remains token-free and opens no HTTP listener.
+
+**Compose/container binding (owner-approved 2026-10-10).** Add
+`--http-container --http-host 0.0.0.0` to allow other containers to connect.
+The default Compose service publishes no ports; put the MCP client or HTTPS
+proxy on the same network. The separate loopback-only override is for a
+host-side private HTTPS proxy such as Tailscale Serve. External clients still
+require private HTTPS. This option is not deployment/exposure approval.
+
+The SDK's DNS-rebinding protection stays enabled. Explicitly add the service
+and proxy Host values using repeatable `--http-allowed-host` flags. For
+clients sending an Origin header, add its exact origin with
+`--http-allowed-origin`; requests without Origin need no origin entry.
+Do not use blanket wildcards. The repository includes a non-root
+[Dockerfile](Dockerfile), [Compose template](compose.yaml) and
+[loopback override](compose.loopback.yaml). See
+[Docker MCP with an external data project](docs/how-to/docker-mcp.md) for
+owner permissions, stable Git/source paths, networking and the smoke check.
+The project stays external and is mounted at the same absolute host/container
+path; the image does not contain project data or credentials. A network client
+uses `http://ariadne:8000/mcp` with the bearer token. A private HTTPS proxy
+must preserve Authorization/MCP headers and streaming; configure it separately.
+
+- `GET /healthz`: `200 {"status":"ok"}` when the HTTP process responds.
+- `GET /readyz`: `200 {"status":"ready"}` for Authoring without requiring a
+  catalog; Analyst/Combined require a usable metric catalog through the shared
+  runtime's discovery/admission checks. Failures return only
+  `503 {"status":"not_ready"}`, never exception details or project paths.
+- Both accept HEAD and require the same bearer token. Readiness does not
+  calculate metrics or test every source/graph operation. A lazy runtime may
+  perform its existing bounded source initialization on the first ready check.
+
+**Optional: MCP.** The base package is local-first and needs no server; MCP registers the
+same operations as tools when you want them. Install the extra into the same Python
+environment as `ariadne`, from the Ariadne package checkout (the directory containing
+`pyproject.toml`):
+
+```bash
+uv pip install --python .venv/bin/python -e '.[mcp]'
+```
+
+Then, from the initialized project root, opt in to its registry-aware `.mcp.json` (an
+existing file is never overwritten) and start the server:
+
+```bash
+ariadne-init --with-mcp .
+ariadne-serve --sources ariadne.yaml
+```
+
+The scaffolded `.mcp.json` wires your agent to `ariadne-serve`; restart the agent after
+config changes.
+
+## Querying: the local commands
+
+All commands print one JSON value; the common options (`--sources`/`--db`, `--served`) follow
+the subcommand. The CLI finds promoted artifacts in the working directory or `./served`.
+MCP registers the same operations as tools when the `mcp` extra is installed.
+
+| Command | What the agent uses it for |
+| --- | --- |
+| `ariadne list-metrics` | Browse the governed catalog: metrics, descriptions, available dimensions |
+| `ariadne get-metric NAME` | Get a governed number — compiled SQL, executed read-only, caveats included |
+| `ariadne search-context QUERY` | Look up business rules, policies, and table caveats before answering |
+| `ariadne resolve-term TERM` | Resolve shared business terms and identifier knowledge (representations, exact field bindings, scope, ambiguity, claim/confirmation status) against the admitted context — read-only guidance, never join authority |
+| `ariadne list-datasets` | See every table across all registered sources |
+| `ariadne query-records` | Bounded record selection from one known table — the latest N rows, no aggregation: explicit filters/order/timestamp contract, `governed: false`, never staged, truthful requested-top-N completeness, and all-or-nothing overflow refusals with narrowing guidance |
+| `ariadne define-metric` | Answer an unmodeled question: draft a metric (JSON via `--file` or stdin), pass the safety checks (ad-hoc SQL may read only registered catalog relations — file, network and table-function reads are refused before execution), run it read-only, label it `governed: false`, and auto-stage it as a proposal |
+| `ariadne search-examples QUERY` | Find explicitly saved answer examples — labeled references (draft/stable, staleness, provenance) for checking a method, never cached answers; never executes a query |
+| `ariadne save-example` | Save one answer example after the user explicitly confirms the bounded record (question, interpretation, method, result excerpt, self-declared attestation); stages a draft under `staging/answer_examples/` for operator review |
+
+Saved examples are references, not caches: a similar question still compares dates, filters,
+grain and cited policies, and the answer is always produced fresh by the runtime. Reviewing
+submissions is an operator command (`ariadne-example-review`), never an agent action.
+
+## Data sources
+
+One read-only connection unifies every source; the agent addresses them as
+`<source>.<table>`:
+
+```yaml
+sources:
+  - name: shop
+    type: sqlite
+    path: data/shop.db
+  - name: events
+    type: parquet
+    path: data/events/*.parquet      # globs become one table
+    table: events
+  - name: warehouse
+    type: postgres
+    connection_env: ARIADNE_PG_DSN   # credentials are env-var names, never inline
+```
+
+Supported: **SQLite, CSV, JSON, Parquet, Delta Lake, Iceberg, DuckLake, PostgreSQL** —
+local files, `https://`, or `s3://`. Document ingestion (optional extra
+`.[documents]`) covers Markdown, text, PDF, Word, PowerPoint, Excel, and HTML.
+
+## How Ariadne compares
+
+| Approach | What it is | Where Ariadne differs |
+| --- | --- | --- |
+| **Raw text-to-SQL agents** | The agent writes SQL directly against your database | No canonical definitions, no business rules, joins can silently inflate numbers. Ariadne compiles every governed metric itself and attaches the rules to the answer. |
+| **Cube** | Mature headless-BI semantic layer you operate as a service | Cube is infrastructure: API gateway, caching, refresh workers, vendor-specific model format. Ariadne is a Python package with embedded DuckDB — no serving infrastructure — using open file formats (OSI YAML, Markdown). |
+| **dbt MetricFlow** | Semantic layer inside the dbt ecosystem | Strong choice if you already run dbt; adopting it otherwise means adopting the whole dbt workflow. Ariadne stands alone and adds the agent-facing context layer and review gate. |
+| **Wren MDL** | AI-oriented semantic modeling spec | Similar niche, but a vendor-specific spec with a weaker governance and join-safety story; Ariadne builds on the vendor-neutral OSI standard. |
+| **Malloy** | Semantic modeling *language* from ex-Looker engineers | A language and toolchain, not a governed metric registry with business context; Ariadne's artifacts are plain YAML/Markdown with a human promotion gate. |
+
+Ariadne is a good fit when AI agents need trustworthy, repeatable numbers from
+heterogeneous data — and a human must stay in the loop on definitions. It is **not** a
+BI dashboard, a transformation pipeline, or a high-concurrency metrics API; if you need
+those, pair it with (or choose) the tools above.
+
+## Technical details
+
+The design decisions, specification, and research behind Ariadne live in the repo:
+
+- Behavior spec: [project/specs/mvp-vertical-slice.md](project/specs/mvp-vertical-slice.md)
+- Architecture decisions: [project/adr/](project/adr/)
+- Domain vocabulary: [CONTEXT.md](CONTEXT.md) · Research: [project/research/](project/research/)
+- Contributing: `python -m pytest` and `ruff check .` must stay green