@sidub-inc/docuoria.cli 1.0.15

package/payload/skill/references/patterns.md

@@ -0,0 +1,97 @@
+# Patterns
+
+The patterns below are *illustrative*. They demonstrate authoring techniques, not copy-paste solutions. Real PDFs have layout quirks (broken words, unicode whitespace, OCR drift) that mean a pattern that matches the visible text will often miss the engine's flattened haystack. Before using any pattern below, run `dotnet script scripts/test-pattern.csx -- <pdf> '<regex>'` and adapt to what `PatternTestResult.Matches` and `PatternTestResult.Gaps` actually report. See `pattern-authoring.md` for techniques.
+
+### 1. ISO 8601 date
+
+- regex: `\b\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])\b`
+- Matches: `2024-01-15`, `1999-12-31`.
+- Non-matches: `2024-13-01`, `24-01-15`, `2024/01/15`.
+- Field type: `DateOnly`.
+- Teaches: alternation inside character classes for valid month/day ranges.
+
+### 2. US ZIP / CA postal code (combined)
+
+- regex: `\b(?:\d{5}(?:-\d{4})?|[A-Z]\d[A-Z] ?\d[A-Z]\d)\b`
+- Matches: `90210`, `90210-1234`, `K1A 0B1`, `K1A0B1`.
+- Non-matches: `1234`, `9021O` (letter O instead of zero).
+- Field type: `string`.
+- Teaches: non-capturing groups and optional whitespace with `?`.
+
+### 3. Currency with symbol
+
+- regex: `(?<currency>[$€£¥])\s?(?<amount>\d{1,3}(?:,\d{3})*(?:\.\d{2})?)`
+- Matches: `$1,234.56`, `€ 99.00`, `£10`.
+- Non-matches: `1234.56` (no symbol), `$1.234,56` (European grouping).
+- Field type: `decimal`.
+- Teaches: named groups (consumed downstream by `PatternExtractionSource.PrimaryGroup`).
+
+### 4. Currency, symbol-stripped numeric only
+
+- regex: `(?<![\$€£¥\d])-?\d{1,3}(?:,\d{3})*\.\d{2}(?![\d])`
+- Matches: `1,234.56`, `-99.00`.
+- Non-matches: `1234` (no decimal), `1,234.5` (one fraction digit).
+- Field type: `decimal`.
+- Teaches: lookbehind/lookahead to reject embedded matches.
+
+### 5. Integer (standalone)
+
+- regex: `(?<![\d.])\d+(?![\d.])`
+- Matches: `42`, `1000`.
+- Non-matches: `3.14`, `12345.67`.
+- Field type: `int`.
+- Teaches: negative lookarounds for "not part of a bigger token".
+
+### 6. Decimal (any precision)
+
+- regex: `(?<![\d.])-?\d+\.\d+(?![\d.])`
+- Matches: `3.14`, `-0.001`.
+- Non-matches: `3.`, `.5`, `3.14.15`.
+- Field type: `decimal`.
+- Teaches: balanced lookarounds preventing partial overlap.
+
+### 7. Percentage
+
+- regex: `(?<value>-?\d+(?:\.\d+)?)\s?%`
+- Matches: `50%`, `12.5 %`, `-3%`.
+- Non-matches: `% 50`, `fifty percent`.
+- Field type: `decimal`.
+- Teaches: optional whitespace plus capture before a literal suffix.
+
+### 8. Email (RFC-pragmatic)
+
+- regex: `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b`
+- Matches: `a@b.co`, `first.last+tag@example.com`.
+- Non-matches: `a@b`, `@example.com`.
+- Field type: `string`.
+- Teaches: word-boundary anchoring and the pragmatic-vs-RFC-strict tradeoff.
+
+### 9. Phone E.164
+
+- regex: `\+[1-9]\d{1,14}\b`
+- Matches: `+14165551212`, `+442071838750`.
+- Non-matches: `416-555-1212`, `+0123` (leading zero after `+`).
+- Field type: `string`.
+- Teaches: format normalisation upstream of extraction (the engine expects already-cleaned input).
+
+### 10. URL (http/https)
+
+- regex: `https?://[^\s<>"']+`
+- Matches: `https://example.com/path?q=1`, `http://a.b`.
+- Non-matches: `ftp://x`, `example.com`.
+- Field type: `string`.
+- Teaches: negated character class for "until whitespace or quote".
+
+### 11. UUID v1–v5
+
+- regex: `\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-5][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}\b`
+- Matches: `550e8400-e29b-41d4-a716-446655440000`.
+- Non-matches: `not-a-uuid`, all-zeros (fails the variant nibble).
+- Field type: `Guid`.
+- Teaches: position-specific character classes to validate format.
+
+## How to verify a pattern
+
+- Run `dotnet script scripts/test-pattern.csx -- <pdf> '<regex>'`. If `PatternTestResult.HasMatches` is `false`, the pattern does not match this PDF's haystack — go to `pattern-authoring.md`.
+- If matches are partial, run `dotnet script scripts/test-groups.csx -- <pdf> '<regex>'` and read `PatternGroupTestResult.Groups[*].MatchesIndependently` to find the failing group.
+- If matches are too many, tighten with lookarounds (patterns 4–6 demonstrate the technique).

package/payload/skill/references/privacy.md

@@ -0,0 +1,36 @@
+# Local-processing privacy guarantee
+
+## Claim
+
+When you call `Docuoria`, the PDF bytes you supply never leave the machine the engine runs on. The library reads, extracts, transforms, and renders the PDF entirely in-process. The only outbound network call any first-party component makes is template JSON read/write against an HTTP template store, and that call is opt-in.
+
+## Evidence — extraction is in-process
+
+Every PDF-consuming primitive on `IDocuoriaEngine` (`InspectAsync`, `TestPatternAsync`, `TestGroupsAsync`, `DryRunAsync`, `ExecuteTemplateAsync`, `EvaluateMatchAsync`, `EvaluateMatchRuleAsync`, `ClassifyAsync`) takes a `Stream` — never a URL or remote handle for the PDF. See `src/libs/Docuoria/Contracts/IDocuoriaEngine.cs`; each method's documentation carries the contract phrase "The PDF stream is opened and disposed within the call (D-13)." The implementation in `src/libs/Docuoria/Engine/DocuoriaEngine.cs` resolves the in-process `IPdfDocumentFactory` and walks the configured extraction/transformation/publish steps without leaving the process.
+
+## Evidence — the one network surface
+
+`src/libs/Docuoria/Storage/ApiTemplateStoreProvider.cs` implements `ITemplateStoreProvider` and uses `IHttpClientFactory` (named client `ApiTemplateStoreProvider.HttpClientName = "Docuoria.TemplateStore"`) to read and write *templates*. Templates are JSON describing how to extract — they do not contain PDF bytes. A host that wants entirely local processing simply does not register `ApiTemplateStoreProvider`; the engine functions identically against a local store.
+
+`src/libs/Docuoria/Pipeline/Retrieval/Http/HttpRetrievalProvider.cs` is an inbound fetch path the *host* opts into for retrieval steps; it does not upload PDFs supplied by the caller — it only downloads PDFs the template explicitly references.
+
+## What this does NOT promise
+
+- If you wire a third-party logger, telemetry sink, or background storage handler around the engine, your hosting code may transmit data. The guarantee is about the library, not your host.
+- If the host uses `HttpRetrievalProvider` to download a PDF before processing, the URL of that PDF is necessarily known to the network. The guarantee is about what happens *after* the engine has the bytes in memory.
+- If you store templates via `ApiTemplateStoreProvider`, template content (which may contain regex patterns derived from the PDF's text) crosses the network. PDFs do not.
+
+## Verifying for yourself
+
+1. Search the library for outbound HTTP usage:
+   `Select-String -Path src/libs/Sidub.PdfPipeline -Pattern 'HttpClient|HttpRequestMessage' -Recurse`.
+   Confirm hits are confined to the template-store and retrieval surfaces:
+   - `Storage/ApiTemplateStoreProvider.cs`
+   - `Storage/Http/TemplateStoreCredentialHandler.cs`
+   - `Registration/HttpRetrievalProviderBuilderExtensions.cs`
+   - `Registration/TemplateStoreBuilderExtensions.cs`
+   - `Pipeline/Retrieval/Http/HttpRetrievalProvider.cs`
+
+   The extra hits beyond `ApiTemplateStoreProvider` and `HttpRetrievalProvider` are credential-handler and DI-registration support for those same two surfaces — they do not introduce new outbound paths.
+2. Confirm `IDocuoriaEngine` only accepts `Stream` for PDF input — open `src/libs/Docuoria/Contracts/IDocuoriaEngine.cs` and read every method signature.
+3. Run `dotnet test` with the network blocked (e.g. firewall rule) and observe extraction tests still pass.

package/payload/skill/references/scripts.md

@@ -0,0 +1,361 @@
+# Agent Scripts
+
+This directory hosts the **agent-facing CLI surface** for `Docuoria`. Each script
+is a [`dotnet-script`](https://github.com/dotnet-script/dotnet-script) `.csx` file that
+binds a single SDK verb to a deterministic JSON contract. The scripts are designed for
+non-interactive automation (LLM agents, CI jobs, shell pipelines) and uphold a strict
+output contract:
+
+- **Successful runs** emit a single line of UTF-8 JSON to **stdout**, exit code `0`.
+- **Errors** emit a single `{"error":{"code","message","detail"}}` line to **stderr**,
+  non-zero exit code.
+- All payloads serialize via `DocuoriaJsonOptions.Default` (camelCase, discriminator
+  `$type` for polymorphic results, `WhenWritingNull` ignore policy — see Classify for
+  the explicit-null exception).
+
+Every script `#load "_common.csx"` to share host bootstrap, argument parsing
+(`Cli.Require` / `Cli.Get` / `Cli.Has`), template-store registration, PDF stream
+loading, and JSON writers.
+
+> v1.4 invariants: confidence is binary (`1.0` / `0.0`), `ClassifyAsync` returns
+> `null` when no template matches (CLS-02), and throws `InvalidOperationException`
+> when no store is registered.
+
+> **Distribution:** this directory is the **source** for the AI plugin's `scripts/`
+> folder. `skills/build.ps1` copies these `.csx` files into `dist/docuoria/scripts/`
+> and rewrites the SDK `#r` line in `_common.csx` to point at the bundled
+> `assets/lib/Docuoria.dll`. In-repo development uses the relative
+> `bin/Release/...dll` path; downstream consumers receive the bundled DLL.
+
+## Installation
+
+```powershell
+dotnet tool install -g dotnet-script
+# build the SDK once so _common.csx can reference the local DLL
+dotnet build src/libs/Docuoria/Docuoria.csproj -c Debug
+```
+
+Run any script with:
+
+```powershell
+dotnet script scripts/<name>.csx -- --pdf path\to\file.pdf
+```
+
+The `--` separator forwards subsequent tokens as script arguments (exposed as the
+`Args` global, an `IList<string>`).
+
+## Common Store Parameters
+
+Scripts that access the template store (`classify`, `evaluate-match`,
+`list-templates`, `load-template`, `save-template`) accept these shared flags
+to configure the store backend. When neither `--store-path` nor `--store-url`
+is provided, the local store defaults to `./templates`.
+
+| Flag           | Default        | Description                                                              |
+| -------------- | -------------- | ------------------------------------------------------------------------ |
+| `--store-path` | `./templates`  | Local file-system template store directory.                              |
+| `--store-url`  | _(none)_       | API template store URL (mutually exclusive with `--store-path`).         |
+| `--store-key`  | _(none)_       | Function key for API store authentication (used with `--store-url`).     |
+
+## Error JSON Shape
+
+```json
+{
+  "error": {
+    "code": "kebab-case-identifier",
+    "message": "Human-readable summary.",
+    "detail": "Optional stack trace or full exception text."
+  }
+}
+```
+
+Common `code` values: `pdf-not-found`, `parse-error`, `already-exists`, `no-store`,
+`unhandled`, `bad-format`.
+
+---
+
+## inspect.csx
+
+**Synopsis.** Report low-level PDF structure (page count, text blocks, candidate
+patterns) for a PDF — primary discovery step before authoring a template.
+
+| Arg       | Required | Description                                  |
+| --------- | -------- | -------------------------------------------- |
+| `--pdf`   | yes      | Path to the source PDF.                      |
+| `--page`  | no       | 1-based page index (default: all pages).     |
+
+**Output schema.** `PdfInspection` payload (`pageCount`, `pages[].blocks[]`, …).
+
+**Exit codes.** `0` success · `1` unhandled · `pdf-not-found` on missing input.
+
+**Example.**
+
+```powershell
+dotnet script scripts/inspect.csx -- --pdf invoice.pdf --page 1
+```
+
+---
+
+## test-pattern.csx
+
+**Synopsis.** Evaluate a single extraction pattern against a PDF and report whether
+it matched, with the captured value(s).
+
+| Arg                  | Required | Description                                                |
+| -------------------- | -------- | ---------------------------------------------------------- |
+| `--pattern`          | yes      | Inline pattern source (regex or DSL block).                |
+| `--pdf`              | yes      | Path to the source PDF.                                    |
+| `--page`             | no       | 1-based page index (default: all pages).                   |
+| `--block-separator`  | no       | Override the block-separator regex used during extraction.  |
+
+**Output schema.** `{ hasMatches: bool, matches: [...] }`.
+
+**Exit codes.** `0` success · `1` unhandled / parse-error · `pdf-not-found`.
+
+**Example.**
+
+```powershell
+dotnet script scripts/test-pattern.csx -- --pattern 'Invoice #(\d+)' --pdf invoice.pdf
+```
+
+---
+
+## test-groups.csx
+
+**Synopsis.** Evaluate a multi-group pattern and emit each named capture group's
+match set — used when authoring repeating-row extractions.
+
+| Arg         | Required | Description                                  |
+| ----------- | -------- | -------------------------------------------- |
+| `--pattern` | yes      | Multi-group pattern source.                  |
+| `--pdf`     | yes      | Path to the source PDF.                      |
+| `--page`    | no       | 1-based page index (default: all pages).     |
+
+**Output schema.** `{ groups: { <name>: [matches...] } }`.
+
+**Exit codes.** `0` success · `1` on extraction failure.
+
+**Example.**
+
+```powershell
+dotnet script scripts/test-groups.csx -- --pattern @rows.txt --pdf invoice.pdf
+```
+
+---
+
+## validate-template.csx
+
+**Synopsis.** Parse a `Template` JSON file and report schema / semantic validation
+results without executing it.
+
+| Arg          | Required | Description                          |
+| ------------ | -------- | ------------------------------------ |
+| `--template` | yes      | Path to the template JSON file.      |
+
+**Output schema.** `{ valid: bool, errors: [string] }`.
+
+**Exit codes.** `0` always (even on `valid:false`) · `1` for `parse-error`.
+
+**Example.**
+
+```powershell
+dotnet script scripts/validate-template.csx -- --template templates/invoice.json
+```
+
+---
+
+## dry-run.csx
+
+**Synopsis.** Execute extraction + publish steps against a PDF **without** producing a
+serialized output payload — useful for end-to-end pipeline validation. Optionally
+preview formatted output with `--preview-as`.
+
+| Arg            | Required | Description                                                |
+| -------------- | -------- | ---------------------------------------------------------- |
+| `--pdf`        | yes      | Path to the source PDF.                                    |
+| `--template`   | yes      | Path to the template JSON file.                            |
+| `--preview-as` | no       | Preview formatted output: `csv` or `json` (no file written). |
+
+**Output schema.** `{ kind: "SucceededResult"|"FailedResult"|"RejectedResult", result }`.
+With `--preview-as`: `{ kind, format, preview }`.
+
+**Exit codes.** `0` success · `1` unhandled.
+
+**Example.**
+
+```powershell
+dotnet script scripts/dry-run.csx -- --pdf invoice.pdf --template templates/invoice.json
+```
+
+---
+
+## execute.csx
+
+**Synopsis.** Full pipeline run with output generation. Writes a CSV or JSON payload
+either to stdout or to `--output`.
+
+| Arg          | Required | Description                                                    |
+| ------------ | -------- | -------------------------------------------------------------- |
+| `--pdf`      | yes      | Path to the source PDF.                                        |
+| `--template` | yes      | Path to the template JSON file.                                |
+| `--format`   | yes      | `csv` or `json`.                                               |
+| `--output`   | no       | Write binary payload to this path. If omitted, stdout text.    |
+
+**Output schema.** Success: `{ status: "ok", format, output? }` (output is base64 / text).
+Failure: `{ status: "rejected"|"failed", result }`.
+
+**Exit codes.** `0` success · `1` rejected/failed · `2` `bad-format`.
+
+**Example.**
+
+```powershell
+dotnet script scripts/execute.csx -- --pdf invoice.pdf --template templates/invoice.json --format json --output out.json
+```
+
+---
+
+## evaluate-match.csx
+
+**Synopsis.** Compute the aggregated match confidence between a PDF and a single
+template. Confidence is `ruleConfidence × extractionProbeScore` (0.0 when either
+fails, 1.0 when both are perfect). Template argument may be a file path **or** a
+template identifier resolved through the configured store.
+
+| Arg            | Required | Description                                                                       |
+| -------------- | -------- | --------------------------------------------------------------------------------- |
+| `--pdf`        | yes      | Path to the source PDF.                                                           |
+| `--template`   | yes      | File path (`.json` / contains path separator) **or** template ID for store lookup. |
+| `--store-path` | no       | Local template store directory (default: `./templates`).                          |
+| `--store-url`  | no       | API template store URL.                                                           |
+| `--store-key`  | no       | Function key for API store authentication.                                        |
+
+**Output schema.** `{ confidence, matchedRules }`.
+
+**Exit codes.** `0` success · `1` template not found / unhandled.
+
+**Example.**
+
+```powershell
+dotnet script scripts/evaluate-match.csx -- --pdf invoice.pdf --template invoice
+```
+
+---
+
+## classify.csx
+
+**Synopsis.** Run ranked classification across **all** registered templates and return
+the top matches sorted by confidence (descending).
+
+| Arg            | Required | Description                                       |
+| -------------- | -------- | ------------------------------------------------- |
+| `--pdf`        | yes      | Path to the source PDF.                           |
+| `--top`        | no       | Maximum number of results to return (default: 5). |
+| `--store-path` | no       | Local template store directory (default: `./templates`). |
+| `--store-url`  | no       | API template store URL.                           |
+| `--store-key`  | no       | Function key for API store authentication.        |
+
+**Output schema.** `{ matches: [{ templateId, confidence }, ...] }`. Only functional
+matches are included (root rule passes AND extraction probe > 0).
+
+**Exit codes.** `0` success (including `match:null`) · `1` `no-store` if no template
+store is registered · `1` unhandled.
+
+**Example.**
+
+```powershell
+dotnet script scripts/classify.csx -- --pdf invoice.pdf
+```
+
+---
+
+## list-templates.csx
+
+**Synopsis.** Enumerate template identifiers from the configured store.
+
+| Arg            | Required | Description                                              |
+| -------------- | -------- | -------------------------------------------------------- |
+| `--store-path` | no       | Local template store directory (default: `./templates`). |
+| `--store-url`  | no       | API template store URL.                                  |
+| `--store-key`  | no       | Function key for API store authentication.               |
+
+**Output schema.** `{ templates: [id, ...] }`.
+
+**Exit codes.** `0` success · `1` unhandled.
+
+**Example.**
+
+```powershell
+dotnet script scripts/list-templates.csx -- --store-path ./templates
+```
+
+---
+
+## load-template.csx
+
+**Synopsis.** Resolve a template by identifier and emit its JSON representation.
+
+| Arg            | Required | Description                                              |
+| -------------- | -------- | -------------------------------------------------------- |
+| `--id`         | yes      | Template identifier.                                     |
+| `--output`     | no       | Write JSON to this file path instead of stdout.          |
+| `--store-path` | no       | Local template store directory (default: `./templates`). |
+| `--store-url`  | no       | API template store URL.                                  |
+| `--store-key`  | no       | Function key for API store authentication.               |
+
+**Output schema.** Without `--output`: full template JSON. With `--output`:
+`{ status: "ok", path }`.
+
+**Exit codes.** `0` success · `1` not-found / unhandled.
+
+**Example.**
+
+```powershell
+dotnet script scripts/load-template.csx -- --id invoice --output templates/invoice.json
+```
+
+---
+
+## save-template.csx
+
+**Synopsis.** Persist a template JSON file to the configured store. Fails with
+`already-exists` unless `--overwrite` is supplied.
+
+| Arg            | Required | Description                                                               |
+| -------------- | -------- | ------------------------------------------------------------------------- |
+| `--template`   | yes      | Path to the template JSON file to persist.                                |
+| `--overwrite`  | no       | Boolean switch — overwrite an existing template with the same identifier. |
+| `--store-path` | no       | Local template store directory (default: `./templates`).                  |
+| `--store-url`  | no       | API template store URL.                                                   |
+| `--store-key`  | no       | Function key for API store authentication.                                |
+
+**Output schema.** `{ status: "ok", identifier }`.
+
+**Exit codes.** `0` success · `1` `already-exists` / parse-error / unhandled.
+
+**Example.**
+
+```powershell
+dotnet script scripts/save-template.csx -- --template templates/invoice.json --overwrite
+```
+
+---
+
+## Internals — `_common.csx`
+
+`_common.csx` is the **single bootstrap** used by every script. It:
+
+1. References the locally-built `Docuoria.dll` and required NuGet packages
+   (`PdfPig`, `Tabula`, `CsvHelper`, `pythonnet`, `Microsoft.Extensions.Hosting` /
+   `DependencyInjection` / `Http`).
+2. Exposes `Cli.Require / Cli.Get / Cli.Has` for argument parsing (renamed from
+   `Args` to avoid shadowing the `dotnet-script` global of the same name).
+3. Builds a Generic Host via `ScriptHost.CreateHost(args, includeStore: bool)` which
+   wires `AddDocuoriaEngine`, `AddBuiltInMatchRules`, the CSV/JSON output
+   generators, and (optionally) the template store selected by the `--store-path` /
+   `--store-url` / `--store-key` flags.
+4. Provides `JsonOut.Write` / `JsonOut.Error` writers backed by
+   `DocuoriaJsonOptions.Default` and a `LoadPdf(path)` helper that exits with
+   `pdf-not-found` when the input is missing.
+
+Scripts must declare `#nullable enable` after `#load "_common.csx"` because the
+nullable context does not propagate across `#load` boundaries.