parseforge 0.2.1__tar.gz → 0.2.3__tar.gz

parseforge-0.2.3/PKG-INFO

@@ -0,0 +1,208 @@
+Metadata-Version: 2.4
+Name: parseforge
+Version: 0.2.3
+Summary: LLM-driven pipeline that forges, validates, and promotes TextFSM templates from network CLI output
+Author-email: Tuyen Mathew Duong <tuyen@geekstrident.com>
+Maintainer-email: Tuyen Mathew Duong <tuyen@geekstrident.com>
+License: MIT
+Keywords: textfsm,network automation,cli parsing,template generator,ai,llm,netmiko,device output parsing
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: System Administrators
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Software Development :: Code Generators
+Classifier: Topic :: Text Processing
+Classifier: Topic :: Utilities
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.1
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: textfsm>=1.1.0
+Requires-Dist: anthropic>=0.40.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: textfsm-ai>=0.5.1
+Provides-Extra: sampling
+Requires-Dist: netmiko>=4.0; extra == "sampling"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Requires-Dist: black>=24.0.0; extra == "dev"
+Requires-Dist: mypy>=1.10.0; extra == "dev"
+Dynamic: license-file
+
+# ParseForge
+
+LLM-driven pipeline that forges, cross-validates, and promotes [TextFSM](https://github.com/google/textfsm)
+templates from network device CLI output.
+
+Full design plan: [SPEC.md](SPEC.md).
+
+## Status
+
+Early beta. The full pipeline is implemented and tested end to end — naming,
+sampling, generation, self-validation, integration (output-schema group/variant
+clustering), promotion (auto and human-reviewed), and drift monitoring — and
+wired into the CLI below. A few things are intentionally not there yet:
+
+- **`USER_REVIEWED` promotion has a library entry point but no CLI command**
+  (`promotion.promote_user_reviewed()` works today; there's no
+  `parseforge promotion --mode user-reviewed` yet). Deferred until real
+  human-reviewed cases exist to show what a CLI/config shape for a list of
+  case/suffix/gate requests should actually look like, rather than guessing
+  ahead of need.
+- **Batch sampling mode** (collect several samples per command before
+  generating, SPEC.md §4) is designed but not built — the simpler
+  per-command loop mode is the only one implemented.
+- **One sampling connector** (Netmiko/SSH). The CLI's `--connector` registry
+  is built to hold more without a redesign, but nothing else is wired in yet.
+
+## Development
+
+```
+pip install -e ".[dev,sampling]"
+pytest
+```
+
+## CLI
+
+Three kinds of commands: single lookups (`name`, `check`), one-shot inspection with
+no persistence (`generate-template`, `canonical`/`readable`/`recognizers`), and the
+config-driven `trial` → `integration` → `promotion` workflow that runs the full
+pipeline end to end (see [Quickstart: end to end](#quickstart-end-to-end) below).
+
+**Naming** — resolve a raw CLI command to its canonical cli-name (cached after the
+first call, per SPEC §2):
+```
+parseforge name --vendor cisco --family catalyst9200 --os ios-xe --version 17.9.1 \
+  show interface GE1.1 status
+```
+Still needs an LLM provider on a cache miss — `--provider` defaults to `anthropic`,
+and `--api-key` falls back to that provider's own env var (`ANTHROPIC_API_KEY`/
+`DEEPSEEK_API_KEY`). A cache hit (a command already seen before) never touches the
+LLM, so no key is needed at all in that case.
+
+**`check`** — validate a connector or provider before spending time/tokens on a real
+run. With neither `--env` nor explicit connection flags, prints what a connector needs
+instead of attempting a connection:
+```
+parseforge check --connector netmiko --env cisco
+parseforge check --provider anthropic
+```
+`--env=<name>` reads `<NAME>_SANDBOX_HOST/USERNAME/PASSWORD/DEVICE_TYPE` from the
+environment (the same convention the test suite uses for `CISCO_SANDBOX_*`).
+
+**`generate-template`** — one-shot template generation, no trial persisted under
+`trials/`:
+```
+parseforge generate-template --sample-file sample.txt \
+  --provider anthropic --api-key $ANTHROPIC_API_KEY --model claude-haiku-4-5-20251001
+```
+Also accepts `--connector`/`--cmdline` (live sample) or `--config <file>` in place of
+`--sample-file`; add `--out <dir>` to also write `template.textfsm`/`readable-dsl.txt`/
+`recognizers.txt` to disk. `parseforge init-generate-template-config [--out <file>]`
+writes a placeholder for that `--config` file, ready to fill in.
+
+**`canonical` / `readable` / `recognizers`** — inspect an existing template file. No
+LLM call; `--sample` is required to build the example records `textfsm-ai`'s DSL
+compiler needs:
+```
+parseforge canonical template.textfsm --sample sample.txt
+parseforge readable template.textfsm --sample sample.txt
+parseforge recognizers template.textfsm --sample sample.txt
+```
+
+**`run`** — a single full trial (sample -> generate -> self-validate), SPEC §5 steps
+1-7:
+```
+parseforge run --vendor cisco --family catalyst9200 --os ios-xe --version 17.9.1 \
+  --host 10.0.0.1 --username admin --device-type cisco_ios \
+  --provider anthropic --api-key $ANTHROPIC_API_KEY \
+  --model claude-haiku-4-5-20251001 \
+  show clock
+```
+`--provider`/`--api-key`/`--model` are for generation. Naming has its own separate
+`--naming-provider`/`--naming-api-key`/`--naming-model`, defaulting independently
+(`--naming-provider` defaults to `anthropic`; the other two fall back to that
+provider's own env var/default model) — set them explicitly if naming needs a
+different provider than generation.
+
+**`init-trial-config`** — write a placeholder `trial.yaml` to fill in, instead of
+writing one by hand:
+```
+parseforge init-trial-config --out trial.yaml
+```
+
+**`trial`** — every command in a YAML config file, optionally in parallel:
+```
+parseforge trial --config trial.yaml
+```
+```yaml
+vendor: cisco
+family: catalyst9200
+os: ios-xe
+version: 17.9.1
+connector: netmiko
+host: 10.0.0.1
+username: admin
+password: secret
+device_type: cisco_ios
+provider: anthropic
+api_key: sk-...
+model: claude-haiku-4-5-20251001
+commands:
+  - show clock
+  - show version
+user: alice
+workers: 1
+```
+`provider`/`api_key`/`model` are one shared LLM source used for both naming and
+generation. Use the `run` command's separate `--naming-*`/`--generation-*` flags
+instead if a trial actually needs two different providers.
+
+**`integration`** — rebuild `integration/` for every case under `trials/` (SPEC §5
+step 8):
+```
+parseforge integration
+```
+
+**`promotion`** — auto-promote every group that clears its gate (SPEC §5 step 9;
+`AUTO_PROMOTED` mode only — `USER_REVIEWED` has no CLI surface yet):
+```
+parseforge promotion --user alice --threshold 1.0 --min-samples 1
+```
+
+`trial`, `integration`, and `promotion` all default to `paths.DEFAULT_STORE_ROOT`
+(`~/.parseforge/tests`); pass `--path <dir>` to point at a different store root.
+
+### Quickstart: end to end
+
+Run `trial` once per device/command batch (repeat as more devices or commands come
+in — each run only adds new evidence, it never discards prior trials), then
+`integration` and `promotion` any time you want the current evidence reflected in
+`authoritative/`:
+
+```bash
+parseforge trial --config trial.yaml
+parseforge integration
+parseforge promotion --user alice
+```
+
+`integration` rebuilds `integration/reference-summary.json` from every trial
+currently on disk, and `promotion` always refreshes integration itself before
+evaluating any gate — so running `promotion` alone after a `trial` run is enough to
+pick up new evidence; a separate `integration` run is only useful if you want to
+inspect `reference-summary.json` without also promoting.
+
+## Reference
+
+- [Documentation site](https://geeks-trident-llc.github.io/parseforge/) ([source](./docs/index.md))
+- [SPEC.md](SPEC.md) — full design plan and open questions

parseforge-0.2.3/README.md

@@ -0,0 +1,167 @@
+# ParseForge
+
+LLM-driven pipeline that forges, cross-validates, and promotes [TextFSM](https://github.com/google/textfsm)
+templates from network device CLI output.
+
+Full design plan: [SPEC.md](SPEC.md).
+
+## Status
+
+Early beta. The full pipeline is implemented and tested end to end — naming,
+sampling, generation, self-validation, integration (output-schema group/variant
+clustering), promotion (auto and human-reviewed), and drift monitoring — and
+wired into the CLI below. A few things are intentionally not there yet:
+
+- **`USER_REVIEWED` promotion has a library entry point but no CLI command**
+  (`promotion.promote_user_reviewed()` works today; there's no
+  `parseforge promotion --mode user-reviewed` yet). Deferred until real
+  human-reviewed cases exist to show what a CLI/config shape for a list of
+  case/suffix/gate requests should actually look like, rather than guessing
+  ahead of need.
+- **Batch sampling mode** (collect several samples per command before
+  generating, SPEC.md §4) is designed but not built — the simpler
+  per-command loop mode is the only one implemented.
+- **One sampling connector** (Netmiko/SSH). The CLI's `--connector` registry
+  is built to hold more without a redesign, but nothing else is wired in yet.
+
+## Development
+
+```
+pip install -e ".[dev,sampling]"
+pytest
+```
+
+## CLI
+
+Three kinds of commands: single lookups (`name`, `check`), one-shot inspection with
+no persistence (`generate-template`, `canonical`/`readable`/`recognizers`), and the
+config-driven `trial` → `integration` → `promotion` workflow that runs the full
+pipeline end to end (see [Quickstart: end to end](#quickstart-end-to-end) below).
+
+**Naming** — resolve a raw CLI command to its canonical cli-name (cached after the
+first call, per SPEC §2):
+```
+parseforge name --vendor cisco --family catalyst9200 --os ios-xe --version 17.9.1 \
+  show interface GE1.1 status
+```
+Still needs an LLM provider on a cache miss — `--provider` defaults to `anthropic`,
+and `--api-key` falls back to that provider's own env var (`ANTHROPIC_API_KEY`/
+`DEEPSEEK_API_KEY`). A cache hit (a command already seen before) never touches the
+LLM, so no key is needed at all in that case.
+
+**`check`** — validate a connector or provider before spending time/tokens on a real
+run. With neither `--env` nor explicit connection flags, prints what a connector needs
+instead of attempting a connection:
+```
+parseforge check --connector netmiko --env cisco
+parseforge check --provider anthropic
+```
+`--env=<name>` reads `<NAME>_SANDBOX_HOST/USERNAME/PASSWORD/DEVICE_TYPE` from the
+environment (the same convention the test suite uses for `CISCO_SANDBOX_*`).
+
+**`generate-template`** — one-shot template generation, no trial persisted under
+`trials/`:
+```
+parseforge generate-template --sample-file sample.txt \
+  --provider anthropic --api-key $ANTHROPIC_API_KEY --model claude-haiku-4-5-20251001
+```
+Also accepts `--connector`/`--cmdline` (live sample) or `--config <file>` in place of
+`--sample-file`; add `--out <dir>` to also write `template.textfsm`/`readable-dsl.txt`/
+`recognizers.txt` to disk. `parseforge init-generate-template-config [--out <file>]`
+writes a placeholder for that `--config` file, ready to fill in.
+
+**`canonical` / `readable` / `recognizers`** — inspect an existing template file. No
+LLM call; `--sample` is required to build the example records `textfsm-ai`'s DSL
+compiler needs:
+```
+parseforge canonical template.textfsm --sample sample.txt
+parseforge readable template.textfsm --sample sample.txt
+parseforge recognizers template.textfsm --sample sample.txt
+```
+
+**`run`** — a single full trial (sample -> generate -> self-validate), SPEC §5 steps
+1-7:
+```
+parseforge run --vendor cisco --family catalyst9200 --os ios-xe --version 17.9.1 \
+  --host 10.0.0.1 --username admin --device-type cisco_ios \
+  --provider anthropic --api-key $ANTHROPIC_API_KEY \
+  --model claude-haiku-4-5-20251001 \
+  show clock
+```
+`--provider`/`--api-key`/`--model` are for generation. Naming has its own separate
+`--naming-provider`/`--naming-api-key`/`--naming-model`, defaulting independently
+(`--naming-provider` defaults to `anthropic`; the other two fall back to that
+provider's own env var/default model) — set them explicitly if naming needs a
+different provider than generation.
+
+**`init-trial-config`** — write a placeholder `trial.yaml` to fill in, instead of
+writing one by hand:
+```
+parseforge init-trial-config --out trial.yaml
+```
+
+**`trial`** — every command in a YAML config file, optionally in parallel:
+```
+parseforge trial --config trial.yaml
+```
+```yaml
+vendor: cisco
+family: catalyst9200
+os: ios-xe
+version: 17.9.1
+connector: netmiko
+host: 10.0.0.1
+username: admin
+password: secret
+device_type: cisco_ios
+provider: anthropic
+api_key: sk-...
+model: claude-haiku-4-5-20251001
+commands:
+  - show clock
+  - show version
+user: alice
+workers: 1
+```
+`provider`/`api_key`/`model` are one shared LLM source used for both naming and
+generation. Use the `run` command's separate `--naming-*`/`--generation-*` flags
+instead if a trial actually needs two different providers.
+
+**`integration`** — rebuild `integration/` for every case under `trials/` (SPEC §5
+step 8):
+```
+parseforge integration
+```
+
+**`promotion`** — auto-promote every group that clears its gate (SPEC §5 step 9;
+`AUTO_PROMOTED` mode only — `USER_REVIEWED` has no CLI surface yet):
+```
+parseforge promotion --user alice --threshold 1.0 --min-samples 1
+```
+
+`trial`, `integration`, and `promotion` all default to `paths.DEFAULT_STORE_ROOT`
+(`~/.parseforge/tests`); pass `--path <dir>` to point at a different store root.
+
+### Quickstart: end to end
+
+Run `trial` once per device/command batch (repeat as more devices or commands come
+in — each run only adds new evidence, it never discards prior trials), then
+`integration` and `promotion` any time you want the current evidence reflected in
+`authoritative/`:
+
+```bash
+parseforge trial --config trial.yaml
+parseforge integration
+parseforge promotion --user alice
+```
+
+`integration` rebuilds `integration/reference-summary.json` from every trial
+currently on disk, and `promotion` always refreshes integration itself before
+evaluating any gate — so running `promotion` alone after a `trial` run is enough to
+pick up new evidence; a separate `integration` run is only useful if you want to
+inspect `reference-summary.json` without also promoting.
+
+## Reference
+
+- [Documentation site](https://geeks-trident-llc.github.io/parseforge/) ([source](./docs/index.md))
+- [SPEC.md](SPEC.md) — full design plan and open questions

parseforge-0.2.3/SPEC.md

@@ -0,0 +1,183 @@
+# Network CLI → TextFSM Template Generator — Project Plan & Requirements
+
+## 1. Project & Repo Naming
+
+| Candidate | Repo name (kebab-case) | Rationale |
+|---|---|---|
+| **ParseForge** | `parseforge` | Short, memorable, clearly about forging parsers. Available-sounding, no CLI/network jargon lock-in if you expand beyond Cisco later. |
+| **CLIFsm** | `cli-fsm-forge` | Literal — CLI + FSM (finite state machine, what TextFSM templates are). Good for discoverability if someone searches "textfsm" + "cli". |
+| **ShowParse AI** | `showparse-ai` | Leans into the fact that almost every input is a `show` command. Very descriptive, less generic. |
+| **NetFSM** | `netfsm` | Shortest option. Good if this becomes a pip-installable library/CLI tool people type often. |
+
+**Recommendation:** `parseforge` as the project name, `parseforge` (or `parseforge-textfsm`) as the repo name. It reads well as a CLI tool name (`parseforge run ...`), isn't locked to Cisco/TextFSM specifically if you extend to other vendors or output formats (e.g., ntc-templates-style YAML, TTP, or Genie parsers) later, and is easy to say out loud.
+
+If you want to keep `textfsm-ai` since that's the term you're already using internally/in docs, that's a perfectly reasonable literal alternative — just know it reads as "TextFSM, but AI" rather than as a product name, which is fine for an internal tool but less distinctive if this is ever open-sourced.
+
+---
+
+## 2. CLI-to-Name Translation
+
+**Rule:** Tokenize the full command string. Classify each token as **literal** (fixed across all invocations of this command type) or **variable** (an identifier — interface name, VLAN ID, IP, hostname, etc.).
+
+**Naming convention — always index variables, even when there's only one:**
+
+```
+show version                         → show-version
+show interface GE1.1 status          → show-interface-var1-status
+show interface GE1.1 counters        → show-interface-var1-counters
+show ip route 10.0.0.0                → show-ip-route-var1
+show bgp neighbors 10.1.1.1 advertised-routes
+                                       → show-bgp-neighbors-var1-advertised-routes
+```
+
+**Why always number, even for a single variable (`var1` not `var`):**
+A naming function that omits the index for single-variable commands has to *already know* the final variable count before naming — but that can change: a newer OS version might add an optional second argument to the same command family, or you might discover a command has two variable slots you didn't notice on first sample. Indexing consistently means:
+- The name-generation code is a pure, stateless function of token position — no lookahead needed.
+- No silent renames/migrations later if a command turns out to have more variables than expected.
+- Multi-variable commands (2+) stay unambiguous by construction — `var1`, `var2` in left-to-right order of appearance.
+
+This differs slightly from the older convention used in the `ntc-templates` community library (which just drops variable tokens entirely rather than placeholding them — e.g. `cisco_ios_show_interfaces.textfsm` regardless of interface name). That works for them because their catalog is hand-curated and collisions are manually resolved. Since your pipeline is LLM-driven and needs to run unattended, explicit indexed placeholders are safer and self-documenting.
+
+**Edge case — literal tokens that look variable-ish:** keywords like `brief`, `status`, `detail`, `counters`, `summary` are still literal (they're fixed sub-command keywords, not user-supplied values) and stay as literal tokens in the name.
+
+---
+
+## 3. Storage Layout — Three-Tier Promotion
+
+This mirrors the candidate → staging → production pattern used for ML model registries, applied to templates: **trials** (raw, unreviewed attempts) → **integration** (cross-validated candidates, pending approval) → **authoritative** (approved, in production use, drift-monitored).
+
+Shared path prefix under all three tiers:
+```
+<vendor>/<device-family>/<os>/<cli-name>/
+```
+e.g. `cisco/catalyst9200/ios-xe/show-interface-var1-status/`
+
+Note: use `ios-xe` / `nx-os` (hyphenated) rather than `xe` / `ios` alone once multiple Cisco OS families share the tree — `xe` alone is ambiguous without the `ios-` prefix once IOS classic, IOS XE, and IOS XR all live under `cisco/`.
+
+Deliberately no `<version>` segment in the path: a cli-name's output structure usually doesn't change across minor OS versions, and when it legitimately does, that's exactly the variance §5 step 8's group clustering is built to catch — lumping versions together under one path gives more evidence per group instead of silently fragmenting it across per-version directories. The OS version a trial was sampled from is recorded per trial instead, in that trial's `summary.json` (`command_info.version`).
+
+### 3.1 `trials/`
+
+```
+trials/<vendor>/<family>/<os>/<cli-name>/
+  <yyyymmdd-HHMMSS-shortid>/
+    samples/
+      sample.txt
+    derive/
+      llm-template.textfsm
+      template.textfsm
+      readable-dsl.txt
+      recognizers.txt
+    summary.json
+```
+Keep the timestamp+shortid directories (not `result1..N`) — chronological ordering and collision-safety in batch mode come for free, and sequential numbers throw both away.
+
+| File | Purpose |
+|---|---|
+| `samples/sample.txt` | Raw CLI output sample fed to the LLM |
+| `derive/llm-template.textfsm` | Template as extracted from the LLM response, pre-cleanup |
+| `derive/template.textfsm` | Cleaned, DSL-compiled TextFSM template candidate (`.textfsm` extension — recognized by TextFSM tooling/linters, unlike `textfsm.template`) |
+| `derive/readable-dsl.txt` | Human-readable description of what the template captures |
+| `derive/recognizers.txt` | Heuristics/signatures for detecting this output type at runtime |
+| `summary.json` | Everything else about the run: `created_at`/`ended_at`/`duration_ms`, `passed`, `error` (when `passed` is false), `metadata` (project/username/email/description), `command_info` (vendor/family/os/version/device_type/command), `usage` (naming + generation token counts/cost), and `provider_info` (the generation provider/model) |
+
+### 3.2 `integration/` (no human review yet)
+
+```
+integration/<vendor>/<family>/<os>/<cli-name>/
+  reference.json              ← every group's clustering evidence for this cli-name
+  group1-template1.textfsm    ← a distinct template variant, copied from trials/
+  group1-template2.textfsm    ← another distinct template text within the same group, if any
+  group2-template1.textfsm    ← a different output-schema group entirely (§6 multi-variant)
+
+integration/
+  reference-summary.json      ← project-wide match-rate ratios aggregated across every cli-name
+```
+
+**How grouping actually works — don't assume one winner.** Every trial whose own `summary.json` says `passed` is clustered by the field-key signature of its parsed records — the columns `derive/template.textfsm` actually produces against `samples/sample.txt` — not by exact template text; two templates that produce the same fields land in the same group even if their regex differs. Within a group, byte-identical template texts are further merged into "variants," so `reference.json` can report `exact_template_count`/`exact_records_count` per variant. This replaces self-validation-only trust: a template that only works against its own source sample still gets clustered on its own field signature, but its group's `ratio_of_passed` (in `reference-summary.json`) reads low if no other trial agrees with it — the overfit signal a human or `promotion.py`'s gate needs, in place of a single hand-picked "winner." The original single-winner design's `artifact/` subdirectory (aggregated `records.json` across cross-validated samples, plus a diff/summary of samples the winner failed on) doesn't carry over: there's no separate evidence folder and no per-sample mismatch diff, since there's no longer a single winner for a sample to fail against — a sample that doesn't fit any existing group's field signature just becomes (or joins) a different group instead.
+
+`reference.json` is `{total_case_count, total_passed_case_count, groups: {group_id: {keys, sample_path, group_case_count, variants: {variant_id: {template_path, exact_template_count, exact_records_count}}}}}` — a full rebuild every time integration runs, not an incremental diff, so it's always a fresh snapshot of every trial currently under `trials/.../<cli-name>/`.
+
+`reference-summary.json` turns those counts into ratios for every cli-name in one report: `ratio_of_total` (a group/variant's share of *every* trial attempted, diluted by raw generation failures) and `ratio_of_passed` (its share of only the trials that actually passed). `promotion.py`'s gate evaluates on `ratio_of_passed`, since promotion should judge template consistency, not raw pipeline reliability.
+
+**Grouping and `recognizers.txt` are two different mechanisms, not one.** §6's note about `recognizers.txt` enabling *one-of-many matching* is about a runtime picking the right authoritative template for new input; it plays no part in how integration itself tells variants apart. Group clustering here never reads `recognizers.txt` — it re-parses `derive/template.textfsm` against `samples/sample.txt` and clusters by the resulting field-key signature, purely evidence-gathering, before any template is authoritative. `recognizers.txt` is still generated per trial and carried through to `authoritative/` per-variant (§3.3), but what consumes it to actually dispatch between promoted variants at runtime is out of parseforge's own scope — still an open question (§7).
+
+### 3.3 `authoritative/` (approved via human review, or confidence-gated auto-promotion)
+
+```
+authoritative/<vendor>/<family>/<os>/<cli-name>/
+  template.textfsm             ← current approved template (the primary variant)
+  template-v2.textfsm          ← an additional simultaneously-valid variant (§6), if any
+  template-v2-<suffix>.textfsm ← a USER_REVIEWED snapshot of that variant, kept alongside its current version
+  recognizers.txt              ← recognizer signature, one per template, same suffix rule
+  data/
+    sample.txt, records.json   ← the sample + parsed output behind each template, same suffix rule
+  golden.hash                  ← sha256 of the most recently promoted template, regardless of variant
+  artifact.json                ← who/when/mode/match-rate/source of the most recent promotion
+  drift-log.json               ← rolling match rate over time per variant, against live production samples
+  history/
+    template-<yyyymmdd-HHMMSS>-<shortid>.textfsm   ← prior content, archived whenever a promotion overwrites it with something different
+
+authoritative/
+  authoritative-log.json       ← project-wide, append-only: every promotion event ever, across every cli-name
+  authoritative-summary.json   ← project-wide snapshot of the most recent promotion run
+```
+
+No per-variant subdirectory — every simultaneously-valid template for a `cli-name` (§6, hardware/firmware variance) lives directly in this one flat directory, distinguished by filename. The first-discovered variant owns the unsuffixed "current version" names; each additional variant owns a stable, permanent `template-v2.textfsm`, `template-v3.textfsm`, ... derived from its own group id and never renumbered. `golden.hash` and `artifact.json` are the one exception to per-variant filenames: both are singular and unsuffixed, always reflecting whichever promotion happened most recently regardless of which variant triggered it.
+
+Promotion defaults to human review, but can be confidence-gated per variant: if a variant's match rate — against only the trials that actually passed, not diluted by raw generation failures — is at or near 100% with enough samples, auto-promote it; otherwise it's queued for review. Two modes cover this: **AUTO_PROMOTED** walks every case and promotes every qualifying variant unsuffixed; **USER_REVIEWED** is scoped to caller-reviewed `(case, suffix)` requests, writing a suffixed snapshot alongside whatever that variant's current auto-promoted files already are, never replacing them. This isn't a replacement for review — it's a filter that keeps the easy, unambiguous cases from waiting on a person while still requiring a human on anything uncertain.
+
+**Drift detection** runs an authoritative template against new production samples on an ongoing basis, tracking a rolling match rate per variant in that cli-name's `drift-log.json`. When the rate drops below threshold, that failing sample is written into `trials/.../<cli-name>/` as a new run, which re-enters the pipeline at §5 step 4 and works its way back through `integration/` to a possible new `authoritative/` version — closing the loop rather than treating drift as a one-off alert.
+
+---
+
+## 4. Execution Modes
+
+**Mode 1 — Batch (gather-then-generate):**
+- Pro: LLM can be shown patterns across multiple samples of the *same* command at once (e.g., three different interface states) in one shot, likely improving template robustness on the first try.
+- Con: Slower feedback loop — you don't know command #3 has a connectivity problem until after collecting #1–#10.
+
+**Mode 2 — Loop (sample → generate, per command):**
+- Pro: Fails fast and isolates problems per-command; simpler to debug; easier to parallelize across commands.
+- Con: Each template is generated from a single sample unless you explicitly loop multiple times per command and merge.
+
+**Recommendation:** Build Mode 2 first as the MVP — it's the simpler pipeline and gives you per-command `summary.json` results immediately. Add Mode 1 as a config flag afterward that changes only the *sampling* stage (collect N samples per command before invoking the LLM) — the generation, storage, and validation stages stay identical between modes if you design sampling as a separable stage up front.
+
+---
+
+## 5. Pipeline Steps
+
+1. **Input intake** — device OS/version/family, auth, command list, mode selection.
+2. **Name generation** — tokenize each command → canonical `cli-name` per §2.
+3. **Path resolution** — compute `<vendor>/<device-family>/<os>/<cli-name>/` per §3.
+4. **Sampling** — connect (Netmiko/similar), run command(s), capture raw output → `trials/.../<run-id>/samples/sample.txt`.
+5. **Generation** — send `sample.txt` (+ prior context if Mode 1) to LLM → `derive/llm-template.textfsm`, token usage recorded in `summary.json`.
+6. **Extraction & cleanup** — pull template from response → cleaned `derive/template.textfsm`, plus `derive/readable-dsl.txt` and `derive/recognizers.txt`.
+7. **Self-validation** — run `derive/template.textfsm` against its own `samples/sample.txt`; record `passed`/`error` in `summary.json`.
+8. **Integration selection** — cluster every passed trial's `derive/template.textfsm` in `trials/.../<cli-name>/` by output-schema signature, cross-validated against every known `samples/sample.txt` for that `cli-name`; write `integration/.../reference.json` and one `group#-template#.textfsm` per distinct variant.
+9. **Authoritative promotion** — auto-promote if the winning candidate clears the confidence threshold across all known samples; otherwise queue for human review. On approval, copy to `authoritative/.../template.textfsm` (or `template-v2.textfsm`, ... for an additional variant), archive the prior content to `history/` if it differs, and write `golden.hash`, `artifact.json`, and `authoritative-log.json`.
+10. **Drift monitoring** — continuously run the authoritative template against new production samples, logging match rate to `drift-log.json`. On breach, feed the failing sample back into `trials/` (→ step 4) to generate a replacement candidate.
+11. **Repeat/loop or batch-complete** depending on mode.
+
+---
+
+## 6. Design Notes: Is Three-Tier Enough?
+
+**Yes, for the promotion pipeline itself.** `trials → integration → authoritative` covers the full lifecycle a template needs to go through to earn trust: generate, cross-validate, approve. Adding a fourth *promotion* tier (e.g. a separate "staging" between integration and authoritative) wouldn't buy you anything distinct — integration already *is* the staging step, since nothing in it is live yet.
+
+Where a fourth tier is tempting but better handled as **metadata instead of a new directory layer:**
+
+- **Quarantine / stale-but-still-serving:** when drift is detected, the authoritative template is still the one in production use — you don't want to yank it out mid-flight while a replacement works its way through the pipeline. Rather than a new top-level tier, this is a `status` field in `drift-log.json` (`ok` / `drifting` / `superseded`) on the existing authoritative entry. The template keeps serving; the status just flags that a replacement is in flight.
+- **Archived / deprecated (EOL device-OS combos):** for template families you no longer actively maintain (e.g. an OS version taken out of production fleet-wide), this is also a status flag rather than a directory move — `history/` already retains every prior version, so "archived" just means `authoritative-log.json` stops getting new entries for that cli-name, not that files need to relocate.
+- **Multiple simultaneously-valid templates for one `cli-name`:** if a command's output legitimately varies by hardware config (e.g. a chassis with vs. without an optional module) rather than by drift, that's not a tier problem — it's a case for storing *multiple* authoritative templates under the same `cli-name` path with distinct `recognizers.txt` signatures, and letting runtime dispatch pick the right one. Worth flagging now so `recognizers.txt` is designed to support one-of-many matching from the start rather than assuming exactly one template per `cli-name`.
+
+**Bottom line:** keep the three directories; push the "in-between" states (drifting-but-live, archived, multi-variant) into metadata/status fields on the existing tiers. Adding more top-level directories per edge case leads to directory sprawl without adding real distinctions in *how* a template is used.
+
+---
+
+## 7. Open Questions for Next Iteration
+
+- Do you want a **registry/index file** (e.g. `catalog.json`) at the repo root listing every `<vendor>/<family>/<os>/<cli-name>` combination that exists, plus its authoritative status, for fast lookup without walking the filesystem?
+- What **confidence threshold** (match-rate %, sample count minimum) should gate auto-promotion vs. human review in step 9 — worth making this configurable per-project rather than hardcoded?
+- Should `recognizers.txt` support **one-of-many matching** from day one (per the multi-variant note in §6), or is that a v2 concern?
+- What's the **LLM provider/model** for generation — worth pinning per-project so `summary.json`'s usage costs are comparable across runs?

parseforge-0.2.3/parseforge/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.3"