parseforge 0.2.1__tar.gz → 0.2.2__tar.gz

{parseforge-0.2.1/parseforge.egg-info → parseforge-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: parseforge
-Version: 0.2.1
+Version: 0.2.2
 Summary: LLM-driven pipeline that forges, validates, and promotes TextFSM templates from network CLI output
 Author-email: Tuyen Mathew Duong <tuyen@geekstrident.com>
 Maintainer-email: Tuyen Mathew Duong <tuyen@geekstrident.com>
@@ -26,6 +26,9 @@ License-File: LICENSE
 Requires-Dist: click>=8.1
 Requires-Dist: PyYAML>=6.0
 Requires-Dist: textfsm>=1.1.0
+Requires-Dist: anthropic>=0.40.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: textfsm-ai>=0.5.1
 Provides-Extra: sampling
 Requires-Dist: netmiko>=4.0; extra == "sampling"
 Provides-Extra: dev
@@ -55,11 +58,12 @@ parseforge/
   promotion.py    authoritative promotion gate + drift status (SPEC §3.3, §5 steps 9-10)
   pipeline.py     orchestrates the above; Mode LOOP (MVP) vs Mode BATCH (SPEC §4)
   cli/            `parseforge` command-line entry point
-
-store/            runtime template tree: trials/ -> integration/ -> authoritative/
-                  (generated at runtime, not checked in — see .gitignore)
 ```
 
+The runtime template tree (trials/ -> integration/ -> authoritative/) is generated at
+`~/.parseforge/tests` by default (see `paths.DEFAULT_STORE_ROOT`), not checked into
+this repo.
+
 `naming.py` and `paths.py` are implemented and tested. `sampling.py`, `generation.py`,
 `validation.py`, `promotion.py`, and `pipeline.py` are stub interfaces — the SPEC.md
 section cited in each docstring is the next thing to implement.

{parseforge-0.2.1 → parseforge-0.2.2}/README.md

@@ -17,11 +17,12 @@ parseforge/
   promotion.py    authoritative promotion gate + drift status (SPEC §3.3, §5 steps 9-10)
   pipeline.py     orchestrates the above; Mode LOOP (MVP) vs Mode BATCH (SPEC §4)
   cli/            `parseforge` command-line entry point
-
-store/            runtime template tree: trials/ -> integration/ -> authoritative/
-                  (generated at runtime, not checked in — see .gitignore)
 ```
 
+The runtime template tree (trials/ -> integration/ -> authoritative/) is generated at
+`~/.parseforge/tests` by default (see `paths.DEFAULT_STORE_ROOT`), not checked into
+this repo.
+
 `naming.py` and `paths.py` are implemented and tested. `sampling.py`, `generation.py`,
 `validation.py`, `promotion.py`, and `pipeline.py` are stub interfaces — the SPEC.md
 section cited in each docstring is the next thing to implement.

{parseforge-0.2.1 → parseforge-0.2.2}/SPEC.md

@@ -48,16 +48,18 @@ This mirrors the candidate → staging → production pattern used for ML model
 
 Shared path prefix under all three tiers:
 ```
-<vendor>/<device-family>/<os>/<version>/<cli-name>/
+<vendor>/<device-family>/<os>/<cli-name>/
 ```
-e.g. `cisco/catalyst9200/ios-xe/17.9.1/show-interface-var1-status/`
+e.g. `cisco/catalyst9200/ios-xe/show-interface-var1-status/`
 
 Note: use `ios-xe` / `nx-os` (hyphenated) rather than `xe` / `ios` alone once multiple Cisco OS families share the tree — `xe` alone is ambiguous without the `ios-` prefix once IOS classic, IOS XE, and IOS XR all live under `cisco/`.
 
+Deliberately no `<version>` segment in the path: a cli-name's output structure usually doesn't change across minor OS versions, and when it legitimately does, that's exactly the variance §5 step 8's group clustering is built to catch — lumping versions together under one path gives more evidence per group instead of silently fragmenting it across per-version directories. The OS version a trial was sampled from is recorded per trial instead, in that trial's `summary.json` (`command_info.version`).
+
 ### 3.1 `trials/`
 
 ```
-trials/<vendor>/<family>/<os>/<version>/<cli-name>/
+trials/<vendor>/<family>/<os>/<cli-name>/
   <yyyymmdd-HHMMSS-shortid>/
     input.txt
     raw-llm-response.txt
@@ -90,7 +92,7 @@ Keep the timestamp+shortid directories (not `result1..N`) — chronological orde
 ### 3.2 `integration/` (no human review yet)
 
 ```
-integration/<vendor>/<family>/<os>/<version>/<cli-name>/
+integration/<vendor>/<family>/<os>/<cli-name>/
   common-result/
     template.textfsm       ← the winning candidate, copied from trials/
     selection-report.json  ← which trials were considered, why this one won
@@ -106,7 +108,7 @@ integration/<vendor>/<family>/<os>/<version>/<cli-name>/
 ### 3.3 `authoritative/` (approved via human review, or confidence-gated auto-promotion)
 
 ```
-authoritative/<vendor>/<family>/<os>/<version>/<cli-name>/
+authoritative/<vendor>/<family>/<os>/<cli-name>/
   template.textfsm          ← current approved template
   artifact/                  ← carried over from the winning integration candidate
   history/
@@ -139,7 +141,7 @@ Promotion from `integration/common-result` to `authoritative/` should default to
 
 1. **Input intake** — device OS/version/family, auth, command list, mode selection.
 2. **Name generation** — tokenize each command → canonical `cli-name` per §2.
-3. **Path resolution** — compute `<vendor>/<device-family>/<os>/<version>/<cli-name>/` per §3.
+3. **Path resolution** — compute `<vendor>/<device-family>/<os>/<cli-name>/` per §3.
 4. **Sampling** — connect (Netmiko/similar), run command(s), capture raw output → `trials/.../<run-id>/input.txt`.
 5. **Generation** — send `input.txt` (+ prior context if Mode 1) to LLM → `raw-llm-response.txt`, `usage.txt`.
 6. **Extraction & cleanup** — pull template from response → `raw-template` → cleaned `template.textfsm`.
@@ -167,7 +169,7 @@ Where a fourth tier is tempting but better handled as **metadata instead of a ne
 
 ## 7. Open Questions for Next Iteration
 
-- Do you want a **registry/index file** (e.g. `catalog.json`) at the repo root listing every `<vendor>/<family>/<os>/<version>/<cli-name>` combination that exists, plus its authoritative status, for fast lookup without walking the filesystem?
+- Do you want a **registry/index file** (e.g. `catalog.json`) at the repo root listing every `<vendor>/<family>/<os>/<cli-name>` combination that exists, plus its authoritative status, for fast lookup without walking the filesystem?
 - What **confidence threshold** (match-rate %, sample count minimum) should gate auto-promotion vs. human review in step 9 — worth making this configurable per-project rather than hardcoded?
 - Should `recognizers.txt` support **one-of-many matching** from day one (per the multi-variant note in §6), or is that a v2 concern?
 - What's the **LLM provider/model** for generation — worth pinning per-project so `usage.txt` costs are comparable across runs?

parseforge-0.2.2/parseforge/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.2.2"

parseforge-0.2.2/parseforge/cli/main.py

@@ -0,0 +1,202 @@
+"""parseforge CLI entry point."""
+
+from __future__ import annotations
+
+import click
+
+from parseforge import naming
+
+_BUILDERS: dict[str, type[naming.RegexBuilder]] = {
+    "anthropic": naming.AnthropicRegexBuilder,
+    "deepseek": naming.DeepSeekRegexBuilder,
+}
+
+
+def _build_regex_builder(
+    provider: str, api_key: str | None, model: str | None
+) -> naming.RegexBuilder:
+    kwargs: dict[str, str] = {}
+    if api_key is not None:
+        kwargs["api_key"] = api_key
+    if model is not None:
+        kwargs["model"] = model
+    return _BUILDERS[provider](**kwargs)
+
+
+@click.group()
+@click.version_option(package_name="parseforge")
+def main() -> None:
+    """ParseForge — forge, validate, and promote TextFSM templates from CLI output."""
+
+
+@main.command("name")
+@click.option("--vendor", required=True)
+@click.option("--family", required=True)
+@click.option("--os", "os_", required=True)
+@click.option("--version", required=True)
+@click.option(
+    "--provider",
+    type=click.Choice(sorted(_BUILDERS)),
+    default="anthropic",
+    show_default=True,
+)
+@click.option(
+    "--api-key",
+    default=None,
+    help="Provider API key. Defaults to that provider's own API key environment "
+    "variable (ANTHROPIC_API_KEY, DEEPSEEK_API_KEY); only needed on a cache miss.",
+)
+@click.option(
+    "--model",
+    default=None,
+    help="Defaults to the selected provider's own default model.",
+)
+@click.argument("command", nargs=-1, required=True)
+def name_cmd(
+    vendor: str,
+    family: str,
+    os_: str,
+    version: str,
+    provider: str,
+    api_key: str | None,
+    model: str | None,
+    command: tuple[str, ...],
+) -> None:
+    """Print the canonical cli-name for a raw CLI COMMAND.
+
+    Looked up in the local cache first; only sent to the selected --provider
+    on a cache miss.
+    """
+    context = naming.CliContext(vendor=vendor, family=family, os=os_, version=version)
+    builder = _build_regex_builder(provider, api_key, model)
+    click.echo(naming.cli_name(" ".join(command), context, builder=builder))
+
+
+@main.command("run")
+@click.option("--vendor", required=True)
+@click.option("--family", required=True)
+@click.option("--os", "os_", required=True)
+@click.option("--version", required=True)
+@click.option("--host", required=True, help="Device host/IP to sample from.")
+@click.option("--username", required=True)
+@click.option(
+    "--password",
+    required=True,
+    envvar="PARSEFORGE_DEVICE_PASSWORD",
+    help="Device password. Defaults to the PARSEFORGE_DEVICE_PASSWORD "
+    "environment variable.",
+)
+@click.option(
+    "--device-type",
+    required=True,
+    help='Netmiko device_type, e.g. "cisco_ios".',
+)
+@click.option(
+    "--naming-provider",
+    type=click.Choice(sorted(_BUILDERS)),
+    default="anthropic",
+    show_default=True,
+)
+@click.option(
+    "--naming-api-key",
+    default=None,
+    help="API key for the naming LLM call. Defaults to that provider's own "
+    "API key environment variable; only needed on a cache miss.",
+)
+@click.option(
+    "--naming-model",
+    default=None,
+    help="Defaults to the naming provider's own default model.",
+)
+@click.option(
+    "--generation-provider",
+    required=True,
+    help="LLM provider for template generation (textfsm-ai's own registry, "
+    'e.g. "anthropic", "deepseek").',
+)
+@click.option(
+    "--generation-api-key",
+    required=True,
+    help="API key for the template-generation LLM call.",
+)
+@click.option(
+    "--generation-model",
+    required=True,
+    help="Model for the template-generation LLM call.",
+)
+@click.option(
+    "--store-root",
+    default=None,
+    help="Root directory for trial output. Defaults to ~/.parseforge/tests.",
+)
+@click.option("--project", default=None)
+@click.option("--email", default=None)
+@click.option("--description", default=None)
+@click.argument("command", nargs=-1, required=True)
+def run_cmd(
+    vendor: str,
+    family: str,
+    os_: str,
+    version: str,
+    host: str,
+    username: str,
+    password: str,
+    device_type: str,
+    naming_provider: str,
+    naming_api_key: str | None,
+    naming_model: str | None,
+    generation_provider: str,
+    generation_api_key: str,
+    generation_model: str,
+    store_root: str | None,
+    project: str | None,
+    email: str | None,
+    description: str | None,
+    command: tuple[str, ...],
+) -> None:
+    """Run a single trial: sample COMMAND from a device, generate a
+    TextFSM template for it, and write the result under a trial
+    directory (SPEC.md §5 steps 1-7)."""
+    from pathlib import Path
+
+    from parseforge import paths
+    from parseforge.pipeline import (
+        LLMProviderConfig,
+        TrialMetadata,
+        run_command_pipeline,
+    )
+    from parseforge.sampling import DeviceConnection
+    from parseforge.sampling.backends import NetmikoSampler
+
+    context = naming.CliContext(vendor=vendor, family=family, os=os_, version=version)
+    connection = DeviceConnection(
+        host=host, username=username, password=password, device_type=device_type
+    )
+    naming_builder = _build_regex_builder(naming_provider, naming_api_key, naming_model)
+    generation_config = LLMProviderConfig(
+        provider=generation_provider, api_key=generation_api_key, model=generation_model
+    )
+    metadata = TrialMetadata(
+        project=project,
+        username=username,
+        email=email,
+        description=description,
+    )
+
+    result = run_command_pipeline(
+        " ".join(command),
+        context,
+        connection,
+        naming_builder,
+        NetmikoSampler(),
+        generation_config,
+        store_root=Path(store_root) if store_root else paths.DEFAULT_STORE_ROOT,
+        metadata=metadata,
+    )
+    click.echo(f"cli_name : {result.cli_name}")
+    click.echo(f"passed   : {result.passed}")
+    click.echo(f"run_dir  : {result.run_dir}")
+
+
+if __name__ == "__main__":
+    main()

parseforge-0.2.2/parseforge/generation.py

@@ -0,0 +1,112 @@
+"""Generation stage — LLM call, extraction, cleanup, and DSL compile
+(SPEC.md §5 steps 5-6), delegated to textfsm-ai's delivery pipeline
+rather than reimplemented here.
+
+textfsm-ai's run_pipeline() already covers extraction + cleanup + DSL
+compilation (canonical template, readable DSL, recognizers) in one
+call — see https://github.com/Geeks-Trident-LLC/textfsm-ai. Always run
+in debug mode with JSON output, for full usage/timing/pipeline detail
+(see GenerationResult.raw).
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from typing import Any
+
+from textfsm_ai import run_pipeline
+
+_SENSITIVE_KEYS = {"api_key"}
+
+
+def _redact(value: Any) -> Any:
+    """Strip credential-shaped fields from a parsed debug payload.
+
+    textfsm-ai's Serializable.to_dict()/to_json() run plain
+    dataclasses.asdict() — masking (mask_middle()) only happens in its
+    *text*-mode output, not JSON. Debug mode's JSON embeds the raw API
+    key unmasked, so it must be stripped before this result is stored
+    or logged anywhere.
+    """
+    if isinstance(value, dict):
+        return {
+            k: "<redacted>" if k in _SENSITIVE_KEYS else _redact(v)
+            for k, v in value.items()
+        }
+    if isinstance(value, list):
+        return [_redact(v) for v in value]
+    return value
+
+
+@dataclass(frozen=True)
+class TokenUsage:
+    input_tokens: int
+    output_tokens: int
+    total_tokens: int
+    estimated_cost: float
+
+
+@dataclass(frozen=True)
+class GenerationResult:
+    """Structured result of generating + DSL-compiling a template from a sample."""
+
+    template: str  # canonical, DSL-compiled TextFSM template
+    raw_template: str  # pre-cleanup template, as the LLM returned it
+    readable_dsl: str
+    recognizers: list[str]
+    records: list[dict[str, str]]
+    usage: TokenUsage
+    duration_ms: float
+    ready: bool
+    reason: str
+    raw: dict[str, Any] = field(repr=False)
+
+
+def generate(
+    sample: str,
+    provider: str,
+    api_key: str,
+    model: str,
+    **kwargs: Any,
+) -> GenerationResult:
+    """Run textfsm-ai's full sample -> template -> DSL-compiled pipeline.
+
+    Always runs in debug mode with JSON output (not overridable — pass
+    ``mode``/``as_json`` in ``kwargs`` and this raises, same as calling
+    ``run_pipeline()`` twice for the same argument would). ``**kwargs``
+    forwards to ``run_pipeline()`` for everything else (``endpoint``,
+    ``region``, ``max_tries``, ...).
+
+    Never raises for a failed generation — check ``.ready``.
+    """
+    result = run_pipeline(
+        sample, provider, api_key, model, mode="debug", as_json=True, **kwargs
+    )
+    debug = _redact(json.loads(result.output))
+
+    usage_data = debug.get("usage") or {}
+    usage = TokenUsage(
+        input_tokens=usage_data.get("input_tokens", 0),
+        output_tokens=usage_data.get("output_tokens", 0),
+        total_tokens=usage_data.get("total_tokens", 0),
+        estimated_cost=usage_data.get("estimated_cost", 0.0),
+    )
+
+    gen_stage = (debug.get("generation_pipeline") or {}).get("last_stage") or {}
+    gen_metadata = gen_stage.get("metadata") or {}
+
+    dsl = (debug.get("dsl_pipeline") or {}).get("dsl") or {}
+
+    return GenerationResult(
+        template=dsl.get("canonical") or "",
+        raw_template=dsl.get("raw_template") or gen_metadata.get("template") or "",
+        readable_dsl=dsl.get("readable") or "",
+        recognizers=dsl.get("recognizers") or [],
+        records=gen_metadata.get("records") or [],
+        usage=usage,
+        duration_ms=debug.get("duration_ms", 0.0),
+        ready=result.passed,
+        reason=result.error,
+        raw=debug,
+    )