labagent 0.1.0__tar.gz → 0.2.0__tar.gz

labagent-0.2.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: labagent
+Version: 0.2.0
+Summary: Open-source CLI and AI agent to launch and tear down hands-on labs for RAG, LLM and Databricks, plus AWS and Terraform sandboxes.
+Author: BeCloudReady
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/becloudready/labagent
+Project-URL: Source, https://github.com/becloudready/labagent
+Project-URL: Issues, https://github.com/becloudready/labagent/issues
+Project-URL: Documentation, https://github.com/becloudready/labagent/tree/main/docs
+Keywords: hands-on-labs,rag,llm,databricks,lakebase,pgvector,text-to-sql,ollama,ai-agent,aws,terraform,training,workshop,sandbox,cleanup,iam,boto3
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Build Tools
+Classifier: Topic :: System :: Systems Administration
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: boto3>=1.34
+Requires-Dist: cryptography>=42
+Provides-Extra: rag
+Requires-Dist: psycopg[binary]>=3.1; extra == "rag"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: moto[all]>=5; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5; extra == "dev"
+Dynamic: license-file
+
+# labagent
+
+labagent is an open-source CLI that launches hands-on labs for RAG, LLM apps and Databricks, then removes them so you do not pay for leftovers. It also sets up and cleans AWS training sandboxes with Terraform.
+
+## Quick start
+
+```bash
+pip install 'labagent[rag]'          # Python 3.9+; labs use the Databricks CLI: databricks auth login
+labagent packs                       # list the labs
+labagent launch rag-sec-filings --name demo
+labagent destroy rag-sec-filings --name demo
+```
+
+Every step is safe to re-run. `launch` resumes where it stopped. `destroy` removes only what the lab created.
+
+## Lab catalog
+
+| Lab | What you learn | Cost | Command |
+|---|---|---|---|
+| `rag-sec-filings` | RAG over real SEC 10-K filings: chunking, hybrid search, citations, a quality gate before deploy | Databricks Free Edition | `labagent launch rag-sec-filings --name demo` |
+| `rag-lakebase` | The same RAG pipeline on a small demo corpus, to smoke-test your setup | Databricks Free Edition | `labagent launch rag-lakebase --name demo` |
+| `rag-sec-filings-live` | Fetch fresh filings from SEC EDGAR at launch | Databricks Free Edition | `labagent launch rag-sec-filings-live --name demo --param "user_agent=Your Name you@example.com"` |
+| `db-agent` | Deploy [db-agent](https://github.com/db-agent/db-agent), a text-to-SQL agent, over a Databricks sample dataset | Databricks Free Edition | `labagent launch db-agent --name demo` |
+
+Free Edition limits apply: one Lakebase project, three apps, and apps stop after 24 hours. Details: [docs/packs.md](docs/packs.md).
+
+## Run a RAG lab on Databricks Lakebase
+
+`rag-sec-filings` builds a RAG app on Postgres with pgvector. It loads a pinned snapshot of 10-K filings, runs a retrieval quality check, and deploys the app only if the check passes. The app has company and year filters, suggested questions and cited answers.
+
+```bash
+labagent launch rag-sec-filings --name demo
+```
+
+The data comes from a checksummed release in [labagent-datasets](https://github.com/becloudready/labagent-datasets). Known gaps are listed in its README.
+
+## Text-to-SQL agent on Databricks
+
+`db-agent` clones [github.com/db-agent/db-agent](https://github.com/db-agent/db-agent) at a pinned commit and deploys it to Databricks Apps. It then asks a test question through the live app. The data is the `samples` catalog every workspace has.
+
+```bash
+labagent launch db-agent --name demo --param dataset=bakehouse
+```
+
+## Use labagent from plain English
+
+A small local model turns a sentence into a command. You see the command and confirm it before anything runs.
+
+```bash
+ollama pull qwen2.5-coder:0.5b
+labagent "deploy rag-sec on databricks"
+```
+
+Any OpenAI-compatible endpoint works. See [docs/packs.md](docs/packs.md#plain-english).
+
+## AWS sandboxes for workshops
+
+labagent creates IAM users, a sandbox policy and a Lambda role per cohort with Terraform. Passwords are returned encrypted. It also tears down what learners created and cleans an untagged shared account from an explicit ruleset.
+
+| Command | What it does |
+|---|---|
+| `provision` | Create a cohort's IAM layer with Terraform |
+| `inventory`, `sweep` | List and delete what learners created. Plan first, then apply that reviewed plan |
+| `reset`, `offboard` | Rotate a password. Destroy a cohort |
+| `audit`, `cleanup` | Clean a long-lived account from a ruleset ([docs](docs/account-cleanup.md)) |
+
+```bash
+labagent --config lab.json run --job job.json --output result.json
+```
+
+Job format: [docs/job-contract.md](docs/job-contract.md). State and secrets: [docs/state-and-secrets.md](docs/state-and-secrets.md).
+
+## Safety
+
+- Plan first, then apply. The plan is bound to the account, its items, an age limit and the hash a human approved.
+- Only a `delete` rule can delete. A `keep` rule always wins. Unmatched resources are reported, never removed.
+- labagent refuses to run in an AWS account that is not on your allow list.
+- No plaintext secrets or emails in output or logs. No personal data in this repo. Tests enforce both.
+
+Early release. Try labs in a throwaway account or workspace first. Apache-2.0. See [SECURITY.md](SECURITY.md) and [CONTRIBUTING.md](CONTRIBUTING.md).
+
+Running labs for a class or team? Private batches and dashboard: https://becloudready.com

labagent-0.2.0/README.md

@@ -0,0 +1,82 @@
+# labagent
+
+labagent is an open-source CLI that launches hands-on labs for RAG, LLM apps and Databricks, then removes them so you do not pay for leftovers. It also sets up and cleans AWS training sandboxes with Terraform.
+
+## Quick start
+
+```bash
+pip install 'labagent[rag]'          # Python 3.9+; labs use the Databricks CLI: databricks auth login
+labagent packs                       # list the labs
+labagent launch rag-sec-filings --name demo
+labagent destroy rag-sec-filings --name demo
+```
+
+Every step is safe to re-run. `launch` resumes where it stopped. `destroy` removes only what the lab created.
+
+## Lab catalog
+
+| Lab | What you learn | Cost | Command |
+|---|---|---|---|
+| `rag-sec-filings` | RAG over real SEC 10-K filings: chunking, hybrid search, citations, a quality gate before deploy | Databricks Free Edition | `labagent launch rag-sec-filings --name demo` |
+| `rag-lakebase` | The same RAG pipeline on a small demo corpus, to smoke-test your setup | Databricks Free Edition | `labagent launch rag-lakebase --name demo` |
+| `rag-sec-filings-live` | Fetch fresh filings from SEC EDGAR at launch | Databricks Free Edition | `labagent launch rag-sec-filings-live --name demo --param "user_agent=Your Name you@example.com"` |
+| `db-agent` | Deploy [db-agent](https://github.com/db-agent/db-agent), a text-to-SQL agent, over a Databricks sample dataset | Databricks Free Edition | `labagent launch db-agent --name demo` |
+
+Free Edition limits apply: one Lakebase project, three apps, and apps stop after 24 hours. Details: [docs/packs.md](docs/packs.md).
+
+## Run a RAG lab on Databricks Lakebase
+
+`rag-sec-filings` builds a RAG app on Postgres with pgvector. It loads a pinned snapshot of 10-K filings, runs a retrieval quality check, and deploys the app only if the check passes. The app has company and year filters, suggested questions and cited answers.
+
+```bash
+labagent launch rag-sec-filings --name demo
+```
+
+The data comes from a checksummed release in [labagent-datasets](https://github.com/becloudready/labagent-datasets). Known gaps are listed in its README.
+
+## Text-to-SQL agent on Databricks
+
+`db-agent` clones [github.com/db-agent/db-agent](https://github.com/db-agent/db-agent) at a pinned commit and deploys it to Databricks Apps. It then asks a test question through the live app. The data is the `samples` catalog every workspace has.
+
+```bash
+labagent launch db-agent --name demo --param dataset=bakehouse
+```
+
+## Use labagent from plain English
+
+A small local model turns a sentence into a command. You see the command and confirm it before anything runs.
+
+```bash
+ollama pull qwen2.5-coder:0.5b
+labagent "deploy rag-sec on databricks"
+```
+
+Any OpenAI-compatible endpoint works. See [docs/packs.md](docs/packs.md#plain-english).
+
+## AWS sandboxes for workshops
+
+labagent creates IAM users, a sandbox policy and a Lambda role per cohort with Terraform. Passwords are returned encrypted. It also tears down what learners created and cleans an untagged shared account from an explicit ruleset.
+
+| Command | What it does |
+|---|---|
+| `provision` | Create a cohort's IAM layer with Terraform |
+| `inventory`, `sweep` | List and delete what learners created. Plan first, then apply that reviewed plan |
+| `reset`, `offboard` | Rotate a password. Destroy a cohort |
+| `audit`, `cleanup` | Clean a long-lived account from a ruleset ([docs](docs/account-cleanup.md)) |
+
+```bash
+labagent --config lab.json run --job job.json --output result.json
+```
+
+Job format: [docs/job-contract.md](docs/job-contract.md). State and secrets: [docs/state-and-secrets.md](docs/state-and-secrets.md).
+
+## Safety
+
+- Plan first, then apply. The plan is bound to the account, its items, an age limit and the hash a human approved.
+- Only a `delete` rule can delete. A `keep` rule always wins. Unmatched resources are reported, never removed.
+- labagent refuses to run in an AWS account that is not on your allow list.
+- No plaintext secrets or emails in output or logs. No personal data in this repo. Tests enforce both.
+
+Early release. Try labs in a throwaway account or workspace first. Apache-2.0. See [SECURITY.md](SECURITY.md) and [CONTRIBUTING.md](CONTRIBUTING.md).
+
+Running labs for a class or team? Private batches and dashboard: https://becloudready.com

labagent-0.2.0/docs/packs.md

@@ -0,0 +1,111 @@
+# Lab packs
+
+A pack is one lab: ordered, idempotent steps behind `launch` and `destroy`. Packs run on the learner's machine with their own
+credentials, or in your CI. Add a lab by writing a `Pack` subclass; the framework, CLI and result envelope do not change.
+
+```bash
+labagent packs                                   # list
+labagent launch rag-lakebase --name demo --dry-run
+labagent launch rag-lakebase --name demo         # re-run to resume after a failure
+labagent destroy rag-lakebase --name demo        # removes only what this pack created
+```
+
+## Writing a pack
+
+```python
+from labagent.packs.base import Pack, Param, Step
+
+class MyLab(Pack):
+    id = "my-lab"; summary = "what it builds"
+    params = [Param("size", "small")]
+    def deploy_steps(self, ctx):
+        return [Step("make", "Create the thing", lambda c: {"thing_id": "..."}, check=lambda c: "thing_id" in c.state)]
+    def destroy_steps(self, ctx): ...
+```
+
+Ship it in your own package and register it: `[project.entry-points."labagent.packs"] my-lab = "mypkg:MyLab"`.
+Rules: `check` is a read-only "already done" test; step outputs are non-secret; tokens stay in memory (`ctx.mem`).
+State is a 0600 file at `~/.labagent/packs/<pack>/<name>.json`.
+
+## RAG packs (Databricks Lakebase)
+
+One reusable pipeline, pluggable data sources. Each stage is idempotent, and `evaluate` gates the deploy:
+
+`preflight, project, locate, build (fetch, parse, chunk), schema, load (embed), index, evaluate, app`
+
+| Pack | Source | Use |
+|---|---|---|
+| `rag-sec-filings` | Pinned snapshot of real SEC 10-Ks ([dataset release](https://github.com/becloudready/labagent-datasets/releases/tag/sec-10k-2026.10)), split into Items. No SEC access or contact needed | Financial research |
+| `rag-sec-filings-live` | The same corpus downloaded fresh from EDGAR at launch (needs a contact for the SEC) | Newer filings, other tickers |
+| `rag-lakebase` | 25 invented passages | Smoke test only |
+
+```bash
+pip install 'labagent[rag]'      # needs the Databricks CLI, then: databricks auth login
+labagent launch rag-sec-filings --name sec --param "user_agent=Full Name name@example.com" \
+    --param tickers=AAPL,JPM,PFE --param filings=2
+```
+
+What makes it more than a toy: section-aware chunking, hybrid retrieval (pgvector + full-text, rank-fused), metadata filters
+(`{"ticker": "AAPL"}`), numbered citations with the filing URL, and an evaluation set that must reach `min_hit_rate`
+(default 0.6) before the app is deployed. The snapshot is downloaded from a GitHub release and verified against a SHA-256 committed in
+`labagent/packs/rag/sources/datasets.json`; a mismatch is refused. The live pack follows EDGAR rules (declared contact, under 10 requests
+per second, cached locally). Filings are never stored in this repo.
+
+Free Edition allows one Lakebase project: pass `--param project=<existing>` to share it. Each source uses its own schema, and
+`destroy` drops only that schema (and the project only if labagent created it). Apps stop after 24 hours.
+
+Add a source by subclassing `labagent.packs.rag.sources.Source` (`chunks()` and `eval_set()`) and registering it under the
+`labagent.rag_sources` entry point.
+
+## Publishing a dataset snapshot (maintainers)
+
+```bash
+labagent dataset --chunks ~/.labagent/cache/rag-sec-filings-live/<name>.jsonl --name sec-10k --version 2026.11 --out out/
+gh release create sec-10k-2026.11 out/* --repo becloudready/labagent-datasets
+```
+
+The build is byte-for-byte reproducible. Then add the release URL and the printed `sha256` to `datasets.json` in a pull request.
+Data limitations (for example, no Item 7 for JPM and CVX) are listed in the dataset README.
+
+## db-agent (text-to-SQL on Databricks)
+
+Deploys [db-agent](https://github.com/db-agent/db-agent), the open-source text-to-SQL agent, to Databricks Apps in your own workspace.
+labagent contains none of db-agent's code: it clones the repo at a pinned commit, writes an `app.yaml` for your workspace, and deploys it.
+
+```bash
+labagent launch db-agent --name demo                              # app: db-agent-demo, data: samples.bakehouse
+labagent launch db-agent --name demo --param dataset=tpch         # any schema in the samples catalog
+labagent destroy db-agent --name demo
+```
+
+Steps: `preflight` (login, SQL warehouse, sample data, serving endpoints), `fetch` (pinned commit), `token`, `app`, `verify`
+(asks a real question through the deployed app: this proves the token, warehouse, LLM and SQL safety layer work together).
+
+- **Pinned code.** `ref` must be a full commit SHA, and the clone is refused if HEAD differs. To move to a newer db-agent, review it and change the pin.
+- **Data.** The `samples` catalog already exists in every workspace, so nothing is downloaded or redistributed.
+- **Credential.** The app needs a token for the warehouse and the LLM. labagent creates one with a 30-day lifetime (`--param token_days`), stores it
+  only as an app secret (via stdin, never on a command line, in state or in output), and `destroy` revokes it. If your workspace blocks
+  token creation, create one yourself and `export LABAGENT_DATABRICKS_TOKEN=...`; labagent then uses it and never revokes it.
+- **Free Edition.** Up to 3 apps per account, and apps stop after 24 hours.
+
+## Plain English
+
+```bash
+labagent "Run and deploy rag-sec in databricks"      # shows the command, asks, then runs it
+labagent "tear down rag-sec" --print-only            # just show the command
+labagent ask what can you deploy                     # 'ask' lets you skip the quotes
+```
+
+A small model reads the sentence and picks an action (launch, destroy, list) and a pack. By default that is `qwen2.5-coder:0.5b`
+(about 400 MB) on local Ollama, so it runs on any laptop: `ollama pull qwen2.5-coder:0.5b`.
+Any OpenAI-compatible endpoint works: `--endpoint`, `--model`, or `LABAGENT_LLM_URL`, `LABAGENT_LLM_MODEL`, `LABAGENT_LLM_API_KEY`.
+A remote endpoint is announced before your sentence is sent to it.
+
+The model never runs anything, and it is not trusted for details:
+- The instance name (`called x`) and parameters (`profile DEFAULT`, `tickers AAPL,JPM`) are read from your sentence by exact rules, so they cannot be invented.
+- A pack you typed (`rag-sec`, or any id or alias) wins over the model's pick. If the model guessed, the pack needs a distinctive word from your sentence.
+- Requests that ask to create and delete in one sentence are refused. A delete needs a delete word, and you confirm by typing the instance name.
+- Without a terminal, nothing runs unless you pass `--yes`.
+
+Measured on `qwen2.5-coder:0.5b` with 42 phrasings (25 used while tuning, 17 unseen): all handled correctly, including refusals.
+That is a small, author-written test set, so expect misses on unusual wording. A refusal is the usual failure, and the command shown is always checked by you.

labagent-0.2.0/labagent/__init__.py

@@ -0,0 +1,3 @@
+"""labagent: launch and tear down hands-on labs (RAG, LLM, Databricks) and AWS training sandboxes."""
+
+__version__ = "0.2.0"

labagent-0.2.0/labagent/cli.py

@@ -0,0 +1,154 @@
+"""labagent command line. stdout is silent unless --output - is given; logs go to stderr (ids and counts only)."""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import boto3
+
+from . import __version__
+from .config import STATE_BACKEND_DIR, load_config
+from .context import Ctx, GuardError
+from .envelope import Envelope
+from .run import COMMANDS, run_job
+
+
+def _emit(env: Envelope, output: str) -> None:
+    text = json.dumps(env.to_dict(), indent=2)
+    if output == "-":
+        print(text)
+    else:
+        Path(output).write_text(text)
+        Path(output).chmod(0o600)
+
+
+def _packs_main(args) -> int:
+    from .packs import registry
+    from .packs.base import run_pack
+    if args.cmd == "packs":
+        for pid, pk in sorted(registry.all_packs().items()):
+            print(f"{pid:16} {pk.summary}")
+        return 0
+    try:
+        pack = registry.resolve(args.pack)
+        given = dict(kv.split("=", 1) for kv in args.param)
+    except (KeyError, ValueError) as e:
+        print(f"error: {e.args[0] if e.args else e}", file=sys.stderr)
+        return 2
+    name = args.name or pack.default_name
+    if not name:
+        print("error: --name is required for this pack", file=sys.stderr)
+        return 2
+    env = run_pack(pack, args.cmd, name, given, dry_run=args.dry_run)
+    _emit(env, args.output)
+    return env.exit_code
+
+
+KNOWN = {"run", "packs", "dataset", "launch", "destroy", "bootstrap-path", "commands"}
+
+
+def _nl_main(argv) -> int:
+    """labagent "deploy rag-sec on databricks": translate with a small model, confirm, then run the real command."""
+    from . import nl
+    from .packs import registry
+    q = argparse.ArgumentParser(prog="labagent", description="Plain-English front end: labagent \"deploy rag-sec on databricks\"")
+    q.add_argument("text", nargs="+")
+    q.add_argument("--endpoint", help=f"OpenAI-compatible base URL (default {nl.DEFAULT_URL}, or LABAGENT_LLM_URL)")
+    q.add_argument("--model", help=f"Model (default {nl.DEFAULT_MODEL}, or LABAGENT_LLM_MODEL)")
+    q.add_argument("-y", "--yes", action="store_true", help="Run without asking for confirmation")
+    q.add_argument("--print-only", action="store_true", help="Show the translated command and stop")
+    a = q.parse_args(argv[1:] if argv and argv[0] == "ask" else argv)
+    cfg = nl.settings(a.endpoint, a.model)
+    if not nl.is_local(cfg["url"]):
+        print(f"note: your request is sent to {cfg['url']}", file=sys.stderr)
+    try:
+        plan = nl.translate(" ".join(a.text), registry.all_packs(), registry.resolve, cfg["url"], cfg["model"], cfg["api_key"])
+    except nl.NLError as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 2
+    for w in plan.warnings:
+        print(f"warning: {w}", file=sys.stderr)
+    print(f"Interpreted as:\n  {plan.command_line()}", file=sys.stderr)
+    if a.print_only:
+        return 0
+    if not a.yes:
+        if not sys.stdin.isatty():
+            print("error: not an interactive terminal: re-run with --yes to execute, or use the command above", file=sys.stderr)
+            return 2
+        if plan.action == "destroy":
+            ans = input(f"This deletes resources. Type the instance name ({plan.name}) to confirm: ").strip()
+            if ans != plan.name:
+                print("cancelled", file=sys.stderr)
+                return 1
+        elif input("Run it? [y/N] ").strip().lower() not in ("y", "yes"):
+            print("cancelled", file=sys.stderr)
+            return 1
+    return main(plan.argv())
+
+
+def main(argv=None) -> int:
+    argv = list(sys.argv[1:] if argv is None else argv)
+    if argv and not argv[0].startswith("-") and (argv[0] not in KNOWN or argv[0] == "ask"):
+        return _nl_main(argv)
+    p = argparse.ArgumentParser(prog="labagent", description=__doc__)
+    p.add_argument("--version", action="version", version=__version__)
+    p.add_argument("--config", help="JSON config file (or LABAGENT_CONFIG)")
+    p.add_argument("--profile", help="AWS profile for local use. CI uses OIDC environment credentials.")
+    sub = p.add_subparsers(dest="cmd", required=True)
+    r = sub.add_parser("run", help="Execute a job document: the CI entry point")
+    r.add_argument("--job", required=True, help="Path to job JSON (contains roster: keep it out of git and logs)")
+    r.add_argument("--output", default="result.json", help="Result envelope path, or - for stdout")
+    r.add_argument("--ruleset", help="Path to a ruleset JSON for audit/cleanup (merged into the job)")
+    sub.add_parser("packs", help="List available lab packs")
+    dp = sub.add_parser("dataset", help="Package a chunk file as a versioned, checksummed snapshot (maintainers)")
+    dp.add_argument("--chunks", required=True, help="JSONL of chunks, e.g. ~/.labagent/cache/rag-sec-filings/<name>.jsonl")
+    dp.add_argument("--name", required=True)
+    dp.add_argument("--version", required=True)
+    dp.add_argument("--out", required=True, help="Output directory")
+    for verb, text in (("launch", "Deploy a lab pack (idempotent: re-run to resume)"), ("destroy", "Remove what a pack created")):
+        lp = sub.add_parser(verb, help=text)
+        lp.add_argument("pack")
+        lp.add_argument("--name", help="Instance name (state key and resource suffix); defaults to the pack's own default")
+        lp.add_argument("--param", action="append", default=[], metavar="KEY=VALUE", help="Pack parameter, repeatable")
+        lp.add_argument("--dry-run", action="store_true", help="Show the steps without running them")
+        lp.add_argument("--output", default="-", help="Result envelope path, or - for stdout")
+    sub.add_parser("bootstrap-path", help="Print the path of the packaged Terraform state-backend module")
+    sub.add_parser("commands", help="List supported job commands")
+    args = p.parse_args(argv)
+
+    if args.cmd == "bootstrap-path":
+        print(STATE_BACKEND_DIR)
+        return 0
+    if args.cmd == "dataset":
+        from .datasets import DatasetError, build_snapshot
+        try:
+            m = build_snapshot(Path(args.chunks), args.name, args.version, Path(args.out))
+        except DatasetError as e:
+            print(f"error: {e}", file=sys.stderr)
+            return 2
+        print(json.dumps({k: m[k] for k in ("file", "chunks", "bytes", "sha256")}, indent=2))
+        return 0
+    if args.cmd in ("packs", "launch", "destroy"):
+        return _packs_main(args)
+    if args.cmd == "commands":
+        print("\n".join(sorted(COMMANDS)))
+        return 0
+    try:
+        cfg = load_config(args.config)
+        job = json.loads(Path(args.job).read_text())
+        session = boto3.Session(profile_name=args.profile, region_name=cfg.region)
+        if getattr(args, "ruleset", None) and "ruleset" not in job:
+            job["ruleset"] = json.loads(Path(args.ruleset).read_text())
+        env = run_job(Ctx(cfg, session), job)
+    except GuardError as e:
+        env = Envelope(command="guard").fail(str(e))
+    except Exception as e:
+        env = Envelope(command="startup").fail(f"{type(e).__name__}: {str(e)[:200]}")
+    _emit(env, args.output)
+    return env.exit_code
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labagent-0.2.0/labagent/datasets.py

@@ -0,0 +1,101 @@
+"""Versioned, checksum-pinned dataset snapshots.
+
+A snapshot is a gzip'd JSONL of chunks (id, title, content, meta) plus a manifest. Snapshots are published as GitHub
+release assets; the pinned URL and SHA-256 live in `labagent/packs/rag/sources/datasets.json`, committed with the code,
+so changing the data is a reviewed code change. A download whose hash differs is refused and deleted.
+"""
+from __future__ import annotations
+
+import gzip
+import hashlib
+import json
+import os
+import urllib.request
+from datetime import date
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional
+
+from . import __version__
+
+
+class DatasetError(Exception):
+    pass
+
+
+def sha256_file(path: Path) -> str:
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for block in iter(lambda: f.read(1 << 20), b""):
+            h.update(block)
+    return h.hexdigest()
+
+
+def build_snapshot(chunks_path: Path, name: str, version: str, out_dir: Path, source: str = "SEC EDGAR",
+                   built: Optional[str] = None) -> Dict[str, Any]:
+    """Write <name>-<version>.chunks.jsonl.gz (byte-for-byte reproducible), a manifest and SHA256SUMS."""
+    rows = [json.loads(l) for l in Path(chunks_path).read_text().splitlines() if l.strip()]
+    if not rows:
+        raise DatasetError("no chunks to package")
+    ids = [r["id"] for r in rows]
+    if len(ids) != len(set(ids)):
+        raise DatasetError("duplicate chunk ids")
+    out_dir.mkdir(parents=True, exist_ok=True)
+    stem = f"{name}-{version}"
+    data = out_dir / f"{stem}.chunks.jsonl.gz"
+    body = "\n".join(json.dumps(r, ensure_ascii=False, sort_keys=True, separators=(",", ":")) for r in rows) + "\n"
+    with open(data, "wb") as raw, gzip.GzipFile(fileobj=raw, mode="wb", mtime=0, compresslevel=9) as gz:
+        gz.write(body.encode("utf-8"))
+    filings = {}
+    for r in rows:
+        m = r["meta"]
+        if "url" in m:
+            filings[m["url"]] = {k: m.get(k) for k in ("ticker", "company", "form", "fiscal_year", "filed", "url")}
+    manifest = {
+        "name": name, "version": version, "source": source, "built": built or date.today().isoformat(),
+        "labagent": __version__, "chunks": len(rows), "file": data.name, "sha256": sha256_file(data), "bytes": data.stat().st_size,
+        "companies": sorted({f["ticker"] for f in filings.values() if f.get("ticker")}),
+        "filings": sorted(filings.values(), key=lambda f: (f["ticker"], f["fiscal_year"])),
+        "items": sorted({r["meta"].get("item") for r in rows if r["meta"].get("item")}),
+    }
+    (out_dir / f"{stem}.manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")
+    (out_dir / "SHA256SUMS").write_text(f"{manifest['sha256']}  {data.name}\n")
+    return manifest
+
+
+def registry() -> Dict[str, Dict[str, Any]]:
+    return json.loads((Path(__file__).parent / "packs" / "rag" / "sources" / "datasets.json").read_text())
+
+
+def default_http(url: str, dest: Path) -> None:
+    req = urllib.request.Request(url, headers={"User-Agent": f"labagent/{__version__}"})
+    with urllib.request.urlopen(req, timeout=120) as r, open(dest, "wb") as f:
+        for block in iter(lambda: r.read(1 << 20), b""):
+            f.write(block)
+
+
+def fetch_snapshot(name: str, cache_dir: Optional[Path] = None, download: Callable[[str, Path], None] = default_http,
+                   entry: Optional[Dict[str, Any]] = None) -> List[Dict[str, Any]]:
+    """Return the chunks of a pinned snapshot, downloading once and verifying the SHA-256 every time it is read."""
+    entry = entry or registry().get(name)
+    if not entry:
+        raise DatasetError(f"unknown dataset '{name}'. Known: {', '.join(sorted(registry()))}")
+    root = cache_dir or Path(os.environ.get("LABAGENT_HOME") or Path.home() / ".labagent") / "cache" / "datasets"
+    root.mkdir(parents=True, exist_ok=True)
+    dest = root / f"{entry['sha256'][:16]}-{Path(entry['url']).name}"
+    if not dest.exists() or sha256_file(dest) != entry["sha256"]:
+        errors = []
+        for url in [entry["url"], *entry.get("mirrors", [])]:
+            tmp = dest.with_suffix(".part")
+            try:
+                download(url, tmp)
+                if sha256_file(tmp) != entry["sha256"]:
+                    raise DatasetError("checksum mismatch")
+                tmp.replace(dest)
+                break
+            except Exception as e:
+                tmp.unlink(missing_ok=True)
+                errors.append(f"{url.split('/')[2]}: {type(e).__name__}")
+        else:
+            raise DatasetError(f"could not obtain dataset '{name}' with a matching checksum ({'; '.join(errors)})")
+    with gzip.open(dest, "rt", encoding="utf-8") as f:
+        return [json.loads(l) for l in f if l.strip()]