aw-spec-cli 0.1.26 → 0.1.27

package/assets/.appweaver/scripts/python/aw-foundation.py

@@ -3,35 +3,91 @@
 aw-foundation — the part of scaffolding that depends on the LANE MODEL.
 
 A foundation pack is a tree of files and an unpacker that writes it under one output
-directory. That output directory is now a LANE'S SOURCE PATH, so everything a pack
-carries lands inside the subtree its lane owns, and nothing a scaffold produces can
-appear outside a declared lane.
-
-A lane owns TWO subtrees, though — its source path and its test path — and for most
-backend stacks they are siblings rather than nested: .NET keeps `tests/` beside `src/`,
-not inside it. A pack that carries its own test project therefore has one directory
-that does NOT belong under the source path, and unpacking it verbatim would bury the
-tests at `src/tests/`, where the lane does not own them, `dotnet test` does not look for
-them, and every later phase that resolves the lane's test path finds nothing.
-
-So the test tree is LIFTED: whatever the pack placed at `<source path>/<test dir>` is
-merged into the lane's declared test path. The directory name comes from the lane's own
-test path rather than from a list here, which is what makes a lane with a custom
-`test_root` work and a stack whose tests live inside its source a no-op — the two paths
-are then the same subtree and there is nothing to lift.
+directory. That output directory is the LANE ROOT FOLDER, and the tree
+lands there exactly as it is packed — build files, source and tests together — so
+nothing a scaffold produces can appear outside a declared lane. The lane runs its
+commands inside that folder and keeps its tests inside it (`lane-files` reports a lane
+that does not, and foundation does not scaffold into it).
+
+`redirect-tests` LIFTS a test tree for a lane whose test path lies OUTSIDE its lane root
+folder: whatever the pack placed at `<lane root folder>/<test dir>` is merged into the lane's
+declared test path. The directory name comes from the lane's own test path rather than
+from a list here. For every lane foundation scaffolds the test path is inside the lane
+root folder, so it reports that the pack's layout is already right and moves nothing.
+
+It also answers WHAT a lane can be scaffolded from. A directory under
+foundations/<ecosystem>/ offers two kinds of entry:
+
+  pack  <Name>/<Name>.pack.txt   a verified, byte-for-byte project, unpacked as-is
+  spec  <Name>/<Stem>.spec.txt   a description of how to build the project, which an
+                                 agent follows in the lane (one entry per spec file)
+
+Either file may open with `# key: value` header lines. Three keys narrow the entry to the
+lanes it fits, because an ecosystem is shared by stacks that cannot use each other's
+scaffold — Next.js, Angular and the Express backend are all `node`, and Blazor and the
+ASP.NET service are both `dotnet`:
+
+  # kind: backend|frontend     absent = backend, which is what every pack shipped before
+                               the key existed scaffolds
+  # profiles: nextjs, ...      stack-profile ids; absent = any profile of that kind
+  # title: <one line>          the label shown in the pick list
+
+A pack may have a RULE FILE beside it, `<pack name>.rules.json` (`VerticalSlice.rules.json`
+for `VerticalSlice.pack.txt`, `VerticalSlice.Reference.rules.json` for its reference sample).
+Every file in a pack is unpacked into the lane root folder as packed, and one that is already
+there is replaced; the rule file names the files and folders that are treated otherwise:
+
+  {"format": "aw-spec-pack-rules/v1",
+   "rules": [{"path": "src/<Sentinel>.Api/AppDbContext.cs", "action": "merge",
+              "instruction": "how to combine the existing file with the pack's"},
+             {"path": "src/<Sentinel>.Api/Features/", "action": "replace"}]}
+
+A path is written as the pack holds it, sentinel and all, and is renamed the way the pack's
+files are. One ending in `/` covers every pack file under that folder; a rule for the file
+itself wins over its folder's. `instruction` may be a string or a list of lines, and a
+`note` on a rule is for the reader only. `rules` works out, before the unpack, which pack
+files land on a file already in the lane and are to be merged, and puts those existing
+files aside under the state folder so the merge can read them once the pack's version is on
+disk; `rules-done` removes them afterwards.
 
 Called as a CLI by both the bash and the PowerShell actions, so the two answer from one
 implementation rather than from two that have to be kept in step.
 
 Usage:
-  aw-foundation.py redirect-tests --out <lane source path> --test-path <lane test path>
+  aw-foundation.py redirect-tests --out <lane root folder> --test-path <lane test path>
                                   [--repo <root>] [--force]
       -> {"moved": n, "conflicts": [...], "from": "...", "to": "...", "message": "..."}
+  aw-foundation.py list [--kind backend|frontend] [--profile <id>] [--ecosystem <e>]...
+                        [--args "<Foundation> <App>"] [--lane-id <id>]
+      -> {"options": [...], "entries": [...], "count": n, "foundations": [...],
+          "preselect_option": "...", "preselect_app": "...", "rejected_out": "...", ...}
+  aw-foundation.py choice --selection <option|name|n> [same filters as list]
+      -> {"ok": bool, "route": "pack|spec|manual|skip", "name": "...", "file": "...", ...}
+  aw-foundation.py spec-capture --lane <id> [--kind <k>] (--file <path> | --input <path|text>)
+      -> {"ok": bool, "spec_path": ".appweaver/foundation/<id>.spec.md", "source": "...", ...}
+  aw-foundation.py lane-files --source <path> [--test <path>] [--workdir <path>] [--profile <id>]
+      -> {"has_files": bool, "count": n, "sample": [...], "generated": [...],
+          "root_files": [...], "folders": [...], "workdir": "...", "summary": "...",
+          "self_contained": bool, "shape_reason": "..."}
+  aw-foundation.py snapshot --out <repo-relative file>
+      -> {"ok": true, "snapshot": "...", "entries": n}
+  aw-foundation.py outside --snapshot <file> --lane <lane root folder> [--undo]
+      -> {"clean": bool, "created": [...], "changed": [...], "deleted": [...],
+          "removed": [...], "summary": "..."}
+  aw-foundation.py rules --name <Foundation> --app <App> --out <lane root folder>
+                         [--reference] [--state <repo-relative folder>]
+      -> {"ok": bool, "reason": "...", "rules_file": "...", "rules": n,
+          "merges": [{"target", "existing", "instruction", "rule"}], "merge_count": n,
+          "stale": [...], "note": "..."}
+  aw-foundation.py rules-done [--state <repo-relative folder>]  -> {"ok": true, "removed": bool}
 """
 
 import argparse
+import datetime
+import glob
 import json
 import os
+import re
 import shutil
 import sys
 
@@ -40,7 +96,7 @@ def norm(path):
     """A repo-relative directory, '/'-separated and '/'-terminated; "" when absent.
 
     Mirrors aw-stack's `norm_root` for the shapes this reads, so the paths compared here
-    are the same strings the lane model stores. "./" — the repo root as a source path —
+    are the same strings the lane model stores. "./" — the repo root as a lane root folder —
     normalises to "" so that joining it yields the repo itself.
     """
     p = (path or "").replace("\\", "/").strip()
@@ -104,7 +160,7 @@ def redirect_tests(repo, out, test_path, force=False):
     # tests it carries are inside the subtree they belong in, and moving them would
     # take them out of it.
     if out_rel and (test_rel == out_rel or test_rel.startswith(out_rel)):
-        return skip("this lane keeps its tests inside its source path (%s), "
+        return skip("this lane keeps its tests inside its lane root folder (%s), "
                     "so the pack's layout is already right" % out_rel)
 
     # WHAT the pack would have called its test tree, had it placed one: the last segment
@@ -133,18 +189,751 @@ def redirect_tests(repo, out, test_path, force=False):
             "redirected": bool(moved), "message": msg}
 
 
+# ─────────────────────────── what a lane can be built from ───────────────────────────
+
+# The option label separator every pick list in the kit uses ("<value>  — <why>"), so a
+# selection read back from the launcher splits the same way a lane option does.
+LABEL_SEP = "  — "
+MANUAL = "[manual spec]"
+SKIP = "[skip]"
+HEADER_KEYS = ("kind", "profiles", "title", "foundation", "sentinel", "files")
+HEADER_RE = re.compile(r"^#\s*([a-z_]+)\s*:\s*(.*?)\s*$")
+
+
+HEADER_LINES = 40
+
+
+def read_header_text(text):
+    """The `# key: value` lines a pack or spec opens with — known keys only, first wins.
+
+    Leading blank lines are skipped; the first line that is not a `#` line ends the
+    header, so a `# key:` line further down the body is content, not header.
+    """
+    head = {}
+    for i, line in enumerate(text.splitlines()):
+        if i >= HEADER_LINES:
+            break
+        s = line.strip()
+        if not s and not head:
+            continue
+        if not s.startswith("#"):
+            break
+        m = HEADER_RE.match(s)
+        if m and m.group(1) in HEADER_KEYS and m.group(1) not in head:
+            head[m.group(1)] = m.group(2)
+    return head
+
+
+def read_header(path):
+    """read_header_text over a file's first lines only — a pack runs to tens of thousands
+    of lines and its header is the first dozen."""
+    lines = []
+    try:
+        with open(path, encoding="utf-8", errors="replace") as fh:
+            for i, line in enumerate(fh):
+                if i >= HEADER_LINES:
+                    break
+                lines.append(line)
+    except OSError:
+        return {}
+    return read_header_text("".join(lines))
+
+
+def scan_entries(repo):
+    """Every pack and spec under foundations/<ecosystem>/<Name>/, unfiltered.
+
+    A directory holding neither (components/) is not a foundation and yields nothing. A
+    Reference pack is never an entry of its own: it rides on the primary pack it extends.
+    """
+    root = os.path.join(repo, "foundations")
+    out = []
+    if not os.path.isdir(root):
+        return out
+    for eco in sorted(os.listdir(root)):
+        edir = os.path.join(root, eco)
+        if not os.path.isdir(edir):
+            continue
+        for name in sorted(os.listdir(edir)):
+            fdir = os.path.join(edir, name)
+            if not os.path.isdir(fdir):
+                continue
+            rel_dir = "foundations/%s/%s" % (eco, name)
+            pack = os.path.join(fdir, name + ".pack.txt")
+            if os.path.isfile(pack):
+                h = read_header(pack)
+                out.append(_entry(name, "pack", eco, "%s/%s.pack.txt" % (rel_dir, name), h,
+                                  has_reference=os.path.isfile(
+                                      os.path.join(fdir, name + ".Reference.pack.txt"))))
+            for f in sorted(os.listdir(fdir)):
+                if f.endswith(".spec.txt") and os.path.isfile(os.path.join(fdir, f)):
+                    h = read_header(os.path.join(fdir, f))
+                    out.append(_entry(f[:-len(".spec.txt")], "spec", eco,
+                                      "%s/%s" % (rel_dir, f), h))
+    return out
+
+
+def _entry(name, route, eco, rel, head, has_reference=False):
+    profiles = [p.strip() for p in re.split(r"[,\s]+", head.get("profiles", "")) if p.strip()]
+    return {"name": name, "route": route, "ecosystem": eco, "file": rel,
+            "kind": (head.get("kind") or "backend").strip().lower(),
+            "profiles": profiles, "title": head.get("title", ""),
+            "files": head.get("files", ""), "has_reference": has_reference}
+
+
+def entry_fits(e, kind, profile, ecosystems):
+    if ecosystems is not None and e["ecosystem"] not in ecosystems:
+        return False
+    if kind and e["kind"] != kind:
+        return False
+    if profile and e["profiles"] and profile not in e["profiles"]:
+        return False
+    return True
+
+
+def entry_option(e):
+    if e["title"]:
+        why = e["title"]
+    elif e["route"] == "pack":
+        why = "unpack a verified, compiling project" + (
+            " (%s files)" % e["files"] if e["files"] else "")
+    else:
+        why = "build the project this spec describes (%s)" % e["file"]
+    return "%s [%s]%s%s" % (e["name"], e["route"], LABEL_SEP, why)
+
+
+def build_list(repo, kind="", profile="", ecosystems=None, raw="", lane_id=""):
+    entries = scan_entries(repo)
+    kept = [e for e in entries if entry_fits(e, kind, profile, ecosystems)]
+    # Packs before specs for the same name: the verified bundle is the safer default.
+    kept.sort(key=lambda e: (e["name"].lower(), 0 if e["route"] == "pack" else 1))
+    lane = ("the %s lane" % lane_id) if lane_id else "this lane"
+    manual = MANUAL + LABEL_SEP + "give the spec yourself: a file path, or the spec text"
+    skip = SKIP + LABEL_SEP + "leave %s as it is" % lane
+    # A lane that already holds files was asked about overwriting before this list is
+    # built (lane-files), so SKIP never needs to come first to protect it.
+    options = [entry_option(e) for e in kept] + [manual, skip]
+
+    pre_name = pre_app = rejected = ""
+    toks = (raw or "").split()
+    if toks:
+        pre_name = toks[0]
+    if len(toks) > 1:
+        pre_app = toks[1]
+    if len(toks) > 2:
+        rejected = toks[2]
+    match = next((e for e in kept if e["name"] == pre_name), None)
+
+    return {
+        "options": options,
+        "entries": kept,
+        "count": len(kept),
+        "foundations": [e["name"] for e in kept if e["route"] == "pack"],
+        "specs": [e["name"] for e in kept if e["route"] == "spec"],
+        "all_foundations": ["%s/%s [%s]" % (e["ecosystem"], e["name"], e["route"])
+                            for e in entries],
+        "all_count": len(entries),
+        "ecosystems": sorted(set(e["ecosystem"] for e in entries)),
+        "lane_ecosystems": list(ecosystems or []),
+        "filtered": ecosystems is not None,
+        "kind": kind,
+        "profile": profile,
+        "preselect_name": match["name"] if match else "",
+        "preselect_option": entry_option(match) if match else "",
+        "preselect_app": pre_app,
+        "rejected_out": rejected,
+    }
+
+
+def resolve_choice(repo, selection, kind="", profile="", ecosystems=None):
+    """Map what the launcher wrote back to one entry.
+
+    Tolerant on purpose: the selection comes back as text the operator or the launching
+    agent wrote, so the full option, the part before the label, a bare name, `skip`,
+    `manual` and a 1-based position in the list all resolve. A bare name that is both a
+    pack and a spec resolves to the pack.
+    """
+    # Same filters as the list the operator saw, so a numbered answer counts the same rows.
+    listing = build_list(repo, kind, profile, ecosystems)
+    options, kept = listing["options"], listing["entries"]
+    sel = (selection or "").strip()
+    head = sel.split(LABEL_SEP, 1)[0].strip()
+    head = re.split(r"\s+—\s+", head, maxsplit=1)[0].strip()
+
+    def result(route, e=None, ok=True, reason=""):
+        return {"ok": ok, "route": route, "name": e["name"] if e else "",
+                "file": e["file"] if e else "", "ecosystem": e["ecosystem"] if e else "",
+                "title": e["title"] if e else "", "selection": sel, "reason": reason}
+
+    if head.isdigit() and 1 <= int(head) <= len(options):
+        head = options[int(head) - 1].split(LABEL_SEP, 1)[0].strip()
+    low = head.lower()
+    if low in (SKIP, "skip"):
+        return result("skip")
+    if low in (MANUAL, "manual", "manual spec", "[manual]"):
+        return result("manual")
+    m = re.match(r"^(.+?)\s*\[(pack|spec)\]$", head)
+    if m:
+        e = next((x for x in kept if x["name"] == m.group(1) and x["route"] == m.group(2)), None)
+    else:
+        e = next((x for x in kept if x["name"] == head and x["route"] == "pack"), None) \
+            or next((x for x in kept if x["name"] == head), None)
+    if e:
+        return result(e["route"], e)
+    return result("", ok=False, reason="`%s` is not one of the options offered for this lane"
+                                        % (sel or "(empty)"))
+
+
+SPEC_LIMIT = 200_000
+
+
+def capture_spec(repo, lane, kind="", file="", text=""):
+    """Copy the spec a lane is being built from to .appweaver/foundation/<lane>.spec.md.
+
+    The copy is the record of what the project was built from: a bundle spec can change
+    after the build and typed text exists nowhere else. `--input` is read as a path when
+    it names a file, and as the spec itself otherwise — except a single word that is
+    shaped like a path (a separator, or a file extension), which is a path that was
+    mistyped: capturing it as a fifteen-character spec would build a project from it.
+    """
+    def fail(why):
+        return {"ok": False, "spec_path": "", "source": "", "chars": 0, "title": "",
+                "reason": why}
+
+    source, body = "", ""
+    cand = (file or "").strip()
+    typed = (text or "").strip()
+    if not cand and typed and "\n" not in typed and len(typed) < 512:
+        maybe = typed.strip("'\"` ")
+        p = maybe if os.path.isabs(maybe) else os.path.join(repo, maybe)
+        if os.path.isfile(p):
+            cand = maybe
+        elif os.path.isdir(p):
+            return fail("`%s` is a directory — give the path of the spec file inside it" % maybe)
+        elif " " not in maybe and re.search(r"[/\\]|\.(md|txt|markdown)$", maybe, re.I):
+            return fail("spec file not found: %s" % maybe)
+    if cand:
+        p = cand if os.path.isabs(cand) else os.path.join(repo, cand)
+        if not os.path.isfile(p):
+            return fail("spec file not found: %s" % cand)
+        if os.path.getsize(p) > SPEC_LIMIT:
+            return fail("spec file %s is larger than %d bytes" % (cand, SPEC_LIMIT))
+        with open(p, encoding="utf-8", errors="replace") as fh:
+            body = fh.read()
+        source = os.path.relpath(p, repo).replace("\\", "/") if not os.path.isabs(cand) else cand
+    elif typed:
+        body, source = typed, "text typed at the prompt"
+        if len(body) > SPEC_LIMIT:
+            return fail("the spec text is larger than %d characters" % SPEC_LIMIT)
+    if not body.strip():
+        return fail("no spec was given — enter a file path or the spec text")
+
+    title = read_header_text(body).get("title", "")
+    if not title:
+        for line in body.splitlines():
+            s = line.strip()
+            if s and not HEADER_RE.match(s):
+                title = s.lstrip("#").strip()[:100]
+                break
+
+    lane_slug = re.sub(r"[^A-Za-z0-9._-]+", "-", lane or kind or "lane").strip("-") or "lane"
+    rel = ".appweaver/foundation/%s.spec.md" % lane_slug
+    dest = os.path.join(repo, rel)
+    os.makedirs(os.path.dirname(dest), exist_ok=True)
+    stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+    with open(dest, "w", encoding="utf-8") as fh:
+        fh.write("<!-- /appweaver foundation: the spec given for the %s `%s`.\n"
+                 "     source: %s · captured %s -->\n\n"
+                 % ((kind + " lane") if kind else "lane", lane_slug, source, stamp))
+        fh.write(body if body.endswith("\n") else body + "\n")
+    return {"ok": True, "spec_path": rel, "source": source, "chars": len(body),
+            "title": title, "reason": ""}
+
+
+# ───────────────────────────── what a lane already holds ─────────────────────────────
+
+# Directories that hold what a build or an install produced rather than code someone
+# wrote. Walking node_modules alone can take longer than the rest of the run, so they are
+# named, never walked — but one being there still means the lane is not empty.
+GENERATED_DIRS = {"node_modules", ".next", "dist", "build", "out", "coverage",
+                  "bin", "obj", "target", ".git"}
+# Files that keep an otherwise empty directory in git, or that an OS drops on its own.
+IGNORED_FILES = {".gitkeep", ".DS_Store", "Thumbs.db"}
+SAMPLE = 5
+
+
+def _profile(repo, profile_id):
+    """The stack profile whose `id` is profile_id — matched on the id, not the file name."""
+    base = os.path.join(repo, ".appweaver", "stack-profiles")
+    for path in sorted(glob.glob(os.path.join(base, "*.json"))):
+        try:
+            with open(path, encoding="utf-8") as fh:
+                prof = json.load(fh)
+        except (ValueError, OSError):
+            continue
+        if prof.get("id") == profile_id:
+            return prof
+    return {}
+
+
+def _inside(path, folder):
+    """True when the normalised directory `path` is `folder` or below it."""
+    return path == folder or path.startswith(folder)
+
+
+def _capped(items):
+    return ", ".join(items[:SAMPLE]) + (" and %d more" % (len(items) - SAMPLE)
+                                        if len(items) > SAMPLE else "")
+
+
+def lane_files(repo, source, test="", workdir="", profile=""):
+    """What a lane already holds — asked before anything is scaffolded into it.
+
+    Walks the lane root folder and, when it is a different subtree, the test path. A path that
+    is the repository root itself is not walked: it would count every other lane and the
+    kit. The lane's build files are looked for where its commands run when that is outside
+    the lane — a .NET lane rooted at src/ keeps its solution file at the repository root —
+    using the profile's `detect` entries that name a file there (the ones with no `/`).
+    """
+    folders = []
+    for f in (norm(source), norm(test)):
+        if f and not any(_inside(f, g) for g in folders):
+            folders = [g for g in folders if not _inside(g, f)] + [f]
+
+    count, sample, generated = 0, [], []
+    for folder in folders:
+        top = os.path.join(repo, folder)
+        if not os.path.isdir(top):
+            continue
+        for dirpath, dirnames, filenames in os.walk(top):
+            rel_dir = os.path.relpath(dirpath, repo).replace("\\", "/")
+            walk = []
+            for d in sorted(dirnames):
+                if d in GENERATED_DIRS:
+                    generated.append("%s/%s/" % (rel_dir, d))
+                else:
+                    walk.append(d)
+            dirnames[:] = walk
+            for name in sorted(filenames):
+                if name in IGNORED_FILES:
+                    continue
+                count += 1
+                if len(sample) < SAMPLE:
+                    sample.append("%s/%s" % (rel_dir, name))
+
+    wd = norm(workdir)
+    root_files = []
+    if not any(_inside(wd, f) for f in folders):
+        for pat in _profile(repo, profile).get("detect", []) if profile else []:
+            if "/" in pat:
+                continue
+            for hit in sorted(glob.glob(os.path.join(repo, wd, pat))):
+                if os.path.isfile(hit):
+                    root_files.append(os.path.relpath(hit, repo).replace("\\", "/"))
+
+    parts = []
+    if count:
+        parts.append("%s: %d file(s), e.g. %s" % (", ".join(folders), count, _capped(sample)))
+    if generated:
+        parts.append("build output / dependencies (not counted): %s" % _capped(generated))
+    if root_files:
+        parts.append("%s: %s" % ("repo root" if not wd else wd, _capped(root_files)))
+    contained, shape_reason = lane_shape(source, test, workdir)
+    return {"has_files": bool(count or generated or root_files), "count": count,
+            "sample": sample, "generated": generated[:SAMPLE], "root_files": root_files,
+            "folders": folders, "workdir": wd or "./", "summary": "; ".join(parts),
+            "self_contained": contained, "shape_reason": shape_reason}
+
+
+def lane_shape(source, test="", workdir=""):
+    """Can a foundation be built into this lane? (self_contained, reason)
+
+    A foundation is a complete project — its build files, its source and its tests — and
+    it lands inside the lane root folder exactly as a pack holds it (a spec build is asked
+    for the same). So the lane's commands have to run inside that folder, or `dotnet
+    build` looks for a solution file at the repository root that is not there; and its
+    tests have to sit inside it too. A lane declared the classic way — code in `src/`,
+    build files and `tests/` at the top — is right for existing code but cannot take a
+    foundation, and is reported rather than scaffolded into a project that does not build.
+    """
+    src, tst, wd = norm(source), norm(test), norm(workdir)
+    where = src or "./"
+    if wd != src:
+        return False, ("the lane's commands run at %s, outside its lane root folder %s — a foundation "
+                       "builds a complete project (build files, source and tests) inside the "
+                       "lane root folder, so its commands have to run there. Re-declare the lane "
+                       "with /appweaver select-stack so they run inside %s (run_from: root)"
+                       % (wd or "the repository root", where, where))
+    if tst and src and not tst.startswith(src):
+        return False, ("the lane keeps its tests at %s, outside its lane root folder %s — a foundation "
+                       "puts the project's tests inside the lane root folder. Re-declare the lane "
+                       "with /appweaver select-stack with its tests inside %s" % (tst, where, where))
+    return True, ""
+
+
+# What the engine itself writes while a build runs: the checkpoint state the /appweaver
+# launcher keeps, with its payload and response sidecars, and the mock transport's output.
+# Never recorded, so never reported and never removed.
+ENGINE_DIRS = (".appweaver/orchestrator/.state/", ".appweaver/orchestrator/.demo-out/")
+# The kit's own folders. A spec build may not write in them even for a lane whose root
+# folder is the repository itself.
+KIT_DIRS = (".appweaver/", "guardrails/", "foundations/", "specs/")
+# Files an operating system drops on its own while a build runs.
+OS_FILES = {".DS_Store", "Thumbs.db"}
+
+
+def _tree(repo):
+    """Every file and folder in the repo: {path: [size, mtime_ns]} for a file, {path/: None}
+    for a folder. A GENERATED_DIRS folder is one entry and is not walked, so a stray
+    node_modules/ still shows up without its contents being read."""
+    entries = {}
+    for dirpath, dirnames, filenames in os.walk(repo):
+        rel = os.path.relpath(dirpath, repo).replace("\\", "/")
+        rel = "" if rel == "." else rel + "/"
+        walk = []
+        for d in dirnames:
+            path = rel + d + "/"
+            if any(path.startswith(e) for e in ENGINE_DIRS):
+                continue
+            entries[path] = None
+            if d not in GENERATED_DIRS:
+                walk.append(d)
+        dirnames[:] = walk
+        for name in filenames:
+            if name in OS_FILES:
+                continue
+            try:
+                st = os.stat(os.path.join(dirpath, name))
+            except OSError:
+                continue
+            entries[rel + name] = [st.st_size, st.st_mtime_ns]
+    return entries
+
+
+def snapshot(repo, out):
+    """Record the repo before a spec build, for `outside` to compare against afterwards."""
+    entries = _tree(repo)
+    path = os.path.join(repo, out)
+    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
+    with open(path, "w", encoding="utf-8") as fh:
+        json.dump({"entries": entries}, fh, separators=(",", ":"))
+    return {"ok": True, "snapshot": out, "entries": len(entries)}
+
+
+def _topmost(paths):
+    """Drop every path that sits inside another folder in the list."""
+    folders = [p for p in paths if p.endswith("/")]
+    return sorted(p for p in paths if not any(p != f and p.startswith(f) for f in folders))
+
+
+def outside(repo, snap, lane, undo=False):
+    """What a spec build did OUTSIDE its lane root folder since `snapshot` ran.
+
+    Outside means anywhere but the lane root folder, and always the kit's own folders. With
+    `undo`, everything the build CREATED out there is removed again — a new folder as a
+    whole, since all of it is new. What it changed or deleted is only listed: there is no
+    copy to put back. The snapshot file is removed once read.
+    """
+    snap_path = os.path.join(repo, snap)
+    try:
+        with open(snap_path, encoding="utf-8") as fh:
+            before = json.load(fh)["entries"]
+    except (OSError, ValueError, KeyError):
+        return {"clean": False, "created": [], "changed": [], "deleted": [], "removed": [],
+                "summary": "no snapshot was taken before the build (%s), so what it wrote "
+                           "outside the lane root folder cannot be checked" % snap}
+    after = _tree(repo)
+    lane_dir = norm(lane)
+
+    def out_of_lane(p):
+        return any(p.startswith(k) for k in KIT_DIRS) or not p.startswith(lane_dir)
+
+    created = _topmost([p for p in after if p not in before and out_of_lane(p)])
+    deleted = _topmost([p for p in before if p not in after and out_of_lane(p)])
+    changed = sorted(p for p in after if p in before and before[p] is not None
+                     and after[p] != before[p] and out_of_lane(p))
+
+    removed = []
+    if undo:
+        for p in created:
+            full = os.path.join(repo, p)
+            try:
+                if p.endswith("/"):
+                    shutil.rmtree(full)
+                else:
+                    os.remove(full)
+                removed.append(p)
+            except OSError:
+                pass
+    try:
+        os.remove(snap_path)
+    except OSError:
+        pass
+
+    parts = []
+    if created:
+        parts.append("created %s%s" % (_capped(created),
+                                       " (removed again)" if removed == created else
+                                       " (removed again: %s)" % _capped(removed) if removed else ""))
+    if changed:
+        parts.append("changed %s (restore them yourself)" % _capped(changed))
+    if deleted:
+        parts.append("deleted %s (restore them yourself)" % _capped(deleted))
+    return {"clean": not (created or changed or deleted), "created": created,
+            "changed": changed, "deleted": deleted, "removed": removed,
+            "summary": "; ".join(parts)}
+
+
+# ──────────────────────────────── a pack's rule file ────────────────────────────────
+
+RULES_FORMAT = "aw-spec-pack-rules/v1"
+RULE_ACTIONS = ("merge", "replace")
+MERGE_STATE = ".appweaver/orchestrator/.state/foundation-merge/"
+PACK_FILE_RE = re.compile(r'^<<<FILE path="([^"]+)"( encoding="([^"]+)")?>>>\s*$')
+PACK_END = "<<<ENDFILE>>>"
+
+
+def _pack_path(repo, name, reference=False):
+    """foundations/<ecosystem>/<name>/<name>[.Reference].pack.txt, repo-relative; "" when
+    there is none."""
+    base = name + (".Reference" if reference else "")
+    for hit in sorted(glob.glob(os.path.join(repo, "foundations", "*", name, base + ".pack.txt"))):
+        return os.path.relpath(hit, repo).replace("\\", "/")
+    return ""
+
+
+def _pack_paths(path):
+    """The path of every file block in a pack, read the way the unpackers read it: a
+    `<<<FILE` line only opens a block outside one."""
+    paths, inside = [], False
+    with open(path, encoding="utf-8", errors="replace") as fh:
+        for line in fh:
+            line = line.rstrip("\r\n")
+            if inside:
+                inside = line != PACK_END
+                continue
+            m = PACK_FILE_RE.match(line)
+            if m:
+                paths.append(m.group(1))
+                inside = True
+    return paths
+
+
+def _rename_forms(sentinel, app):
+    """The replacement contract in a pack's header, as the unpackers apply it: the dotted,
+    underscored, lower-hyphen and lower-dot forms, no-ops dropped, longest first."""
+    s_low, a_low = sentinel.lower(), app.lower()
+    forms, seen = [], set()
+    for src, dst in ((sentinel, app),
+                     (sentinel.replace(".", "_"), app.replace(".", "_")),
+                     (s_low.replace(".", "-"), a_low.replace(".", "-")),
+                     (s_low, a_low)):
+        if src and src != dst and src not in seen:
+            seen.add(src)
+            forms.append((src, dst))
+    return sorted(forms, key=lambda f: -len(f[0]))
+
+
+def _renamed(path, forms):
+    for src, dst in forms:
+        path = path.replace(src, dst)
+    return path
+
+
+def load_rules(path):
+    """(rules, error) from a rule file. No file is no rules, not an error."""
+    if not os.path.isfile(path):
+        return [], ""
+    try:
+        with open(path, encoding="utf-8") as fh:
+            data = json.load(fh)
+    except ValueError as e:
+        return [], "is not valid JSON (%s)" % e
+    except OSError as e:
+        return [], "cannot be read (%s)" % e
+    if not isinstance(data, dict) or not isinstance(data.get("rules"), list):
+        return [], 'needs a "rules" list'
+    fmt = data.get("format", RULES_FORMAT)
+    if fmt != RULES_FORMAT:
+        return [], 'has format "%s"; this kit reads "%s"' % (fmt, RULES_FORMAT)
+    rules = []
+    for i, r in enumerate(data["rules"], 1):
+        if not isinstance(r, dict):
+            return [], "has rule %d that is not an object" % i
+        p = str(r.get("path", "")).replace("\\", "/").strip()
+        while p.startswith("./"):
+            p = p[2:]
+        p = p.lstrip("/")
+        action = str(r.get("action", "")).strip().lower()
+        instr = r.get("instruction", "")
+        if isinstance(instr, list):
+            instr = "\n".join(str(x) for x in instr)
+        instr = str(instr or "").strip()
+        if not p:
+            return [], "has rule %d with no path" % i
+        if action not in RULE_ACTIONS:
+            return [], 'has rule %d (%s) whose action is "%s"; it must be "merge" or "replace"' % (
+                i, p, action)
+        if action == "merge" and not instr:
+            return [], "has rule %d (%s) that merges with no instruction for how" % (i, p)
+        rules.append({"path": p, "action": action, "instruction": instr})
+    return rules, ""
+
+
+def _state_dir(state):
+    """The folder merges are staged in — only ever below the engine's state folder, since
+    it is emptied before and removed after."""
+    s = norm(state) or MERGE_STATE
+    return s if s.startswith(ENGINE_DIRS[0]) and s != ENGINE_DIRS[0] else ""
+
+
+def _rule_for(path, rules):
+    """The rule for one pack file: its own, else the deepest folder rule above it."""
+    exact = next((r for r in rules if r["path"] == path), None)
+    if exact:
+        return exact
+    folders = [r for r in rules if r["path"].endswith("/") and path.startswith(r["path"])]
+    return max(folders, key=lambda r: len(r["path"])) if folders else None
+
+
+def pack_rules(repo, name, app, out, reference=False, state=MERGE_STATE):
+    """What a pack's rule file asks of THIS unpack, worked out before it runs.
+
+    A pack file that will land on a file already in the lane and that a `merge` rule
+    covers is a merge: the file there now is copied under `state`, where the merge reads
+    it once the pack's version has been written over it. Everything else — a `replace`
+    rule, no rule, or nothing there yet — is simply written as packed. A rule whose path
+    the pack does not hold is reported, so a rule file left behind by a re-pack is seen.
+    """
+    pack = _pack_path(repo, name, reference)
+    label = "%s%s" % (name, " reference sample" if reference else " pack")
+    result = {"ok": True, "reason": "", "rules_file": "", "rules": 0, "merges": [],
+              "merge_count": 0, "stale": [], "note": ""}
+    if not pack:
+        return dict(result, ok=False, reason="the %s was not found" % label)
+    rules_rel = pack[:-len(".pack.txt")] + ".rules.json"
+    rules, err = load_rules(os.path.join(repo, rules_rel))
+    if err:
+        return dict(result, ok=False, rules_file=rules_rel,
+                    reason="the rule file %s %s" % (rules_rel, err))
+    if not rules:
+        return result
+
+    stash = _state_dir(state)
+    if not stash:
+        return dict(result, ok=False, rules_file=rules_rel,
+                    reason="--state %s is not a folder under %s" % (state, ENGINE_DIRS[0]))
+    forms = _rename_forms(read_header(os.path.join(repo, pack)).get("sentinel", ""), app)
+    lane = norm(out)
+    shutil.rmtree(os.path.join(repo, stash), ignore_errors=True)
+
+    merges, used = [], set()
+    for packed in _pack_paths(os.path.join(repo, pack)):
+        rule = _rule_for(packed, rules)
+        if not rule:
+            continue
+        used.add(rule["path"])
+        rel = _renamed(packed, forms)
+        target = lane + rel
+        if rule["action"] != "merge" or not os.path.isfile(os.path.join(repo, target)):
+            continue
+        existing = stash + rel
+        os.makedirs(os.path.dirname(os.path.join(repo, existing)), exist_ok=True)
+        shutil.copy2(os.path.join(repo, target), os.path.join(repo, existing))
+        merges.append({"target": target, "existing": existing,
+                       "instruction": rule["instruction"], "rule": rule["path"]})
+    stale = [r["path"] for r in rules if r["path"] not in used]
+
+    parts = []
+    if merges:
+        parts.append("merged %d existing file(s) as %s asks: %s"
+                     % (len(merges), rules_rel, _capped([m["target"] for m in merges])))
+    if stale:
+        parts.append("%s names path(s) the pack does not hold: %s" % (rules_rel, _capped(stale)))
+    return dict(result, rules_file=rules_rel, rules=len(rules), merges=merges,
+                merge_count=len(merges), stale=stale,
+                note="".join("; " + p for p in parts))
+
+
+def rules_done(repo, state=MERGE_STATE):
+    """Remove the files `rules` put aside, once the merges have read them."""
+    stash = _state_dir(state)
+    if not stash:
+        return {"ok": False, "removed": False,
+                "reason": "--state %s is not a folder under %s" % (state, ENGINE_DIRS[0])}
+    full = os.path.join(repo, stash)
+    there = os.path.isdir(full)
+    shutil.rmtree(full, ignore_errors=True)
+    return {"ok": True, "removed": there}
+
+
+def _filters(p):
+    p.add_argument("--kind", default="", choices=["", "backend", "frontend"])
+    p.add_argument("--profile", default="")
+    # Repeatable. Absent = unfiltered; passed EMPTY = filtered to nothing, which is what a
+    # lane with no known ecosystem gets — no bundle is known to fit it.
+    p.add_argument("--ecosystem", action="append", default=None)
+
+
 def main():
     ap = argparse.ArgumentParser(description="lane-aware scaffolding helpers")
     ap.add_argument("--repo", default=os.getcwd())
     sub = ap.add_subparsers(dest="cmd", required=True)
     r = sub.add_parser("redirect-tests")
-    r.add_argument("--out", required=True, help="the lane's source path, where the pack was unpacked")
+    r.add_argument("--out", required=True, help="the lane root folder, where the pack was unpacked")
     r.add_argument("--test-path", default="", help="the lane's declared test path")
     r.add_argument("--force", action="store_true")
+    ls = sub.add_parser("list")
+    _filters(ls)
+    ls.add_argument("--args", default="", help="fast path: '<Foundation> <App>'")
+    ls.add_argument("--lane-id", default="")
+    ch = sub.add_parser("choice")
+    _filters(ch)
+    ch.add_argument("--selection", default="")
+    sc = sub.add_parser("spec-capture")
+    sc.add_argument("--lane", default="")
+    sc.add_argument("--kind", default="")
+    sc.add_argument("--file", default="")
+    sc.add_argument("--input", default="")
+    lf = sub.add_parser("lane-files")
+    lf.add_argument("--source", default="", help="the lane root folder")
+    lf.add_argument("--test", default="", help="the lane's test path")
+    lf.add_argument("--workdir", default="", help="where the lane's commands run")
+    lf.add_argument("--profile", default="", help="stack-profile id, for its build files")
+    sn = sub.add_parser("snapshot")
+    sn.add_argument("--out", required=True, help="repo-relative file to record the repo in")
+    os_ = sub.add_parser("outside")
+    os_.add_argument("--snapshot", required=True, help="the file `snapshot` wrote")
+    os_.add_argument("--lane", default="", help="the lane root folder")
+    os_.add_argument("--undo", action="store_true", help="remove what the build created outside it")
+    ru = sub.add_parser("rules")
+    ru.add_argument("--name", required=True, help="the foundation")
+    ru.add_argument("--app", required=True, help="the app name the sentinel is renamed to")
+    ru.add_argument("--out", default="", help="the lane root folder the pack unpacks into")
+    ru.add_argument("--reference", action="store_true", help="the pack's reference sample")
+    ru.add_argument("--state", default=MERGE_STATE, help="where existing files are put aside")
+    rd = sub.add_parser("rules-done")
+    rd.add_argument("--state", default=MERGE_STATE)
     args = ap.parse_args()
 
     if args.cmd == "redirect-tests":
         emit(redirect_tests(args.repo, args.out, args.test_path, args.force))
+    elif args.cmd == "list":
+        emit(build_list(args.repo, args.kind, args.profile, args.ecosystem, args.args,
+                        args.lane_id))
+    elif args.cmd == "choice":
+        emit(resolve_choice(args.repo, args.selection, args.kind, args.profile, args.ecosystem))
+    elif args.cmd == "spec-capture":
+        emit(capture_spec(args.repo, args.lane, args.kind, args.file, args.input))
+    elif args.cmd == "lane-files":
+        emit(lane_files(args.repo, args.source, args.test, args.workdir, args.profile))
+    elif args.cmd == "snapshot":
+        emit(snapshot(args.repo, args.out))
+    elif args.cmd == "outside":
+        emit(outside(args.repo, args.snapshot, args.lane, args.undo))
+    elif args.cmd == "rules":
+        emit(pack_rules(args.repo, args.name, args.app, args.out, args.reference, args.state))
+    elif args.cmd == "rules-done":
+        emit(rules_done(args.repo, args.state))
 
 
 if __name__ == "__main__":