@sitar_fiercer4c/skills 0.1.0

package/skills/datalab-api/references/table-selection.md

@@ -0,0 +1,35 @@
+# Automating table selection with a model call
+
+Read this when you need to pick the line-item table across many workbooks
+programmatically. For a one-off, just read the previews and choose — the
+principles in SKILL.md are the whole method, and this file only adds the
+plumbing.
+
+Send a prose description of the target plus a short preview of each table, and
+constrain the output to ids so what comes back is something you can look up:
+
+```python
+from pydantic import BaseModel
+
+class Selection(BaseModel):
+    table_ids: list[str]
+    reasoning: str
+
+response = client.messages.parse(
+    model="claude-opus-4-7",
+    max_tokens=4000,
+    system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}],
+    thinking={"type": "adaptive"},
+    output_config={"effort": "high"},
+    messages=[{"role": "user", "content": prompt}],
+    output_format=Selection,
+)
+```
+
+Build `prompt` from the `--preview` output: each table labelled with its exact
+`id`, five rows, cells truncated to ~40 characters. Cache the system prompt when
+looping over a directory — the target description is identical across files and
+only the previews change.
+
+`reasoning` is worth keeping even though you don't act on it. When a selection
+looks wrong, it tells you whether the description or the preview was at fault.

package/skills/datalab-api/scripts/datalab_tables.py

@@ -0,0 +1,365 @@
+#!/usr/bin/env python3
+"""Extract table blocks from spreadsheets via the Datalab Convert API.
+
+    datalab_tables.py <file-or-dir> [--json out.json] [--preview out.md]
+
+For each input file: submit to Datalab, poll until done, walk the block tree
+for `block_type: "Table"`, and report each table's id, source cell range, and
+size. Raw responses are cached on disk, so re-runs cost no credits.
+
+Needs DATALAB_API_KEY in the environment or in a .env file found by walking up
+from the current directory.
+"""
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sys
+import time
+from html.parser import HTMLParser
+from pathlib import Path
+
+import requests
+
+API_URL = "https://www.datalab.to/api/v1/convert"
+MODES = ("fast", "balanced", "accurate")
+POLL_SECONDS = 2
+POLL_ATTEMPTS = 300
+
+MIME_TYPES = {
+    ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
+    ".xlsm": "application/vnd.ms-excel.sheet.macroEnabled.12",
+    ".xltx": "application/vnd.openxmlformats-officedocument.spreadsheetml.template",
+    ".xls": "application/vnd.ms-excel",
+    ".ods": "application/vnd.oasis.opendocument.spreadsheet",
+    ".csv": "text/csv",
+}
+
+PREVIEW_HEAD = 5
+PREVIEW_TAIL = 3
+CELL_CHARS = 28
+
+
+# ----------------------------------------------------------------- auth
+
+def api_key() -> str:
+    key = os.environ.get("DATALAB_API_KEY")
+    if key:
+        return key
+    for directory in [Path.cwd(), *Path.cwd().parents]:
+        for name in (".env.local", ".env"):
+            env = directory / name
+            if not env.exists():
+                continue
+            for line in env.read_text().splitlines():
+                if line.startswith("DATALAB_API_KEY="):
+                    return line.split("=", 1)[1].strip().strip("'\"")
+    raise SystemExit("DATALAB_API_KEY not found in environment or any parent .env file")
+
+
+# ------------------------------------------------------------ api access
+
+def strip_images(node):
+    """Drop base64 image payloads.
+
+    Datalab always returns images for spreadsheets and `disable_image_extraction`
+    does not change that, so this is the only way to keep responses small. They
+    are stored twice: under the top-level `images` key and inside each Picture
+    block. On image-heavy sheets this is ~98% of the bytes.
+
+    Images are one-per-row product photos on many BOQ-style sheets. If you need
+    them, keep the raw response instead of stripping.
+    """
+    if isinstance(node, list):
+        for item in node:
+            strip_images(item)
+    elif isinstance(node, dict):
+        node.pop("images", None)
+        strip_images(node.get("children") or [])
+        strip_images(node.get("json") or [])
+    return node
+
+
+def cache_path(path: Path, mode: str, cache_dir: Path, keep_images: bool) -> Path:
+    """Cache filename that can't serve one file's response for another.
+
+    The stem alone collides: a/report.xlsx, b/report.xlsx and report.csv would
+    share an entry and silently return each other's tables. keep_images belongs
+    in the key too, or the flag returns stripped data from an earlier run and
+    appears to do nothing.
+    """
+    digest = hashlib.sha1(str(path.resolve()).encode()).hexdigest()[:8]
+    images = ".img" if keep_images else ""
+    return cache_dir / f"{path.stem}.{mode}{images}.{digest}.json"
+
+
+def convert(path: Path, key: str, mode: str, cache_dir: Path, keep_images: bool) -> dict:
+    """Submit one file and poll until the conversion completes."""
+    cache = cache_path(path, mode, cache_dir, keep_images)
+    if cache.exists():
+        return json.loads(cache.read_text())
+
+    mime = MIME_TYPES.get(path.suffix.lower(), "application/octet-stream")
+    headers = {"X-API-Key": key}
+    with path.open("rb") as fh:
+        resp = requests.post(
+            API_URL,
+            headers=headers,
+            files={"file": (path.name, fh, mime)},
+            data={"output_format": "json", "mode": mode},
+            timeout=120,
+        )
+    resp.raise_for_status()
+    submitted = resp.json()
+    if not submitted.get("success"):
+        raise SystemExit(f"{path.name}: submit failed: {submitted.get('error')}")
+
+    check_url = submitted["request_check_url"]
+    for _ in range(POLL_ATTEMPTS):
+        result = requests.get(check_url, headers=headers, timeout=60).json()
+        if result.get("status") == "complete":
+            if not keep_images:
+                strip_images(result)
+            cache_dir.mkdir(parents=True, exist_ok=True)
+            cache.write_text(json.dumps(result, indent=2))
+            return result
+        if result.get("status") == "failed" or result.get("error"):
+            raise SystemExit(f"{path.name}: conversion failed: {result.get('error')}")
+        time.sleep(POLL_SECONDS)
+    raise SystemExit(f"{path.name}: timed out after {POLL_ATTEMPTS * POLL_SECONDS}s")
+
+
+# ---------------------------------------------------------- block walking
+
+def find_tables(node, page=None, out=None):
+    """Depth-first walk of the block tree, collecting Table blocks.
+
+    Blocks nest arbitrarily, so recursion is the reliable way to reach them.
+    TableOfContents is included because Datalab sometimes classifies a plain
+    lead-in table that way.
+    """
+    if out is None:
+        out = []
+    if isinstance(node, list):
+        for item in node:
+            find_tables(item, page, out)
+        return out
+    if not isinstance(node, dict):
+        return out
+
+    block_type = node.get("block_type")
+    if block_type == "Page":
+        page = node.get("id", page)
+    if block_type in ("Table", "TableOfContents") and node.get("html"):
+        out.append({
+            "id": node.get("id"),
+            "page": page,
+            "html": node["html"],
+            "bbox": node.get("bbox"),
+        })
+    find_tables(node.get("children") or [], page, out)
+    return out
+
+
+# ----------------------------------------------------------- html -> grid
+
+class TableParser(HTMLParser):
+    """Flatten an HTML table into a list of row-lists of cell strings.
+
+    Cells are placed by occupancy rather than appended, so `colspan` and
+    `rowspan` consume the columns they actually cover. These sheets merge
+    header cells constantly, and appending would shift every later cell left —
+    a header spanning A:C followed by one over D would report that second
+    header as column 2 and misalign it against the data beneath.
+
+    A merged cell puts its text in the first covered position and leaves the
+    rest empty, which is how openpyxl reads the same merge in the source.
+    """
+
+    def __init__(self):
+        super().__init__()
+        self.cells = {}
+        self.filled_rows = set()
+        self.row_idx = -1
+        self.col_idx = 0
+        self.cell = None
+        self.span = (1, 1)
+
+    @staticmethod
+    def _span(attrs, name):
+        try:
+            return max(1, int(dict(attrs).get(name, 1)))
+        except (TypeError, ValueError):
+            return 1
+
+    def handle_starttag(self, tag, attrs):
+        if tag == "tr":
+            self.row_idx += 1
+            self.col_idx = 0
+        elif tag in ("td", "th"):
+            self.cell = []
+            self.span = (self._span(attrs, "colspan"), self._span(attrs, "rowspan"))
+
+    def handle_endtag(self, tag):
+        if tag not in ("td", "th") or self.cell is None:
+            return
+        text = re.sub(r"\s+", " ", "".join(self.cell)).strip()
+        self.cell = None
+        colspan, rowspan = self.span
+        while (self.row_idx, self.col_idx) in self.cells:
+            self.col_idx += 1
+        for dr in range(rowspan):
+            for dc in range(colspan):
+                self.cells[(self.row_idx + dr, self.col_idx + dc)] = text if dr == dc == 0 else ""
+        self.filled_rows.add(self.row_idx)
+        self.col_idx += colspan
+
+    def handle_data(self, data):
+        if self.cell is not None:
+            self.cell.append(data)
+
+    def grid(self) -> list[list[str]]:
+        if not self.cells:
+            return []
+        width = max(c for _, c in self.cells) + 1
+        rows = sorted({r for r, _ in self.cells} | self.filled_rows)
+        return [[self.cells.get((r, c), "") for c in range(width)] for r in rows]
+
+
+def to_grid(html: str) -> list[list[str]]:
+    parser = TableParser()
+    parser.feed(html)
+    return parser.grid()
+
+
+# ------------------------------------------------------------- bbox -> A1
+
+def col_letter(n: int) -> str:
+    """1 -> A, 27 -> AA."""
+    out = ""
+    while n > 0:
+        n, rem = divmod(n - 1, 26)
+        out = chr(65 + rem) + out
+    return out
+
+
+def excel_range(bbox) -> str:
+    """Spreadsheet bboxes are [col0, row0, col1, row1], 1-indexed inclusive.
+
+    This is cell coordinates, not pixels or points. Undocumented, but verified
+    against openpyxl on real workbooks.
+
+    Only Table and Text blocks use that convention. A Page bbox is a sheet
+    extent starting [0, 0, ...] and a Picture bbox is continuous, so neither is
+    a range. Both are rejected here rather than converted, because col_letter(0)
+    is the empty string and would otherwise yield a plausible-looking "0:J19".
+    """
+    if not bbox or len(bbox) != 4:
+        return "n/a"
+    c0, r0, c1, r1 = (int(v) for v in bbox)
+    if min(c0, r0, c1, r1) < 1:
+        return "n/a"
+    return f"{col_letter(c0)}{r0}:{col_letter(c1)}{r1}"
+
+
+# -------------------------------------------------------------- reporting
+
+def md_row(cells: list[str], index: str = "") -> str:
+    safe = [c.replace("|", "\\|").replace("\n", " ")[:CELL_CHARS] or " " for c in cells]
+    return "| " + index + " | " + " | ".join(safe) + " |"
+
+
+def render(grid: list[list[str]]) -> list[str]:
+    """Markdown preview: header, first rows and last rows with a gap marker."""
+    if not grid:
+        return ["_empty table_"]
+    headers = [h or f"col_{i}" for i, h in enumerate(grid[0])]
+    body = grid[1:]
+
+    lines = [md_row(headers, "#"), md_row(["---"] * len(headers), "---")]
+    if len(body) <= PREVIEW_HEAD + PREVIEW_TAIL:
+        shown, gap_after = [(i, r) for i, r in enumerate(body, 1)], None
+    else:
+        shown = [(i, r) for i, r in enumerate(body[:PREVIEW_HEAD], 1)]
+        shown += [(i, r) for i, r in enumerate(body[-PREVIEW_TAIL:], len(body) - PREVIEW_TAIL + 1)]
+        gap_after = PREVIEW_HEAD
+
+    for pos, (num, row) in enumerate(shown):
+        if gap_after is not None and pos == gap_after:
+            lines.append(md_row(["…"] * len(headers), "…"))
+        lines.append(md_row(row, str(num)))
+    return lines
+
+
+def describe(path: Path, tables: list[dict]) -> list[dict]:
+    out = []
+    for table in tables:
+        grid = to_grid(table["html"])
+        out.append({
+            "id": table["id"],
+            "bbox": table["bbox"],
+            "excel_range": excel_range(table["bbox"]),
+            "rows": max(len(grid) - 1, 0),
+            "columns": len(grid[0]) if grid else 0,
+            "headers": grid[0] if grid else [],
+        })
+    return out
+
+
+def collect_inputs(source: Path) -> list[Path]:
+    if source.is_dir():
+        return sorted(p for p in source.iterdir() if p.suffix.lower() in MIME_TYPES)
+    return [source]
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__,
+                                     formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("input", type=Path, help="a spreadsheet file or a directory of them")
+    parser.add_argument("--json", type=Path, help="write the table index as JSON")
+    parser.add_argument("--preview", type=Path, help="write a markdown preview of every table")
+    parser.add_argument("--mode", choices=MODES, default="fast",
+                        help="Datalab mode (default: fast; has no effect on spreadsheets)")
+    parser.add_argument("--cache-dir", type=Path, default=Path("datalab_raw"),
+                        help="where to cache raw API responses (default: ./datalab_raw)")
+    parser.add_argument("--keep-images", action="store_true",
+                        help="keep base64 image payloads instead of stripping them")
+    args = parser.parse_args()
+
+    files = collect_inputs(args.input)
+    if not files:
+        parser.error(f"no spreadsheet files found at {args.input}")
+
+    key = api_key()
+    index, preview_lines = {}, ["# Datalab table extraction\n"]
+
+    for path in files:
+        print(f"{path.name}", file=sys.stderr)
+        result = convert(path, key, args.mode, args.cache_dir, args.keep_images)
+        tables = find_tables(result.get("json") or {})
+        index[path.name] = describe(path, tables)
+        print(f"  {len(tables)} table block(s)", file=sys.stderr)
+
+        preview_lines.append(f"\n## {path.name}\n")
+        for n, table in enumerate(tables):
+            grid = to_grid(table["html"])
+            preview_lines.append(f"\n### Table {n} — `{table['id']}`\n")
+            preview_lines.append(
+                f"{max(len(grid) - 1, 0)} data rows x {len(grid[0]) if grid else 0} columns"
+                f" — bbox `{table['bbox']}` — range `{excel_range(table['bbox'])}`\n")
+            preview_lines.extend(render(grid))
+
+    if args.preview:
+        args.preview.write_text("\n".join(preview_lines) + "\n")
+        print(f"preview -> {args.preview}", file=sys.stderr)
+    if args.json:
+        args.json.write_text(json.dumps(index, indent=2) + "\n")
+        print(f"index -> {args.json}", file=sys.stderr)
+    if not args.json and not args.preview:
+        print(json.dumps(index, indent=2))
+
+
+if __name__ == "__main__":
+    main()

package/skills/find-test-seam/SKILL.md

@@ -0,0 +1,41 @@
+---
+name: find-test-seam
+description: Find the input and output seams in existing code that a characterization test can be pinned to before a business-logic refactor. Use this skill when a business-goal node/edge graph and an API recording already exist and only the seam is wanted — if the graph does not exist yet, refac-wrt-business-goal builds it and names the seams in one pass. It names and ranks seams; it does not write tests and does not change code.
+---
+
+# Find the test seam
+
+Given a happy-path API recording and a business-goal node/edge graph, name the input and output seam
+a characterization test can be pinned to. You name seams. You do not write tests or change code.
+
+## Hard constraint
+
+1. Touch only files in backend — `services/`, `utils/`.
+2. Find a seam which is currently existing in code. Do not create a new one.
+
+## Method
+
+- Filter the recording to state-changing 2xx calls; ignore OPTIONS, 304s and polling GETs.
+- A `202` means the work is fire-and-forget: the response body is not the outcome, and the seam
+  does not await it. Name the signal the test polls for completion.
+- Input seam: narrowest existing function with every `E` node on its input side. Count what it
+  fetches, reads and imports, not just its parameters.
+- Check it is already reachable — exported, on the default export. If you would have to export it,
+  you are creating a seam, not finding one.
+- Output seam: where every outcome becomes durable. Read the `catch` block — the refusal lives there.
+- Reject any seam the refactor would dissolve. Pure helpers die; orchestrator entry and terminal
+  write survive.
+- Grep every call site of the output seam; find what distinguishes the in-graph ones.
+- Read that function's body — retry or pruning logic silently voids the assertions you plan.
+- Map each node's output to a field on the output-seam payload. An unmappable node means either a
+  misplaced seam or a node not implemented — say which, and what can be checked instead.
+- Rank 2–3 seam pairs on coverage versus testability, checking each can be driven today with the
+  fixtures and injection points that exist. An existing script that already invokes the pair is the
+  strongest evidence it is drivable. Recommend one.
+
+## Output format
+
+First answer the exact question the user is asking. No extra context or extra info. Fewer words is
+better. Use progressive disclosure — keep the headings `## Input seam`, `## Output seam`,
+`## Candidates`, `## Blockers`, and be brief under each: a 30–60 word sentence, or 10–15 word
+bullets for technical points. Cite `file:line` for every claim.