@ketecode/loom 0.1.0 → 0.2.0

package/skills/design-brainstorming/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "I want to add some kind of team sharing to our notes app but I haven't worked out how it should behave. Can we think it through together first?",
+    "Let's brainstorm the design for a recurring-invoices feature before anyone writes code",
+    "I have a rough idea for an offline mode in our field-inspection app, help me shape it into a design we agree on",
+    "Before we build the new notification preferences, ask me what you need to know and propose a couple of approaches",
+    "Help me figure out what our internal deploy dashboard should actually do, then write the design down in the repo"
+  ],
+  "should_not_trigger": [
+    "Write a PRD for the recurring invoices feature with goals, metrics and rollout for the leadership review",
+    "Here's the approved design doc for offline mode, break it into bite-sized implementation tasks",
+    "Should our notifications live in a separate service or stay in the monolith? Weigh the options",
+    "Record our decision to use Postgres row-level security as an architecture decision record",
+    "Implement the team sharing endpoints from the plan in docs/plans, test first"
+  ]
+}

package/skills/design-brainstorming/references/visual-mockups.md

@@ -0,0 +1,34 @@
+# Visual mockups during design
+
+Use these when the user has to choose between layouts, flows or visual treatments, and words would leave them guessing. A mockup is there to help them decide, not to be the product: throw it away once the design doc records the choice.
+
+## When to offer one
+
+- The question is about a screen, a flow across screens, a chart or dashboard, or the density and hierarchy of a page.
+- Two approaches differ mainly in how they look or feel.
+- The user says "show me", or keeps asking what something would look like.
+
+Don't offer one for back-end, data or API decisions; a short table or sequence of steps is clearer there. Ask before making mockups: "Want me to sketch A and B as HTML pages you can open side by side?"
+
+## How to make them
+
+1. Put them outside the source tree, in a folder that's git-ignored, such as `.design/mockups/<topic>/`. Check with `git check-ignore -q .design/` first. If it isn't ignored, ask before adding it to `.gitignore`, or use the system temp folder.
+2. Write **one self-contained HTML file per option**: inline CSS, no build step, no framework, no external scripts, fonts or images. Name the files `a-<short-name>.html`, `b-<short-name>.html`.
+3. Use realistic but obviously fake content: plausible names and numbers that are not real people or data. Show the awkward cases too: a long name, an empty list, an error state.
+4. Match only what the decision needs. Use grey boxes for anything that isn't being decided, and the project's real colours and type only if the choice depends on them.
+5. Add a short header on each page: the option letter, a one-line summary, and what to look at ("compare how the filters sit relative to the table").
+6. When there are more than two options, or one flow across several screens, add an `index.html` that links them, or shows them side by side in iframes.
+7. Tell the user the paths and how to open them (`open` on macOS, `xdg-open` on Linux, `start` on Windows, or "open this file in your browser"). Opening a browser yourself is fine when the user asked for it.
+
+## Taking feedback
+
+- Ask one question per round: "Which layout, A or B?" then "What would you change about B?"
+- Edit the mockup the user picked, and keep the earlier versions (`b-v2.html`) so you can go back.
+- Once a section of the design is agreed, record the decision in words, with a screenshot if the user wants one. Never put the mockup HTML into the design doc or the source tree.
+- When the design doc is written, offer to delete the mockup folder.
+
+## Never
+
+- Wire mockups to real APIs, real customer data or production credentials.
+- Turn a mockup into production code by copying it into the app. Build the real thing from the design doc, test-first.
+- Run a local server for mockups unless the user asks. Static files are enough, and leave nothing running.

package/skills/desktop-app-testing/SKILL.md

@@ -0,0 +1,104 @@
+---
+name: desktop-app-testing
+description: Automates end-to-end UI tests for native desktop applications (WPF, WinForms, Win32, Qt and Electron) by driving them through UI Automation with pywinauto, through the Appium Windows driver backed by WinAppDriver, or through Playwright's Electron support, with stable automation IDs instead of screen coordinates, isolated app data per test, screenshots on failure, and CI runners that have a real interactive desktop session or a virtual display. Use when writing UI tests for a Windows or cross-platform desktop app, a desktop test passes locally but cannot find windows in CI, choosing between pywinauto, Appium and Playwright for a desktop app, or replacing coordinate-click scripts that break on every resolution change.
+license: MIT
+compatibility: Any Agent Skills host. Windows targets need a Windows machine or runner with an unlocked interactive session; Electron tests run on Windows, macOS or Linux (Linux needs a display, for example Xvfb).
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: testing
+  kete-loom/risk: runs-commands
+  kete-loom/owner: "Kete-org"
+---
+
+# Desktop app testing
+
+Write desktop UI tests that find controls by identity, start from a known state, and give a screenshot and log when they fail, in CI as well as on a developer machine.
+
+Browser apps belong to `e2e-testing-playwright`. Unit and component tests belong to the language's testing skill (`csharp-testing`, `python-testing`, `cpp-testing`). This skill is for driving the packaged, running desktop app.
+
+## 0. Before you start
+
+1. Identify the UI framework and target OS: WPF, WinForms, Win32/MFC, WinUI/UWP, Qt (Widgets or QML), Electron, or other.
+2. Find how the app is built and launched (path to the built `.exe`, command-line flags, config or data directories).
+3. Check for existing UI tests and their tool. Extend that tool unless it is unworkable; tell the user before switching.
+4. Inspect the UI tree with Accessibility Insights for Windows or `Inspect.exe` (Windows SDK) to see which controls expose `AutomationId`, `Name` and `ControlType`. For Electron, use DevTools.
+
+## 1. Pick the driver
+
+| App | Tool | Why |
+|---|---|---|
+| Electron | Playwright `_electron.launch` | Same API as web tests, access to the main process |
+| WPF, WinForms, WinUI, Win32, Qt on Windows, test code in Python | pywinauto with `backend="uia"` | No server process, direct UI Automation |
+| Same apps, team wants WebDriver or non-Python language | Appium with the Windows driver (`appium driver install windows`) | WebDriver protocol, any client language; needs WinAppDriver and Windows Developer Mode |
+| Old Win32/MFC controls with poor UIA support | pywinauto `backend="win32"` | Reads native Win32 controls |
+| Qt on Linux or macOS | the platform accessibility layer, or Squish (commercial) if the team owns it; **stop and ask** | Few free options; confirm the choice |
+| macOS native (AppKit/SwiftUI) | XCUITest, or Appium Mac2 driver | Out of scope here beyond this pointer |
+
+WinAppDriver has seen no active development for years; it still works for many apps but check its status and the Appium Windows driver's current requirements before committing a new suite to it. Prefer pywinauto for new Python suites.
+
+## 2. Make the app testable
+
+1. Give every control a test needs a stable automation identifier:
+   - WPF: `AutomationProperties.AutomationId="SaveButton"` (or `x:Name`, which UIA exposes).
+   - WinForms: the control's `Name` property becomes the `AutomationId`.
+   - Qt: `setObjectName("saveButton")` and accessible names; check what UIA shows.
+   - Electron: `data-testid` or accessible roles and names in the renderer.
+2. Add a launch flag or environment variable for tests: a separate data directory, disabled auto-update, telemetry off, animations off, a fixed locale. Never point tests at the user's real profile.
+3. Where a test needs data, seed it through files or an API before launch, not by clicking through setup screens.
+
+Changing app code to add IDs is in scope; ask before changing behaviour.
+
+## 3. Write the tests
+
+### Locators
+
+- Order of preference: automation ID, then control type plus accessible name, then a path inside a uniquely identified parent. Never screen coordinates, image matching or tab-order key presses as the main strategy.
+- Scope searches to the app's top window, not the desktop root; desktop-wide searches are slow and pick up other apps.
+
+### Waiting
+
+- Wait for a condition (window exists, control enabled, text equals) with a timeout. pywinauto: `.wait("visible enabled", timeout=10)`; Appium: explicit WebDriver waits; Playwright: web-first `expect` assertions.
+- Never use fixed sleeps except as a last resort with a comment explaining what is missing.
+
+### Structure
+
+- One fresh app instance per test or per test class, launched with a clean temp data directory; close it in teardown even when the test fails, and kill leftover processes by PID.
+- Page-object style: one class per window or dialog wrapping its locators and actions.
+- Assert on visible outcomes (a list row appears, a file is written, a status label changes), not on internal state.
+- On failure, save a screenshot of the app window and the app's log file as test artifacts.
+
+Code templates for each tool are in `references/drivers.md`.
+
+## 4. CI
+
+1. Windows UI automation needs an **interactive, unlocked desktop session**. A runner installed as a Windows service, a locked screen or a disconnected RDP session makes window searches fail or hang.
+   - GitHub-hosted Windows runners: try them first; check that windows are found and screenshots are not black.
+   - Self-hosted: run the runner as a normal user with auto-logon and the screensaver and lock disabled, not as a service. Keep the screen resolution fixed.
+2. Linux (Electron, Qt): run under a virtual display, `xvfb-run -a <test command>`.
+3. Install the app from the same build artifact the release uses, not a debug build, at least for the smoke subset.
+4. Run UI tests serially on one desktop; parallel instances fight over focus. Shard across machines instead.
+5. Upload screenshots, app logs and test reports on failure.
+
+## 5. Flakiness
+
+When a test is flaky, find the cause before adding retries: a missing wait, focus stolen by another window (update prompts, notifications), shared state between tests, or animations. One CI retry is acceptable while the cause is investigated; report each flaky test to the user.
+
+## Never
+
+- Never automate the user's real profile, real accounts or production back ends.
+- Never rely on screen coordinates, pixel matching or fixed sleeps as the main technique.
+- Never disable OS security features (UAC, Defender) on shared machines to make tests pass; use a dedicated test VM.
+- Never leave app processes running after the suite.
+
+## Report back
+
+```
+## Desktop UI tests: <app>
+Framework / OS: <WPF on Windows 11, ...>
+Tool: <pywinauto uia | Appium Windows | Playwright Electron> and why
+Tests added: <names, what each covers>
+App changes: <automation IDs or test flags added>
+CI: <runner type, session setup, artifacts>
+Result: <command, pass/fail counts, runs repeated for flakiness>
+Gaps: <controls without IDs, untestable areas>
+```

package/skills/desktop-app-testing/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Write UI tests for our WPF invoicing app that click through creating a customer",
+    "Our pywinauto tests pass locally but can't find the main window on the CI Windows runner",
+    "Should we use pywinauto or Appium with WinAppDriver to test our WinForms client?",
+    "Add end-to-end tests for our Electron app that launch it and check the main window",
+    "Replace our AutoIt scripts that click screen coordinates with something that survives a resolution change"
+  ],
+  "should_not_trigger": [
+    "Write Playwright tests for the login page of our React web app",
+    "Add xUnit unit tests for the tax calculation class in our C# library",
+    "Package our Python Qt app into a signed Windows installer",
+    "Check our web app's forms for screen reader and keyboard accessibility",
+    "Test our iOS app's onboarding flow with XCUITest on a simulator"
+  ]
+}

package/skills/desktop-app-testing/references/drivers.md

@@ -0,0 +1,81 @@
+# Desktop driver templates
+
+Read this when writing the first test with a tool. Check the tool's current documentation for version-specific capability names.
+
+## pywinauto (Python, UI Automation backend)
+
+```python
+import subprocess, tempfile, pytest
+from pywinauto import Application
+
+APP = r"C:\build\MyApp\MyApp.exe"
+
+@pytest.fixture
+def app():
+    data_dir = tempfile.mkdtemp()
+    application = Application(backend="uia").start(f'"{APP}" --data-dir "{data_dir}" --no-update')
+    main = application.window(title_re="MyApp.*")
+    main.wait("visible ready", timeout=20)
+    yield application, main
+    application.kill()
+
+def test_save_creates_record(app, request):
+    _, main = app
+    try:
+        main.child_window(auto_id="NameBox", control_type="Edit").set_edit_text("Ama")
+        main.child_window(auto_id="SaveButton", control_type="Button").click_input()
+        row = main.child_window(title="Ama", control_type="DataItem")
+        row.wait("visible", timeout=10)
+    except Exception:
+        main.capture_as_image().save(f"{request.node.name}.png")
+        raise
+```
+
+Use `print_control_identifiers()` on a window once to explore its tree; do not leave it in tests. `click_input()` moves the real mouse (needs focus); `invoke()` uses the UIA Invoke pattern and is often steadier.
+
+## Appium Windows driver (WebDriver)
+
+Install: `npm install -g appium`, `appium driver install windows`, install WinAppDriver, enable Windows Developer Mode, start `appium`.
+
+```python
+from appium import webdriver
+from appium.options.windows import WindowsOptions
+from appium.webdriver.common.appiumby import AppiumBy
+
+opts = WindowsOptions()
+opts.app = r"C:\build\MyApp\MyApp.exe"
+opts.set_capability("appium:appArguments", "--data-dir C:\\temp\\t1")
+driver = webdriver.Remote("http://127.0.0.1:4723", options=opts)
+try:
+    driver.find_element(AppiumBy.ACCESSIBILITY_ID, "NameBox").send_keys("Ama")
+    driver.find_element(AppiumBy.ACCESSIBILITY_ID, "SaveButton").click()
+finally:
+    driver.quit()
+```
+
+`ACCESSIBILITY_ID` maps to UIA `AutomationId`. Use `"appium:app": "Root"` to attach to the desktop for apps you start yourself, then find the window by name.
+
+## Playwright for Electron (TypeScript)
+
+```ts
+import { test, expect, _electron as electron } from '@playwright/test';
+import os from 'node:os'; import path from 'node:path'; import fs from 'node:fs';
+
+test('save creates a record', async () => {
+  const userData = fs.mkdtempSync(path.join(os.tmpdir(), 'app-'));
+  // Pass the temp dir however the app accepts it (a flag or env var that calls app.setPath('userData', ...)).
+  const app = await electron.launch({ args: ['.'], env: { ...process.env, MYAPP_USER_DATA: userData } });
+  try {
+    const win = await app.firstWindow();
+    await win.getByLabel('Name').fill('Ama');
+    await win.getByRole('button', { name: 'Save' }).click();
+    await expect(win.getByRole('row', { name: /Ama/ })).toBeVisible();
+    const version = await app.evaluate(({ app }) => app.getVersion());
+    expect(version).toMatch(/\d+\.\d+\.\d+/);
+  } finally {
+    await app.close();
+  }
+});
+```
+
+For a packaged build pass `executablePath` to the built binary. The config's `use.trace` option does not cover a context you launch yourself; record a trace with `await app.context().tracing.start({ screenshots: true, snapshots: true })` after launch and `tracing.stop({ path })` in the `finally` block when the test failed.

package/skills/document-processing/SKILL.md

@@ -0,0 +1,100 @@
+---
+name: document-processing
+description: Pipelines for PDFs and scanned paperwork built on open tools - triage each file as born-digital, scanned or mixed, pull text and tables with pdfplumber or pdftotext, add a searchable text layer with OCRmyPDF and Tesseract language packs, split, merge, rotate, decrypt and repair with qpdf, read and fill AcroForm fields with pypdf or pdftk, redact so the underlying text and metadata are truly gone and prove it, and translate a document while keeping its layout. Use when someone has a folder of PDFs or scans to extract data from, needs a scanned contract made searchable, wants a PDF form filled from a spreadsheet, or must hand over a redacted document that has to survive copy-paste and text extraction.
+license: MIT
+compatibility: Any Agent Skills host. Uses Python 3 with pdfplumber and pypdf, plus the qpdf, ocrmypdf, tesseract (with the needed language data), pdftotext/pdfinfo (poppler-utils) and optionally pdftk CLIs. Redaction examples use PyMuPDF, which is AGPL-licensed.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: data
+  kete-loom/risk: runs-commands
+  kete-loom/requires: "python3, qpdf, ocrmypdf, tesseract, pdftotext"
+  kete-loom/owner: "Kete-org"
+---
+
+# Document processing
+
+Treat every document as evidence: never change the original, keep each step's output, and check results against the page images, not just the extracted text. Command and code detail is in `references/extraction-and-ocr.md`, `references/forms-and-conversion.md` and `references/redaction.md`.
+
+## 0. Before you start
+
+1. Copy inputs to `work/in/` and treat them as read-only. Outputs go to `work/out/`.
+2. Ask the user what the result is for (search, a spreadsheet, a filled form, a redacted copy for disclosure, a translation) and the languages in the documents.
+3. **Sensitive content** (IDs, health, financial, legal): confirm the user may process it and that no cloud OCR, translation or AI service will be used without their approval. Default to local tools.
+4. Do not open files from untrusted sources with tools that execute embedded content; work with command-line tools and libraries, not by opening JavaScript-enabled viewers.
+
+## 1. Triage each file
+
+Run `pdfinfo file.pdf` (pages, encryption, PDF version, tagged or not) and `pdffonts file.pdf`, then sample text with `pdftotext -f 1 -l 3 -layout file.pdf -`.
+
+| Finding | Class | Next step |
+|---|---|---|
+| Text comes out readable, fonts listed | Born-digital | Extract directly (step 2) |
+| No text, no fonts, pages are images | Scanned | OCR first (step 3) |
+| Some pages have text, others not | Mixed | OCR with skip mode for pages that already have text |
+| Text is garbled (wrong glyphs, `(cid:123)`) | Broken font encoding | OCR with force mode, or redo |
+| `qpdf --check` reports damage | Damaged | Repair with `qpdf in.pdf repaired.pdf`, then re-triage |
+| Encrypted | Protected | Decrypt with `qpdf --decrypt` only with the password and permission from the user |
+
+## 2. Extract text and tables (born-digital)
+
+1. Plain text in reading order: `pdftotext -layout` for column-preserving text, or pdfplumber `page.extract_text()` for per-page control.
+2. Tables: pdfplumber `page.extract_tables()` with explicit `table_settings` (lines strategy for ruled tables, text strategy for whitespace tables). Crop to the table region with `page.crop(bbox)` when a page has more than one table.
+3. Keep provenance on every row: file, page number and, for tables, the bounding box.
+4. Validate: row counts per page, column totals against the printed totals, and a manual spot check of at least three rows against the page image. Report any mismatch rather than "fixing" numbers.
+
+## 3. OCR scans
+
+1. Run OCRmyPDF with the right languages (`-l eng+fra`; install the Tesseract language data first, check with `tesseract --list-langs`).
+2. For phone photos or skewed scans add `--rotate-pages --deskew --clean`. These cannot be combined with redo mode.
+3. For mixed files use skip mode (`--skip-text`, or `--mode skip` in OCRmyPDF 17+); for an existing bad text layer use `--redo-ocr` (`--mode redo`); `--force-ocr` (`--mode force`) rasterises everything and loses vector quality.
+4. Write PDF/A for archiving (`--output-type pdfa`) and a `--sidecar out.txt` for the plain text.
+5. Check quality: sample pages, compare OCR text with the image, and note low-confidence areas (handwriting, stamps, tables). OCR is never 100% accurate; say so in the report and keep the images as the source of truth.
+
+## 4. Convert, split, merge, rotate
+
+Use qpdf for structural work (no re-rendering, so quality is kept): `--pages` to select, split or merge, `--rotate`, `--linearize` for web viewing, `--split-pages` for one file per page. Office formats to PDF via `libreoffice --headless --convert-to pdf` if installed. Recipes in `references/forms-and-conversion.md`.
+
+## 5. Fill forms
+
+1. List the fields first (`pypdf` `reader.get_fields()` or `pdftk in.pdf dump_data_fields_utf8`) and map them to the user's data. **Show the user the mapping** before filling.
+2. Fill with pypdf (`writer.update_page_form_field_values`) or pdftk (`fill_form data.fdf output out.pdf`), one output per record.
+3. Checkboxes and radio buttons need their export value (often `/Yes` or `/On`), not `True`.
+4. Flatten (`pdftk ... flatten`, or render) only when the user wants a non-editable copy; keep the unflattened version.
+5. XFA forms (dynamic forms from Adobe LiveCycle) are not AcroForms; most open tools cannot fill them. Stop and tell the user.
+6. Open each output and check a few fields by eye.
+
+## 6. Redact (must be real)
+
+Drawing a black box over text does **not** redact it: the text stays in the content stream and is copied, searched and extracted. Follow `references/redaction.md` exactly:
+
+1. Find every occurrence (search the text layer and the OCR layer; patterns for IDs, emails, phone numbers; ask the user for names and terms).
+2. Apply true redaction that removes the text, vector graphics and image pixels under each area (PyMuPDF `add_redact_annot` then `apply_redactions`), or rasterise the pages and re-OCR the result.
+3. Remove metadata, XMP, attachments, annotations, form field values, bookmarks and earlier revisions (incremental updates) by rewriting the file.
+4. **Verify**: extract all text from the output and search for every redacted term; inspect metadata; check that the file has no incremental updates left. Any hit means the redaction failed.
+5. A person must review redactions before disclosure. Legal, court or regulatory redaction rules (what must be withheld, how it is marked) are outside this skill; the user's legal team decides.
+
+## 7. Translate with layout
+
+1. Extract text per block with its position, font size and page (pdfplumber words or PyMuPDF blocks).
+2. Translate block by block, with the tool or service the user approves; keep numbers, names, codes and dates unchanged and give the translator a glossary.
+3. Rebuild: either an overlay PDF that writes translated text into each block's box (shrink font to fit, flag blocks that overflow) or a bilingual table (source, translation, page) when layout matters less than accuracy.
+4. Fonts must cover the target script (for example Noto fonts for Bengali, Arabic, CJK); check that no characters render as boxes. Complex scripts need a shaping-capable text engine.
+5. Machine translation of legal, medical or official documents needs review by a qualified translator before use; certified translations are out of scope.
+
+## Never
+
+- Never modify or delete the original files.
+- Never present OCR or extracted numbers as exact without a spot check.
+- Never send documents to an online service without the user's explicit approval.
+- Never call a document redacted until the verification step passes.
+
+## Report back
+
+```
+Files: n (born-digital / scanned / mixed / damaged)
+Outputs: <paths>   Tools and versions: ...
+Extraction: rows or pages extracted, spot checks done, mismatches
+OCR: languages, pages OCRed, known weak areas
+Redaction: terms and patterns, hits after verification (must be 0), metadata cleared
+Needs a human: legal review of redactions, translation review, unreadable pages
+```

package/skills/document-processing/evals/triggers.json

@@ -0,0 +1,17 @@
+{
+  "should_trigger": [
+    "Pull the transaction tables out of these 200 bank statement PDFs into one CSV",
+    "Make this folder of scanned contracts searchable with OCR, some are in French",
+    "Fill this PDF application form for each of the 50 rows in our spreadsheet",
+    "Redact every name and account number from this PDF and prove the text can't be copied back out",
+    "Translate this German PDF brochure into English but keep the layout of each page",
+    "The text I extract from this PDF comes out as (cid:34) gibberish, how do I get real text"
+  ],
+  "should_not_trigger": [
+    "Create a Word document report with a table of contents and page numbers",
+    "Translate the app's UI strings into French and German with message catalogs",
+    "Write the user guide for our CLI tool",
+    "Build a RAG pipeline that chunks and embeds our product manuals",
+    "Normalise the Bengali text in our search index so the same word always matches"
+  ]
+}

package/skills/document-processing/references/extraction-and-ocr.md

@@ -0,0 +1,71 @@
+# Extraction and OCR
+
+## Triage commands
+
+```bash
+pdfinfo in.pdf                      # pages, encryption, producer, PDF version
+pdffonts in.pdf                     # no fonts => likely scanned
+pdftotext -f 1 -l 3 -layout in.pdf - | head -50
+qpdf --check in.pdf                 # structural problems
+pdfimages -list in.pdf | head       # image resolution per page (OCR needs ~300 dpi)
+```
+
+## Text with pdfplumber
+
+```python
+import pdfplumber, json
+rows = []
+with pdfplumber.open("work/in/report.pdf") as pdf:
+    for i, page in enumerate(pdf.pages, start=1):
+        text = page.extract_text(layout=False) or ""
+        rows.append({"page": i, "text": text})
+json.dump(rows, open("work/out/report.text.json", "w"), ensure_ascii=False, indent=1)
+```
+
+Words with positions (for layout-aware work): `page.extract_words(keep_blank_chars=False, use_text_flow=True)` returns `x0, top, x1, bottom, text`.
+
+## Tables with pdfplumber
+
+```python
+ruled = {"vertical_strategy": "lines", "horizontal_strategy": "lines"}
+whitespace = {"vertical_strategy": "text", "horizontal_strategy": "text", "snap_tolerance": 3}
+
+with pdfplumber.open("work/in/statement.pdf") as pdf:
+    page = pdf.pages[0]
+    region = page.crop((0, 120, page.width, 700))   # x0, top, x1, bottom in PDF points
+    for table in region.extract_tables(ruled):
+        for row in table:
+            print(row)
+```
+
+Debug a table: `page.to_image(resolution=150).debug_tablefinder(ruled).save("debug.png")` and look at the detected lines.
+
+Tips:
+- Header rows that repeat on every page: drop them after the first page by matching the header.
+- Cells that wrap onto two lines become `"line1\nline2"`; join them, do not split into rows.
+- Numbers: strip thousands separators according to the document's locale (1.234,56 vs 1,234.56); keep the original string next to the parsed value.
+
+## OCRmyPDF
+
+```bash
+tesseract --list-langs                         # confirm language data is installed
+ocrmypdf -l eng --rotate-pages --deskew --clean --output-type pdfa --sidecar out.txt in.pdf out.pdf
+ocrmypdf -l eng+deu --skip-text in.pdf out.pdf     # mixed: only OCR pages without text
+ocrmypdf -l eng --redo-ocr in.pdf out.pdf          # replace a bad hidden text layer
+ocrmypdf -l ben --force-ocr in.pdf out.pdf         # rasterise and OCR everything
+```
+
+- OCRmyPDF 17 added `--mode skip|redo|force`; the legacy flags still work.
+- `--clean` needs the `unpaper` program installed; it only cleans the image Tesseract sees, not the output page (that is `--clean-final`).
+- `--redo-ocr` cannot be combined with `--deskew`, `--rotate-pages` or `--clean`.
+- Tesseract language codes are ISO 639-2 style (`eng`, `fra`, `deu`, `ben`, `hin`, `ara`, `chi_sim`, `jpn`). Choose `script/` models (for example `script/Devanagari`) when the language is unknown but the script is.
+- Very low-resolution scans: upscale is no substitute for a rescan at 300 dpi; tell the user.
+
+## Plain Tesseract on images
+
+```bash
+tesseract page.png out -l eng --psm 6       # writes out.txt; --psm 6 = single uniform block
+tesseract page.png out -l eng tsv           # word-level confidence in out.tsv
+```
+
+Use the `conf` column of the TSV to flag words under about 60 for human checking.

package/skills/document-processing/references/forms-and-conversion.md

@@ -0,0 +1,69 @@
+# Forms and structural conversion
+
+## qpdf
+
+```bash
+qpdf in.pdf repaired.pdf                                   # rewrite, fixes many damaged files
+qpdf --decrypt --password-file=pw.txt locked.pdf open.pdf  # only with the owner's permission; a file keeps the password out of shell history
+qpdf --empty --pages a.pdf b.pdf 1-3 -- merged.pdf          # all of a, pages 1-3 of b
+qpdf in.pdf --pages . 5-10 -- part.pdf                      # extract pages
+qpdf --split-pages in.pdf page-%d.pdf                       # one file per page
+qpdf in.pdf rotated.pdf --rotate=+90:2,4                    # rotate pages 2 and 4 by 90 degrees clockwise
+qpdf --linearize in.pdf web.pdf                             # fast web view
+qpdf --qdf --object-streams=disable in.pdf inspect.pdf      # human-readable objects for debugging
+```
+
+Office documents to PDF (if LibreOffice is installed):
+
+```bash
+libreoffice --headless --convert-to pdf --outdir work/out work/in/*.docx
+```
+
+## List form fields
+
+```bash
+pdftk form.pdf dump_data_fields_utf8
+```
+
+```python
+from pypdf import PdfReader
+reader = PdfReader("form.pdf")
+for name, f in (reader.get_fields() or {}).items():
+    print(name, f.get("/FT"), f.get("/V"), f.get("/_States_", ""))   # /_States_ lists checkbox values
+```
+
+If `get_fields()` is empty but the PDF looks like a form, it is either flat (no fields) or XFA. Check with `acro = reader.trailer["/Root"].get("/AcroForm")` then `acro is not None and "/XFA" in acro.get_object()`. Report which.
+
+## Fill with pypdf
+
+```python
+from pypdf import PdfReader, PdfWriter
+reader = PdfReader("form.pdf")
+writer = PdfWriter(clone_from=reader)
+values = {"FullName": "Ama Mensah", "Date": "2026-03-02", "Agree": "/Yes"}
+for page in writer.pages:
+    writer.update_page_form_field_values(page, values, auto_regenerate=False)
+writer.set_need_appearances_writer(True)   # ask viewers to regenerate appearances
+with open("filled.pdf", "wb") as fh:
+    writer.write(fh)
+```
+
+Check the pypdf version's docs: the signature of `update_page_form_field_values` has gained options (such as `flatten`) over releases.
+
+## Fill with pdftk
+
+Write an FDF or XFDF with the values, then:
+
+```bash
+pdftk form.pdf fill_form data.xfdf output filled.pdf            # keeps fields editable
+pdftk form.pdf fill_form data.xfdf output filled_flat.pdf flatten
+```
+
+Use `need_appearances` if the viewer shows empty fields after filling.
+
+## Batch filling from a CSV
+
+1. Read the CSV with the csv module (UTF-8, keep leading zeros as strings).
+2. Map CSV columns to field names with an explicit dict the user approved.
+3. One output per row named from a key column, sanitised to a safe filename.
+4. Log each row: output path, fields filled, fields missing.

package/skills/document-processing/references/redaction.md

@@ -0,0 +1,61 @@
+# Redaction that holds up
+
+A redaction has failed if any redacted value can be recovered by selecting, copying, searching, extracting text, reading metadata, opening an earlier revision or lifting an overlay. Check every one.
+
+## 1. Find targets
+
+```python
+import fitz, re   # PyMuPDF (AGPL; confirm the licence suits the user)
+patterns = [r"\b\d{3}-\d{2}-\d{4}\b", r"[\w.+-]+@[\w-]+\.[\w.]+"]
+terms = ["Ama Mensah", "Account 0012345678"]   # from the user
+doc = fitz.open("work/in/case.pdf")
+hits = []
+for page in doc:
+    text = page.get_text()
+    found = set(terms) | {m.group(0) for p in patterns for m in re.finditer(p, text)}
+    for t in found:
+        for rect in page.search_for(t):
+            hits.append((page.number + 1, t, rect))
+print(len(hits), "hits")
+```
+
+Scanned pages have no text to search: OCR them first, then search the OCR layer, and redact the image region as well as the hidden text.
+
+Show the user the hit list (page, term, count) before applying.
+
+## 2. Apply
+
+```python
+for page_no, _term, rect in hits:                    # the approved hit list from step 1
+    doc[page_no - 1].add_redact_annot(rect, fill=(0, 0, 0))
+for page in doc:
+    page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_PIXELS)   # removes text and the image pixels under each box; be explicit, defaults changed across versions
+doc.set_metadata({})
+doc.del_xml_metadata()
+for name in doc.embfile_names():
+    doc.embfile_del(name)
+doc.save("work/out/case.redacted.pdf", garbage=4, deflate=True, clean=True)   # full rewrite, no incremental update
+```
+
+Also remove, where present: form field values, annotations and comments, bookmarks that quote redacted text, and document-level JavaScript.
+
+Stronger alternative when the layout does not need to stay searchable: rasterise each page to an image at 200 to 300 dpi after drawing the boxes, build a new PDF from the images, then OCR it if search is needed. This leaves no original text objects at all.
+
+## 3. Verify (all must pass)
+
+```bash
+pdftotext work/out/case.redacted.pdf - | grep -i -E "Ama Mensah|0012345678|[0-9]{3}-[0-9]{2}-[0-9]{4}" && echo "FAIL: text still present"
+pdfinfo work/out/case.redacted.pdf            # author, title, subject, keywords cleared
+exiftool work/out/case.redacted.pdf           # if installed: XMP and other metadata
+grep -c "%%EOF" work/out/case.redacted.pdf    # more than 1 suggests incremental updates remain (a linearized file also has 2)
+qpdf --check work/out/case.redacted.pdf
+```
+
+pdftotext does not read annotation or form-field text: also list fields with pypdf `get_fields()` and annotations with PyMuPDF `page.annots()` and confirm none hold a redacted value.
+
+Then open the output, select all text on a redacted page, and paste it into a text file; search again. Check a few redacted areas zoomed in for half-covered characters.
+
+## 4. Hand-off
+
+- Report: terms and patterns used, hits found, hits after verification (must be zero), what metadata was removed.
+- A person must review every redacted page before release. What must legally be withheld, and how redactions must be labelled (for example exemption codes), is decided by the user's legal or records team, not by this process.