slopkeeper 0.1.0__py3-none-any.whl

slopkeeper/AGENT_GUIDE.md

@@ -0,0 +1,305 @@
+<!--
+SPDX-FileCopyrightText: 2026 Greg Brandt <brandt.greg@gmail.com>
+
+SPDX-License-Identifier: MIT
+-->
+
+# AGENT_GUIDE — how to drive SlopKeeper
+
+You are the calling agent. The user said something like "organize my
+directory" or "clean up my workspace". This tool is your hands and your
+seatbelt: it inventories, validates, moves and scaffolds. You classify.
+The human approves. **Never generate or execute a plan without showing the
+rendered markdown to the human first.**
+
+(Humans can also drive slopkeeper themselves: bare `slopkeeper` with no subcommand
+starts an interactive `/command` REPL. That is not your surface — drive
+the JSON subcommands documented here; they are scriptable and stable.)
+
+## Hard rules
+
+1. **No plan, no moves.** Every file operation goes through a validated
+   plan JSON. There is no other write path.
+2. **Dry-run is the default.** `apply` and `scaffold` and `undo` do nothing
+   destructive without `--execute`.
+3. **The human gate.** Render the plan (`plan render`) and present it.
+   The human edits the plan JSON or approves. Re-validate after any edit.
+4. **Respect reference risks.** `reference_risks` lists files cited from
+   files that are NOT moving. Prefer declaring the rewrite IN the plan
+   (`edits: [{file, find, replace}]` — applied after the moves, journaled,
+   undo-able; `edit.file` is the move-source path if the file moves,
+   else its current path, and `find` must occur exactly once). Otherwise:
+   move the citing files too, or get explicit human sign-off on the
+   breakage. Never silently accept. Citations from OUTSIDE the corpus
+   (skills, cron scripts, scratch harnesses) are never listed there —
+   apply scans the corpus only; sweep the directories that cite your
+   corpus with `refs --scan-path` (see "Authoring reference rewrites").
+   Recall limits: the scan is not exhaustive — basenames under 3
+   characters are never scanned; a citation from a file that shares the
+   cited file's basename is skipped (attribution would be ambiguous);
+   extensionless files and non-text extensions are never read. So sweep
+   critical directories with `slopkeeper refs --scan-path` too.
+5. **Unresolved is honest.** Files you can't classify go in
+   `unresolved` with a reason — never guess. `keep` is for top-level
+   entries that stay (e.g. `docs/`, a watchlist).
+6. **Corpus changed?** If apply raises `corpus_changed`, re-inventory and
+   regenerate the plan. `--force` only if the changes were yours and
+   expected. Staleness is content-based: a bare `touch` won't force a
+   re-plan, but a real edit — even one that keeps the byte size — will.
+
+## The protocol
+
+```
+1. slopkeeper doctor <root>
+     → strategy suggestion, token estimate, advice. Read it.
+
+2. slopkeeper inventory <root> [--fast --summary | --out inv.json]
+     → <800 files: full JSON with heads + cross-references (read it whole)
+       >800 files: --fast --summary, then per-dir inventories as needed
+
+3. slopkeeper clusters <root>
+     → similarity HINTS: content clusters, scrape pairs, dupes, hubs.
+       Name each cluster, merge/split by judgment. Never apply a cluster
+       as-is; and check `hubs` before moving a file cited by many others.
+
+4. slopkeeper strategies show <suggested-id>   (or your choice)
+     → the partitioning rubric. Apply it. Strategies:
+       role-bots | topics | projects | hybrid (default for mixed corpora)
+
+5. Write the plan JSON. Get the template:
+     slopkeeper plan schema
+   Record the inventory's fingerprint as based_on.corpus_fingerprint.
+   Cover every top-level entry: moves | keep | unresolved.
+   (keep accepts "name" or {"path": "name", "reason": "..."}.)
+
+6. slopkeeper plan validate <plan.json> <root>
+     → fix every error code it gives you. Then:
+   slopkeeper plan render <plan.json> <root> --out plan.md
+     → HUMAN GATE: show plan.md. Wait for approval.
+
+7. slopkeeper apply <plan.json> <root>            (dry-run)
+   slopkeeper apply <plan.json> <root> --execute  (for real)
+     → note reference_risks; do rewrite steps now (plain edits, no tool).
+       A plan whose bytes differ from the last `plan render --out`
+       (e.g. hand-edited JSON) gets a `render_warning` in the result —
+       informational only: re-render or get explicit sign-off if the
+       human approved the earlier render.
+
+8. slopkeeper scaffold <plan.json> <root> --execute
+     → writes AGENTS.md roster + per-folder AGENTS.md/STATE.md.
+       Existing files are never overwritten; you get *.slopkeeper-suggested
+       to merge by hand.
+
+9. Report: moved counts, risk resolutions, suggested-file merges needed,
+   journal path (for undo). Offer: slopkeeper undo <journal> <root>
+   (dry-run is the default; add --execute to actually restore)
+
+   PARTIAL APPLY: `apply --execute` can fail MID-RUN (an OSError on one
+   hop — permissions, disk full). The envelope then carries
+   "partial": true, the process exits non-zero, and next.command is
+   the exact undo for this journal. Read `errors`, fix the cause, run
+   the suggested `slopkeeper undo <journal> <root>` (dry-run is the default;
+   add --execute to actually restore), then re-apply. A source that
+   vanishes mid-apply is the same path: it lands in `errors` and the
+   run exits non-zero — never silent. Earlier hops stay journaled and
+   undoable throughout.
+```
+
+## Re-tending (the recurring loop)
+
+After the first sweep, `apply --execute` and `scaffold --execute` both
+write `.slopkeeper/manifest.json`. From then on, a tending pass costs
+tokens proportional to the slop shot in, not the corpus size:
+
+```
+1. slopkeeper status <root>
+     → new_files (with title+head: triage them without opening anything),
+       modified, removed, stale folders, still-unresolved items.
+
+2. slopkeeper draft <root> --out draft-plan.json
+     → OPTIONAL but usually the fastest path: auto-drafts a PARTIAL plan
+       by nearest filed neighbor (TF-IDF cosine). Confident files get a
+       move with the neighbor + score as the reason; everything else is
+       unresolved WITH the reason ("best neighbor only 0.03"). Draft is
+       a guess with a citation — review every move, then edit.
+       The stdout envelope carries COUNTS only ({drafted_count,
+       unresolved_count, new_count, undrafted, out_path}) — the plan
+       body lives only in the --out file. The default path resolves
+       against your CWD and is REFUSED if it lands inside the corpus
+       (`draft_out_inside_root`) — a draft written into the root would
+       shift its fingerprint and the next apply would refuse with
+       corpus_changed. When new files exceed what the draft could see
+       (status's 2000-new_files cap), `undrafted` carries the count and
+       the reason, and the next-step hint says so. A zero-move draft
+       (everything went unresolved) is still a valid plan: validate it
+       and apply --execute — the backlog is recorded for the human.
+
+3. Triage. Fix the draft: reclassify wrong moves, resolve the
+   unresolved ones (they're the actual judgment work), add folder
+   entries for any new folder the roster needs.
+
+4. apply --execute → scaffold --execute  (both record the manifest)
+```
+
+Notes:
+- A delta plan should carry `"partial": true` at the top level (draft
+  plans already do) — no --allow-partial flags needed.
+- `still_unresolved` in the status output is the human-decides backlog;
+  surface it every tend until it's empty.
+- `skipped_bad_unresolved` counts backlog entries whose `path` was
+  null or missing — they are skipped (state-file content is data,
+  never a crash); the next re-record drops them from the manifest and
+  records the same count there.
+- `modified` files are leads, not chores: a changed STATE.md or AGENTS.md
+  is the owner tending by hand — leave it alone; a changed runbook may
+  deserve a fresh look in your triage note.
+
+## Tending several corpora on a schedule
+
+Register the corpora you tend once; the registry is the loop's to-do
+list, so a scheduled job is one call instead of a shell script. It lives
+at `$XDG_CONFIG_HOME/slopkeeper/roots.json` (set `SLOPKEEPER_HOME` to
+relocate it, e.g. for tests); content is sorted, unique, absolute paths.
+
+```
+slopkeeper roots add <dir>     → register a corpus (idempotent; must exist)
+slopkeeper roots remove <dir>  → unregister (the corpus itself is untouched)
+slopkeeper roots list          → what is registered
+
+slopkeeper status --all        → ONE digest over every registered root:
+  per root {root, ok, never_tended, new_count, modified_count,
+  removed_count, stale_folders, unresolved_count}
+```
+
+The digest is counts, not listings — keep it skimmable, then drill into
+whichever roots are nonzero with the ordinary per-root `status`. A
+deleted root, a never-tended corpus, or a root whose status fails is a
+row (`ok: false` carries the error), never a failed run — the other
+roots always report. `--all` with an empty registry is a clean error:
+add roots first.
+
+## Authoring reference rewrites
+
+When a move risks breaking citations from stay-put files:
+
+```
+1. slopkeeper refs <root> <basename>
+     → exact citation sites: {file, line, text}.
+
+2. Citations also live OUTSIDE the corpus — skills, AGENT_GUIDE-style
+   guides, cron scripts, scratch harnesses. reference_risks never lists
+   them (apply scans the corpus only), so sweep the dirs that cite your
+   corpus explicitly:
+
+     slopkeeper refs <root> <basename> --scan-path ~/skills --scan-path ~/cron
+      → same {file, line, text} sites; every hit outside the corpus
+        carries "external": true and an absolute `file`. Dot-directories
+        and non-text files are skipped; the same per-file cap and size
+        limit as the in-tree scan apply (corpus_too_big refuses an
+        oversized dir, scan_path_missing a bad one).
+
+3. Add to the plan:
+     "edits": [{"file": "<citing file (move-source path if it moves)>",
+                "find": "<exact text from the citation line, unique>",
+                "replace": "<new path after the move>",
+                "reason": "fix citation"}]
+   find must occur exactly once in the file (edit_ambiguous otherwise);
+   if the content drifted, you get edit_stale. The target must be valid
+   UTF-8 (edit_not_utf8 otherwise) — non-UTF-8 files are refused, never
+   re-encoded. Edits run after the moves,
+   are journaled, and undo reverses them.
+
+4. To rewrite a citation in a file OUTSIDE the corpus, declare the
+   directory it lives under and use an ABSOLUTE edit path:
+
+     "external_roots": ["/abs/dir"],
+     "edits": [{"file": "/abs/dir/harness.sh",
+                "find": "tmp/docs/infra/config.yaml",
+                "replace": "docs/infra/config.yaml",
+                "reason": "external citation"}]
+
+   Every external_roots entry must be an absolute, existing directory
+   (plan_bad_external_root otherwise). With no external_roots declared,
+   an edit outside the root is rejected (path_escape) — the old rule,
+   unchanged. External edits run after the in-root ones, are journaled
+   under their absolute path, and undo reverses them. `plan render`
+   marks external edits so the human sees writes outside the corpus.
+```
+
+A citing file covered by an edit no longer counts as a reference risk,
+so `--strict-references` is usable when all rewrites are declared —
+in-root and external alike.
+
+## Error codes you will see
+
+| code | meaning | your move |
+|---|---|---|
+| `plan_incomplete` | top-level entries uncovered | add to moves/keep/unresolved |
+| `plan_no_moves` | `moves` is not a list, or is empty on a non-partial plan | a plan needs at least one move — unless it carries `"partial": true` (a zero-move partial plan is valid: the draft filed everything as unresolved; validate it and `apply --execute` records the backlog) |
+| `plan_stale_src` | a source vanished | re-inventory, regenerate |
+| `corpus_changed` | fingerprint mismatch | re-inventory; `--force` only if yours |
+| `dst_exists` | destination occupied | resolve collision in plan |
+| `referenced_moves` | strict mode: refs would break | rewrite refs first |
+| `plan_nested_moves` | dir + its contents both moved | move dir whole OR its files |
+| `plan_bad_keep` | keep entry is neither string nor {path} object | use `"name"` or `{"path": "name", "reason": "..."}` |
+| `plan_bad_unresolved` | an unresolved entry has no non-empty string `path` (null, missing, or blank) | every entry is `{"path": "<file>", "reason": "<why it waits>"}` — fix or drop it |
+| `edit_stale` | edit target missing or find-text gone | update the edit in the plan |
+| `edit_ambiguous` | find text occurs more than once | make find unique; split the edit |
+| `edit_not_utf8` | edit target is not valid UTF-8 (e.g. latin-1); editing would corrupt it | convert the file to UTF-8 first, or rewrite the citation by hand |
+| `plan_bad_external_root` | external_roots entry is not an absolute, existing directory | create or fix the directory, or omit external_roots |
+| `scan_path_missing` | `refs --scan-path` directory does not exist | create it or fix the path |
+| `no_manifest` | draft with no prior tend | run a first tend first |
+| `draft_out_inside_root` | `draft --out` resolves inside the corpus root — a draft written into the corpus shifts its fingerprint (the next apply would fail `corpus_changed`) | write the draft outside the corpus: run from another directory (the default `draft-plan.json` resolves against your CWD) or pass an explicit path elsewhere |
+| `path_escape` | path outside root, contains a null byte, or is the root itself | use relative paths to real files; for edits outside the root, declare `external_roots` |
+| `manifest_unreadable` / `manifest_bad_schema` | `.slopkeeper/manifest.json` corrupt | delete it; `status --record` rebuilds |
+| `tending_in_progress` | another tend holds `.slopkeeper/tend.lock` (execute-mode apply/scaffold/undo, `status --record`) | wait for the other tend, or check for a dead agent's session; the OS frees the lock on process death |
+| `root_not_dir` | `roots add` target is not an existing directory | register corpora that exist; SlopKeeper doesn't create them |
+| `root_not_registered` | `roots remove` for a path not in the registry | check `slopkeeper roots list` |
+| `root_missing` (in `status --all` data) | a registered root is no longer a directory | re-add its new path, or `roots remove` if it's gone |
+| `roots_unreadable` / `roots_bad_schema` | roots registry file corrupt/foreign | delete it; re-add roots |
+| `no_roots` | `status --all` with an empty registry | `roots add` first, or run `status <root>` directly |
+| `no_root` | `status` with neither a root nor `--all` | name a root or use `--all` |
+| `similarity_cap` | clusters handed more than 3000 files (the O(n²) matcher would crawl) | partition per-directory: `slopkeeper clusters <root>/<subdir>`, or organize smaller scopes |
+| `plan_not_json` | plan file is not valid JSON | fix the JSON syntax and re-validate |
+| `plan_unreadable` | plan file cannot be read | check the path exists and is readable |
+| `plan_not_object` | plan JSON parses but is not an object | the plan is a JSON object with `schema` and `moves` — not a bare list |
+| `plan_missing_field` | plan is missing a required field | add it; see `slopkeeper plan schema` |
+| `plan_bad_schema` | `plan.schema` is not the current schema | regenerate from the current template: `slopkeeper plan schema` |
+| `plan_wrong_root` | the plan's `corpus_root` names a different corpus | re-target or regenerate the plan for this root |
+| `plan_bad_move` | a move lacks `from`/`to` | every move is `{"from": "...", "to": "..."}` |
+| `plan_noop_move` | a move's `from` equals its `to` | drop the no-op move |
+| `plan_dup_src` | two moves share a source | one move per file |
+| `plan_dup_dst` | two moves share a destination | resolve the collision — chains (a→b, b→c) are fine, two files onto one path are not |
+| `plan_dst_under_src` | a destination is inside another move's source | move the directory whole, or flatten the plan |
+| `plan_bad_edit` | an edit lacks `file`/`find`/`replace` | every edit carries all three; see `slopkeeper plan schema` |
+| `plan_bad_folder` | a folder entry has no `path` | every folder entry names a path |
+| `plan_bad_kind` | folder `kind` is unknown | kind is one of bot, case, topic, project, archive, scratch |
+| `move_cycle` | the moves form a cycle (a→b→a) | remove the circular moves from the plan |
+| `symlink_escape` | a path resolves outside the root via a symlink | remove the symlink or move its target explicitly |
+| `no_corpus` | the argument to `doctor` is not a directory | pass the directory you want to organize |
+| `no_plan` | REPL: no compiled plan to act on (`/apply`/`/render`/`/scaffold` before `/plan`, or decisions changed since) | `/plan` first — it compiles decisions, validates and saves; `/load` + `/plan` for an agent-authored plan |
+| `no_such_move` | REPL: not a move number | `/moves` lists the working plan's moves |
+| `no_basename` | REPL: `/refs` called with no basename | `/refs <basename>` |
+| `no_path` | REPL: a required path argument is missing or empty | see the usage line in the message |
+| `no_such_journal` | REPL: not a journal index (or a non-numeric one) | `/undo` lists journals; `/undo <n> [!]` |
+| `cat_unreadable` | REPL: the `/cat` target cannot be read | check the path (it is relative to the corpus root) |
+| `edit_unreadable` | an edit target cannot be read | check the file exists and is readable |
+| `journal_unreadable` | the journal file cannot be read | check the journal path |
+| `journal_empty` | the journal contains no entries | nothing to undo |
+| `corpus_too_big` | a corpus or scan dir exceeds its cap (2000 for the refs scan, 3000 filed files for draft) | partition per-directory, scan a narrower dir, or grep by hand |
+| `unknown_strategy` | no strategy with that id | `slopkeeper strategies` for the available ids |
+
+## Sizing
+
+- Inventory: at ≤ 800 files the full JSON (heads + cross-references)
+  is one read — read it whole. Above that a full read is too big
+  (~1000 files ≈ 150k tokens): use `--fast --summary`, then
+  `inventory <subdir>` for the dirs you plan to reorganize, plus
+  title/head from the summary. Inventory refuses past 8000 files.
+- Reference scanning: apply and `refs` scan at most 2000 files — past
+  that the result is `reference_scan: "skipped_corpus_too_big"`,
+  never a silent empty risk set. (The inventory's embedded
+  cross-reference scan only runs at ≤ 800 files.)
+- Clusters and draft: 3000 files. Past it `clusters` refuses with
+  `similarity_cap` (draft: `corpus_too_big`) — partition
+  per-directory instead.

slopkeeper/__init__.py

@@ -0,0 +1,12 @@
+# SPDX-FileCopyrightText: 2026 Greg Brandt <brandt.greg@gmail.com>
+#
+# SPDX-License-Identifier: MIT
+
+"""slopkeeper — inventory, partition and scaffold messy file corpora.
+
+Agent-first CLI: the calling agent is the brain; this tool is the hands
+and the seatbelt. No LLM calls, no network, stdlib only.
+"""
+
+__version__ = "0.1.0"
+VERSION = __version__

slopkeeper/__main__.py

@@ -0,0 +1,8 @@
+# SPDX-FileCopyrightText: 2026 Greg Brandt <brandt.greg@gmail.com>
+#
+# SPDX-License-Identifier: MIT
+
+from .cli import main
+import sys
+
+sys.exit(main())