textus 0.54.1 → 0.54.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '08d1e232ff648999588cfcfe0cccafd7954a020805f3e344aabdbbb2798b51af'
-  data.tar.gz: 4b8545d90d8aa990d63565f1d2caaff7e3e519d7504ae22a04dad68507c90949
+  metadata.gz: 53f01828833f9c5e6cb0249fe513b92d6afbb210d90cfca10f51c560c8b2e2ca
+  data.tar.gz: 1956232435c941479290225e20168979beb270a6e2f5874df37c302e170a43e7
 SHA512:
-  metadata.gz: 28b8de7a9e84208008c7082cd597d6905b743fe035b0344953cc82d5581f8a6ce98abf4bae40e5f1040780a1874743e7403695831709a539ea2cd4fd7dbc4c79
-  data.tar.gz: 9be159077a4f4e17e0118b3efea9085d3e5f6f463d7f558d6fcfdc45912c07cde6fe0824de213171806245cbca3b38f689f6b6922a2f7ddf9fd4dcb0c0bdc2a7
+  metadata.gz: c086755b90faab7ecebdeebd722e9458dbede54c5efa9be212d6d861c999655093166a1f61826aa9e09ebee8cd1cf57996d958bd410c3b682763d911ed44abe5
+  data.tar.gz: aad04f9470c3de178ad6c762ef3102528cbdc94922fedb79d121ffbadc23acc5dd3f52871f6b3c0c75664afdb968c12fa814b88fbedcbd0d914161e4e1f337ab

data/CHANGELOG.md

@@ -9,6 +9,19 @@ The **gem version** (`0.x.y`) is distinct from the **protocol version**
 bump is a breaking change that requires a store migration; the gem version
 tracks both additive improvements and breaking protocol bumps independently.
 
+## 0.54.2 — 2026-06-18 — ingest dedup, .run → .state rename
+
+### Added
+
+- **Content-hash dedup in `Action::Ingest`**: re-ingesting the same URL or file bytes silently creates a supersede chain — new entry with full content, old stripped to tombstone with a `superseded_by` pointer.
+- **`Ports::RawIndex`**: lightweight YAML-based O(1) index for raw lane content-hash/URL lookups, stored at `.textus/.state/indexes/raw.yaml`.
+- **`content_hash` field**: every new raw entry stores a `sha256:` content hash for fast dedup lookups.
+- **Asset file move on re-ingest**: asset files are moved (not copied) to the new date path on supersede.
+
+### Changed
+
+- **`.run` → `.state` rename**: the runtime artifact directory is renamed from `.textus/.run/` to `.textus/.state/`. The `state/` subdirectory becomes `cursors/`. Layout path methods follow automatically. Manual migration step: `mv .textus/.run .textus/.state`.
+
 ## 0.54.1 — 2026-06-17 — MCP SDK, dry ecosystem, ERB templates
 
 ### Added
@@ -406,7 +419,7 @@ No `textus/3` wire-format change. Manifest schema gains one optional, backward-c
   segment-wise globstar — `**` spans zero or more path segments; within a
   segment `*` is anchored and `{a,b}` alternates (stdlib `File.fnmatch`,
   no new dependency). Documented in
-  [`docs/reference/zones.md`](docs/reference/zones.md#nested-entries).
+  [`docs/reference/zones.md`](docs/reference/lanes.md#nested-entries).
 
 ### Internal
 

data/README.md

@@ -39,6 +39,9 @@ flowchart LR
     agent -->|keep| notebook["notebook<br/>(workspace)"]
     agent -->|propose| proposals["proposals<br/>(queue)"]
     automation -->|drain| artifacts["artifacts<br/>(machine)"]
+    human -->|ingest| raw["raw<br/>(intake)"]
+    agent -->|ingest| raw
+    automation -->|ingest| raw
 
     proposals ==>|human accept| knowledge
     knowledge -.->|projection source| artifacts
@@ -46,13 +49,15 @@ flowchart LR
     classDef actor fill:#238636,stroke:#2ea043,color:#fff;
     classDef gate fill:#9e6a03,stroke:#bb8009,color:#fff;
     classDef anchor fill:#1f6feb,stroke:#388bfd,color:#fff;
+    classDef intake fill:#6e40c9,stroke:#8b5cf6,color:#fff;
     class human,agent,automation actor;
     class proposals gate;
     class knowledge anchor;
+    class raw intake;
 ```
 
 *Each actor writes only into its own lane; low-trust input climbs to authoritative lanes only by passing a guarded transition (an agent's proposal needs a human `accept`).*
-*Colour legend: **green** = writers · **amber** = the review gate (`proposals`) · **blue** = the trust anchor (`knowledge`).*
+*Colour legend: **green** = writers · **amber** = the review gate (`proposals`) · **blue** = the trust anchor (`knowledge`) · **purple** = write-once intake (`raw`).*
 
 The point of those lanes is to **build context you can trust**. Place each lane on two axes — how durable it is, and how much you can rely on it without review — and the value shows up as a climb: the high-trust corner (durable *and* authoritative = `knowledge`) is the one place nothing is *written* directly. It's *earned* by crossing the `accept` gate.
 
@@ -64,9 +69,10 @@ DURABLE       │  notebook                │  knowledge  ★ the goal
 (kept)        │  agent's working truth   │  canon — a human authors      │
               │  durable, but low-trust  │  here · the context you ship  │
               ├──────────────────────────┼───────────────────────────────┤
-TRANSIENT     │  artifacts.*             │  proposals  (queue)           │
-(staging)     │  computed outputs and    │  a candidate, in review       │
-              │  external inputs         │  ▲ climbs via human accept    │
+TRANSIENT     │  artifacts  (outputs)    │  proposals  (queue)           │
+(staging)     │  computed, machine-made  │  a candidate, in review       │
+              │  raw        (inputs)     │  ▲ climbs via human accept    │
+              │  ingested, write-once    │                               │
               └──────────────────────────┴───────────────────────────────┘
                 raw material ──── propose ────► a human accept lifts it to canon
 ```
@@ -135,7 +141,7 @@ bundle exec exe/textus --help
 
 ## What `textus init` gives you
 
-You get `.textus/` with all four lane directories under `data/`, baseline schemas, a starter manifest, and a gitignored `.run/` for disposable runtime state (the audit log, per-role cursors, produce locks). Roles declare capabilities; each lane declares a `kind:`, and write authority is derived from the role's capabilities crossed with the lane's kind:
+You get `.textus/` with all five lane directories under `data/`, baseline schemas, a starter manifest, and a gitignored `.state/` for disposable runtime state (the audit log, per-role cursors, produce locks). Roles declare capabilities; each lane declares a `kind:`, and write authority is derived from the role's capabilities crossed with the lane's kind:
 
 ```yaml
 roles:
@@ -156,17 +162,18 @@ lanes:
   schemas/               # YAML field shapes per entry family
   templates/             # ERB templates for produced entries
   workflows/             # Ruby workflow files (Textus.workflow DSL) for data acquisition
-  .gitignore             # generated — ignores .run/ and any tracked:false entries
+  .gitignore             # generated — ignores .state/ and any tracked:false entries
   data/                  # one dir per lane; kinds + capabilities are in the manifest above
     knowledge/           # e.g. identity (knowledge.identity.*), voice, decisions, notes
     notebook/
     proposals/
     artifacts/           # machine lane: computed outputs + external inputs
-  .run/                  # disposable runtime state — gitignored, safe to delete (ADR 0038)
+  .state/                  # disposable runtime state — gitignored, safe to delete (ADR 0038)
     audit/audit.log      # append-only NDJSON event ledger, every write (rotates at ~10 MB)
-    state/cursor.<role>  # per-role pulse cursor — where `pulse --since` resumes
+    cursors/<role>       # per-role pulse cursor — where `pulse --since` resumes
     locks/               # per-key produce locks + the produce mutex
     sentinels/           # publish bookkeeping (target sha) — regenerated on drain (ADR 0070)
+    indexes/raw.yaml     # raw lane content-hash/URL index — regenerable cache
 ```
 
 Manifest `path:` fields are relative to `.textus/data/`. So `knowledge.notes.org.jane` lives at `.textus/data/knowledge/notes/org/jane.md`.
@@ -195,14 +202,14 @@ For a worked store — knowledge entries, a staged proposal, schemas, ERB templa
 - **Agent loop.** `textus boot` orients a fresh session; `textus pulse --since=N` is the per-turn heartbeat (changed entries, pending proposals, index etag for catalog drift detection). ([docs/how-to/agents-mcp.md](docs/how-to/agents-mcp.md))
 - **MCP surface.** The official `mcp` Ruby SDK drives the stdio JSON-RPC server; protocol version auto-negotiated up to `2025-11-25`. Wire textus into Claude Code, Cursor, or any MCP host in one config block.
 - **`textus doctor`.** Health checks across schemas, workflow registrations, keys, sentinels, and the audit log.
-- **`raw` lane kind.** A write-once intake lane for external data that hasn't been reviewed yet; carries the `ingest` capability.
+- **`raw` lane and `ingest` verb.** Write-once intake lane for external URL bookmarks, files, and binary assets. Three source kinds (`url`/`file`/`asset`); daily key derivation; notebook stub per ingest. See "Intake and ingest" section below.
 
 ## CLI and lanes
 
 Every command operates on one store, located in this order: `--root <path>` flag → **`TEXTUS_ROOT`** env → walk up from the working directory for a `.textus/` ([SPEC §3.1](SPEC.md)). Write verbs require `--as=<role>`, resolved as: `--as` flag → **`TEXTUS_ROLE`** env → `.textus/role` file → default `human` ([SPEC §5.1](SPEC.md)). Default roles: `human`, `agent`, `automation` (rename or add your own in the manifest's `roles:` block). All verbs accept `--output=json` and return the envelope defined in [SPEC §8](SPEC.md).
 
 - Full verb table — read, write, health, scaffolding — is in [SPEC §9](SPEC.md).
-- Lane semantics and the capability × lane-kind mapping live in [SPEC §5](SPEC.md), with the reference in [`docs/reference/zones.md`](docs/reference/zones.md).
+- Lane semantics and the capability × lane-kind mapping live in [SPEC §5](SPEC.md), with the reference in [`docs/reference/lanes.md`](docs/reference/lanes.md).
 
 `textus boot` prints the same information for the current store: lanes, entry families with schemas, registered workflows, write flows, and the verb catalog. Run it inside a store and you get the live picture; reach for the SPEC when you want the contract.
 
@@ -213,7 +220,7 @@ Produced entries (`kind: produced`) declare how they're acquired in one `source:
 - **`source: { from: external, command: "...", sources: [...] }`** — *externally managed*: an out-of-band command or workflow writes the file; textus tracks staleness via declared `sources`.
 - **`source: { from: external, command: "true", sources: [] }` + a workflow** — *workflow-driven*: a `Textus.workflow` block (in `.textus/workflows/`) acquires and shapes the data on `drain`.
 
-Publishing is one typed `publish:` block (ADR 0052/0094). Each target is either `{ to: path, template?: name }` for a single file (optionally rendered through an ERB template) or `{ tree: "dir" }` to mirror a whole stored subtree. Sentinels for every published file live under `.textus/.run/sentinels/` (git-ignored, regenerated on drain). See SPEC §5.2, §5.3, §5.12.
+Publishing is one typed `publish:` block (ADR 0052/0094). Each target is either `{ to: path, template?: name }` for a single file (optionally rendered through an ERB template) or `{ tree: "dir" }` to mirror a whole stored subtree. Sentinels for every published file live under `.textus/.state/sentinels/` (git-ignored, regenerated on drain). See SPEC §5.2, §5.3, §5.12.
 
 Templates live in `.textus/templates/` as ERB files (`.erb`). The template receives the entry's `content` hash as local variables via `ERB#result_with_hash`. If `inject_boot: true`, a `boot` variable is also available with the live orientation context.
 
@@ -222,7 +229,7 @@ Templates live in `.textus/templates/` as ERB files (`.erb`). The template recei
 textus extends through **workflows** — a `Textus.workflow` block placed in `.textus/workflows/**/*.rb`. Each workflow matches a produced entry by key glob, then runs one or more named steps to acquire its data:
 
 ```ruby
-# .textus/workflows/artifacts/my_report.rb
+# .textus/workflows/docs/my_report.rb
 Textus.workflow "my_report" do
   match "artifacts.my-report"
 
@@ -238,27 +245,47 @@ end
 
 `drain` discovers all workflow files, matches them against produced entries, and runs the steps. The result is written back to the entry's data path; `publish:` then copies it to its consumer paths.
 
-## Hooks
+## Intake and ingest
 
-Out-of-band event reactions use the hook DSL. Drop a file anywhere in `.textus/workflows/` and use `Textus.hook`:
+The `raw` lane is the inbound counterpart to `artifacts`: where `drain` materialises
+**outbound** computed outputs, `ingest` receives **inbound** external source material.
 
-```ruby
-Textus.hook do |reg|
-  reg.on(:entry_written) do |key:, envelope:, **|
-    # fire-and-forget — runs after every successful write
-    $stderr.puts "wrote #{key} (etag #{envelope.etag[0, 8]})"
-  end
-end
-```
+**The ingest principle:** prefer a reference over a copy. Store body or asset only when the
+content itself is the value — human-authored notes, brainstorm outputs, context you want to
+annotate. For everything else, the URL is enough. If the source is private or
+access-restricted, set `access: private` in `source:` so downstream workflows can handle it
+appropriately.
+
+**Three source kinds:**
+
+| Kind | Stores | Use when |
+|------|--------|----------|
+| `url` | URL reference only (`body: null`) | Bookmarking a page, skill, or doc for later annotation |
+| `file` | File body text | Valuable human-authored content (brainstorm notes, meeting summaries) |
+| `asset` | Binary at `assets/raw/` | Screenshots, PDFs — only when the asset itself is the artefact |
 
-Observable events: `:entry_written`, `:entry_deleted`, `:entry_fetched`, `:entry_renamed`, `:entry_produced`, `:entry_published`, `:produce_failed`, `:proposal_accepted`, `:proposal_rejected`, `:store_loaded`, `:session_opened`, `:entry_fetch_started`, `:entry_fetch_failed`.
+**Write-once** — the same slug on the same day cannot be overwritten. Delete and re-ingest to replace.
+
+```sh
+# bookmark a skill reference — URL only, body stays null
+textus ingest url agentskills-io-brainstorming \
+  --url=https://agentskills.io/skills/brainstorming \
+  --label="brainstorming skill" \
+  --as=agent
+
+# see what landed in the raw lane
+textus list --lane=raw
+
+# a notebook stub was created alongside — annotate it
+textus get notebook.notes.raw
+```
 
-Stale produced entries are re-materialised by `drain`, not by reads — `get` is a pure read that annotates the returned envelope with a freshness verdict (ADR 0089).
+Stale produced entries are re-materialised by `drain`, not by reads — `get` is a pure read (ADR 0089).
 
 ```sh
 textus drain --as=automation                  # re-materialise every stale produced entry
-textus drain artifacts.feeds --as=automation  # scope to one prefix
-textus get artifacts.feeds.calendar.events    # a pure read; carries a freshness verdict
+textus drain artifacts.feeds.skills --as=automation # scope to one prefix
+textus get artifacts.feeds.skills                  # a pure read; carries a freshness verdict
 ```
 
 Schemas (`.textus/schemas/<name>.yaml`) declare field shapes, per-field `maintained_by:` ownership, and an `evolution:` block (`added_in`, `deprecated_at`, `migrate_from`). Full contract in SPEC §5.8.