instar 1.3.889 → 1.3.891

package/src/scaffold/templates/jobs/instar/feedback-factory-process.md

@@ -1,12 +1,12 @@
 ---
-name: Feedback-Factory Processor
-description: "Cadenced clustering pass over the canonical feedback-factory store. Runs the already-parity'd processUnprocessed pipeline via POST /feedback-factory/process: it reads unprocessed fleet reports, groups them into dedup clusters (similarity/Jaccard), auto-reopens a cluster on a possible-regression merge, and flips each processed item unprocessed→processing. Appends LOCAL JSONL only — no external action, no force-close of a curated cluster. Dev-gated dark (feedbackFactory.processing): LIVE on a development agent, the route 503s on the fleet so this job exits silently. Tier-1 supervised (this haiku job wraps the deterministic endpoint and sanity-checks the pass result against the post-pass stats). Spec docs/specs/feedback-factory-migration.md §191 (the wiring that makes the ported processor not dead code)."
+name: Feedback-Factory Operating Drain
+description: "Cadenced end-to-end feedback drain. POST /feedback-factory/drain/tick clusters canonical input, runs registered frontier-model readiness judgment inside deterministic floors, enqueues one durable outbox row per readiness epoch, and—after separate consumer promotion—creates and reads back one Initiative task. Development-agent processing/drain is live; fleet remains dark; consumer ships simulation-first. Spec: docs/specs/feedback-factory-operating-drain.md."
 schedule: "*/30 * * * *"
 priority: low
 expectedDurationMinutes: 2
 model: haiku
 supervision: tier1
-enabled: false
+enabled: true
 tags:
   - cat:feedback-factory
   - role:worker
@@ -16,20 +16,21 @@ toolAllowlist: "*"
 unrestrictedTools: true
 mcpAccess: none
 ---
-Run one feedback-factory clustering pass. This is a mechanical, near-silent watchdog — do NOT message the user. It exists because the canonical store ingests new fleet reports continuously, but nothing clusters/triages them without a production trigger; this job IS that trigger on a cadence.
+Run one feedback-factory operating-drain tick. This is a near-silent operated cadence — do NOT message the user. The server owns every transition; this job only triggers and sanity-checks one bounded run.
 
 AUTH="${INSTAR_AUTH_TOKEN:-$(python3 -c "import json; v=json.load(open('.instar/config.json')).get('authToken',''); print(v if isinstance(v, str) else '')" 2>/dev/null)}"
 AGENT_ID="${INSTAR_AGENT_ID:-$(python3 -c "import json; print(json.load(open('.instar/config.json')).get('projectName',''))" 2>/dev/null)}"
 PORT="${INSTAR_PORT:-4042}"
+NONCE="feedback-drain-$(date +%s)-$$"
 
-1. Confirm the capability is live. Read the stats first:
-   `curl -s -H "Authorization: Bearer $AUTH" -H "X-Instar-AgentId: $AGENT_ID" http://localhost:$PORT/feedback-factory/stats`
-   A 503 means the feature is dark for this agent (`feedbackFactory.processing` gated off) — exit silently, there is nothing to do. On 200 the body is `{ total, byStatus, clusterCount, dispatchCount, lastWriteAt }`. Note `byStatus.unprocessed` — that is the work in front of this pass.
+1. Confirm the operated drain is live:
+   `curl -s -H "Authorization: Bearer $AUTH" -H "X-Instar-AgentId: $AGENT_ID" http://localhost:$PORT/feedback-factory/drain/status`
+   A 503 is expected only on a fleet-dark install. Read `.instar/config.json`: if `developmentAgent` is not true, exit silently. If it is true, a 503 is degradation—fail this job run so JobRun history and the server audit expose it; never treat it as healthy.
 
-2. Trigger one clustering pass:
-   `curl -s -X POST -H "Authorization: Bearer $AUTH" -H "X-Instar-AgentId: $AGENT_ID" http://localhost:$PORT/feedback-factory/process`
-   The response is `{ processed, metrics: { captured, created, merged, reopened }, stats: { total, byStatus, clusterCount, dispatchCount, lastWriteAt } }`. The endpoint does the deterministic work: it reads unprocessed items + active clusters, decides merge/create per item (parity'd logic), auto-reopens on a possible-regression merge, and marks each item processed. It appends LOCAL JSONL only — it takes no external action and never force-closes a curated cluster.
+2. Trigger one bounded drain tick:
+   `curl -s -X POST -H "Authorization: Bearer $AUTH" -H "X-Instar-AgentId: $AGENT_ID" -H "X-Instar-Request: 1" -H "X-Instar-Request-Nonce: $NONCE" http://localhost:$PORT/feedback-factory/drain/tick`
+   A 202 response reports `{ runId, accepted, reason? }`. Concurrent triggers return the active run rather than starting a second writer. Poll the status route, bounded to 90 seconds, until `lastRun.runId` matches and `lastRun.state` is no longer `accepted` or `running`; never start a second tick while polling.
 
-3. **Tier-1 supervision (your job).** Sanity-check the pass against its own stats before concluding: `processed` should equal the number of items that were `unprocessed` going in (step 1's `byStatus.unprocessed`), and the post-pass `stats.byStatus.unprocessed` should have dropped by `processed`. `created + merged` from `metrics` should account for the processed items. If the numbers are internally inconsistent (e.g. `processed > 0` but `unprocessed` did not drop), do NOT retry-flood — note it once and exit; the next tick re-attempts.
+3. **Tier-1 supervision.** Accept terminal `succeeded` and `no-op`. A `degraded` run must carry a nonempty reason. Never retry in the same run; the durable queue and next cadence own retry. In simulation, canonical claimed/completed counts must not advance. In live mode, completed may never exceed the durable claimed/linked history or the configured batch bound.
 
-4. Exit silently. This job is just the cadence — it produces dedup-cluster signals, not user messages. Do NOT relay anything to Telegram and do NOT summarize. If a curl fails, that failure is itself recorded server-side; do not retry-flood.
+4. Exit silently. Do NOT relay anything to Telegram and do NOT summarize. The drain's durable run row, metrics, and bounded self-heal/attention path own observability.

package/src/scaffold/templates.ts

@@ -1680,6 +1680,10 @@ A proactive backstop that posts ONE purely-observational liveness line when an a
 - **It can't spam you (three LOCAL brakes, NOT dedup):** a long user-silence gate that ANY outbound (including your own normal reply) resets; a per-topic emit-cooldown; and a widening per-run backoff (25→40→60→90m) with a hard cap (~6 lines per run). A 24h silent-but-working run yields a handful of hedged lines, never dozens. Output advancing proves only LIVENESS, never that work is progressing (a log/retry loop also advances output) — which is exactly why the wording is liveness-only.
 - **Signal-only:** it only ever ADDS a line — never blocks, delays, or rewrites your real messages. Every predicate fails CLOSED (no emit) on any uncertainty (can't read history, the shared snapshot is unavailable, the run is mid-move to another machine). The interpolated \`focus\` is scrubbed for credentials/secrets/paths (drop-to-generic on any match), length-clamped, and HTML-escaped before it's ever shown.
 - **Status:** \`curl -H "Authorization: Bearer $AUTH" http://localhost:4042/autonomous-heartbeat\` → \`{ enabled, dryRun, silenceThresholdMinutes, lastTickAt, topicsConsidered, lastEmits }\` (503 when dark). Ships dark on the fleet + \`dryRun: true\` on a dev agent (logs "would emit" on the SAME cooldown/budget as live). Tune/disable: \`monitoring.autonomousHeartbeat\` (\`dryRun\`, \`silenceThresholdMinutes\`, \`tickIntervalMs\`, \`maxHeartbeatsPerRun\`). Spec: \`docs/specs/autonomous-progress-heartbeat.md\`.
+
+## Autonomous Throughput Floor
+
+A pull/audit-only view measures project PR movement and manager outbound silence for active autonomous runs. It never notifies, dispatches, remediates, or creates attention. Read \`GET /autonomous/throughput-floor\` when investigating a quiet run. HOLD still requires an actual open approval gate plus authoritative saturation of every non-gated lane; v1 has no lane authority and cannot grant HOLD. A proactive surface is follow-on work gated on a separately converged SelfHealGate.
 `;
 
   if (hasTelegram) {

package/upgrades/1.3.890.md

@@ -0,0 +1,19 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+Feedback Factory now has an operated, development-gated end-to-end drain: bounded incremental source projection and clustering, registered frontier-model agent readiness, a durable fenced queue, simulation-first consumer promotion, and exact-key Initiative handoff with crash recovery. Canonical accepted-intake pipelines also gain a registered completeness standard plus structural and runtime evidence guards, so future pipelines cannot silently stop after persistence.
+
+## What to Tell Your User
+
+Feedback Factory can now carry accepted feedback all the way into durable, agent-approved work while staying simulated until its consumer is deliberately promoted.
+
+## Summary of New Capabilities
+
+- Operated feedback intake, clustering, readiness approval, queueing, and Initiative handoff.
+- Agent-first frontier-model readiness decisions with a human escalation path.
+- Fenced ownership, cancellation, backup and restore, failover recovery, and bounded self-healing.
+- A dashboard view for backlog, readiness, consumer posture, and recovery health.

package/upgrades/1.3.891.md

@@ -0,0 +1,30 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+Adds a fleet-dark, pull/audit-only view for active autonomous runs. It measures bounded project PR
+movement and manager outbound silence, persists a restart-safe per-run read breaker, and exposes the
+scrubbed result at `GET /autonomous/throughput-floor` for the dashboard and operator.
+
+It does not notify, create attention, dispatch, restart, remediate, or mutate a repository or run.
+Missing or uncertain evidence is reported as unknown/ineligible. Proactive attention is explicitly
+follow-on work gated on a separately converged SelfHealGate.
+
+## What to Tell Your User
+
+There is now a read-only dashboard observation for autonomous runs that have shown neither project PR
+movement nor manager communication. It does not send alerts or change the run.
+
+## Summary of New Capabilities
+
+- Pull scrubbed per-run throughput observations from the Machines dashboard or authenticated API.
+- Retain the observation baseline and bounded-read breaker across server restarts.
+
+## Evidence
+
+- Unit coverage for deliverable semantics, dual-clock flatline, fresh baseline, and restart breaker.
+- Authenticated route integration coverage.
+- Lifecycle ratchet proving there is no self-action/governor or action-bearing source seam.

package/upgrades/side-effects/autonomous-throughput-floor.md

@@ -0,0 +1,36 @@
+# Side-effects review: Autonomous Throughput Floor
+
+## Changed
+
+- Adds bounded Git/GitHub and Telegram-history reads for eligible single-machine autonomous runs.
+- Adds one constant-size 0600 machine-local sidecar per run plus a scrubbed JSONL audit row.
+- Adds authenticated `GET /autonomous/throughput-floor` and dashboard consumption of that pull surface.
+- Adds fleet-dark config, migration, guard posture, agent awareness, and release documentation.
+
+## Explicitly unchanged
+
+- No Telegram or Slack message, attention item, notification, A2A message, dispatch, restart, remediation,
+  scheduler mutation, PR mutation, git fetch/checkout, or route write is possible from this feature.
+- No personal SSH, repository worktree, conversation content, autonomous run document, or pairing state is written.
+- HOLD authority is not added. Missing lane-saturation truth remains false/unavailable.
+
+## Resource and failure posture
+
+- One tick at a time; PR/history bounds are fixed; failures back off 15m/30m/60m then open a persisted 6h breaker.
+- Missing/corrupt/future state, incomplete history, invalid scope, moves, and multi-machine runs fail to
+  `unknown`/`ineligible`, never to a flatline claim.
+- Audit/status expose only ids, durations, decision/failure enums, and timestamps—no raw messages, stderr,
+  repository paths, URLs, refs, or PR prose.
+- The append-only observation audit is explicitly registered as machine-local state with bounded streamed
+  retention; the source annotation binds the write site to that declaration without changing runtime behavior.
+- Expected bounded-read and fleet-dark initialization failures carry explicit fallback annotations: reads fail
+  closed to `unknown`, breaker accounting remains durable, and initialization failure is logged.
+- The hand-authored dark-gate attribution map is advanced by the five ConfigDefaults lines introduced by this
+  feature; its 25-path set is unchanged, and the real-default resolver test verifies every attribution.
+
+## Follow-on gate
+
+Proactive attention is not part of this change. It is named only as follow-on work gated on a separately
+converged SelfHealGate; there is no dormant config path or callback that can enable it.
+
+Decision-audit evidence is carried with the implementation commit and covers every behavior-changing file.

package/upgrades/side-effects/feedback-factory-operating-drain.md

@@ -0,0 +1,110 @@
+# Side-Effects Review — Feedback Factory Operating Drain
+
+**Version / slug:** `feedback-factory-operating-drain`
+**Date:** `2026-07-20`
+**Author:** `Instar-codey`
+**Second-pass reviewer:** required — durable queue, model authority, PIN mutations, multi-process fencing, and canonical-pipeline guard
+**Status:** Concurred after correction-pass re-review. The independent reviewer initially withheld concurrence across ownership, source-generation, isolation, scrubbing, authority-failure, API, performance, cancellation, and restore/failover boundaries. Each finding received a concrete fix and positive/negative control; the final re-review concurred.
+
+## Summary and decision points
+
+This change registers Canonical Pipeline Operational Completeness and its structural/runtime evidence guards, then completes the Feedback Factory path from canonical report storage through bounded clustering, registered frontier-model readiness, a fenced SQLite outbox, and exact-key Initiative handoff. Development processing/drain defaults live; the consumer remains simulation-only until a durable PIN-approved promotion. Fleet defaults remain dark.
+
+Decision points added or changed:
+
+- Canonical-pipeline declaration coverage: deterministic structural signal; CI can reject missing declarations/citations, not semantic product decisions.
+- Feedback readiness: a registered frontier-model Instar agent is the default bounded authority. Deterministic evidence, identity, generation, model/provider, prompt/schema, confidence, spend, and injection floors constrain it. Human authority is escalation/break-glass only.
+- Consumer promotion, authority mutation, and break-glass hold: PIN-rooted operator authority with mutation intent and same-origin checks. A registered agent may release only after the named deterministic source/projection/authority predicate passes.
+- Queue claim/ack: deterministic SQLite lease/token/owner-epoch authority.
+- Generated-default drift is visible through posture and startup diagnostics but is not automatically repaired in this landing. Runtime recovery is deferred to the separately converged reusable SelfHealGate/P19 foundation lane; this PR contains no feature-local retry engine.
+
+## 1. Over-block
+
+- A valid agent readiness batch is refused if its registered provider/model family or prompt/schema canary differs from the resolved runtime. This is intentional fail-closed behavior; the cluster remains collecting rather than disappearing.
+- A legitimate retry with a reused request nonce is rejected. The scheduled job creates a new bounded nonce each invocation; a rejected replay does not mutate state.
+- A non-holder machine cannot admit local work; each mutating/ack stage is re-fenced against the live lease epoch, and request replay protection is durable across restart. The correction pass adds the authenticated one-hop owner proxy and exercises its refusal/replay path before concurrence.
+- Corrupt, truncated, or generation-drifted source projection holds progress. It does not guess an offset or manufacture readiness.
+
+## 2. Under-block
+
+- Metadata-only readiness can still approve work whose underlying product interpretation later proves wrong. Completion means only that a readable Initiative handoff exists; it never closes the legacy cluster or claims the product fix is complete.
+- The title-level injection detector is a deterministic floor, not a complete semantic detector. The frontier model is also instructed to treat all packet fields as untrusted; output is constrained and cannot emit `held`.
+- Novel or obfuscated sensitive content outside the deterministic credential-pattern floor could still reach bounded metadata. Raw report bodies remain excluded entirely, and known credential shapes are scrubbed before queue persistence.
+
+## 3. Level-of-abstraction fit
+
+- Pipeline completeness lives in one typed intake registry plus one manifest, not scattered comments.
+- Runtime evidence is separate from structural lint: constructed consumer, enabled cadence, effective idempotency, progress, and real read-back have their own verifier/E2E.
+- SQLite owns cross-store queue truth, leases, authority generations, audit, source cursor/checksums, and artifact links. InitiativeTracker owns the user-visible artifact. The boundary is an idempotent saga, not a fake distributed transaction.
+- The existing FeedbackProcessingService and InitiativeTracker are extended instead of replaced.
+
+## 4. Signal vs authority compliance
+
+Compliant with `docs/signal-vs-authority.md`.
+
+- Structural and runtime pipeline guards produce deterministic evidence about construction/completeness; they do not decide product readiness.
+- The readiness judgment is explicitly routed to a registered frontier-model authority with provenance and deterministic floors.
+- The model cannot hold or clear a hold by judgment. Corruption/integrity logic and the PIN-rooted human break-glass surface own hold; a registered agent can release only when the named deterministic revalidation predicate independently passes.
+- Consumer promotion remains an operator decision, separate from agent readiness approval.
+
+## 5. Interactions and feedback loops
+
+- Concurrent ticks collapse onto one durable active run; concurrent OS processes converge through SQLite uniqueness and busy fencing.
+- Repeated readiness/enqueue/consume uses exact keys, so cadence retries do not multiply work or Initiatives.
+- A crash after Initiative creation re-reads by `feedbackWorkKey` and completes the same work; it never recreates the artifact.
+- Simulation clones a bounded queue snapshot into a separate ephemeral SQLite database and exercises the real claim/link/ack FSM there; canonical attempts, leases, retry times, dead letters, bytes, and artifacts remain untouched.
+- Source compaction is serialized with all append writers, fsyncs the next immutable generation and checksummed handoff manifest, retains the old generation, and requires the durable SQLite tail cursor to acknowledge the exact boundary before following it. Before/after append, projection/cursor transaction, and manifest publication crash points replay without duplicate projection.
+- Spend, batch, attempt, lease, due-review, source-projection, reconciliation, and full-tick work are bounded. No self-triggering loop schedules another tick.
+- Cancellation is durable and owner-fenced, is observed only at explicit stage boundaries, and intentionally abandons the run without entering the self-heal loop. Terminal history pruning is owner-fenced, capped at 500 rows per tick, and preserves immutable idempotency tombstones.
+- Production source compaction is stage-budgeted on a durable 24-hour cadence. The first observation establishes the baseline instead of compacting a large historical source at startup; empty generations are not rotated.
+- The feedback job is enabled on development agents and treats an unexpected dev 503 as degradation, preventing silent “healthy” darkness.
+
+## 6. External and operator surfaces
+
+- New authenticated status/tick, run-cancellation, bounded backlog-analysis, failover-finalization, authority, promotion/revoke, and break-glass hold/release routes.
+- Normal readiness ticks require Bearer auth, registered `X-Instar-AgentId`, mutation intent, and a one-use bounded nonce; no PIN or human readiness mutation is required.
+- Operator mutations require PIN, `X-Instar-Request`, same-origin when an Origin is present, and the existing durable PIN-attempt brake.
+- Responses expose bounded identifiers/counts/state only; raw report bodies, prompts, transcripts, credentials, and claim tokens are absent.
+- Historical backlog analysis is metadata-only, registered-agent/nonce bound, capped by the authority batch limit (and an absolute 100-row ceiling), and cannot classify or dispatch work.
+- The standing operator surface remains the phone-capable dashboard/API control plane. Primary actions are promotion/revocation and break-glass repair; destructive state deletion is not exposed. No raw filesystem paths or internal stack traces are returned. Phone-width behavior inherits the existing dashboard/API controls; no new desktop-only flow is introduced.
+
+## 7. Multi-machine posture
+
+**Cluster-shared, canonical-holder writes with authenticated one-hop forwarding.** The mutating routes are registered `cluster-shared` in `WriteDomainRegistry`; the canonical host owns the local SQLite DB, source cursor, authority generation, promotion record, and Initiative handoff. Peer machines do not self-elect or run competing drains. Live lease epochs fence admission, stage heartbeats, claims, retries, and artifact acknowledgement. Backup recursively includes the operated feedback directory. PIN-rooted failover finalization verifies the successor epoch, restores/increments the durable owner epoch, reconciles Initiative links by immutable `feedbackWorkKey`, and forces a checkpointed failover backup before cadence resumes. The destructive restore fixture removes the entire operated state directory and rejects stale claims after restoration.
+
+A destructively restored snapshot boots in `restorePending`: canonical ownership, ticks/proxying, backup cadence, nonce admission, cancellation, authority/readiness mutations, and consumer promotion/revocation remain fenced until checksum-bound PIN finalization succeeds. Routine same-file checkpoint restart remains live. The restore fixture proves pre-finalization database, promotion, run/work, and Initiative invariance and post-finalization successor activation.
+
+## 6b. Operator-surface quality
+
+The dashboard addition is a read-only Feedback Drain status tab; it adds no operator mutation or destructive action.
+
+1. **Leads with the primary action:** yes. The tab opens directly on the current drain posture, followed by backlog, progress, and source-integrity summaries; no explanatory toggle hides the status.
+2. **Zero raw internals as primary content:** yes. The primary content uses plain state labels and bounded counts/ages. It does not render JSON, hashes, fingerprints, UUIDs, file paths, or claim tokens.
+3. **Destructive actions de-emphasized:** yes. There are no destructive or mutating actions in this surface.
+4. **Plain language + phone width:** yes. The tab reuses the existing responsive Process Health card layout, vertically stacks its three short sections, uses normal dashboard tap targets, and introduces no table, fixed width, horizontal scroll, or truncated identifier column.
+
+## 8. Rollback cost
+
+- Immediate rollback: disable `feedbackFactory.drain.enabled`; processing can remain independently available. Disable/revoke the consumer promotion to return to simulation without deleting queue state.
+- Revoke the authority generation to stop readiness decisions while preserving every audit row.
+- Reverting code leaves additive SQLite tables and optional Initiative fields readable/ignorable; no destructive migration is required.
+- Fleet is dark by default, bounding blast radius to development agents until an explicit future promotion.
+
+## Verification evidence
+
+- Repository-wide delivery ratchets classify the dashboard tab on the glance floor, account for the process-lifetime SQLite handle, declare cancellation and failover as cluster-shared writes, preserve the keyword-intent baseline, and keep the no-silent-fallback baseline unchanged.
+- Structural guard negative fixtures: missing intake/stage/owner/handoff/cadence/metrics/citation/CI collection.
+- Runtime guard negative fixtures: unconstructed consumer, disabled cadence, ineffective idempotency, missing progress, fake wiring.
+- Unit/integration: authority generations/restore/spend/demotion/canaries, proposal-set nonce binding, token fencing, stale owner epochs, exact idempotency, isolated simulation, crash-after-artifact recovery, source append/projection/handoff crash recovery, generated-default/posture visibility, cancellation boundaries, retention tombstones, backlog analysis, write-domain census, and destructive restore/failover finalization.
+- E2E: canonical feedback intake records are projected and clustered by the real processing service, then traverse real AgentServer + SQLite + InitiativeTracker with no PIN on normal agent readiness.
+- Multi-process/machine: two real OS processes create exactly one work/link pair; two conflicting approval writers create one approval plus an integrity hold; concurrent append+compaction loses no accepted source row. A separate network fixture uses two state trees, two SQLite/Initiative stores, asymmetric Ed25519 identities, real HTTP/fetch, and real services to prove only the registry owner executes and stale epoch/owner envelopes cannot mutate either side.
+- Performance: one real drain tick over a 150,000-row source under concurrent ingest projects/processes 500, performs readiness/enqueue/live bounded consumption, proves the oldest pre-existing eligible row is claimed first within one tick, stays under 90 seconds and 512 MiB, and exposes remaining lag.
+- CI correction: the destructive-restore fixture copies the exact checkpoint-authenticated SQLite bytes before close, and restored-file identity includes birth time as well as device/inode because Linux may immediately recycle a deleted inode. This preserves the production checksum/identity guard while distinguishing a real replacement from routine reopen. The operated-drain initialization fallback is explicitly annotated because it logs, exposes unavailable posture, and raises an urgent item for integrity failures. The exact push-config shard-4 invocation passes the destructive-restore case.
+
+## Second-pass review
+
+Independent correction-pass review: **CONCUR**. The final review inspected the shared restore-pending gate, destructive restore invariance, checksum/identity-bound finalization, routine restart distinction, stale-owner fencing, and in-flight cancellation linkage, and ran the focused high-risk fixtures green.
+
+## Class-Closure Declaration
+
+No feedback-drain self-action controller ships in this landing. Runtime recovery is owned by the separately tracked shared SelfHealGate/P19 foundation lane, so this feature does not claim an unbounded-self-action closure.