@useshifu/coding-harness 0.3.2 → 0.3.3

package/README.md

@@ -14,7 +14,7 @@ Connect installs the local instruction and scheduled runner, saves the connectio
 - automatic coding-session sync (manual approval remains available for coding sessions);
 - one input folder created at `~/Desktop/Shifu` on macOS, with a `README.md` explaining what it is for and shortcuts at `~/Downloads/Shifu` and `~/Documents/Shifu` when those parent folders exist.
 
-Place `.txt`, `.md`, `.rtf`, or `.docx` files directly in the input folder. Its `README.md` is never synced and an existing one is not overwritten. Screenshots and images are not processed. Existing files are picked up on the next scheduled run; unchanged files are not sent again. A failed file stays pending for retry. Folder files never require a claim-review step, even when coding-session sync is in manual mode. Shifu creates claims only when the file supports a specific contribution by the connected user; meeting action items or another person's work are not treated as the user's claims.
+Place `.txt`, `.md`, `.rtf`, or `.docx` files up to 20 MB directly in the input folder. Its `README.md` is never synced and an existing one is not overwritten. Screenshots, images, links, subfolders, and unknown formats are not uploaded; unsupported regular files are counted in sync status. Existing files are picked up on the next scheduled run; unchanged files are not sent again. A failed file stays pending for retry. Folder files never require a claim-review step, even when coding-session sync is in manual mode. Shifu creates claims only when the file supports a specific contribution by the connected user; meeting action items or another person's work are not treated as the user's claims.
 
 For named meeting transcripts, set your display name in Shifu Profile so your speaker turns can be identified. If the speaker cannot be matched, uncertain contributions are skipped.
 
@@ -41,7 +41,7 @@ npx @useshifu/coding-harness sync-files --harness codex
 
 Automatic scheduled runs send each accepted segment and advance its local checkpoint only after the server returns a matching receipt. Manual scheduled runs rebuild an accumulated local queue without advancing checkpoints and create or resume one content-free harness session when the pending count changes; one approval sends the queue in order. Network failures and server throttling are retried, while rejected segments stay pending.
 
-The same scheduled run checks the input folder. `sync-files` checks it immediately without changing the schedule. It sends supported file bytes over the configured HTTPS connection (or localhost HTTP for development); the server keeps the raw bytes transient and returns a final processing receipt before the local file checkpoint advances. Files larger than 5 MB are left pending with an error. The scanner reads regular files directly in the chosen folder and ignores symlinks and subfolders.
+The same scheduled run checks the input folder. `sync-files` checks it immediately without changing the schedule. Before sending new or changed files, the runner polls unfinished jobs from its local checkpoints. It sends supported file bytes over the configured HTTPS connection (or localhost HTTP for development). The server accepts each file as a background job and processes its text in chunks; upload does not wait for claim extraction. A later scheduled or manual sync polls the job's result. Only `processed`, `no_claims`, or `duplicate` finalizes the local checkpoint; `partial` reopens failed chunks on the next sync. Existing file checkpoints are checked once again after upgrading to this job-based protocol. Files larger than 20 MB remain pending with an error. The scanner reads regular files directly in the chosen folder and ignores symlinks and subfolders. Unknown formats are reported as unsupported rather than treated as successful syncs.
 
 The connector captures any useful harness activity, including engineering, product discovery, writing, interview feedback, research, and planning. Version-four payloads support generic `activity` items without a technical scope and accept a safe session-level summary when no item survives redaction. Version 1–3 clients remain supported.
 

package/bin/shifu-harness.js

@@ -41,15 +41,19 @@ const MIN_INTERVAL_HOURS = 0.25;
 const MAX_INTERVAL_HOURS = 720;
 const SCHEDULER_TICK_MINUTES = 15;
 const MAX_RETRIES = 4;
-const MAX_WORK_FILE_BYTES = 5 * 1024 * 1024;
-const WORK_FILE_SYNC_VERSION = 2;
+const MAX_WORK_FILE_BYTES = 20 * 1024 * 1024;
+const WORK_FILE_SYNC_VERSION = 3;
 const WORK_FILE_EXTENSIONS = new Set([".txt", ".md", ".rtf", ".docx"]);
+const FINAL_WORK_FILE_STATUSES = new Set(["processed", "no_claims", "duplicate"]);
+const ACTIVE_WORK_FILE_STATUSES = new Set(["queued", "processing", "retrying"]);
 const INPUT_FOLDER_README_NAME = "README.md";
 const INPUT_FOLDER_README = `# Shifu sync folder
 
 Put work logs, meeting transcripts, and other work documents directly in this folder. Shifu checks it every hour by default and keeps claims only when the file supports a contribution you made. Other people's work and meeting action items alone are not your claims.
 
-Supported files: .txt, .md, .rtf, and .docx. Subfolders, links, images, and screenshots are not processed. This README is not synced.
+Supported files: .txt, .md, .rtf, and .docx, up to 20 MB each. Subfolders, links, images, and screenshots are not processed. This README is not synced. Unsupported files are counted but never uploaded.
+
+Uploading starts a background job. Shifu checks unfinished jobs before the next scheduled or manual sync; a file is complete only when the server reports its final result. A partial result stays pending for retry.
 
 Files you add are sent to Shifu for parsing and sanitization; they are not redacted on this device. Only place files here if you want them processed. For named meeting transcripts, set your display name in Shifu Profile so your speaker turns can be matched.
 `;
@@ -990,18 +994,25 @@ async function postSync(config, input) {
   throw lastError;
 }
 
+function scanWorkFiles(folder) {
+  const entries = fs.readdirSync(validateInputFolder(folder), { withFileTypes: true })
+    .filter((entry) => entry.isFile() && entry.name !== INPUT_FOLDER_README_NAME);
+  return {
+    files: entries.filter((entry) => WORK_FILE_EXTENSIONS.has(path.extname(entry.name).toLowerCase()))
+      .map((entry) => path.join(folder, entry.name)).sort(),
+    unsupported: entries.filter((entry) => !WORK_FILE_EXTENSIONS.has(path.extname(entry.name).toLowerCase())).length,
+  };
+}
+
 function workFiles(folder) {
-  return fs.readdirSync(validateInputFolder(folder), { withFileTypes: true })
-    .filter((entry) => entry.isFile() && entry.name !== INPUT_FOLDER_README_NAME && WORK_FILE_EXTENSIONS.has(path.extname(entry.name).toLowerCase()))
-    .map((entry) => path.join(folder, entry.name))
-    .sort();
+  return scanWorkFiles(folder).files;
 }
 
 function readWorkFile(file) {
   const descriptor = fs.openSync(file, fs.constants.O_RDONLY | (fs.constants.O_NOFOLLOW || 0));
   try {
     const before = fs.fstatSync(descriptor);
-    if (!before.isFile() || before.size > MAX_WORK_FILE_BYTES) throw new Error("Only regular files up to 5 MB are supported.");
+    if (!before.isFile() || before.size > MAX_WORK_FILE_BYTES) throw new Error("Only regular files up to 20 MB are supported.");
     const bytes = fs.readFileSync(descriptor);
     const after = fs.fstatSync(descriptor);
     if (bytes.length > MAX_WORK_FILE_BYTES || before.size !== after.size || before.mtimeMs !== after.mtimeMs) {
@@ -1021,11 +1032,11 @@ async function postWorkFile(config, file, contents) {
       const form = new FormData();
       form.append("file", new Blob([contents.bytes]), path.basename(file));
       form.append("sourceModifiedAt", contents.modifiedAt);
-      response = await fetch(`${config.apiUrl}/v1/connectors/work-files`, {
+      response = await fetch(`${config.apiUrl}/v1/connectors/work-files/jobs`, {
         method: "POST",
         headers: { Authorization: `Bearer ${config.token}` },
         body: form,
-        signal: AbortSignal.timeout(15 * 60_000),
+        signal: AbortSignal.timeout(2 * 60_000),
       });
     } catch (error) {
       lastError = new Error(`Could not send ${path.basename(file)} to Shifu: ${error?.cause?.code || error?.message || "unknown transport error"}.`);
@@ -1035,7 +1046,7 @@ async function postWorkFile(config, file, contents) {
     const payload = await response.json().catch(() => undefined);
     if (response.ok) {
       const receipt = payload?.data;
-      if (["processed", "no_claims", "duplicate"].includes(receipt?.status) && Number.isInteger(receipt?.claimsCreated) && receipt.claimsCreated >= 0) return receipt;
+      if (validWorkFileJob(receipt)) return receipt;
       throw new Error(`Shifu returned an invalid receipt for ${path.basename(file)}. Local state was not advanced.`);
     }
     lastError = new Error(payload?.error?.message || `Shifu rejected ${path.basename(file)} (${response.status}).`);
@@ -1045,32 +1056,97 @@ async function postWorkFile(config, file, contents) {
   throw lastError;
 }
 
+function validWorkFileJob(receipt) {
+  return typeof receipt?.id === "string" && receipt.id.length > 0 &&
+    (ACTIVE_WORK_FILE_STATUSES.has(receipt.status) || FINAL_WORK_FILE_STATUSES.has(receipt.status) || receipt.status === "partial") &&
+    Number.isInteger(receipt.claimsCreated) && receipt.claimsCreated >= 0;
+}
+
+async function getWorkFileJob(config, jobId) {
+  let lastError;
+  for (let attempt = 0; attempt < MAX_RETRIES; attempt += 1) {
+    let response;
+    try {
+      response = await fetch(`${config.apiUrl}/v1/connectors/work-files/jobs/${encodeURIComponent(jobId)}`, {
+        headers: { Authorization: `Bearer ${config.token}` },
+        signal: AbortSignal.timeout(30_000),
+      });
+    } catch (error) {
+      lastError = new Error(`Could not check a Shifu file job: ${error?.cause?.code || error?.message || "unknown transport error"}.`);
+      if (attempt + 1 < MAX_RETRIES) await sleep(retryDelay(undefined, attempt));
+      continue;
+    }
+    const payload = await response.json().catch(() => undefined);
+    if (response.ok) {
+      if (validWorkFileJob(payload?.data) && payload.data.id === jobId) return payload.data;
+      throw new Error("Shifu returned an invalid file-job status. Local state was not advanced.");
+    }
+    lastError = new Error(payload?.error?.message || `Shifu could not check a file job (${response.status}).`);
+    if (response.status !== 429 && response.status < 500) throw lastError;
+    if (attempt + 1 < MAX_RETRIES) await sleep(retryDelay(response, attempt));
+  }
+  throw lastError;
+}
+
+function saveWorkFileEntry(harness, folder, name, entry) {
+  const state = readState(harness);
+  const files = state.workFiles?.folder === folder && state.workFiles.version === WORK_FILE_SYNC_VERSION ? state.workFiles.files || {} : {};
+  state.workFiles = { folder, version: WORK_FILE_SYNC_VERSION, files: { ...files, [name]: entry } };
+  saveState(harness, state);
+}
+
 async function syncWorkFiles(harness, config = readConfig(harness)) {
   if (typeof config.inputFolder !== "string") throw new Error("No input folder is configured. Run connect again to enable file sync.");
   const folder = validateInputFolder(configuredInputFolder(config));
   const state = readState(harness);
   const prior = state.workFiles?.folder === folder && state.workFiles.version === WORK_FILE_SYNC_VERSION ? state.workFiles.files || {} : {};
-  let synced = 0;
-  let failures = 0;
-  for (const file of workFiles(folder)) {
+  let submitted = 0;
+  const failedFiles = new Set();
+  const unverifiedJobs = new Set();
+  for (const [name, entry] of Object.entries(prior)) {
+    if (typeof entry !== "object" || !entry?.jobId) continue;
+    try {
+      const job = await getWorkFileJob(config, entry.jobId);
+      if (FINAL_WORK_FILE_STATUSES.has(job.status)) {
+        saveWorkFileEntry(harness, folder, name, entry.sha256);
+        prior[name] = entry.sha256;
+        console.error(`${name}: ${job.status} (${job.claimsCreated} claim${job.claimsCreated === 1 ? "" : "s"}).`);
+      } else {
+        prior[name] = { ...entry, status: job.status };
+        saveWorkFileEntry(harness, folder, name, prior[name]);
+        if (job.status === "partial") console.error(`${name}: partially processed; failed chunks will be retried on this sync.`);
+      }
+    } catch (error) {
+      failedFiles.add(name);
+      unverifiedJobs.add(name);
+      console.error(`${name} status remains pending: ${error.message}`);
+    }
+  }
+  const scan = scanWorkFiles(folder);
+  if (scan.unsupported) console.error(`${scan.unsupported} unsupported file${scan.unsupported === 1 ? "" : "s"} skipped (only .txt, .md, .rtf, and .docx are uploaded).`);
+  for (const file of scan.files) {
     const name = path.basename(file);
     try {
+      if (unverifiedJobs.has(name)) continue;
       const contents = readWorkFile(file);
-      if (prior[name] === contents.sha256) continue;
-      const receipt = await postWorkFile(config, file, contents);
-      const latest = readState(harness);
-      const files = latest.workFiles?.folder === folder && latest.workFiles.version === WORK_FILE_SYNC_VERSION ? latest.workFiles.files || {} : {};
-      latest.workFiles = { folder, version: WORK_FILE_SYNC_VERSION, files: { ...files, [name]: contents.sha256 } };
-      saveState(harness, latest);
-      prior[name] = contents.sha256;
-      synced += 1;
-      console.error(`${name}: ${receipt.status} (${receipt.claimsCreated} claim${receipt.claimsCreated === 1 ? "" : "s"}).`);
+      const previous = prior[name];
+      if (previous === contents.sha256 || (previous?.sha256 === contents.sha256 && previous.status !== "partial")) continue;
+      const job = await postWorkFile(config, file, contents);
+      const entry = FINAL_WORK_FILE_STATUSES.has(job.status) ? contents.sha256 : { sha256: contents.sha256, jobId: job.id, status: job.status };
+      saveWorkFileEntry(harness, folder, name, entry);
+      prior[name] = entry;
+      submitted += 1;
+      console.error(`${name}: ${job.status}${FINAL_WORK_FILE_STATUSES.has(job.status) ? ` (${job.claimsCreated} claim${job.claimsCreated === 1 ? "" : "s"})` : " (processing in background)"}.`);
     } catch (error) {
-      failures += 1;
+      failedFiles.add(name);
       console.error(`${name} remains pending: ${error.message}`);
     }
   }
-  return { synced, failures };
+  const pending = Object.values(prior).filter((entry) => typeof entry === "object" && entry?.jobId).length;
+  for (const [name, entry] of Object.entries(prior)) {
+    if (entry?.status === "partial") failedFiles.add(name);
+  }
+  return { submitted, pending, failures: failedFiles.size, unsupported: scan.unsupported };
 }
 
 async function sendSync(harness, input, config = readConfig(harness)) {
@@ -1223,13 +1299,17 @@ async function scheduledSync(harness) {
     const startedAt = Date.now();
     let failures = 0;
     let synced = 0;
-    let syncedFiles = 0;
+    let submittedFiles = 0;
+    let pendingFiles = 0;
     let failedFiles = 0;
+    let unsupportedFiles = 0;
     if (config.inputFolder) {
       try {
         const files = await syncWorkFiles(harness, config);
         failedFiles = files.failures;
-        syncedFiles = files.synced;
+        submittedFiles = files.submitted;
+        pendingFiles = files.pending;
+        unsupportedFiles = files.unsupported;
       } catch (error) {
         failedFiles = 1;
         console.error(`Shifu input folder remains pending: ${error.message}`);
@@ -1256,8 +1336,10 @@ async function scheduledSync(harness) {
         ...state.schedule,
         lastAttemptAt: startedAt,
         lastResult: notificationError ? "awaiting_approval_notification_failed" : failedFiles ? "partial_failure" : "awaiting_approval",
-        syncedFiles,
+        submittedFiles,
+        pendingFiles,
         failedFiles,
+        unsupportedFiles,
         pendingSegments: queue.segments.length,
         pendingFingerprint,
         approvalSessionRef,
@@ -1282,8 +1364,10 @@ async function scheduledSync(harness) {
         lastResult: failures || failedFiles ? "partial_failure" : "complete",
         syncedSegments: synced,
         failedSessions: failures,
-        syncedFiles,
+        submittedFiles,
+        pendingFiles,
         failedFiles,
+        unsupportedFiles,
         nextRunAt: startedAt + (failures || failedFiles ? SCHEDULER_TICK_MINUTES / 60 : policy.intervalHours) * 3_600_000,
       };
     }
@@ -1298,14 +1382,21 @@ function status(harness) {
   const config = readConfig(harness);
   const state = readState(harness);
   const pending = readJSON(pendingPath(harness), { segments: [] });
-  console.log(JSON.stringify({ harness, apiUrl: config.apiUrl, inputFolder: config.inputFolder || null, syncPolicy: normalizedPolicy(config), schedule: state.schedule || null, pendingSegments: pending.segments.length, syncedFiles: Object.keys(state.workFiles?.files || {}).length, sessions: state.sessions }, null, 2));
+  const folder = config.inputFolder ? configuredInputFolder(config) : null;
+  const entries = state.workFiles?.version === WORK_FILE_SYNC_VERSION && state.workFiles.folder === folder ? Object.values(state.workFiles.files || {}) : [];
+  let unsupportedFiles = null;
+  let inputFolderError = null;
+  if (folder) {
+    try { unsupportedFiles = scanWorkFiles(folder).unsupported; } catch (error) { inputFolderError = error.message; }
+  }
+  console.log(JSON.stringify({ harness, apiUrl: config.apiUrl, inputFolder: folder, inputFolderError, syncPolicy: normalizedPolicy(config), schedule: state.schedule || null, pendingSegments: pending.segments.length, completedFiles: entries.filter((entry) => typeof entry === "string").length, pendingFiles: entries.filter((entry) => typeof entry === "object" && entry?.jobId).length, unsupportedFiles, sessions: state.sessions }, null, 2));
 }
 
 async function syncFilesNow(harness) {
   const result = await withSyncLock(harness, () => syncWorkFiles(harness));
   if (result?.skipped) throw new Error("Another Shifu sync is already running. Files remain pending.");
-  console.error(`Checked the input folder: ${result.synced} sent, ${result.failures} pending.`);
-  if (result.failures) throw new Error("Some input files remain pending. Retry sync-files after resolving the errors above.");
+  console.error(`Checked the input folder: ${result.submitted} submitted, ${result.pending} processing, ${result.failures} needing retry, ${result.unsupported} unsupported.`);
+  if (result.failures) throw new Error("Some input files need retry. They will be checked on the next sync.");
 }
 
 function sessions(harness) {

package/commands/shifu-sync.md

@@ -4,7 +4,7 @@ description: Inspect or sync redacted OpenCode activity using the saved Shifu po
 
 Run `npx @useshifu/coding-harness status --harness opencode` first.
 
-For an explicit Shifu input-folder sync, run `sync-files --harness opencode`. It processes supported `.txt`, `.md`, `.rtf`, and `.docx` files in the configured folder without a claim-review step. Do not treat another speaker's work or a meeting action item as the user's contribution. The server parses and sanitizes files; the local runner does not redact file contents.
+For an explicit Shifu input-folder sync, run `sync-files --harness opencode`. It polls unfinished jobs before submitting new or changed `.txt`, `.md`, `.rtf`, and `.docx` files up to 20 MB. Upload starts asynchronous server processing; check status on the next manual or scheduled sync. Only a final `processed`, `no_claims`, or `duplicate` result completes the local checkpoint; a `partial` result remains retryable. Unsupported files are counted but not uploaded. No claim-review step is needed. Do not treat another speaker's work or a meeting action item as the user's contribution. The server parses and sanitizes files; the local runner does not redact file contents.
 
 If manual approval has queued scheduled work, run `pending --harness opencode`, show the complete redacted list, and ask once. After approval, run `approve --approved --harness opencode`.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@useshifu/coding-harness",
-  "version": "0.3.2",
+  "version": "0.3.3",
   "description": "Scheduled, redacted coding-harness activity sync for Shifu",
   "bin": {
     "coding-harness": "bin/shifu-harness.js",

package/skills/shifu-sync/SKILL.md

@@ -19,7 +19,7 @@ When the user asks to connect, run:
 npx @useshifu/coding-harness connect --harness codex
 ```
 
-`connect` installs the current runner and a local schedule with no policy questions. New connections default to hourly automatic sync and `~/Desktop/Shifu` as the single input folder on macOS. It creates shortcuts at `~/Downloads/Shifu` and `~/Documents/Shifu` when possible without replacing conflicting items. Supported files are `.txt`, `.md`, `.rtf`, and `.docx` directly inside the folder; images and screenshots are not supported. If a connection already exists, `connect` keeps its key, checkpoints, and chosen policy. It also excludes the setup session when the harness exposes its current opaque session reference. No content is sent during setup.
+`connect` installs the current runner and a local schedule with no policy questions. New connections default to hourly automatic sync and `~/Desktop/Shifu` as the single input folder on macOS. It creates shortcuts at `~/Downloads/Shifu` and `~/Documents/Shifu` when possible without replacing conflicting items. Supported files are `.txt`, `.md`, `.rtf`, and `.docx` up to 20 MB directly inside the folder; images and screenshots are not supported. Unknown formats are counted but never uploaded. If a connection already exists, `connect` keeps its key, checkpoints, and chosen policy. It also excludes the setup session when the harness exposes its current opaque session reference. No content is sent during setup.
 
 When the user asks to change the interval, approval mode, or input folder, run:
 
@@ -33,7 +33,9 @@ Use only the flags the user wants to change; existing settings are preserved. Th
 
 Run `npx @useshifu/coding-harness status --harness codex` to inspect the saved policy, next scheduled run, pending count, and checkpoints.
 
-For an explicit input-folder sync, run `npx @useshifu/coding-harness sync-files --harness codex`. It sends supported files automatically and advances each local checkpoint only after a final server receipt. No claim review or manual approval is needed for files. If a file fails, leave it pending and report the error; do not claim it was synced.
+For an explicit input-folder sync, run `npx @useshifu/coding-harness sync-files --harness codex`. It first polls previously accepted file jobs, then submits new or changed supported files. Uploading starts background processing; it does not wait for extraction. A later manual or scheduled sync polls status again. Only `processed`, `no_claims`, or `duplicate` advances a file's completed checkpoint; `partial` is retried on the next sync. No claim review or manual approval is needed for files. Report submitted, processing, retry-needed, and unsupported counts separately. Do not claim a submitted file was fully processed. If a file fails, leave it pending and report the error.
+
+The folder runner accepts nonempty `.txt`, `.md`, `.rtf`, and `.docx` files up to 20 MB; the server accepts at most 500,000 extractable characters per file and splits that text into bounded chunks. Empty, malformed, binary, and renamed unsupported files cannot yield claims. A `no_claims` result means processing finished but no sufficiently grounded contribution could be attributed to the user; it does not mean upload failed. For a named transcript, confirm the user's Shifu Profile display name matches their speaker label before expecting automated attribution. Never promise a claim from every file or infer one from an action item. The upload has a transport deadline, but model processing has no upload-request timeout; failed chunks back off and can be retried without republishing completed chunks.
 
 In manual mode, scheduled runs accumulate redacted segments locally. Run: