@indigoai-us/hq-cli 5.20.0 → 5.22.0

package/CHANGELOG.md

@@ -2,6 +2,86 @@
 
 ## [Unreleased]
 
+## [5.22.0] — 2026-05-25
+
+### Changed
+
+- Bumps `@indigoai-us/hq-cloud` to `~5.37.0` (was `~5.34.0`). Picks up
+  the 5.35 → 5.37 chain on top of yesterday's 5.34.0 cleanup.
+
+  - **5.35.0** — `.claude/state/` + `.claude/audit/` in
+    `DEFAULT_IGNORES`. Removes ~85% of the conflict-mirror class
+    observed during the 5.34.0 live cross-machine test. See
+    [hq-cloud#25](https://github.com/indigoai-us/hq-cloud/pull/25).
+  - **5.36.0** — two sync speedups: an lstat fast-path that skips
+    SHA-256 when `(size, mtimeMs)` match the journal baseline
+    (~5–10× on no-op syncs, which are most syncs), and a bounded-
+    parallel transfer pool (default 16, knob
+    `HQ_SYNC_TRANSFER_CONCURRENCY`) for uploads + downloads (4–8× on
+    transfer-heavy syncs). Codex P1 follow-ups serialize the
+    interactive conflict prompt under the pool and drain in-flight
+    transfers on worker error. See
+    [hq-cloud#26](https://github.com/indigoai-us/hq-cloud/pull/26).
+  - **5.37.0** — file mtime / birthtime preservation across sync.
+    Push stamps source-side `lstat.mtimeMs` (+ `birthtimeMs` when the
+    filesystem supports it AND differs from mtime) into S3 metadata as
+    `hq-mtime` / `hq-btime`. Pull applies the stamped value via
+    `utimesSync` after the byte write, falling back to write-time
+    when metadata is absent (back-compat). Symlinks skipped on both
+    sides. Composes cleanly with the 5.36.0 fast-path — the journal
+    records the post-utimes mtime, so the next sync correctly skips
+    re-hashing. Codex P2 widened the accepted mtime domain to include
+    `0` (Unix epoch, reproducible-builds clamp value) and negative
+    epochs (pre-1970). See
+    [hq-cloud#27](https://github.com/indigoai-us/hq-cloud/pull/27).
+
+  Live verified end-to-end on the macOS desktop ↔ Lightsail outpost
+  pair: bilateral convergence in 2–3 sync rounds, mtime round-trip
+  exact for 2020 / 2030 / epoch-0 / pre-epoch-negative test cases,
+  fast-path correctly skips post-utimes files on the next no-op.
+
+## [5.21.0] — 2026-05-24
+
+### Changed
+
+- Bumps `@indigoai-us/hq-cloud` to `~5.34.0` (was `~5.25.0`). 5.34.0
+  closes the 10-bug cross-machine sync cleanup from
+  [hq-cloud#24](https://github.com/indigoai-us/hq-cloud/pull/24) plus
+  the Codex P1/P2 follow-up safeties.
+
+  Highest-impact fixes that ride this bump:
+
+  - **Bug #9** (drifted files never converge): the pull walker on
+    `~5.25.0` had no tombstone-consumption mechanism, so cross-machine
+    deletes never propagated — every host kept the union of "every
+    file ever created anywhere." 5.34.0 lands journal-vs-LIST diff +
+    HEAD-verify STS-scope guard + local-edit divergence detector for
+    both files and symlinks. The Drift counter can finally reach zero.
+  - **Bug #7** (concurrent edit data loss): two open laptops editing
+    the same file before either had synced silently overwrote the
+    slower side on `~5.25.0` — no conflict event, no tray badge,
+    invisible data loss. 5.34.0 writes a mirror to
+    `<orig>.conflict-<ts>-<short>` instead.
+  - **Bug #10** (`ENOTDIR` wedges the whole company sync): one
+    dir-vs-file collision on a stale path on `~5.25.0` threw
+    `ENOTDIR` and aborted the whole company — `partial: true`,
+    `filesSkipped: 0`. 5.34.0 handles both `(local-file, cloud-dir)`
+    and `(local-dir, cloud-file)` topologies symmetrically.
+
+  Plus #1/#6/#8 (`.hq/` directory leak channel — including the
+  alternate `.hq/machine.json` and `.hq/install-manifest.json` paths
+  that were cross-polluting per-host identity), #2 (PULL walker now
+  applies `EPHEMERAL_PATH_PATTERN` — legacy `.conflict-*` litter no
+  longer ratchets), #3 (`conflictPaths` deduped within a company), #4
+  (dir-vs-file warning + structured event), and #5 (file mode
+  preserved across upload/download — was previously collapsing every
+  mode to receiver umask default).
+
+  New `SyncResult` fields surface via the runner's `complete` event:
+  `filesTombstoned`, `filesExcludedByPolicy`,
+  `filesRefusedStalePaths` (cap 50). hq-cli forwards them through
+  unchanged; the menubar can opt in via UI work in a follow-up.
+
 ## [5.20.0] — 2026-05-21
 
 ### Added

package/dist/commands/pack-install.d.ts

@@ -54,6 +54,30 @@ export declare function sourceMatchesPackPattern(source: string): boolean;
  * contract this function pins.
  */
 export declare function installToPackages(payloadDir: string, pkg: PackManifest, hqRoot: string): string;
+/**
+ * Stamp the install-time `source:` into the destination package.yaml. This is
+ * the only on-disk record of which `recommended_packages[].source` produced
+ * each installed pack. `core/scripts/setup.sh` (hq-core >=14.2.x) reads
+ * `^source:` from every installed `core/packages/<name>/package.yaml` to dedup
+ * against `core/core.yaml:recommended_packages` — without this stamp, the
+ * dedup set is empty and every recommended pack is re-prompted on every
+ * setup.sh run.
+ *
+ * Idempotent: re-installs (and `hq update`) replace any pre-existing
+ * top-level `source:` line rather than duplicating it. Nested `source:`
+ * fields under `contributes:` (or any other block) are left untouched.
+ *
+ * The value is YAML double-quoted so `github:` sources containing `#`
+ * survive round-trip (an unquoted `#` would be parsed as a comment).
+ *
+ * If the manifest opens with a `---` YAML document marker, the source line is
+ * inserted *after* the marker — otherwise the prepend would create a
+ * two-document stream that single-doc `yaml.load` callers in this codebase
+ * cannot read.
+ *
+ * Exported for tests.
+ */
+export declare function stampInstallSource(destDir: string, source: string): void;
 /**
  * Run `<hqRoot>/core/scripts/scan-packages.sh` to wire the newly installed
  * pack's contributions into the host paths (skills, hooks, policies, etc.).

package/dist/commands/pack-install.js

@@ -34,7 +34,7 @@
  * from each pack's package.yaml; rationale lives in the layout-fix PR.)
  */
 
-!function(){try{var e="undefined"!=typeof window?window:"undefined"!=typeof global?global:"undefined"!=typeof globalThis?globalThis:"undefined"!=typeof self?self:{},n=(new e.Error).stack;n&&(e._sentryDebugIds=e._sentryDebugIds||{},e._sentryDebugIds[n]="5dea7752-f10c-553c-a419-cf05441b3c13")}catch(e){}}();
+!function(){try{var e="undefined"!=typeof window?window:"undefined"!=typeof global?global:"undefined"!=typeof globalThis?globalThis:"undefined"!=typeof self?self:{},n=(new e.Error).stack;n&&(e._sentryDebugIds=e._sentryDebugIds||{},e._sentryDebugIds[n]="845e5459-a7ba-53ca-a04c-dd055fabe52d")}catch(e){}}();
 import * as fs from 'fs';
 import * as os from 'os';
 import * as path from 'path';
@@ -446,6 +446,61 @@ export function installToPackages(payloadDir, pkg, hqRoot) {
     execFileSync('rsync', ['-a', srcSlashed, destSlashed], { stdio: 'inherit' });
     return destDir;
 }
+/**
+ * Stamp the install-time `source:` into the destination package.yaml. This is
+ * the only on-disk record of which `recommended_packages[].source` produced
+ * each installed pack. `core/scripts/setup.sh` (hq-core >=14.2.x) reads
+ * `^source:` from every installed `core/packages/<name>/package.yaml` to dedup
+ * against `core/core.yaml:recommended_packages` — without this stamp, the
+ * dedup set is empty and every recommended pack is re-prompted on every
+ * setup.sh run.
+ *
+ * Idempotent: re-installs (and `hq update`) replace any pre-existing
+ * top-level `source:` line rather than duplicating it. Nested `source:`
+ * fields under `contributes:` (or any other block) are left untouched.
+ *
+ * The value is YAML double-quoted so `github:` sources containing `#`
+ * survive round-trip (an unquoted `#` would be parsed as a comment).
+ *
+ * If the manifest opens with a `---` YAML document marker, the source line is
+ * inserted *after* the marker — otherwise the prepend would create a
+ * two-document stream that single-doc `yaml.load` callers in this codebase
+ * cannot read.
+ *
+ * Exported for tests.
+ */
+export function stampInstallSource(destDir, source) {
+    const manifestPath = path.join(destDir, 'package.yaml');
+    if (!fs.existsSync(manifestPath)) {
+        // installToPackages already validated the payload contained package.yaml.
+        // If it's gone by the time we stamp, something deleted it between calls —
+        // fail loud so the regression surfaces in tests.
+        throw new Error(`stampInstallSource: ${manifestPath} not found`);
+    }
+    const original = fs.readFileSync(manifestPath, 'utf-8');
+    // Strip any existing top-level `source:` line. Match only lines starting at
+    // column 0 so nested `source:` fields (none in spec today, but defensive)
+    // are not touched.
+    const lines = original.split('\n').filter((line) => !/^source\s*:/.test(line));
+    const quoted = `"${source.replace(/\\/g, '\\\\').replace(/"/g, '\\"')}"`;
+    const sourceLine = `source: ${quoted}`;
+    // Detect a leading YAML document marker (`---`, optionally followed by
+    // trailing whitespace or a comment). If present, the source line goes
+    // *after* it so the manifest stays a single document.
+    let insertAt = 0;
+    while (insertAt < lines.length && lines[insertAt].trim() === '')
+        insertAt++;
+    const opensWithDocMarker = insertAt < lines.length && /^---(\s|#|$)/.test(lines[insertAt]);
+    if (opensWithDocMarker) {
+        lines.splice(insertAt + 1, 0, sourceLine);
+    }
+    else {
+        // Drop any leading blank lines we walked past so the file doesn't grow a
+        // blank-line gutter on every re-stamp.
+        lines.splice(0, insertAt, sourceLine);
+    }
+    fs.writeFileSync(manifestPath, lines.join('\n'));
+}
 /**
  * Run `<hqRoot>/core/scripts/scan-packages.sh` to wire the newly installed
  * pack's contributions into the host paths (skills, hooks, policies, etc.).
@@ -509,12 +564,15 @@ export async function installPack(source, opts = {}) {
             return;
         }
         const destDir = installToPackages(fetched.payloadDir, pkg, hqRoot);
-        // Under the v12 HQ layout, packs live at `core/packages/<name>/` and are
-        // tracked by filesystem presence alone — no `modules.yaml` write. That
-        // removes the side effect that created a top-level `modules/` directory
-        // alongside the canonical `core/`. `hq update <pack>` will re-resolve a
-        // pack's source from its on-disk package.yaml or prompt for it, but that
-        // tradeoff is intentional — see the layout-fix PR for rationale.
+        // Under the v12+ HQ layout, packs live at `core/packages/<name>/` and
+        // are tracked by filesystem presence — no `modules.yaml` write (that
+        // removed the orphan top-level `modules/` tree alongside the canonical
+        // `core/`). The one piece of install-time provenance we DO record is the
+        // input `source` string, stamped into the destination `package.yaml` so
+        // setup.sh can dedup against `core/core.yaml:recommended_packages` on
+        // re-runs. Stamping the literal input (not the resolved SHA/version)
+        // matches the verbatim equality check in setup.sh.
+        stampInstallSource(destDir, source);
         runScanPackages(hqRoot);
         console.log(chalk.green(`\n✓ Installed ${pkg.name}@${pkg.version} → ${path.relative(hqRoot, destDir)}/`));
         console.log(chalk.dim(`  Wired ${Object.values(pkg.contributes).flat().filter(Boolean).length} ` +
@@ -525,4 +583,4 @@ export async function installPack(source, opts = {}) {
     }
 }
 //# sourceMappingURL=pack-install.js.map
-//# debugId=5dea7752-f10c-553c-a419-cf05441b3c13
+//# debugId=845e5459-a7ba-53ca-a04c-dd055fabe52d

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@indigoai-us/hq-cli",
-  "version": "5.20.0",
+  "version": "5.22.0",
   "description": "HQ by Indigo management CLI — modules and cloud sync",
   "main": "dist/index.js",
   "bin": {
@@ -15,7 +15,7 @@
     "clean": "rm -rf dist"
   },
   "dependencies": {
-    "@indigoai-us/hq-cloud": "~5.25.0",
+    "@indigoai-us/hq-cloud": "~5.37.0",
     "@indigoai-us/hq-onboarding": "^0.1.0",
     "@sentry/node": "^10.49.0",
     "chalk": "^5.3.0",

package/src/commands/pack-install.test.ts

@@ -21,7 +21,7 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
 import * as fs from 'fs';
 import * as os from 'os';
 import * as path from 'path';
-import { installToPackages, runScanPackages } from './pack-install.js';
+import { installToPackages, runScanPackages, stampInstallSource } from './pack-install.js';
 import type { PackManifest } from '../types.js';
 
 // ---------------------------------------------------------------------------
@@ -145,4 +145,118 @@ describe('pack-install: install path layout', () => {
     expect(fs.existsSync(path.join(hqRoot, 'modules'))).toBe(false);
     expect(fs.existsSync(path.join(hqRoot, 'modules.yaml'))).toBe(false);
   });
+
+  // ---- 5. stampInstallSource — setup.sh dedup contract --------------------
+  // setup.sh in hq-core ≥14.2.x reads `^source:` from every installed
+  // core/packages/*/package.yaml and uses verbatim equality against
+  // core.yaml:recommended_packages[].source to skip "already installed" packs.
+  // Without the stamp the dedup set is empty and every recommended pack gets
+  // re-prompted on every setup.sh run — the bug this stamp closes.
+  it('stampInstallSource writes a top-level source: line into package.yaml', () => {
+    const dest = installToPackages(payload, fakeManifest({ name: 'hq-pack-test' }), hqRoot);
+    const src = '@indigoai-us/hq-pack-test@1.2.3';
+    stampInstallSource(dest, src);
+
+    const written = fs.readFileSync(path.join(dest, 'package.yaml'), 'utf-8');
+    // First line so a `grep ^source:` style scan never misses it.
+    expect(written.split('\n')[0]).toBe(`source: "${src}"`);
+    // Original fields preserved verbatim — the stamp must not clobber name/version.
+    expect(written).toContain('name: hq-pack-test');
+    expect(written).toContain('version: 1.0.0');
+  });
+
+  it('stampInstallSource is idempotent — re-install replaces (not duplicates) the source line', () => {
+    const dest = installToPackages(payload, fakeManifest({ name: 'hq-pack-test' }), hqRoot);
+    stampInstallSource(dest, '@indigoai-us/hq-pack-test@1.0.0');
+    stampInstallSource(dest, '@indigoai-us/hq-pack-test@1.2.3');
+
+    const written = fs.readFileSync(path.join(dest, 'package.yaml'), 'utf-8');
+    // Exactly one top-level source: line, and it carries the latest value.
+    const sourceLines = written.split('\n').filter((l) => /^source\s*:/.test(l));
+    expect(sourceLines).toHaveLength(1);
+    expect(sourceLines[0]).toBe('source: "@indigoai-us/hq-pack-test@1.2.3"');
+  });
+
+  it('stampInstallSource quotes github: sources so the # is not parsed as a YAML comment', () => {
+    const dest = installToPackages(payload, fakeManifest({ name: 'hq-pack-test' }), hqRoot);
+    const src = 'github:indigoai-us/hq-packages#packages/hq-pack-test';
+    stampInstallSource(dest, src);
+
+    const written = fs.readFileSync(path.join(dest, 'package.yaml'), 'utf-8');
+    // Double-quoted form survives the '#' that would otherwise truncate the
+    // value at the first YAML comment marker.
+    expect(written.split('\n')[0]).toBe(`source: "${src}"`);
+    // Round-trip through js-yaml to confirm a real parser reads the full value.
+    const yamlLib = require('js-yaml');
+    const parsed = yamlLib.load(written) as { source: string };
+    expect(parsed.source).toBe(src);
+  });
+
+  it('stampInstallSource only strips top-level source: lines, not nested ones', () => {
+    // Defensive: today's spec has no nested `source:` keys, but if a pack
+    // author adds e.g. `metadata: { source: ... }` we should leave it alone.
+    const dest = installToPackages(payload, fakeManifest({ name: 'hq-pack-test' }), hqRoot);
+    fs.writeFileSync(
+      path.join(dest, 'package.yaml'),
+      [
+        'name: hq-pack-test',
+        'version: 1.0.0',
+        'metadata:',
+        '  source: nested-value-keep-me',
+        '',
+      ].join('\n'),
+    );
+
+    stampInstallSource(dest, '@indigoai-us/hq-pack-test@1.0.0');
+
+    const written = fs.readFileSync(path.join(dest, 'package.yaml'), 'utf-8');
+    expect(written.split('\n')[0]).toBe('source: "@indigoai-us/hq-pack-test@1.0.0"');
+    // Nested source line survives.
+    expect(written).toContain('  source: nested-value-keep-me');
+  });
+
+  it('stampInstallSource throws when package.yaml is missing — surfaces upstream regressions loudly', () => {
+    const dest = installToPackages(payload, fakeManifest({ name: 'hq-pack-test' }), hqRoot);
+    fs.unlinkSync(path.join(dest, 'package.yaml'));
+
+    expect(() => stampInstallSource(dest, '@scope/x@1.0.0')).toThrow(/package\.yaml.*not found/);
+  });
+
+  it('stampInstallSource preserves a leading `---` document marker — no multi-doc YAML stream', () => {
+    // Regression: prepending `source:` before a `---` would split the file
+    // into two YAML documents, and single-doc `yaml.load` callers downstream
+    // (in this codebase + hq-core) silently misread or fail. Insert AFTER
+    // the marker so the manifest stays a single document.
+    const dest = installToPackages(payload, fakeManifest({ name: 'hq-pack-test' }), hqRoot);
+    fs.writeFileSync(
+      path.join(dest, 'package.yaml'),
+      ['---', 'name: hq-pack-test', 'version: 1.0.0', ''].join('\n'),
+    );
+    const src = 'github:indigoai-us/hq-packages#packages/hq-pack-test';
+
+    stampInstallSource(dest, src);
+
+    const written = fs.readFileSync(path.join(dest, 'package.yaml'), 'utf-8');
+    // `---` stays on line 0, `source:` follows immediately so `grep ^source:`
+    // dedup scans still match.
+    const lines = written.split('\n');
+    expect(lines[0]).toBe('---');
+    expect(lines[1]).toBe(`source: "${src}"`);
+
+    // Critical assertion: js-yaml's single-doc loader returns the merged
+    // manifest, not just the first of two documents.
+    const yamlLib = require('js-yaml');
+    const parsed = yamlLib.load(written) as { source: string; name: string; version: string };
+    expect(parsed.source).toBe(src);
+    expect(parsed.name).toBe('hq-pack-test');
+    expect(parsed.version).toBe('1.0.0');
+
+    // Re-stamp is still idempotent in the doc-marker case.
+    stampInstallSource(dest, '@scope/x@2.0.0');
+    const rewritten = fs.readFileSync(path.join(dest, 'package.yaml'), 'utf-8');
+    const sourceLines = rewritten.split('\n').filter((l) => /^source\s*:/.test(l));
+    expect(sourceLines).toHaveLength(1);
+    expect(sourceLines[0]).toBe('source: "@scope/x@2.0.0"');
+    expect(rewritten.split('\n')[0]).toBe('---');
+  });
 });

package/src/commands/pack-install.ts

@@ -547,6 +547,62 @@ export function installToPackages(
   return destDir;
 }
 
+/**
+ * Stamp the install-time `source:` into the destination package.yaml. This is
+ * the only on-disk record of which `recommended_packages[].source` produced
+ * each installed pack. `core/scripts/setup.sh` (hq-core >=14.2.x) reads
+ * `^source:` from every installed `core/packages/<name>/package.yaml` to dedup
+ * against `core/core.yaml:recommended_packages` — without this stamp, the
+ * dedup set is empty and every recommended pack is re-prompted on every
+ * setup.sh run.
+ *
+ * Idempotent: re-installs (and `hq update`) replace any pre-existing
+ * top-level `source:` line rather than duplicating it. Nested `source:`
+ * fields under `contributes:` (or any other block) are left untouched.
+ *
+ * The value is YAML double-quoted so `github:` sources containing `#`
+ * survive round-trip (an unquoted `#` would be parsed as a comment).
+ *
+ * If the manifest opens with a `---` YAML document marker, the source line is
+ * inserted *after* the marker — otherwise the prepend would create a
+ * two-document stream that single-doc `yaml.load` callers in this codebase
+ * cannot read.
+ *
+ * Exported for tests.
+ */
+export function stampInstallSource(destDir: string, source: string): void {
+  const manifestPath = path.join(destDir, 'package.yaml');
+  if (!fs.existsSync(manifestPath)) {
+    // installToPackages already validated the payload contained package.yaml.
+    // If it's gone by the time we stamp, something deleted it between calls —
+    // fail loud so the regression surfaces in tests.
+    throw new Error(`stampInstallSource: ${manifestPath} not found`);
+  }
+  const original = fs.readFileSync(manifestPath, 'utf-8');
+  // Strip any existing top-level `source:` line. Match only lines starting at
+  // column 0 so nested `source:` fields (none in spec today, but defensive)
+  // are not touched.
+  const lines = original.split('\n').filter((line) => !/^source\s*:/.test(line));
+  const quoted = `"${source.replace(/\\/g, '\\\\').replace(/"/g, '\\"')}"`;
+  const sourceLine = `source: ${quoted}`;
+
+  // Detect a leading YAML document marker (`---`, optionally followed by
+  // trailing whitespace or a comment). If present, the source line goes
+  // *after* it so the manifest stays a single document.
+  let insertAt = 0;
+  while (insertAt < lines.length && lines[insertAt].trim() === '') insertAt++;
+  const opensWithDocMarker =
+    insertAt < lines.length && /^---(\s|#|$)/.test(lines[insertAt]);
+  if (opensWithDocMarker) {
+    lines.splice(insertAt + 1, 0, sourceLine);
+  } else {
+    // Drop any leading blank lines we walked past so the file doesn't grow a
+    // blank-line gutter on every re-stamp.
+    lines.splice(0, insertAt, sourceLine);
+  }
+  fs.writeFileSync(manifestPath, lines.join('\n'));
+}
+
 /**
  * Run `<hqRoot>/core/scripts/scan-packages.sh` to wire the newly installed
  * pack's contributions into the host paths (skills, hooks, policies, etc.).
@@ -640,12 +696,15 @@ export async function installPack(
     }
 
     const destDir = installToPackages(fetched.payloadDir, pkg, hqRoot);
-    // Under the v12 HQ layout, packs live at `core/packages/<name>/` and are
-    // tracked by filesystem presence alone — no `modules.yaml` write. That
-    // removes the side effect that created a top-level `modules/` directory
-    // alongside the canonical `core/`. `hq update <pack>` will re-resolve a
-    // pack's source from its on-disk package.yaml or prompt for it, but that
-    // tradeoff is intentional — see the layout-fix PR for rationale.
+    // Under the v12+ HQ layout, packs live at `core/packages/<name>/` and
+    // are tracked by filesystem presence — no `modules.yaml` write (that
+    // removed the orphan top-level `modules/` tree alongside the canonical
+    // `core/`). The one piece of install-time provenance we DO record is the
+    // input `source` string, stamped into the destination `package.yaml` so
+    // setup.sh can dedup against `core/core.yaml:recommended_packages` on
+    // re-runs. Stamping the literal input (not the resolved SHA/version)
+    // matches the verbatim equality check in setup.sh.
+    stampInstallSource(destDir, source);
     runScanPackages(hqRoot);
 
     console.log(