@abarcenas/ai-workflow-template 1.39.0 → 1.41.0

package/.agents/instructions/learned-knowledge.instructions.md

@@ -184,3 +184,46 @@
 - **orchestrator/CI**: Make CI a required check on `main` (branch protection) so a 4-minute PR merge window cannot bypass it. Any fix must go through a PR that runs CI, not be pushed directly to a branch.
 - **researcher/coder**: Verify contested library behavior claims against the INSTALLED source before persisting them. The spike endorsed "pgvector accepts typed arrays" based on an unverified note in a fix commit; the installed `pgvector@0.3.0` source and an empirical repro prove the opposite. When a knowledgebase contains contradictory sessions about library behavior (2026-07-30 says Float32Array fails, 2026-08-01 spike says it works), run a minimal repro to settle it — never trust the newer claim by default.
 - **coder**: When fixing a type mismatch, check BOTH consumers of the value. `embed()`'s result feeds both the test assertion AND pgvector's `toSql()`. A fix that satisfies the test but breaks the DB boundary (or vice versa) is not a fix — convert at the boundary instead of changing the function's contract.
+
+## Session: 2026-08-02 — Fix npm E404 + Setup `.env` Loading (release v1.40.0)
+
+**Pipeline:** implementer (bootstrap) → researcher (npm E404) → researcher (DATABASE_URL/.env) → implementer (plan) → coder (3 parallel batches A/B/C, 12 tasks) → coder (MCP server fix) → deployer (release) → tracker
+**Coverage:** 160/160 tests passing across 9 test files, 0 failures; `index.js` coverage 90.75% stmts / 90.9% lines (above plan's 90% threshold)
+**TDD Iterations:** 0 (feature pipeline, not TDD)
+
+**New knowledge:**
+- **The setup CLI never loaded the consumer `.env` → `DATABASE_URL` appeared unset** — `scripts/setup/knowledgebase.js:64` checks `process.env.DATABASE_URL` directly, and nothing in `scripts/setup/` loaded the consumer's `.env`. The child `knowledgebase-cli.js` had `import 'dotenv/config'` but was never spawned because the parent check short-circuited. **Fix pattern:** move `dotenv` from `devDependencies` → `dependencies` (devDeps are NOT installed at `npx` time) and add `import 'dotenv/config'` at the top of the CLI entry point (`bin/setup.js`). ESM static side-effect imports are hoisted, so `dotenv.config()` runs before any other module code — including dynamic `await import()` of the orchestrator. `dotenv.config()` resolves `.env` from `process.cwd()`, which is the consumer root in the documented `npx` use case.
+- **The knowledgebase MCP server had the SAME bug class — the CLI got dotenv but the MCP server didn't** — `scripts/mcp-knowledgebase-server.js` never imported dotenv, so `DATABASE_URL` was `undefined` in the MCP process even though `.env` had it. The opencode.json knowledgebase MCP entry also had NO `env` block (unlike the `github` entry which passes `$GITHUB_PERSONAL_ACCESS_TOKEN`). **Universal pattern:** ANY Node.js script (CLI, MCP server, child process) that reads `process.env` variables sourced from `.env` MUST load dotenv itself or receive the env via an explicit config/env block. `process.env` does not auto-load `.env`. Belt-and-suspenders fix: add `import 'dotenv/config'` to the script AND an `env` block (`"DATABASE_URL": "$DATABASE_URL"`) to the MCP entry in opencode.json.
+- **`--knowledgebase` flag semantics: run ONLY the knowledgebase phase** — implemented by setting ALL the existing skip flags true (`flags.skipHooks = true; flags.skipPrepare = true; flags.skipSync = true; flags.skipKnowledgebase = false`) so the existing phase loop runs only Phase 1 (discovery, always runs) + Phase 6 (knowledgebase). This reuses the existing skip-gate infrastructure (ALT-04 design) instead of adding a new control-flow path — 4-line change, fully covered by tests. The `skipKnowledgebase = false` guard handles the pathological `--knowledgebase --skip-knowledgebase` combination.
+- **Warning text bug: recommended a command that doesn't exist** — the old warning said `npx ai-workflow-template setup --knowledgebase`, which is wrong THREE ways: (1) unscoped package name (`@abarcenas/` scope required), (2) bogus `setup` positional (the CLI parser silently ignores unknown args), (3) `--knowledgebase` wasn't even a flag (only `--skip-knowledgebase` existed). Correct message now: `DATABASE_URL not configured. Set DATABASE_URL in your .env file, then re-run: npx @abarcenas/ai-workflow-template --knowledgebase`. **Lesson: when a warning tells the user how to fix the problem, verify every part of that instruction actually works.**
+- **Version is bumped by a Husky pre-commit hook (`scripts/bump-version.js`) which ALWAYS does a minor bump** — every commit auto-bumps `package.json` (1.40.0 → 1.41.0 on the deployer's first attempt). Release engineers must use `git commit --no-verify` (or `git reset --soft HEAD~1` + restore + re-commit) to keep the intended version. **Recommendation: add a `SKIP_BUMP` env guard** to `bump-version.js` so `SKIP_BUMP=1 git commit` bypasses the bump without skipping ALL hooks (which also skips memory-schema validation).
+- **npm-publish GitHub workflow triggers on push to `main`, NOT on tags** — `.github/workflows/npm-publish.yml` runs `npm publish` when a branch merges into `main` (via PR). Pushing a tag alone (`git push origin --tags`) does NOT trigger publish. **Deploy flow for this repo:** commit + tag + push branch, then open a PR to `main`; publish happens on merge.
+- **SECURITY: `.env` was tracked in git with a real DATABASE_URL password** — the file contains a live 32-char PostgreSQL credential (`postgres@192.168.31.200`). It should have been gitignored from day one. **Remediation:** `git rm --cached .env`, add `.env` to `.gitignore`, rotate the credential (it is in git history regardless). Secrets-check the template: `.env.example` is safe, `.env` is not.
+- **npx behavior for scoped packages with a bin** — `npx @scope/package` resolves the scoped package, detects the single `bin` entry (`ai-workflow-setup`), and runs it. Positional subcommands like `setup` are silently ignored by this CLI's parser (`parseCliArgs` maps only known flags; unrecognized args fall through harmlessly). The unscoped `npx ai-workflow-template` returns 404 because that name doesn't exist on the registry — scoped packages must be invoked with their full `@scope/name`.
+
+**Agent tuning notes:**
+- **implementer (bootstrap)**: Verify the package identity facts (name, scope, bin, publishability, npmrc auth) BEFORE the researcher investigates — ground truth prevents wasted research on wrong assumptions (e.g., whether the package was even published).
+- **researcher**: When a consumer reports "X not found"/"not configured", check the code path for env loading before blaming the user's config — the consumer's `.env` was valid and correctly placed; the parent process just never read it. Also always validate the error/warning message itself (it contained a command that couldn't possibly work). Look for the SAME bug class in sibling entry points (CLI fixed earlier, MCP server missed) — search all scripts that read `process.env`.
+- **implementer (planning)**: When two tasks edit the same file, merge them into one task to avoid parallel write races (the coder had to adapt T1+T5 → one task, T7+T8+T9 → one task). Keep the entry-point import (`bin/setup.js`) rather than per-module imports — it guarantees ALL phases benefit from env loading, and ESM hoisting makes the timing safe.
+- **coder (parallel batches)**: Before executing a batch, detect same-file collisions and merge them — this pipeline's planned 5+5 parallel tasks were safely collapsed to 4+3. When adding a flag to `SUPPORTED_FLAGS`, remember to update THREE places: constants.js (flag map), index.js (handler), ui.js (help text) — plus tests. When restoring coverage, add tests that assert the new behavior (not just that code runs).
+- **coder (MCP/env fixes)**: If one script in a family needed a dotenv fix, audit the whole family. The knowledgebase-cli.js fix (2026-07-30) should have prompted checking mcp-knowledgebase-server.js at the same time. For opencode MCP entries that need env vars, the `github` entry is the reference pattern (`"env": { "VAR": "$VAR" }`).
+- **deployer**: Before committing a release, check for auto-bump hooks (`scripts/bump-version.js` via Husky pre-commit) and confirm the publish trigger in the workflow file (`npm-publish.yml` on push to `main`) — neither the version bump nor npm publish happens where you'd naively expect. Never commit a release without running `git diff` to confirm the version didn't change. Scan `git ls-files` for tracked secrets before release.
+- **tracker**: When a pipeline produces spikes + a plan + a release, record the fix pattern (env loading via `dotenv/config` at entry points) as a UNIVERSAL rule, not a one-off — this is the third time this project hit the "script reads process.env without loading .env" bug (knowledgebase-cli 2026-07-30, setup pipeline 2026-08-01, MCP server 2026-08-02).
+
+## Session: 2026-08-02 — Knowledgebase MCP Search Threshold Fix
+
+**Pipeline:** Follow-up fix after the npm E404 + `.env` loading pipeline (research spike → coder → verify after restart → tracker)
+**Coverage:** N/A (one-line production fix + verification; all 160 tests still passing)
+**TDD Iterations:** 0 (ad-hoc follow-up fix, not a pipeline run)
+
+**New knowledge:**
+- **An empty search result does NOT mean the index is empty — check the similarity threshold against the embedding model's actual score range** — `scripts/mcp-knowledgebase-server.js` defaulted `threshold` to `0.6`, but the `all-MiniLM-L6-v2` (384-dim) embedding model produces cosine similarities in the range ~0.01–0.41 for this corpus. So ALL results were filtered out and `knowledgebase_knowledgebase_search` returned "No results found. The knowledgebase may be empty." even though 9 chunks were indexed and `knowledgebase_knowledgebase_stats` showed them. The CLI worked because it passes `undefined` → core engine default `0.0`. **Diagnostic rule:** when search returns empty but stats show chunks, run the same query through the CLI at threshold 0.0 to see the real similarity distribution, then compare against the tool's default threshold.
+- **`||` vs `??` falsy-coercion bug** — `args.threshold || 0.6` coerces an explicit `threshold: 0` to `0.6` because `0` is falsy. Fixed with nullish coalescing: `args.threshold ?? 0.1`. **Universal pattern:** use `??` (nullish coalescing) for default values wherever `0`/`false`/`''` are legitimate explicit inputs — `||` silently overwrites them.
+- **Code changes to an MCP server require a server/opencode restart to take effect** — after fixing the MCP server code, the running opencode session's MCP server process still had the old code in memory; the live `knowledgebase_knowledgebase_search` kept returning "No results" until opencode was restarted. **Universal pattern:** during development, verify a fix against a freshly spawned server process (spawn → `initialize` → `tools/list` → `tools/call`), then restart the host process to load the fix into the live tool. A working fresh handshake is NOT proof the live tool is fixed.
+- **Verification workflow that worked** — the fix was verified two ways: (a) CLI search (`node scripts/knowledgebase-cli.js search "..."`) returned ranked results with real similarities (top 0.396), and (b) a full MCP handshake (spawn server → `initialize` → `tools/list` → `tools/call knowledgebase_search`) returned "### Result 1 (similarity: 0.396)" with no explicit threshold passed. After restarting opencode, the live `knowledgebase_search` returned 5 results (top 0.396) — confirming the fix end-to-end.
+- **Threshold tuning is model-specific** — the `0.6` default was an arbitrary high value (likely copied from a different embedding model). For `all-MiniLM-L6-v2`, realistic thresholds are 0.1–0.2 for "somewhat related", 0.25–0.4 for "moderately related", 0.4+ for "strongly related". A threshold default must be chosen for the embedding model actually in use.
+
+**Agent tuning notes:**
+- **researcher**: CLI-search diagnostics are the fastest way to compare thresholds — running the SAME query through `knowledgebase-cli.js` at `--threshold 0.0` vs `--threshold 0.6` reproduced the bug exactly and quantified the real similarity range (0.01–0.41). When an MCP tool returns empty but the data exists, use the CLI to isolate whether the problem is the tool's defaults vs. the data itself.
+- **coder**: When fixing a default-value bug (`||` → `??`), scan sibling call sites for the same falsy-coercion pattern — the same class of bug can hide in `args.limit || 5` (also in this handler) and any other `||` default.
+- **tracker**: Record MCP-server restart requirements as a universal gotcha — live MCP tools run long-lived processes that do NOT hot-reload code changes; document the restart step in verification notes.

package/bin/setup.js

@@ -13,6 +13,9 @@
 //   2 — Fatal error (discovery failure or unexpected exception)
 // ═════════════════════════════════════════════════════════════════════════════
 
+// Load consumer's .env into process.env before any phase runs
+import 'dotenv/config'
+
 try {
   const { main } = await import('../scripts/setup/index.js')
   const exitCode = await main(process.argv.slice(2))

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@abarcenas/ai-workflow-template",
-  "version": "1.39.0",
+  "version": "1.41.0",
   "description": "A comprehensive template repository for AI-assisted development workflows using GitHub Copilot. This template provides a structured approach to leveraging AI assistance through pre-configured skills, prompts, agents, and coding standards.",
   "main": "index.js",
   "bin": {
@@ -57,14 +57,14 @@
   "devDependencies": {
     "@playwright/test": "^1.59.1",
     "@vitest/coverage-v8": "^4.1.8",
-    "dotenv": "^17.4.2",
     "husky": "^9.0.0",
     "vitest": "^4.1.8"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.0.0",
     "better-sqlite3": "^12.11.0",
-    "sqlite-vec": "^0.1.9"
+    "sqlite-vec": "^0.1.9",
+    "dotenv": "^17.4.2"
   },
   "optionalDependencies": {
     "@xenova/transformers": "^2.17.0",

package/scripts/mcp-knowledgebase-server.js

@@ -20,6 +20,10 @@ import {
   CallToolRequestSchema,
   ListToolsRequestSchema,
 } from '@modelcontextprotocol/sdk/types.js';
+
+// Load .env file into process.env before any configuration reads
+import 'dotenv/config';
+
 import {
   search,
   upsertChunks,
@@ -136,7 +140,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
       case 'knowledgebase_search': {
         const results = await search(args.query, {
           project_id: args.projectId || undefined,
-          threshold: args.threshold || 0.6,
+          threshold: args.threshold ?? 0.1,
           limit: args.limit || 5,
         });
 

package/scripts/setup/constants.js

@@ -139,6 +139,7 @@ export const SUPPORTED_FLAGS = {
   '--skip-prepare': 'skipPrepare',
   '--skip-sync': 'skipSync',
   '--skip-knowledgebase': 'skipKnowledgebase',
+  '--knowledgebase': 'knowledgebase',
   '--help': 'help',
   '-h': 'help',
   '--version': 'version',

package/scripts/setup/index.js

@@ -141,6 +141,14 @@ export async function main(argv) {
     return EXIT_CODES.SUCCESS
   }
 
+  // When --knowledgebase is passed, run ONLY the knowledgebase phase
+  if (flags.knowledgebase) {
+    flags.skipHooks = true
+    flags.skipPrepare = true
+    flags.skipSync = true
+    flags.skipKnowledgebase = false
+  }
+
   // ── 3. Display header banner ────────────────────────────────────────────
   if (!flags.quiet) {
     header()

package/scripts/setup/index.test.js

@@ -116,6 +116,7 @@ const defaultFlags = {
   skipPrepare: false,
   skipSync: false,
   skipKnowledgebase: false,
+  knowledgebase: false,
   help: false,
   version: false,
   quiet: false,
@@ -549,6 +550,48 @@ describe('main() — setup orchestrator', () => {
     expect(registerKnowledgebase).toHaveBeenCalledTimes(1)
   })
 
+  // ── 16b. --knowledgebase flag (T6: run ONLY the knowledgebase phase) ─────
+
+  it('--knowledgebase: runs only discovery + knowledgebase phases, exits 0', async () => {
+    parseCliArgs.mockReturnValue({ ...defaultFlags, knowledgebase: true })
+
+    const code = await main(['--knowledgebase'])
+
+    expect(code).toBe(EXIT_CODES.SUCCESS)
+    expect(parseCliArgs).toHaveBeenCalledWith(['--knowledgebase'])
+    // Discovery always runs
+    expect(discover).toHaveBeenCalledTimes(1)
+    // All other phases are skipped by the --knowledgebase shortcut
+    expect(installHooks).not.toHaveBeenCalled()
+    expect(handlePrepare).not.toHaveBeenCalled()
+    expect(initHusky).not.toHaveBeenCalled()
+    expect(runSyncPhase).not.toHaveBeenCalled()
+    // Knowledgebase phase runs
+    expect(registerKnowledgebase).toHaveBeenCalledTimes(1)
+    // Header and summary shown (not quiet)
+    expect(header).toHaveBeenCalledTimes(1)
+    expect(summary).toHaveBeenCalledTimes(1)
+  })
+
+  it('--knowledgebase: still runs knowledgebase phase when skipKnowledgebase was also passed', async () => {
+    // --knowledgebase forces skipKnowledgebase back to false so the phase runs
+    parseCliArgs.mockReturnValue({
+      ...defaultFlags,
+      knowledgebase: true,
+      skipKnowledgebase: true,
+    })
+
+    const code = await main(['--knowledgebase', '--skip-knowledgebase'])
+
+    expect(code).toBe(EXIT_CODES.SUCCESS)
+    expect(discover).toHaveBeenCalledTimes(1)
+    expect(installHooks).not.toHaveBeenCalled()
+    expect(handlePrepare).not.toHaveBeenCalled()
+    expect(initHusky).not.toHaveBeenCalled()
+    expect(runSyncPhase).not.toHaveBeenCalled()
+    expect(registerKnowledgebase).toHaveBeenCalledTimes(1)
+  })
+
   // ── 17. Verify parseCliArgs receives the argv ───────────────────────────
 
   it('passes argv array to parseCliArgs unchanged', async () => {

package/scripts/setup/knowledgebase.js

@@ -63,8 +63,8 @@ export async function registerKnowledgebase(context) {
   // ── DATABASE_URL check ───────────────────────────────────────────────
   if (!process.env.DATABASE_URL) {
     const message =
-      'DATABASE_URL not configured. Set up later with: ' +
-      'npx ai-workflow-template setup --knowledgebase'
+      'DATABASE_URL not configured. Set DATABASE_URL in your .env file, ' +
+      'then re-run: npx @abarcenas/ai-workflow-template --knowledgebase'
     logWarn(
       `[knowledgebase] Skipping Phase 6 \u2014 ${message}`,
     )

package/scripts/setup/knowledgebase.test.js

@@ -161,8 +161,12 @@ describe('registerKnowledgebase', () => {
 
     expect(result).toEqual({
       action: 'skipped',
-      message: expect.stringContaining('DATABASE_URL'),
+      message: expect.stringContaining('DATABASE_URL not configured'),
     })
+    // The warning must guide the consumer to set DATABASE_URL and re-run
+    // with the scoped package name and the --knowledgebase flag.
+    expect(result.message).toContain('@abarcenas/ai-workflow-template')
+    expect(result.message).toContain('--knowledgebase')
     expect(spawn).not.toHaveBeenCalled()
   })
 

package/scripts/setup/ui.js

@@ -422,6 +422,8 @@ export function help() {
     skipHooks: 'Skip git hook installation phase',
     skipPrepare: 'Skip prepare script modification phase',
     skipSync: 'Skip file sync phase',
+    skipKnowledgebase: 'Skip knowledgebase registration phase',
+    knowledgebase: 'Run ONLY the knowledgebase registration phase',
     help: 'Show this help message and exit',
     version: 'Show the version number and exit',
     quiet: 'Suppress non-error output (quiet mode)',
@@ -435,6 +437,8 @@ export function help() {
     'skipHooks',
     'skipPrepare',
     'skipSync',
+    'skipKnowledgebase',
+    'knowledgebase',
     'help',
     'version',
     'quiet',
@@ -446,7 +450,7 @@ export function help() {
   write(`${BLD}${PACKAGE_NAME}${RST} ${GRN}v${version}${RST}`)
   write()
   write(`  ${BLD}Usage:${RST}`)
-  write(`    npx ${PACKAGE_NAME} setup [options]`)
+  write(`    npx ${PACKAGE_NAME} [options]`)
   write(`    ${BIN_NAME} [options]`)
   write()
   write(`  ${BLD}Description:${RST}`)
@@ -468,9 +472,10 @@ export function help() {
 
   write()
   write(`  ${BLD}Examples:${RST}`)
-  write(`    npx ${PACKAGE_NAME} setup`)
-  write(`    npx ${PACKAGE_NAME} setup --dry-run --verbose`)
-  write(`    npx ${PACKAGE_NAME} setup --force --skip-hooks`)
+  write(`    npx ${PACKAGE_NAME}`)
+  write(`    npx ${PACKAGE_NAME} --dry-run --verbose`)
+  write(`    npx ${PACKAGE_NAME} --knowledgebase`)
+  write(`    npx ${PACKAGE_NAME} --force --skip-hooks`)
   write(`    ${BIN_NAME} --help`)
   write()
 }