coaiajs 0.1.2 → 0.2.0

package/articles/technical/commander-cli-framework.md

@@ -0,0 +1,262 @@
+# Commander.js CLI Framework: Technical Assessment for CoAiA.js
+
+> Package selection brief — CLI framework for coaiajs's multi-subcommand interface (tash, fetch, fuse, pipeline, env, gh, narrative, pde, planning)
+
+## Summary & Recommendation
+
+**Use `commander` v12.x** for the coaiajs CLI. Commander's Git-style subcommand model maps directly to our 9+ subcommand structure. It's the most widely adopted Node.js CLI framework (weekly downloads dwarf alternatives), has zero dependencies, generates help automatically, and every team member already knows it. Clipanion's class-based approach is elegant but niche; yargs is powerful but over-engineered for our tree.
+
+**Pin:** `"commander": "^12.1.0"`
+
+## What We're Replacing
+
+**Coaiapy** uses Python's `argparse` with subparsers:
+
+```python
+# coaiapy/coaiacli.py — argparse subcommand pattern
+import argparse
+parser = argparse.ArgumentParser(description='CoAiA CLI')
+subparsers = parser.add_subparsers(dest='command')
+parser_fuse = subparsers.add_parser('fuse', help='Manage Langfuse integrations.')
+parser_gh = subparsers.add_parser('gh', help='GitHub operations.')
+# ... 9+ subcommands
+```
+
+**coaia-narrative** uses `minimist` — bare-bones argv parsing with no help generation:
+
+```typescript
+// coaia-narrative/src/cli.ts — minimist pattern
+import minimist from 'minimist';
+const args = minimist(process.argv.slice(2));
+// Manual flag handling: --memory-path, -M, --json, --no-color, --interactive
+```
+
+**coaia-planning** also uses `minimist` for `--plans-dir` and `--output-dir`.
+
+## Options Compared
+
+| Feature | Commander v12 | Yargs v17 | Clipanion v4 | minimist v1.2 |
+|---------|--------------|-----------|-------------|--------------|
+| Weekly downloads | ~125M | ~80M | ~4M | ~50M |
+| Dependencies | 0 | 5+ | 0 | 0 |
+| Subcommand model | Git-style `.command()` | Builder callbacks | Class-based `paths` | None (manual) |
+| Auto-help | ✅ Excellent | ✅ Excellent | ✅ Good | ❌ None |
+| TypeScript | Good (DefinitelyTyped) | Good | Excellent (first-class) | Minimal |
+| Learning curve | Low | Medium | Medium-high | Very low |
+| Argument parsing | Options, args, variadic | All types + coercion | All types + validation | Raw key-value |
+| Completion (bash/zsh) | Plugin available | Built-in | Not built-in | ❌ |
+| Version handling | Built-in `.version()` | Built-in | Manual | ❌ |
+| Nested subcommands | `.command()` nesting | Recursive builders | Arbitrary `paths` depth | ❌ |
+| Exit override | `.exitOverride()` for testing | `fail()` handler | Exception-based | N/A |
+
+## API Overview
+
+### Main CLI Entry Point
+
+```typescript
+// src/cli.ts
+import { Command } from 'commander';
+import { version } from '../package.json';
+
+const program = new Command()
+  .name('coaia')
+  .version(version)
+  .description('CoAiA.js — Structural Tension Agent Framework');
+
+// Register subcommands
+program.addCommand(createTashCommand());
+program.addCommand(createFetchCommand());
+program.addCommand(createFuseCommand());
+program.addCommand(createPipelineCommand());
+program.addCommand(createEnvCommand());
+program.addCommand(createGhCommand());
+program.addCommand(createNarrativeCommand());
+program.addCommand(createPdeCommand());
+program.addCommand(createPlanningCommand());
+
+program.parse();
+```
+
+### Subcommand Definition Pattern
+
+```typescript
+// src/commands/fuse.ts
+import { Command } from 'commander';
+
+export function createFuseCommand(): Command {
+  const fuse = new Command('fuse')
+    .description('Manage Langfuse integrations');
+
+  fuse
+    .command('trace')
+    .description('Create or list traces')
+    .option('-s, --session <id>', 'Session ID')
+    .option('--list', 'List recent traces')
+    .option('--json', 'Output as JSON')
+    .action(async (opts) => {
+      if (opts.list) {
+        const traces = await listTraces(opts);
+        console.log(opts.json ? JSON.stringify(traces) : formatTable(traces));
+      } else {
+        await createTrace(opts);
+      }
+    });
+
+  fuse
+    .command('score')
+    .description('Score a trace')
+    .requiredOption('-t, --trace-id <id>', 'Trace ID to score')
+    .requiredOption('-n, --name <name>', 'Score name')
+    .requiredOption('-v, --value <number>', 'Score value', parseFloat)
+    .action(async (opts) => {
+      await scoreTrace(opts.traceId, opts.name, opts.value);
+    });
+
+  return fuse;
+}
+```
+
+### GitHub Subcommand (replacing cogh.py)
+
+```typescript
+// src/commands/gh.ts
+import { Command } from 'commander';
+
+export function createGhCommand(): Command {
+  const gh = new Command('gh')
+    .description('GitHub operations');
+
+  gh.command('issues')
+    .description('List repository issues')
+    .argument('<owner>', 'Repository owner')
+    .argument('<repo>', 'Repository name')
+    .option('--state <state>', 'Filter by state', 'open')
+    .option('--labels <labels...>', 'Filter by labels')
+    .option('--json', 'JSON output')
+    .action(async (owner, repo, opts) => {
+      const issues = await listIssues(owner, repo, opts);
+      console.log(opts.json ? JSON.stringify(issues) : formatIssuesTable(issues));
+    });
+
+  gh.command('issue <owner> <repo> <number>')
+    .description('Get a specific issue')
+    .action(async (owner, repo, number) => {
+      const issue = await getIssue(owner, repo, parseInt(number));
+      console.log(JSON.stringify(issue, null, 2));
+    });
+
+  return gh;
+}
+```
+
+### Pipeline Subcommand (with YAML template loading)
+
+```typescript
+// src/commands/pipeline.ts
+import { Command } from 'commander';
+
+export function createPipelineCommand(): Command {
+  const pipeline = new Command('pipeline')
+    .description('Execute pipeline templates');
+
+  pipeline
+    .command('run <template>')
+    .description('Run a pipeline template')
+    .option('-v, --var <key=value...>', 'Template variables')
+    .option('--dry-run', 'Show execution plan without running')
+    .option('--template-dir <dir>', 'Template search directory')
+    .action(async (template, opts) => {
+      const vars = parseVars(opts.var ?? []);
+      await runPipeline(template, vars, opts);
+    });
+
+  pipeline
+    .command('list')
+    .description('List available templates')
+    .action(async () => {
+      const templates = await listTemplates();
+      templates.forEach(t => console.log(`  ${t.name} (v${t.version})`));
+    });
+
+  return pipeline;
+}
+```
+
+### Global Options Pattern
+
+```typescript
+// Shared options across all subcommands
+program
+  .option('--json', 'Output as JSON')
+  .option('--no-color', 'Disable colored output')
+  .option('--verbose', 'Verbose logging')
+  .option('-M, --memory-path <path>', 'Memory file path')
+  .option('--env <file>', 'Custom env file');
+
+// Access in any subcommand action
+function getGlobalOpts(cmd: Command) {
+  const root = cmd.parent ?? cmd;
+  return root.opts() as { json?: boolean; color?: boolean; verbose?: boolean; memoryPath?: string };
+}
+```
+
+## Integration Plan
+
+1. **Entry point:** `src/cli.ts` — Commander program with `.parse()`
+2. **Command directory:** `src/commands/` — one file per subcommand
+   - `tash.ts` — Task management (structural tension)
+   - `fetch.ts` — Data fetching operations
+   - `fuse.ts` — Langfuse integration
+   - `pipeline.ts` — Pipeline template execution
+   - `env.ts` — Environment management
+   - `gh.ts` — GitHub operations
+   - `narrative.ts` — Narrative/chart operations
+   - `pde.ts` — Prompt decomposition
+   - `planning.ts` — Plan-to-STC operations
+3. **Bin entry:** `package.json` → `"bin": { "coaia": "./dist/src/cli.js" }`
+4. **Testing:** Commander's `.exitOverride()` + `.configureOutput()` for unit tests
+
+### Migration Path from minimist
+
+```typescript
+// BEFORE (coaia-narrative minimist pattern)
+const args = minimist(process.argv.slice(2));
+const memoryPath = args['memory-path'] || args.M || process.env.COAIA_MEMORY_PATH;
+
+// AFTER (Commander)
+program.option('-M, --memory-path <path>', 'Memory file path', process.env.COAIA_MEMORY_PATH);
+// memoryPath is now typed and auto-documented in --help
+```
+
+## Version & Ecosystem
+
+| Metric | Value |
+|--------|-------|
+| Current version | 12.1.0 |
+| Weekly downloads | ~125M |
+| Dependencies | 0 |
+| TypeScript | `@types/commander` (DefinitelyTyped) |
+| GitHub stars | ~27,000 |
+| Used by | npm CLI, Vue CLI, create-react-app |
+| Node.js compat | ≥16 (we target ≥20) |
+| License | MIT |
+
+## Why Not Yargs?
+
+Yargs is powerful but overkill. Our subcommand tree is wide (9 commands) but not deeply nested. Commander's chainable API makes each command file self-contained and readable. Yargs' middleware system and callback-based builders add complexity we don't need.
+
+## Why Not Clipanion?
+
+Clipanion's TypeScript-first class-based approach is architecturally beautiful, but:
+1. ~4M weekly downloads vs Commander's ~125M = smaller help ecosystem
+2. Class-based commands are less familiar to contributors
+3. Yarn is its primary consumer; documentation coverage is thin for edge cases
+4. Commander is already in our `package.json`
+
+## References
+
+- npm: https://www.npmjs.com/package/commander
+- GitHub: https://github.com/tj/commander.js
+- Guide: https://betterstack.com/community/guides/scaling-nodejs/commander-explained/
+- Comparison: https://npm-compare.com/commander,yargs
+- Clipanion: https://github.com/arcanis/clipanion

package/articles/technical/dotenv-config-patterns.md

@@ -0,0 +1,360 @@
+# Config Management in Node.js: Technical Assessment for CoAiA.js
+
+> Package selection brief — Configuration cascade replacing coaiapy's multi-file config (env vars > .env > coaia.json > defaults)
+
+## Summary & Recommendation
+
+**Use `dotenv` v16.x + `cosmiconfig` v9.x + `zod` for validation.** This triple replaces coaiapy's hand-rolled config cascade with a standard, validated pattern:
+- `dotenv` loads `.env` files (matching coaiapy's `.coaia-env`)
+- `cosmiconfig` discovers `coaia.json`, `coaia.yaml`, `.coaiarc`, or `coaia.config.js` (matching coaiapy's multi-format loading)
+- `zod` validates the merged config at startup (what coaiapy lacks entirely)
+
+**Pin:**
+```json
+"dotenv": "^16.4.0",
+"cosmiconfig": "^9.0.0",
+"zod": "^4.0.0"
+```
+
+## What We're Replacing
+
+Coaiapy has a complex config cascade across multiple modules:
+
+### environment.py (EnvironmentManager)
+```python
+@dataclass
+class EnvironmentConfig:
+    project_env_file: str = ".coaia-env"
+    global_env_file: str = "~/.coaia/global.env"
+    supported_formats: List[str] = ["json", "env"]
+
+class EnvironmentManager:
+    def _read_env_file(self, file_path: Path) -> Dict[str, Any]:
+        if content.startswith('{'):
+            return json.loads(content)
+        # Parse KEY=value format
+```
+
+### coaiamodule.py (read_config)
+```python
+def read_config():
+    # Priority: env vars > .env > coaia.json > defaults
+    # Searches: cwd, parent dirs, ~/.coaia/
+    # Supports: JSON, .env, shell env vars
+```
+
+### coaia-narrative CLI (minimist + dotenv)
+```typescript
+// Priority order:
+// 1. Command-line flags (--memory-path, -M)
+// 2. Custom env file (--env)
+// 3. .env in cwd
+// 4. _env.sh (fallback)
+// 5. Environment variables
+// 6. Defaults
+```
+
+The Python approach has no validation — missing keys fail at runtime deep in execution. The TypeScript approach in coaia-narrative works but is ad-hoc per module.
+
+## Options Compared
+
+| Feature | dotenv + cosmiconfig + zod | dotenv only | convict | nconf | node-config |
+|---------|--------------------------|-------------|---------|-------|-------------|
+| .env loading | ✅ dotenv | ✅ | ⚠️ Manual | ✅ | ❌ |
+| Multi-format (JSON/YAML/JS) | ✅ cosmiconfig | ❌ | ✅ JSON | ✅ | ✅ |
+| Parent dir search | ✅ cosmiconfig | ❌ | ❌ | ❌ | ❌ |
+| Schema validation | ✅ zod | ❌ | ✅ convict | ❌ | ❌ |
+| Type inference | ✅ zod infer | ❌ | ⚠️ Manual | ❌ | ❌ |
+| Variable expansion | ✅ dotenv-expand | ✅ dotenv-expand | ❌ | ❌ | ❌ |
+| Env override | ✅ process.env priority | ✅ | ✅ | ✅ | ✅ |
+| Weekly downloads | ~93M + ~30M + ~100M | ~93M | ~900K | ~900K | ~1.5M |
+| Bundle overhead | Minimal | Tiny | Medium | Medium | Medium |
+| TypeScript | All have types | ✅ | ⚠️ | ⚠️ | ⚠️ |
+
+## API Overview
+
+### Config Schema (Zod)
+
+```typescript
+// src/config/schema.ts
+import { z } from 'zod';
+
+export const CoaiaConfigSchema = z.object({
+  // Redis
+  redisUrl: z.string().url().default('redis://localhost:6379'),
+  
+  // Langfuse
+  langfusePublicKey: z.string().optional(),
+  langfuseSecretKey: z.string().optional(),
+  langfuseBaseUrl: z.string().url().default('https://cloud.langfuse.com'),
+  
+  // OpenAI
+  openaiApiKey: z.string().optional(),
+  openaiBaseUrl: z.string().url().optional(),
+  openaiModel: z.string().default('gpt-5.4'),
+  
+  // AWS Polly
+  awsRegion: z.string().default('us-east-1'),
+  pollyKey: z.string().optional(),
+  pollySecret: z.string().optional(),
+  
+  // GitHub
+  githubToken: z.string().optional(),
+  
+  // Paths
+  memoryPath: z.string().default('.coaia/memory.jsonl'),
+  plansDir: z.string().default('.coaia/plans'),
+  outputDir: z.string().default('.coaia/output'),
+  templateDir: z.string().default('.coaia/templates'),
+  
+  // Behavior
+  jsonOutput: z.boolean().default(false),
+  noColor: z.boolean().default(false),
+  verbose: z.boolean().default(false),
+});
+
+export type CoaiaConfig = z.infer<typeof CoaiaConfigSchema>;
+```
+
+### Config Loader (cosmiconfig + dotenv)
+
+```typescript
+// src/config/loader.ts
+import { cosmiconfig } from 'cosmiconfig';
+import dotenv from 'dotenv';
+import { CoaiaConfigSchema, type CoaiaConfig } from './schema.js';
+import { existsSync } from 'fs';
+import { resolve } from 'path';
+
+// cosmiconfig explorer — searches for config files
+const explorer = cosmiconfig('coaia', {
+  searchPlaces: [
+    'coaia.config.js',
+    'coaia.config.ts',
+    'coaia.json',
+    'coaia.yaml',
+    'coaia.yml',
+    '.coaiarc',
+    '.coaiarc.json',
+    '.coaiarc.yaml',
+    'package.json',  // "coaia" key in package.json
+  ],
+});
+
+let _config: CoaiaConfig | null = null;
+
+export async function loadConfig(overrides?: Partial<CoaiaConfig>): Promise<CoaiaConfig> {
+  if (_config && !overrides) return _config;
+
+  // Step 1: Load .env files (matches coaiapy's .coaia-env pattern)
+  const envFiles = [
+    '.env',
+    '.coaia-env',
+    resolve(process.env.HOME ?? '', '.coaia', 'global.env'),
+  ];
+  
+  for (const envFile of envFiles) {
+    if (existsSync(envFile)) {
+      dotenv.config({ path: envFile });
+    }
+  }
+
+  // Step 2: Load config file via cosmiconfig (searches parent dirs)
+  const result = await explorer.search();
+  const fileConfig = result?.config ?? {};
+
+  // Step 3: Merge sources (priority: CLI overrides > env vars > config file > defaults)
+  const merged = {
+    ...fileConfig,
+    // Env var overrides (matching coaiapy's env var names)
+    ...(process.env.REDIS_URL && { redisUrl: process.env.REDIS_URL }),
+    ...(process.env.UPSTASH_REDIS_URL && { redisUrl: process.env.UPSTASH_REDIS_URL }),
+    ...(process.env.LANGFUSE_PUBLIC_KEY && { langfusePublicKey: process.env.LANGFUSE_PUBLIC_KEY }),
+    ...(process.env.LANGFUSE_SECRET_KEY && { langfuseSecretKey: process.env.LANGFUSE_SECRET_KEY }),
+    ...(process.env.LANGFUSE_BASE_URL && { langfuseBaseUrl: process.env.LANGFUSE_BASE_URL }),
+    ...(process.env.OPENAI_API_KEY && { openaiApiKey: process.env.OPENAI_API_KEY }),
+    ...(process.env.OPENAI_BASE_URL && { openaiBaseUrl: process.env.OPENAI_BASE_URL }),
+    ...(process.env.AWS_REGION && { awsRegion: process.env.AWS_REGION }),
+    ...(process.env.GH_TOKEN && { githubToken: process.env.GH_TOKEN }),
+    ...(process.env.GITHUB_TOKEN && { githubToken: process.env.GITHUB_TOKEN }),
+    ...(process.env.COAIA_MEMORY_PATH && { memoryPath: process.env.COAIA_MEMORY_PATH }),
+    ...(process.env.COAIA_PLANS_DIR && { plansDir: process.env.COAIA_PLANS_DIR }),
+    ...(process.env.COAIA_OUTPUT_DIR && { outputDir: process.env.COAIA_OUTPUT_DIR }),
+    // CLI overrides (highest priority)
+    ...overrides,
+  };
+
+  // Step 4: Validate with Zod
+  _config = CoaiaConfigSchema.parse(merged);
+  return _config;
+}
+
+// Synchronous getter after initial load
+export function getConfig(): CoaiaConfig {
+  if (!_config) throw new Error('Config not loaded. Call loadConfig() first.');
+  return _config;
+}
+```
+
+### CLI Integration with Commander
+
+```typescript
+// src/cli.ts — CLI flags feed into config overrides
+import { Command } from 'commander';
+import { loadConfig } from './config/loader.js';
+
+const program = new Command()
+  .option('-M, --memory-path <path>', 'Memory file path')
+  .option('--env <file>', 'Custom env file path')
+  .option('--json', 'Output as JSON')
+  .option('--no-color', 'Disable colored output')
+  .option('--verbose', 'Verbose logging');
+
+program.hook('preAction', async (thisCommand) => {
+  const opts = thisCommand.opts();
+  
+  // Load custom env file if specified
+  if (opts.env) {
+    dotenv.config({ path: opts.env });
+  }
+  
+  // Initialize config with CLI overrides
+  await loadConfig({
+    ...(opts.memoryPath && { memoryPath: opts.memoryPath }),
+    ...(opts.json && { jsonOutput: true }),
+    ...(opts.color === false && { noColor: true }),
+    ...(opts.verbose && { verbose: true }),
+  });
+});
+```
+
+### Variable Expansion (dotenv-expand)
+
+```typescript
+// For .env files that reference other variables
+// .env:
+// BASE_DIR=/opt/coaia
+// MEMORY_PATH=${BASE_DIR}/memory.jsonl
+// PLANS_DIR=${BASE_DIR}/plans
+
+import dotenv from 'dotenv';
+import { expand } from 'dotenv-expand';
+
+const env = dotenv.config({ path: '.env' });
+expand(env);  // Resolves ${BASE_DIR} references
+```
+
+### Config File Examples
+
+```yaml
+# coaia.yaml — project-level config
+redis:
+  url: redis://localhost:6379
+
+langfuse:
+  publicKey: pk-lf-xxx
+  secretKey: sk-lf-xxx
+  baseUrl: https://cloud.langfuse.com
+
+openai:
+  apiKey: sk-xxx
+  model: gpt-5.4
+
+paths:
+  memory: .coaia/memory.jsonl
+  plans: .coaia/plans
+  templates: .coaia/templates
+```
+
+```json
+// coaia.json — alternative JSON format
+{
+  "redisUrl": "redis://localhost:6379",
+  "langfusePublicKey": "pk-lf-xxx",
+  "openaiModel": "gpt-5.4",
+  "memoryPath": ".coaia/memory.jsonl"
+}
+```
+
+```ini
+# .coaia-env — environment variable format (coaiapy compatible)
+REDIS_URL=redis://localhost:6379
+LANGFUSE_PUBLIC_KEY=pk-lf-xxx
+LANGFUSE_SECRET_KEY=sk-lf-xxx
+OPENAI_API_KEY=sk-xxx
+GH_TOKEN=ghp_xxx
+AWS_REGION=us-east-1
+COAIA_MEMORY_PATH=.coaia/memory.jsonl
+```
+
+### Validation Error Messages
+
+```typescript
+// On startup with invalid config:
+try {
+  await loadConfig();
+} catch (error) {
+  if (error instanceof z.ZodError) {
+    console.error('Configuration errors:');
+    error.issues.forEach(issue => {
+      console.error(`  ${issue.path.join('.')}: ${issue.message}`);
+    });
+    process.exit(1);
+  }
+  throw error;
+}
+
+// Output:
+// Configuration errors:
+//   redisUrl: Invalid url
+//   langfuseBaseUrl: Invalid url
+```
+
+## Integration Plan
+
+1. **Schema:** `src/config/schema.ts` — Zod schema for all config values
+2. **Loader:** `src/config/loader.ts` — dotenv + cosmiconfig + Zod validation
+3. **Export:** `src/config/index.ts` — `loadConfig()` and `getConfig()`
+4. **CLI hook:** Commander `preAction` hook loads config with CLI overrides
+5. **Testing:** Config loader is mockable via overrides parameter
+6. **Migration:** Support both `coaia.json` and `.coaia-env` formats (backward compat)
+
+### Config Priority (matches coaiapy's cascade)
+
+```
+1. CLI flags              (--memory-path, --json)     ← Highest
+2. Environment variables  (REDIS_URL, GH_TOKEN)
+3. .env / .coaia-env      (project root)
+4. ~/.coaia/global.env    (global user config)
+5. coaia.json / coaia.yaml (cosmiconfig search)
+6. Zod defaults           (schema defaults)            ← Lowest
+```
+
+## Version & Ecosystem
+
+| Package | Version | Weekly Downloads | Purpose |
+|---------|---------|-----------------|---------|
+| dotenv | 16.4.x | ~93M | .env file loading |
+| dotenv-expand | 12.0.x | ~30M | Variable expansion in .env |
+| cosmiconfig | 9.0.x | ~30M | Multi-format config discovery |
+| zod | 4.3.x | ~100M+ | Schema validation + type inference |
+
+All packages: MIT licensed, zero or minimal dependencies, TypeScript supported.
+
+## Why Not Convict?
+
+Convict (from Mozilla) offers built-in schema validation, but:
+1. ~900K downloads vs dotenv's ~93M — smaller ecosystem
+2. Doesn't support YAML config files natively
+3. Doesn't search parent directories (cosmiconfig does)
+4. We already have Zod for validation (no need for convict's schema DSL)
+5. Convict's API is less TypeScript-friendly than Zod's type inference
+
+## References
+
+- dotenv npm: https://www.npmjs.com/package/dotenv
+- cosmiconfig npm: https://www.npmjs.com/package/cosmiconfig
+- cosmiconfig GitHub: https://github.com/cosmiconfig/cosmiconfig
+- dotenv security: https://www.johal.in/secrets-dotenv-secure-config-management-in-production-2026/
+- Config comparison: https://npm-compare.com/config,convict,dotenv,nconf
+- dotenv usage: https://oneuptime.com/blog/post/2026-01-25-dotenv-configuration-nodejs/view