arboris-cli 1.0.0 → 1.1.1

package/prisma/skills/data-throughput-accelerator/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: data-throughput-accelerator
+description: Use when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness.
+metadata:
+  origin: ECC
+tools: Read, Write, Edit, Bash, Grep, Glob
+---
+
+# Data Throughput Accelerator
+
+Use this skill when the bottleneck is moving, transforming, or saving lots of
+data. The goal is not just speed. The goal is faster correct data landing in the
+right place with proof.
+
+## First Distinction
+
+Separate these before optimizing:
+
+- source extraction speed;
+- network transfer speed;
+- warehouse/load speed;
+- transform speed;
+- serving-table freshness;
+- live tail growth while the job runs.
+
+A pipeline can be "fast" and still appear behind if new data arrives faster
+than the final catch-up window.
+
+## Fast Path Heuristics
+
+- Move compute to where the data already is.
+- Prefer warehouse-native scans, joins, and appends for large landed files.
+- Use manifests or checkpoints so completed files/partitions are skipped.
+- Use partitioning and clustering that match the read and append pattern.
+- Batch small files, requests, and writes.
+- Make writes idempotent through unique keys, manifests, or replaceable staging.
+- Keep raw, derived, and serving tables separately accountable.
+
+## Workflow
+
+1. Read the current source, target, and manifest contracts.
+2. Measure backlog: external files, manifest rows, raw rows, derived rows,
+   min/max timestamps, and unprocessed counts.
+3. Run a safe catch-up or sample benchmark.
+4. Compare variants: batch size, worker count, warehouse SQL, file grouping,
+   staging shape, and manifest update method.
+5. Promote only the fastest path that keeps counts and timestamps coherent.
+6. Codify the path as a CLI, scheduled job, workflow, or runbook.
+7. Rerun final accounting after the codified path executes.
+
+## Accounting Output
+
+Use a hard accounting block:
+
+```text
+Data throughput result:
+- Source files discovered: 294
+- Files processed this run: 294
+- Raw rows added: 9,683,598
+- Derived rows added: 8,917,585
+- Remaining tail: 24 files at readback time
+- Runtime: 38.7s
+- Correctness gate: manifest counts and table max timestamps match
+```
+
+## Guardrails
+
+- Do not delete raw data to make a metric look better.
+- Do not skip failed files silently.
+- Do not mix historical backfill status with live-tail freshness.
+- Do not call a pipeline complete until the target tables and manifest agree.
+- For finance, healthcare, regulated, or customer-impacting data, preserve
+  replay evidence and approval gates.

package/prisma/skills/database-migrations/SKILL.md

@@ -0,0 +1,430 @@
+---
+name: database-migrations
+description: Database migration best practices for schema changes, data migrations, rollbacks, and zero-downtime deployments across PostgreSQL, MySQL, and common ORMs (Prisma, Drizzle, Kysely, Django, TypeORM, golang-migrate).
+metadata:
+  origin: ECC
+---
+
+# Database Migration Patterns
+
+Safe, reversible database schema changes for production systems.
+
+## When to Activate
+
+- Creating or altering database tables
+- Adding/removing columns or indexes
+- Running data migrations (backfill, transform)
+- Planning zero-downtime schema changes
+- Setting up migration tooling for a new project
+
+## Core Principles
+
+1. **Every change is a migration** — never alter production databases manually
+2. **Migrations are forward-only in production** — rollbacks use new forward migrations
+3. **Schema and data migrations are separate** — never mix DDL and DML in one migration
+4. **Test migrations against production-sized data** — a migration that works on 100 rows may lock on 10M
+5. **Migrations are immutable once deployed** — never edit a migration that has run in production
+
+## Migration Safety Checklist
+
+Before applying any migration:
+
+- [ ] Migration has both UP and DOWN (or is explicitly marked irreversible)
+- [ ] No full table locks on large tables (use concurrent operations)
+- [ ] New columns have defaults or are nullable (never add NOT NULL without default)
+- [ ] Indexes created concurrently (not inline with CREATE TABLE for existing tables)
+- [ ] Data backfill is a separate migration from schema change
+- [ ] Tested against a copy of production data
+- [ ] Rollback plan documented
+
+## PostgreSQL Patterns
+
+### Adding a Column Safely
+
+```sql
+-- GOOD: Nullable column, no lock
+ALTER TABLE users ADD COLUMN avatar_url TEXT;
+
+-- GOOD: Column with default (Postgres 11+ is instant, no rewrite)
+ALTER TABLE users ADD COLUMN is_active BOOLEAN NOT NULL DEFAULT true;
+
+-- BAD: NOT NULL without default on existing table (requires full rewrite)
+ALTER TABLE users ADD COLUMN role TEXT NOT NULL;
+-- This locks the table and rewrites every row
+```
+
+### Adding an Index Without Downtime
+
+```sql
+-- BAD: Blocks writes on large tables
+CREATE INDEX idx_users_email ON users (email);
+
+-- GOOD: Non-blocking, allows concurrent writes
+CREATE INDEX CONCURRENTLY idx_users_email ON users (email);
+
+-- Note: CONCURRENTLY cannot run inside a transaction block
+-- Most migration tools need special handling for this
+```
+
+### Renaming a Column (Zero-Downtime)
+
+Never rename directly in production. Use the expand-contract pattern:
+
+```sql
+-- Step 1: Add new column (migration 001)
+ALTER TABLE users ADD COLUMN display_name TEXT;
+
+-- Step 2: Backfill data (migration 002, data migration)
+UPDATE users SET display_name = username WHERE display_name IS NULL;
+
+-- Step 3: Update application code to read/write both columns
+-- Deploy application changes
+
+-- Step 4: Stop writing to old column, drop it (migration 003)
+ALTER TABLE users DROP COLUMN username;
+```
+
+### Removing a Column Safely
+
+```sql
+-- Step 1: Remove all application references to the column
+-- Step 2: Deploy application without the column reference
+-- Step 3: Drop column in next migration
+ALTER TABLE orders DROP COLUMN legacy_status;
+
+-- For Django: use SeparateDatabaseAndState to remove from model
+-- without generating DROP COLUMN (then drop in next migration)
+```
+
+### Large Data Migrations
+
+```sql
+-- BAD: Updates all rows in one transaction (locks table)
+UPDATE users SET normalized_email = LOWER(email);
+
+-- GOOD: Batch update with progress
+DO $$
+DECLARE
+  batch_size INT := 10000;
+  rows_updated INT;
+BEGIN
+  LOOP
+    UPDATE users
+    SET normalized_email = LOWER(email)
+    WHERE id IN (
+      SELECT id FROM users
+      WHERE normalized_email IS NULL
+      LIMIT batch_size
+      FOR UPDATE SKIP LOCKED
+    );
+    GET DIAGNOSTICS rows_updated = ROW_COUNT;
+    RAISE NOTICE 'Updated % rows', rows_updated;
+    EXIT WHEN rows_updated = 0;
+    COMMIT;
+  END LOOP;
+END $$;
+```
+
+## Prisma (TypeScript/Node.js)
+
+### Workflow
+
+```bash
+# Create migration from schema changes
+npx prisma migrate dev --name add_user_avatar
+
+# Apply pending migrations in production
+npx prisma migrate deploy
+
+# Reset database (dev only)
+npx prisma migrate reset
+
+# Generate client after schema changes
+npx prisma generate
+```
+
+### Schema Example
+
+```prisma
+model User {
+  id        String   @id @default(cuid())
+  email     String   @unique
+  name      String?
+  avatarUrl String?  @map("avatar_url")
+  createdAt DateTime @default(now()) @map("created_at")
+  updatedAt DateTime @updatedAt @map("updated_at")
+  orders    Order[]
+
+  @@map("users")
+  @@index([email])
+}
+```
+
+### Custom SQL Migration
+
+For operations Prisma cannot express (concurrent indexes, data backfills):
+
+```bash
+# Create empty migration, then edit the SQL manually
+npx prisma migrate dev --create-only --name add_email_index
+```
+
+```sql
+-- migrations/20240115_add_email_index/migration.sql
+-- Prisma cannot generate CONCURRENTLY, so we write it manually
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_email ON users (email);
+```
+
+## Drizzle (TypeScript/Node.js)
+
+### Workflow
+
+```bash
+# Generate migration from schema changes
+npx drizzle-kit generate
+
+# Apply migrations
+npx drizzle-kit migrate
+
+# Push schema directly (dev only, no migration file)
+npx drizzle-kit push
+```
+
+### Schema Example
+
+```typescript
+import { pgTable, text, timestamp, uuid, boolean } from "drizzle-orm/pg-core";
+
+export const users = pgTable("users", {
+  id: uuid("id").primaryKey().defaultRandom(),
+  email: text("email").notNull().unique(),
+  name: text("name"),
+  isActive: boolean("is_active").notNull().default(true),
+  createdAt: timestamp("created_at").notNull().defaultNow(),
+  updatedAt: timestamp("updated_at").notNull().defaultNow(),
+});
+```
+
+## Kysely (TypeScript/Node.js)
+
+### Workflow (kysely-ctl)
+
+```bash
+# Initialize config file (kysely.config.ts)
+kysely init
+
+# Create a new migration file
+kysely migrate make add_user_avatar
+
+# Apply all pending migrations
+kysely migrate latest
+
+# Rollback last migration
+kysely migrate down
+
+# Show migration status
+kysely migrate list
+```
+
+### Migration File
+
+```typescript
+// migrations/2024_01_15_001_create_user_profile.ts
+import { type Kysely, sql } from 'kysely'
+
+// IMPORTANT: Always use Kysely<any>, not your typed DB interface.
+// Migrations are frozen in time and must not depend on current schema types.
+export async function up(db: Kysely<any>): Promise<void> {
+  await db.schema
+    .createTable('user_profile')
+    .addColumn('id', 'serial', (col) => col.primaryKey())
+    .addColumn('email', 'varchar(255)', (col) => col.notNull().unique())
+    .addColumn('avatar_url', 'text')
+    .addColumn('created_at', 'timestamp', (col) =>
+      col.defaultTo(sql`now()`).notNull()
+    )
+    .execute()
+
+  await db.schema
+    .createIndex('idx_user_profile_avatar')
+    .on('user_profile')
+    .column('avatar_url')
+    .execute()
+}
+
+export async function down(db: Kysely<any>): Promise<void> {
+  await db.schema.dropTable('user_profile').execute()
+}
+```
+
+### Programmatic Migrator
+
+```typescript
+import { Migrator, FileMigrationProvider } from 'kysely'
+import { promises as fs } from 'fs'
+import * as path from 'path'
+// ESM only — CJS can use __dirname directly
+import { fileURLToPath } from 'url'
+const migrationFolder = path.join(
+  path.dirname(fileURLToPath(import.meta.url)),
+  './migrations',
+)
+
+// `db` is your Kysely<any> database instance
+const migrator = new Migrator({
+  db,
+  provider: new FileMigrationProvider({
+    fs,
+    path,
+    migrationFolder,
+  }),
+  // WARNING: Only enable in development. Disables timestamp-ordering
+  // validation, which can cause schema drift between environments.
+  // allowUnorderedMigrations: true,
+})
+
+const { error, results } = await migrator.migrateToLatest()
+
+results?.forEach((it) => {
+  if (it.status === 'Success') {
+    console.log(`migration "${it.migrationName}" executed successfully`)
+  } else if (it.status === 'Error') {
+    console.error(`failed to execute migration "${it.migrationName}"`)
+  }
+})
+
+if (error) {
+  console.error('migration failed', error)
+  process.exit(1)
+}
+```
+
+## Django (Python)
+
+### Workflow
+
+```bash
+# Generate migration from model changes
+python manage.py makemigrations
+
+# Apply migrations
+python manage.py migrate
+
+# Show migration status
+python manage.py showmigrations
+
+# Generate empty migration for custom SQL
+python manage.py makemigrations --empty app_name -n description
+```
+
+### Data Migration
+
+```python
+from django.db import migrations
+
+def backfill_display_names(apps, schema_editor):
+    User = apps.get_model("accounts", "User")
+    batch_size = 5000
+    users = User.objects.filter(display_name="")
+    while users.exists():
+        batch = list(users[:batch_size])
+        for user in batch:
+            user.display_name = user.username
+        User.objects.bulk_update(batch, ["display_name"], batch_size=batch_size)
+
+def reverse_backfill(apps, schema_editor):
+    pass  # Data migration, no reverse needed
+
+class Migration(migrations.Migration):
+    dependencies = [("accounts", "0015_add_display_name")]
+
+    operations = [
+        migrations.RunPython(backfill_display_names, reverse_backfill),
+    ]
+```
+
+### SeparateDatabaseAndState
+
+Remove a column from the Django model without dropping it from the database immediately:
+
+```python
+class Migration(migrations.Migration):
+    operations = [
+        migrations.SeparateDatabaseAndState(
+            state_operations=[
+                migrations.RemoveField(model_name="user", name="legacy_field"),
+            ],
+            database_operations=[],  # Don't touch the DB yet
+        ),
+    ]
+```
+
+## golang-migrate (Go)
+
+### Workflow
+
+```bash
+# Create migration pair
+migrate create -ext sql -dir migrations -seq add_user_avatar
+
+# Apply all pending migrations
+migrate -path migrations -database "$DATABASE_URL" up
+
+# Rollback last migration
+migrate -path migrations -database "$DATABASE_URL" down 1
+
+# Force version (fix dirty state)
+migrate -path migrations -database "$DATABASE_URL" force VERSION
+```
+
+### Migration Files
+
+```sql
+-- migrations/000003_add_user_avatar.up.sql
+ALTER TABLE users ADD COLUMN avatar_url TEXT;
+CREATE INDEX CONCURRENTLY idx_users_avatar ON users (avatar_url) WHERE avatar_url IS NOT NULL;
+
+-- migrations/000003_add_user_avatar.down.sql
+DROP INDEX IF EXISTS idx_users_avatar;
+ALTER TABLE users DROP COLUMN IF EXISTS avatar_url;
+```
+
+## Zero-Downtime Migration Strategy
+
+For critical production changes, follow the expand-contract pattern:
+
+```
+Phase 1: EXPAND
+  - Add new column/table (nullable or with default)
+  - Deploy: app writes to BOTH old and new
+  - Backfill existing data
+
+Phase 2: MIGRATE
+  - Deploy: app reads from NEW, writes to BOTH
+  - Verify data consistency
+
+Phase 3: CONTRACT
+  - Deploy: app only uses NEW
+  - Drop old column/table in separate migration
+```
+
+### Timeline Example
+
+```
+Day 1: Migration adds new_status column (nullable)
+Day 1: Deploy app v2 — writes to both status and new_status
+Day 2: Run backfill migration for existing rows
+Day 3: Deploy app v3 — reads from new_status only
+Day 7: Migration drops old status column
+```
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It Fails | Better Approach |
+|-------------|-------------|-----------------|
+| Manual SQL in production | No audit trail, unrepeatable | Always use migration files |
+| Editing deployed migrations | Causes drift between environments | Create new migration instead |
+| NOT NULL without default | Locks table, rewrites all rows | Add nullable, backfill, then add constraint |
+| Inline index on large table | Blocks writes during build | CREATE INDEX CONCURRENTLY |
+| Schema + data in one migration | Hard to rollback, long transactions | Separate migrations |
+| Dropping column before removing code | Application errors on missing column | Remove code first, drop column next deploy |

package/prisma/skills/deep-research/SKILL.md

@@ -0,0 +1,160 @@
+---
+name: deep-research
+description: Multi-source deep research using firecrawl and exa MCPs. Searches the web, synthesizes findings, and delivers cited reports with source attribution. Use when the user wants thorough research on any topic with evidence and citations.
+metadata:
+  origin: ECC
+---
+
+# Deep Research
+
+> **Drift-prone skill.** Firecrawl/Exa MCP tool names, quotas, and result
+> shapes change. Verify the configured MCP tools and current API docs before
+> promising coverage or quoting live source counts.
+
+Produce thorough, cited research reports from multiple web sources using firecrawl and exa MCP tools.
+
+## When to Activate
+
+- User asks to research any topic in depth
+- Competitive analysis, technology evaluation, or market sizing
+- Due diligence on companies, investors, or technologies
+- Any question requiring synthesis from multiple sources
+- User says "research", "deep dive", "investigate", or "what's the current state of"
+
+## MCP Requirements
+
+At least one of:
+- **firecrawl** — `firecrawl_search`, `firecrawl_scrape`, `firecrawl_crawl`
+- **exa** — `web_search_exa`, `web_search_advanced_exa`, `crawling_exa`
+
+Both together give the best coverage. Configure in `~/.claude.json` or `~/.codex/config.toml`.
+
+## Workflow
+
+### Step 1: Understand the Goal
+
+Ask 1-2 quick clarifying questions:
+- "What's your goal — learning, making a decision, or writing something?"
+- "Any specific angle or depth you want?"
+
+If the user says "just research it" — skip ahead with reasonable defaults.
+
+### Step 2: Plan the Research
+
+Break the topic into 3-5 research sub-questions. Example:
+- Topic: "Impact of AI on healthcare"
+  - What are the main AI applications in healthcare today?
+  - What clinical outcomes have been measured?
+  - What are the regulatory challenges?
+  - What companies are leading this space?
+  - What's the market size and growth trajectory?
+
+### Step 3: Execute Multi-Source Search
+
+For EACH sub-question, search using available MCP tools:
+
+**With firecrawl:**
+```
+firecrawl_search(query: "<sub-question keywords>", limit: 8)
+```
+
+**With exa:**
+```
+web_search_exa(query: "<sub-question keywords>", numResults: 8)
+web_search_advanced_exa(query: "<keywords>", numResults: 5, startPublishedDate: "2025-01-01")
+```
+
+**Search strategy:**
+- Use 2-3 different keyword variations per sub-question
+- Mix general and news-focused queries
+- Aim for 15-30 unique sources total
+- Prioritize: academic, official, reputable news > blogs > forums
+
+### Step 4: Deep-Read Key Sources
+
+For the most promising URLs, fetch full content:
+
+**With firecrawl:**
+```
+firecrawl_scrape(url: "<url>")
+```
+
+**With exa:**
+```
+crawling_exa(url: "<url>", tokensNum: 5000)
+```
+
+Read 3-5 key sources in full for depth. Do not rely only on search snippets.
+
+### Step 5: Synthesize and Write Report
+
+Structure the report:
+
+```markdown
+# [Topic]: Research Report
+*Generated: [date] | Sources: [N] | Confidence: [High/Medium/Low]*
+
+## Executive Summary
+[3-5 sentence overview of key findings]
+
+## 1. [First Major Theme]
+[Findings with inline citations]
+- Key point ([Source Name](url))
+- Supporting data ([Source Name](url))
+
+## 2. [Second Major Theme]
+...
+
+## 3. [Third Major Theme]
+...
+
+## Key Takeaways
+- [Actionable insight 1]
+- [Actionable insight 2]
+- [Actionable insight 3]
+
+## Sources
+1. [Title](url) — [one-line summary]
+2. ...
+
+## Methodology
+Searched [N] queries across web and news. Analyzed [M] sources.
+Sub-questions investigated: [list]
+```
+
+### Step 6: Deliver
+
+- **Short topics**: Post the full report in chat
+- **Long reports**: Post the executive summary + key takeaways, save full report to a file
+
+## Parallel Research with Subagents
+
+For broad topics, use Claude Code's Task tool to parallelize:
+
+```
+Launch 3 research agents in parallel:
+1. Agent 1: Research sub-questions 1-2
+2. Agent 2: Research sub-questions 3-4
+3. Agent 3: Research sub-question 5 + cross-cutting themes
+```
+
+Each agent searches, reads sources, and returns findings. The main session synthesizes into the final report.
+
+## Quality Rules
+
+1. **Every claim needs a source.** No unsourced assertions.
+2. **Cross-reference.** If only one source says it, flag it as unverified.
+3. **Recency matters.** Prefer sources from the last 12 months.
+4. **Acknowledge gaps.** If you couldn't find good info on a sub-question, say so.
+5. **No hallucination.** If you don't know, say "insufficient data found."
+6. **Separate fact from inference.** Label estimates, projections, and opinions clearly.
+
+## Examples
+
+```
+"Research the current state of nuclear fusion energy"
+"Deep dive into Rust vs Go for backend services in 2026"
+"Research the best strategies for bootstrapping a SaaS business"
+"What's happening with the US housing market right now?"
+"Investigate the competitive landscape for AI code editors"
+```