oh-my-agent-commandcode 0.0.0-stage → 20.38.3

package/skills/oma-db/resources/execution-protocol.md

@@ -0,0 +1,127 @@
+# DB Agent - Execution Protocol
+
+## Preparation
+Use the task's scope, existing project conventions, and acceptance criteria. Follow `../../_shared/core/execution-policy.md` when it has not already been supplied. Read only references needed by the selected operation; consult lessons or recovery guides for an observed issue. Expand planning depth only when the change requires it.
+
+## Step 1: Explore
+- Identify actors and external views that need data
+- Capture entities, aggregates, reference data, events, and lifecycle states
+- List access patterns:
+  - point lookup
+  - range scan
+  - aggregation/reporting
+  - full-text/search
+  - semantic retrieval / RAG
+  - batch ingestion
+- Decide relational vs non-relational:
+  - Prefer relational when joins, strong integrity, and transactional consistency dominate
+  - Prefer NoSQL when aggregate reads/writes, massive scale, flexible schema, or distribution tolerance dominate
+  - Prefer a graph model only when variable-depth traversals or relationship-pattern queries dominate the workload; adjacency (`parent_id`, junction tables) in a relational engine handles shallow, fixed-depth traversals fine
+  - Prefer vector retrieval only for semantic similarity workloads; do not replace keyword/search engines blindly
+- Screen for anti-pattern signals early:
+  - comma-separated values in one column
+  - tag1/tag2/tag3 style multi-column attributes
+  - generic EAV tables
+  - polymorphic foreign keys
+  - missing FK constraints
+  - blind surrogate-key usage
+  - business policies, implicit workflows, or external side effects in affected triggers, stored procedures/functions, views, DB jobs, or ORM hooks
+- For the affected logic, identify callers and all writers (API, batch, admin, import, direct SQL); use `resources/anti-patterns.md` §5 to distinguish application responsibilities from DB enforcement
+- Record assumptions and open risks
+- If vector retrieval is needed, define:
+  - hybrid vs pure vector retrieval
+  - exact-filter requirements
+  - embedding model and dimension
+  - chunking strategy
+  - reranking need
+
+## Step 2: Design
+- Produce **external schema**:
+  - user-facing views, APIs, reports, or producer/consumer contracts
+- Produce **conceptual schema**:
+  - entities/aggregates, relationships, cardinality, optionality, ownership
+- Produce **internal schema**:
+  - tables/collections, keys, indexes, partitions, tablespaces, storage layout
+- SQL path:
+  - normalize to **3NF** minimum
+  - define PK, FK, unique, check, default, nullability
+  - document transaction boundaries and target isolation level
+  - apply the application ownership policy in `resources/anti-patterns.md` §5 to both new and existing business logic; recommend alternatives and transition priorities, and document technical evidence for any retention
+- NoSQL path:
+  - model around access patterns and aggregate boundaries
+  - define sharding/partition keys, secondary indexes, TTL, duplication policy
+  - document consistency model, conflict strategy, and repair/reconciliation approach
+- Graph path:
+  - model nodes/edges from the traversal questions the business actually asks, not from the ERD
+  - define edge direction, cardinality, and properties; decide which lookups still need secondary indexes on node properties
+  - document traversal depth expectations and whether the engine's strength (index-free adjacency) is actually exercised
+  - keep source-of-truth entities in the primary store when the graph is a derived projection; document the sync path
+- Vector path:
+  - define embedding model, dimension, normalization, and metadata versioning
+  - define chunking policy, overlap, and document-type-specific chunk rules
+  - design hybrid retrieval: lexical + ANN + metadata filtering + reranking
+  - define canonical document store vs vector index boundary
+  - document ANN index choice and key tuning parameters
+- Apply data standards:
+  - naming, definition, format, code set, validation rule
+- Review security/compliance posture:
+  - least privilege support
+  - audit trail needs
+  - encryption at rest / in transit expectations
+  - backup/restore traceability
+  - retention and deletion obligations
+  - control guidance alignment for ISO 27001 / 27002
+  - continuity and recovery alignment for ISO 22301
+
+## Step 3: Optimize
+- Validate hot paths against indexes and partitions; tune with `resources/query-tuning.md` (measure -> explain -> nominate -> test, never guesswork)
+- If optimization requires schema changes on live tables, plan them via `resources/migration-playbook.md` (expand-contract, lock-aware DDL, batched backfill)
+- If moving existing DB business logic into the application, use the same playbook's ownership cutover steps to preserve writer coverage and avoid duplicate side effects
+- For vector systems:
+  - benchmark recall vs latency on production-like queries
+  - measure ANN parameter tradeoffs, filter pushdown cost, and write amplification
+  - design re-embedding and shadow-index migration flow
+  - separate ingestion/search clusters or hot/cold vector tiers when write load is high
+- Separate hot and cold data:
+  - recent operational data stays online
+  - historical/archive data moves to cheaper storage or archive partitions
+- Define backup and recovery:
+  - full backup cadence
+  - incremental backup cadence
+  - retention window
+  - restore validation frequency
+- Review anti-patterns in five buckets:
+  - logical design
+  - physical design
+  - query design
+  - application integration
+  - vector retrieval / RAG
+- Update capacity estimation:
+  - object-by-object volume
+  - tablespace/storage allocation
+  - disk estimate
+  - online workload
+  - batch workload
+  - backup volume
+
+## Step 4: Verify
+- Run `resources/checklist.md`
+- For a schema-design task, confirm affected existing artifacts were updated;
+  for scoped tuning, record SQL/plan evidence instead of creating this full set:
+  - glossary
+  - standards table
+  - capacity estimate
+  - backup/recovery notes
+- For vector systems, confirm updated:
+  - embedding version metadata
+  - chunking policy
+  - retrieval evaluation set
+  - re-index / re-embedding migration notes
+- Confirm anti-pattern findings were either fixed or explicitly accepted with rationale
+- For affected DB business logic, confirm a strong improvement recommendation and concrete application alternative were given even when implementation is deferred; retention needs technical evidence, and planned transitions need writer coverage, preserved DB guarantees, effect ownership, and rollback
+- Call out ISO 27001 / 27002-relevant risks and recommended controls when applicable
+- Call out ISO 22301-relevant continuity and recovery gaps when applicable
+- Call out any intentional denormalization or weaker consistency with reason
+
+## On Error
+See `resources/error-playbook.md` for recovery steps.

package/skills/oma-db/resources/iso-controls.md

@@ -0,0 +1,98 @@
+# DB Agent - ISO 27001 / 27002 / 22301 Guide
+
+Use this file when the task involves database controls, audit readiness, resilience, or continuity requirements.
+
+## Positioning
+
+- **ISO 27001**: management-system and control objectives view
+- **ISO 27002**: practical control guidance view
+- **ISO 22301**: business continuity and recovery view
+
+Do not claim compliance from schema work alone. Use these standards to improve database design decisions, control coverage, and operational evidence.
+
+## 1. ISO 27001 / 27002 for Databases
+
+Recommend improvements when the database design weakens:
+
+- access control and least privilege
+- separation of duties for admin, operator, developer, and read-only access
+- encryption at rest and in transit
+- key and secret management
+- audit logging and tamper-resistant log retention
+- change management and schema migration traceability
+- backup security and retention handling
+- secure configuration and baseline hardening
+- data classification, masking, and PII handling
+
+Typical DB-level suggestions:
+
+- role-based access with least privilege
+- dedicated break-glass admin path
+- column or application-layer protection for sensitive fields where needed
+- immutable or centralized audit log export
+- secret rotation and non-human service identities
+- explicit approval path for destructive schema changes
+
+## 2. ISO 22301 for Databases
+
+Recommend improvements when the design weakens business continuity:
+
+- RTO / RPO definition
+- failover readiness
+- backup coverage and retention
+- restore validation frequency
+- dependency mapping for upstream/downstream systems
+- data replication strategy
+- hot/warm/cold standby decisions
+- archival and cold-data recovery assumptions
+
+Typical DB-level suggestions:
+
+- define tiered continuity expectations by database criticality
+- document backup cadence aligned to RPO
+- document restore and failover process aligned to RTO
+- test restore drills regularly, not only backup completion
+- make derived indexes rebuildable from canonical data
+- separate source-of-truth data from caches and secondary indexes
+
+## 3. Review Questions
+
+Ask these when standards matter:
+
+- Who can read, write, alter schema, and administer the database?
+- Are privileged actions logged and retained?
+- Are secrets and keys managed outside code and migration files?
+- What evidence exists that backups can actually be restored?
+- What is the target RPO and RTO?
+- Which systems depend on this database to continue service?
+- Can cold/archive data be recovered within required time?
+- Are vector indexes or search indexes rebuildable from canonical data?
+
+## 4. Output Pattern
+
+When relevant, add a short section like:
+
+```md
+## ISO Control Notes
+
+### ISO 27001 / 27002
+- Access control gap:
+- Logging / traceability gap:
+- Encryption / secret gap:
+- Recommended control:
+
+### ISO 22301
+- Criticality:
+- RPO / RTO gap:
+- Backup / restore gap:
+- Failover / dependency gap:
+- Recommended continuity improvement:
+```
+
+## 5. Guardrails
+
+- Keep standards comments practical and database-specific
+- Prefer implementable controls over abstract policy language
+- Distinguish between source-of-truth data and rebuildable derived data
+- If the issue is infrastructure-only, involve `tf-infra-engineer` (`oma-tf-infra`)
+- If the issue is org-wide audit/process ownership, involve QA or PM

package/skills/oma-db/resources/migration-playbook.md

@@ -0,0 +1,93 @@
+# DB Agent - Live Schema & Data Migration Playbook
+
+Use this file when a schema or data change must be applied to a live database: altering hot tables, backfilling data, changing types, splitting/merging tables, or cutting over to a new structure.
+
+Scope boundary: `oma-refactor` plans the code-side expand-contract sequence and routes the **database mechanics** here. This file owns lock behavior, backfill execution, cutover verification, and rollback of the data layer.
+
+## Core Position
+
+- Deployment, not commit, is the unit of incrementality for stateful change.
+- `git revert` does not restore data. Every phase must be independently deployable and reversible until the contract step.
+- Never ship a destructive schema change in the same deploy as the code that requires it.
+- A migration that cannot state its lock impact on the largest affected table is not ready to run.
+
+## Expand-Contract for Schemas
+
+Run live changes in four separately deployed phases:
+
+1. **Expand** - additive only
+   - Add new columns as nullable (or with a non-volatile default), new tables, new indexes
+   - Old code keeps working untouched; new structures are invisible to it
+2. **Migrate** - dual-write + backfill
+   - New code writes to both old and new structures
+   - Batched backfill copies historical data (see below)
+   - Verify parity: row counts, checksums, or sampled field comparison between old and new
+3. **Switch** - move reads
+   - Flip reads to the new structure behind a feature flag
+   - Keep dual-write and the old read path available for instant rollback
+   - Soak: hold the flag on for an agreed window while monitoring errors and drift
+4. **Contract** - remove old
+   - Only after the soak window passes and no consumer reads the old structure
+   - Drop old columns/tables in a later, separate deploy; take a final backup first
+
+Rollback rule per phase: Expand -> drop the additions; Migrate -> stop dual-write, keep old as truth; Switch -> flip the flag back; Contract -> restore from backup only (this is why contract waits).
+
+## Moving Business Logic into the Application
+
+Use these steps when replacing existing trigger/procedure/function/job business behavior. The ownership recommendation and retention criteria live in `resources/anti-patterns.md` §5; this section owns the cutover mechanics.
+
+1. **Inventory the affected execution paths**: identify API, batch, admin, import, and direct SQL writers, chained DB routines, privileges, transaction boundaries, and externally visible effects. Resolve writers that would bypass the new policy before cutover.
+2. **Prepare the application path**: implement explicit commands, domain policies, or jobs with equivalent invariants, policy/decision versioning where required, idempotency, and bounded retry. Keep required DB constraints, RLS, auditing, atomic updates, and isolation. Write outbox events in the business transaction when external delivery is needed.
+3. **Verify without duplicating effects**: compare policy decisions on representative inputs and test concurrency, direct writers, rollback, and failure recovery. Shadow evaluation must not execute payments, notifications, or other external effects.
+4. **Switch execution ownership**: coordinate application rollout with DB behavior changes so each effect has one owner. Account for old app instances, in-flight transactions, queued jobs/events, and deduplication keys. Do not let the trigger and application both execute the same effect. Dual-writing old/new storage representations within one transaction may still be needed; it is distinct from executing effects twice.
+5. **Soak and recover**: monitor errors, decision parity, processing lag, and missing/duplicate effects. Define how to restore the old execution owner and reconcile already committed or published work; disabling a feature flag alone cannot undo external effects.
+6. **Contract later**: remove obsolete DB routines only after callers/writers and queued work have migrated and the soak window passes, in a separate deployment with versioned definitions and the required backup/recovery evidence.
+
+## Lock-Aware DDL
+
+Know what each statement locks on your engine and version before running it against a hot table.
+
+- Always set a lock timeout so DDL waits fail fast instead of queueing behind long transactions and blocking all traffic behind them (e.g. PostgreSQL `SET lock_timeout = '2s'` + retry loop).
+- PostgreSQL highlights:
+  - `CREATE INDEX CONCURRENTLY` instead of plain `CREATE INDEX` on live tables (not inside a transaction)
+  - Add `NOT NULL` via `CHECK (... IS NOT NULL) NOT VALID` then `VALIDATE CONSTRAINT` (validation takes a weaker lock)
+  - Column type changes that rewrite the table (`ALTER TYPE`) need a dual-column expand-contract instead
+  - Adding a column with a constant default is metadata-only on modern versions; volatile defaults rewrite
+- MySQL highlights:
+  - Check whether the change supports `ALGORITHM=INSTANT` or `INPLACE`; anything forcing `COPY` on a hot table should go through an online schema change tool (gh-ost, pt-online-schema-change)
+- Any engine:
+  - Run risky DDL at low-traffic windows even when "online"
+  - One DDL concern per migration file; do not mix heavy DDL and DML in one transaction
+
+## Batched Backfill
+
+- Batch by primary-key range with a bounded batch size; never one `UPDATE ... WHERE new_col IS NULL` over the whole table
+- Make each batch idempotent and record a checkpoint (last key processed) so the job is resumable after failure
+- Throttle between batches and watch replication lag; pause when lag exceeds the agreed threshold
+- Do not wrap the whole backfill in one transaction: long transactions block vacuum/purge and hold locks
+- Backfill runs as application-level code or a migration job, not inside the DDL migration file
+
+## Cutover Verification
+
+Before the switch phase:
+
+- Parity check old vs new (counts, checksums, or sampled diffs) with results recorded
+- Read-path comparison in shadow mode where feasible (serve old, compare new, log mismatches)
+- Confirm every consumer of the old structure is identified: application paths, reports, ETL, triggers, views
+
+## Migration Hygiene
+
+- Prefer forward-only migrations; if down migrations exist, they must be tested, not decorative
+- Test the migration against production-scale data volume, not an empty dev schema
+- Record expected duration and lock impact in the migration description
+- Schema change management and traceability requirements from `resources/iso-controls.md` apply: approvals for destructive changes, audit trail of who ran what when
+
+## Failure Modes Quick Table
+
+| Failure | Cause | Prevention |
+| --- | --- | --- |
+| Site-wide stall during DDL | DDL queued behind a long transaction, everything queues behind the DDL | lock timeout + retry, kill long transactions first |
+| Backfill causes outage | unbatched update, lock escalation, replication lag | bounded batches, throttle, lag monitoring |
+| Cannot roll back after cutover | old structure dropped too early | contract only after soak window + final backup |
+| Data drift between old and new | dual-write bug or missed writer | parity checks before switch, single write path per field owner |
+| Migration dies halfway | non-idempotent steps, no checkpoint | idempotent batches, resumable checkpointing, transactional DDL where the engine supports it |

package/skills/oma-db/resources/query-tuning.md

@@ -0,0 +1,57 @@
+# DB Agent - Query Tuning & Index Design Guide
+
+Use this file when the task involves slow queries, execution plans, index selection, or query rewrites. Anti-pattern smells (`SELECT *`, `ORDER BY RAND()`, poor-man's search) live in `resources/anti-patterns.md`; this file owns the tuning method.
+
+## Triage Order
+
+Never tune from guesswork. Follow MENTOR (Measure, Explain, Nominate, Test, Optimize, Rebuild) concretely:
+
+1. **Measure** - find the actual offender
+   - PostgreSQL: `pg_stat_statements` by total/mean time; MySQL: slow query log / `performance_schema`
+   - Rank by total time, not just per-call time: a 10ms query at 1k QPS beats a 2s report query
+2. **Explain** - read the real plan
+   - PostgreSQL: `EXPLAIN (ANALYZE, BUFFERS)`; MySQL: `EXPLAIN ANALYZE`
+   - Run with production-representative parameters; plans change with parameter selectivity
+3. **Nominate** - identify the dominant cost node before changing anything
+4. **Test** - reproduce the slowness on production-like data volume
+5. **Optimize** - index, rewrite, or schema change (in that order of preference)
+6. **Rebuild** - re-measure p95/p99 after the change, not just the single-run time
+
+## Reading a Plan
+
+Look for these, roughly in order:
+
+- **Estimate vs actual row mismatch** (e.g. estimated 100, actual 2M): stale statistics or correlated columns -> `ANALYZE` the table, consider extended statistics; a fix based on a wrong estimate fixes nothing
+- **The most expensive node**: tune the node that dominates total time, not the first scan you see
+- **Seq Scan on a large table** with a selective filter: missing or unusable index
+- **Index Scan with a large `Filter:` line**: the index matches only part of the predicate; the rest is filtered row-by-row -> extend the index (`Index Cond` good, `Filter` suspicious)
+- **Nested Loop over many outer rows** with an inner Seq Scan: usually a missing index on the join key
+- **Sort spilling to disk** (`external merge`): missing index for `ORDER BY`, or work_mem-class tuning question
+- **Repeated identical subplan execution**: correlated subquery to rewrite as join or lateral
+
+## Index Design Heuristics
+
+- **Composite column order**: equality predicates first, then the range/sort column. `WHERE tenant_id = ? AND created_at > ?` wants `(tenant_id, created_at)`, not the reverse
+- **Leftmost prefix rule**: a composite index serves queries filtering on its leading columns only
+- **Selectivity**: index columns that narrow rows sharply; a standalone index on a low-cardinality column (status, boolean) rarely helps — combine it into a composite or use a partial index
+- **Covering / index-only**: include the selected columns (`INCLUDE` / suffix columns) so hot queries never touch the heap
+- **Partial indexes** for skewed predicates: `WHERE status = 'PENDING'` when 99% of rows are done
+- **Functional indexes** when queries filter on an expression (`lower(email)`) — or make the predicate sargable instead
+- **Index every FK** used in joins or cascading deletes; missing FK indexes cause seq-scan deletes and lock storms
+- Every index taxes writes and storage: after adding, check for now-redundant indexes (see index shotgun in `resources/anti-patterns.md`) and drop unused ones using index usage stats
+
+## Rewrite Patterns
+
+- **Sargability**: never wrap the indexed column in a function or cast in the predicate; move the transformation to the constant side (`created_at >= date '2026-01-01'`, not `date(created_at) = ...`)
+- **Keyset pagination** over `OFFSET`: `WHERE (created_at, id) < (?, ?) ORDER BY created_at DESC, id DESC LIMIT n`; OFFSET scans and discards everything it skips
+- **`OR` across different columns** often blocks index use: use a semantics-preserving `UNION` over row identity, or `UNION ALL` only when branches are provably disjoint (including NULL behavior) of index-friendly branches
+- **`EXISTS` vs `IN`**: prefer `EXISTS` for correlated membership checks against large subquery results
+- **Batch `IN` lists** instead of N+1 single-row queries from the application (see `oma-observability` DB span conventions for detecting N+1)
+- **Pre-aggregate** heavy repeated reporting queries into materialized/summary tables instead of tuning an inherently large scan
+
+## Verification
+
+- Compare before/after with `EXPLAIN (ANALYZE, BUFFERS)` on the same data and parameters; record both plans
+- Watch p95/p99 under concurrency, not a single warm-cache run
+- Check plan stability across parameter values (skewed tenants, empty vs huge accounts)
+- Confirm write-path latency did not regress after adding indexes

package/skills/oma-db/resources/vector-db.md

@@ -0,0 +1,159 @@
+# DB Agent - Vector DB & RAG Design Guide
+
+Use this file when the task involves vector databases, semantic retrieval, ANN tuning, or RAG.
+
+## Core Position
+
+- Vector DB is not a drop-in replacement for search engines or relational databases.
+- Vector DB is usually an **index layer** for semantic retrieval.
+- Canonical documents should live in an object store or primary database.
+- Production retrieval systems are usually **hybrid**:
+  - lexical retrieval for exactness and explainability
+  - vector retrieval for semantic proximity
+  - metadata filtering for governance and scope
+  - reranking for precision
+
+## Design Sequence
+
+1. Define the retrieval problem
+   - semantic discovery
+   - exact term retrieval
+   - compliance / tenant filtering
+   - recommendation
+   - agent context retrieval
+2. Decide whether vector retrieval is justified
+3. Design chunking and embedding strategy
+4. Choose ANN index and benchmark it
+5. Design filtering, reranking, and observability
+6. Plan lifecycle:
+   - re-embedding
+   - re-indexing
+   - hot/cold vector tiers
+   - backup / restore
+
+## Best Practices
+
+### 1. Use Hybrid Retrieval by Default
+
+- Keep lexical retrieval when:
+  - exact term match matters
+  - user expects deterministic ranking
+  - compliance or audit filters must be explicit
+  - explainability matters
+- Good default pipeline:
+  - filter scope
+  - lexical retrieval
+  - ANN retrieval
+  - merge / dedupe
+  - rerank
+  - context compression
+
+### 2. Govern Embeddings Like Schema
+
+Store in metadata:
+
+- embedding model name
+- dimension
+- normalization method
+- chunking policy version
+- preprocessing version
+- creation timestamp
+
+Plan for:
+
+- rolling re-embedding
+- shadow indexes
+- side-by-side evaluation
+- backfill and cutover
+
+### 3. Chunking is a First-Class Design Decision
+
+Choose chunking based on document type:
+
+- contracts / policies: heading-aware chunks with overlap
+- tickets / chats: conversation-turn boundaries
+- code / config: syntax-aware or file-structure-aware chunking
+- FAQs / short docs: smaller direct-answer chunks
+
+Watch for:
+
+- over-chunking -> weak context and low recall
+- under-chunking -> noisy retrieval and token waste
+
+### 4. Tune ANN with Real Data
+
+Common ANN choices:
+
+- HNSW: strong recall, high memory
+- IVF: faster partitioned retrieval, recall tradeoffs
+- PQ: smaller memory footprint, lower precision
+- DiskANN: disk-oriented large-scale retrieval
+
+Benchmark on production-like distributions, not synthetic random vectors.
+
+Measure:
+
+- recall@k
+- latency p50 / p95 / p99
+- memory footprint
+- index build time
+- update latency
+
+### 5. Design Filtering Deliberately
+
+Metadata filtering gets expensive with high-cardinality dimensions.
+
+Check:
+
+- whether filters are pushed down before ANN or after
+- whether approximate filtering affects correctness
+- whether tenant / user / region isolation needs separate partitions
+
+Patterns:
+
+- partition by tenant or region
+- pre-segment indexes for extreme-cardinality filters
+- maintain dedicated collections for very large tenants
+
+### 6. Plan Writes and Lifecycle
+
+High-write vector workloads need:
+
+- ingestion buffers
+- async indexing
+- compaction
+- periodic rebuild strategy
+- separation of ingestion and search responsibilities when load is high
+
+Consider:
+
+- hot vectors in fast mutable index
+- cold vectors in slower compacted tier
+
+### 7. Observe Semantics, Not Just Infra
+
+Track more than QPS and latency:
+
+- recall drift
+- semantic drift after model update
+- similarity score distribution
+- false positives
+- empty-result rate
+- prompt-token waste from irrelevant retrieval
+
+Maintain:
+
+- golden query sets
+- offline evaluation pipelines
+- before/after comparison when changing model or chunking
+
+## Review Questions
+
+- Why is vector retrieval needed here?
+- Why is pure lexical retrieval insufficient?
+- Why is pure vector retrieval insufficient?
+- What is the source of truth?
+- What invalidates the embeddings?
+- How will re-embedding be rolled out safely?
+- How are filters enforced?
+- How is retrieval quality measured over time?