agileflow 4.0.0-alpha.2 → 4.0.0-alpha.21

package/content/plugins/database/skills/agileflow-database/workflows/optimize-query.md

@@ -0,0 +1,253 @@
+# Workflow: Optimize Query
+
+Follow this workflow when the user has a slow query or reports database performance problems. Diagnose before prescribing. Collect evidence, identify the bottleneck, apply the minimum effective fix, and verify the improvement.
+
+---
+
+## Step 1: Capture the Slow Query
+
+Ask the user to provide:
+
+1. **The query** — exact SQL, including WHERE clauses, ORDER BY, LIMIT, JOINs
+2. **EXPLAIN ANALYZE output** — run `EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) <query>;`
+3. **Table row counts** — approximate sizes of the tables involved
+4. **Context** — is this a background job, user-facing API endpoint, report? What latency is acceptable?
+5. **How slow** — current execution time, target execution time
+
+If EXPLAIN ANALYZE output is not available, ask why (read replica, permissions) and adapt.
+
+---
+
+## Step 2: Read EXPLAIN ANALYZE
+
+Parse the output from the bottom up (innermost nodes first). Identify:
+
+**What to look for:**
+
+| Signal                                         | Meaning                                                                |
+| ---------------------------------------------- | ---------------------------------------------------------------------- |
+| `Seq Scan` on a large table                    | Missing index on a filter column                                       |
+| `actual rows` >> `rows` estimate (10x or more) | Stale statistics — `ANALYZE` needed                                    |
+| `Sort` node with high cost                     | Consider index to serve ORDER BY                                       |
+| `Hash Join` with `Batches: N > 1`              | Hash doesn't fit in memory — increase `work_mem` or rewrite            |
+| `Nested Loop` with large outer rows            | N×M iterations — may need different join type or index                 |
+| High `shared read` buffers                     | Data not cached — cold cache or table too large for cache              |
+| `Filter: (...)` rows removed after index       | Index chosen but lots of rows filtered post-scan — better index needed |
+| `Index Scan` with high loops                   | Called repeatedly in a nested loop — may be N+1                        |
+
+**Cost anatomy:**
+
+```
+Seq Scan on orders  (cost=0.00..58432.10 rows=2000000 width=128)
+                     (actual time=0.012..820.341 rows=2000000 loops=1)
+```
+
+- `cost=0.00..58432.10`: startup cost..total cost (planner's estimate)
+- `rows=2000000`: planner's row estimate
+- `actual time=0.012..820.341`: real ms (startup..total)
+- `actual rows=2000000`: real rows returned
+- `loops=1`: how many times this node ran
+
+---
+
+## Step 3: Identify the Bottleneck
+
+Categorize the problem before choosing a fix:
+
+### Missing Index (most common)
+
+**Signs:** `Seq Scan` on a large table with low selectivity filter, or `Filter` removing most rows after a scan.
+
+**Fix:** Add an index on the filter column(s). Use the column order rules from `references/indexing-guide.md`.
+
+```sql
+-- Confirm the index would be used by checking with set enable_seqscan = off:
+SET enable_seqscan = off;
+EXPLAIN ANALYZE <your query>;
+SET enable_seqscan = on;
+```
+
+### Wrong Index (planner chose incorrectly)
+
+**Signs:** An index exists but isn't being used, or a different index is chosen than expected.
+
+**Causes:**
+
+- Stale statistics: planner row estimates are far off actual rows
+- Index isn't selective enough for the data distribution
+- Function applied to indexed column: `WHERE LOWER(email) = $1` won't use index on `email`
+
+**Fix:**
+
+```sql
+-- Refresh stats
+ANALYZE table_name;
+
+-- For function on column, create functional index
+CREATE INDEX idx_users_email_lower ON users (LOWER(email));
+-- Query must match exactly:
+SELECT * FROM users WHERE LOWER(email) = LOWER($1);
+```
+
+### N+1 Query Pattern
+
+**Signs:** A query node shows `loops=N` where N equals the number of parent rows. 1000 index lookups for 1000 parent rows.
+
+**Fix:** Rewrite to use a JOIN or a `WHERE id = ANY($1)` batch fetch.
+
+```sql
+-- Bad pattern (application-level N+1):
+-- SELECT * FROM orders WHERE user_id = $1   -- loop N times
+
+-- Good: one query with JOIN
+SELECT o.*, u.name, u.email
+FROM orders o
+JOIN users u ON u.id = o.user_id
+WHERE o.status = 'pending'
+  AND o.created_at > NOW() - INTERVAL '7 days';
+
+-- Or batch fetch with ANY
+SELECT * FROM users WHERE id = ANY($1::BIGINT[]);
+```
+
+### Slow Sort / ORDER BY
+
+**Signs:** `Sort` node in EXPLAIN with high cost, or sort is the dominant time.
+
+**Fix:** Create an index that serves the ORDER BY:
+
+```sql
+-- Query: WHERE status = 'active' ORDER BY created_at DESC
+CREATE INDEX CONCURRENTLY idx_orders_status_created
+  ON orders (status, created_at DESC);
+```
+
+The planner can use this index and return rows in sorted order without an explicit sort step.
+
+### Inefficient JOIN
+
+**Signs:** `Hash Join` batching to disk, `Nested Loop` with large row counts, `Merge Join` on unsorted input requiring expensive sort.
+
+**Fix options:**
+
+- Ensure FK columns are indexed
+- Add a more selective filter to reduce the row count before the join
+- Rewrite with a CTE to force evaluation order (use sparingly — can prevent optimization)
+- Increase `work_mem` if hash join is batching: `SET work_mem = '256MB';`
+
+### Stale Statistics
+
+**Signs:** `rows=100` estimate but `actual rows=500000`, or vice versa.
+
+**Fix:**
+
+```sql
+ANALYZE table_name;
+
+-- After bulk loads, VACUUM ANALYZE to reclaim dead tuples and update stats
+VACUUM ANALYZE table_name;
+
+-- Check when last analyzed:
+SELECT relname, last_analyze, last_autoanalyze, n_live_tup, n_dead_tup
+FROM pg_stat_user_tables
+WHERE relname = 'orders';
+```
+
+### Large Result Set (Missing LIMIT)
+
+**Signs:** Query returns millions of rows when only a page is needed.
+
+**Fix:** Add LIMIT/OFFSET pagination, or use cursor-based pagination:
+
+```sql
+-- Offset pagination (simple but slow for large offsets)
+SELECT * FROM orders
+WHERE user_id = $1
+ORDER BY created_at DESC
+LIMIT 20 OFFSET $2;
+
+-- Cursor-based pagination (fast even on page 10000)
+SELECT * FROM orders
+WHERE user_id = $1
+  AND created_at < $last_cursor_value  -- cursor = last seen created_at
+ORDER BY created_at DESC
+LIMIT 20;
+```
+
+---
+
+## Step 4: Apply the Fix
+
+Apply the minimum effective change:
+
+1. **Index fix:** write the `CREATE INDEX CONCURRENTLY` statement
+2. **Query rewrite:** rewrite the query, keeping semantics identical
+3. **Stats fix:** `ANALYZE table_name`
+4. **Schema change:** if a column type or missing column is the root cause, follow `workflows/design-schema.md` or `references/migration-guide.md`
+
+Always implement index additions as a separate migration using `CREATE INDEX CONCURRENTLY`.
+
+---
+
+## Step 5: Verify the Improvement
+
+After applying the fix, run EXPLAIN ANALYZE again on the same query with the same parameters.
+
+**What to confirm:**
+
+- [ ] Seq Scan replaced by Index Scan or Index Only Scan
+- [ ] `actual time` is within the target latency
+- [ ] `actual rows` is close to `rows` estimate (planner is accurate now)
+- [ ] No new bottleneck introduced elsewhere in the plan
+- [ ] Query returns the same results as before
+
+Compare before and after:
+
+```
+Before: Seq Scan on orders  actual time=820.341ms  rows=2000000
+After:  Index Scan using idx_orders_user_id  actual time=0.081ms  rows=12
+```
+
+---
+
+## Step 6: Document
+
+For any significant change (new index, query rewrite, schema change):
+
+- Add a comment to the migration explaining why the index was added
+- Note the before/after execution time in the PR or commit message
+- If the root cause was stale statistics, check autovacuum settings:
+
+```sql
+-- Check autovacuum settings for a specific table
+SELECT relname, reloptions FROM pg_class WHERE relname = 'orders';
+
+-- Check if autovacuum is running frequently enough
+SELECT schemaname, relname, last_autovacuum, last_autoanalyze
+FROM pg_stat_user_tables
+ORDER BY last_autovacuum DESC NULLS LAST;
+```
+
+If autovacuum is too infrequent on a busy table, tune it:
+
+```sql
+ALTER TABLE orders SET (
+  autovacuum_vacuum_scale_factor = 0.01,  -- trigger at 1% dead tuples (default 20%)
+  autovacuum_analyze_scale_factor = 0.005 -- trigger analyze at 0.5% changes
+);
+```
+
+---
+
+## Quick Diagnosis Reference
+
+| Symptom                               | First thing to check                                            |
+| ------------------------------------- | --------------------------------------------------------------- |
+| Query suddenly slow after data load   | ANALYZE — statistics are stale                                  |
+| Slow on large table with WHERE clause | Missing index on filter column                                  |
+| Slow ORDER BY                         | Index that serves the ORDER BY clause                           |
+| API endpoint slow under load          | N+1 pattern — count queries in one request                      |
+| Report query times out                | Pagination, partial index, materialized view                    |
+| Deadlock errors                       | Lock ordering inconsistency — acquire locks in same order       |
+| Connection pool exhausted             | Long-running transactions, missing indexes causing slow queries |
+| Replication lag spike                 | Large write operation or bulk update without batching           |

package/content/plugins/debugging/plugin.yaml

@@ -0,0 +1,14 @@
+id: debugging
+name: Debugging
+description: Hypothesis-driven debugging for systematic issue resolution.
+version: 1.0.0
+enabledByDefault: false
+cannotDisable: false
+depends: []
+provides:
+  skills:
+    - id: agileflow-debug
+      dir: skills/agileflow-debug
+  agents: []
+  hooks: []
+  templates: []

package/content/plugins/debugging/skills/agileflow-debug/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: agileflow-debug
+version: 1.0.0
+category: agileflow/debugging
+description: |
+  Use when the user reports a bug, error, unexpected behaviour, or
+  wants to understand why something isn't working. Applies the
+  scientific debugging method: form a hypothesis, reproduce the
+  issue, isolate the cause, verify the fix, and prevent regression.
+triggers:
+  keywords:
+    - debug
+    - not working
+    - error
+    - bug
+    - broken
+    - why is this failing
+    - investigate
+    - something's wrong
+    - unexpected behaviour
+    - exception
+    - crash
+    - failing test
+    - stack trace
+    - cannot figure out
+    - help me understand why
+  priority: 50
+  exclude:
+    - debug mode (configuration context, not debugging a bug)
+    - bug report (filing a ticket, not debugging code)
+provides:
+  agents: []
+learns:
+  enabled: true
+  file: _learnings/debug.yaml
+  maxEntries: 50
+depends:
+  skills: []
+  plugins: [core]
+---
+
+# AgileFlow Debug
+
+Applies the scientific debugging method to any bug or error: reproduce → isolate → hypothesise → verify → fix → prevent regression.
+
+## When this skill activates
+
+- User reports something isn't working and wants help finding the cause
+- User pastes an error message, stack trace, or logs and wants an explanation
+- User describes unexpected behaviour ("this should return X but returns Y")
+- A test is failing and the user isn't sure why
+- User is stuck and wants a systematic approach to a hard bug
+- User wants to know the root cause before applying a fix
+
+## Opening discovery flow
+
+**When invoked without enough context, ask once to gather what's needed.**
+
+```xml
+<invoke name="AskUserQuestion">
+<parameter name="questions">[
+  {
+    "question": "What's happening?",
+    "header": "Bug description",
+    "multiSelect": false,
+    "options": [
+      {"label": "I have an error message or stack trace — I'll paste it", "description": "Share the full error and I'll analyse it, identify the root cause, and suggest a fix"},
+      {"label": "Something behaves wrong — not what I expect (Recommended)", "description": "Describe what you expect vs what actually happens — I'll form hypotheses and help isolate the cause"},
+      {"label": "A test is failing and I don't know why", "description": "Paste the failing test + error output and I'll trace through the failure"},
+      {"label": "The bug is intermittent / flaky", "description": "Intermittent bugs are usually race conditions, timing issues, or shared state — I'll help diagnose"},
+      {"label": "I know what's wrong but need help fixing it safely", "description": "Describe the root cause and I'll help you fix it without introducing new issues"}
+    ]
+  }
+]</parameter>
+</invoke>
+```
+
+## The scientific debugging method
+
+Never guess at a fix. Always follow: **Reproduce → Isolate → Hypothesise → Test the hypothesis → Fix → Verify → Prevent regression.**
+
+```
+Reproduce
+  └─ Can you make it fail consistently?
+     ├─ No → Gather more data (logs, conditions)
+     └─ Yes → Isolate
+
+Isolate
+  └─ What is the smallest failing case?
+     └─ Binary-search the code path to find where good data goes bad
+
+Hypothesise
+  └─ Form a specific, testable hypothesis:
+     "I believe X is causing Y because of Z"
+
+Test
+  └─ Prove or disprove the hypothesis without changing production code
+     ├─ Disproved → form the next hypothesis
+     └─ Proved → Fix
+
+Fix
+  └─ Change the minimum code needed to fix the root cause
+     (not a workaround, not patching the symptom)
+
+Verify
+  └─ Confirm the original issue is gone
+  └─ Confirm nothing else broke (run tests)
+
+Prevent
+  └─ Add a test that would have caught this bug
+  └─ Ask: could this same class of bug exist elsewhere?
+```
+
+## Hypothesis formation
+
+A good hypothesis is:
+
+- **Specific**: not "something is wrong with the database" but "the query returns stale data because the Redis cache is not invalidated when a user updates their profile"
+- **Testable**: you can prove or disprove it with a specific action
+- **Falsifiable**: if X is the cause, then Y should be true; if Y is not true, X is not the cause
+
+**Bad hypothesis:** "The API might be broken"
+**Good hypothesis:** "The `getUser` function returns the wrong user because the session `userId` is not being updated after an email change"
+
+Form 1–3 hypotheses in priority order. Test the most likely one first.
+
+## Binary search debugging
+
+When a bug is somewhere in a large code path, use binary search to find it:
+
+1. Identify the start of the bad path (where input is definitely correct)
+2. Identify the end (where output is definitely wrong)
+3. Add a probe (log, assertion, breakpoint) at the midpoint
+4. Is the data correct at the midpoint?
+   - Yes → bug is in the second half
+   - No → bug is in the first half
+5. Repeat until the exact line is found
+
+This is almost always faster than reading the code top-to-bottom trying to spot the bug.
+
+## Error message analysis
+
+When the user provides an error message or stack trace:
+
+1. **Read the error type** — TypeError, NullPointerException, IndexError → tells you the category
+2. **Find the first line that is in the application code** — ignore framework/library frames above it
+3. **Read the message literally** — "Cannot read properties of undefined (reading 'email')" means a variable is undefined; ask what it should have been at that point
+4. **Trace backward** — where was that variable set? What could make it undefined?
+
+Common error → root cause translations:
+
+| Error                                 | Likely root cause                                                 |
+| ------------------------------------- | ----------------------------------------------------------------- |
+| `Cannot read properties of undefined` | Object that should have been fetched from DB is null/undefined    |
+| `ECONNREFUSED`                        | Service not running, wrong port, or firewall                      |
+| `UNIQUE constraint failed`            | Duplicate insert — race condition or missing check                |
+| `JWT expired`                         | Token expiry handling missing or clock skew                       |
+| `TypeError: X is not a function`      | Wrong import, wrong export type, or circular dependency           |
+| `Maximum call stack size exceeded`    | Infinite recursion — function calls itself directly or indirectly |
+| `Promise rejected with non-Error`     | `.catch(err => ...)` where `err` is a string, not an Error object |
+
+## Intermittent bug strategies
+
+Flaky bugs are often:
+
+| Category                      | Diagnosis approach                                                               |
+| ----------------------------- | -------------------------------------------------------------------------------- |
+| **Race condition**            | Add logging around concurrent operations; use `--runInBand` to serialise tests   |
+| **Timer / clock dependency**  | Check if code depends on `Date.now()` or `setTimeout` — use fake timers in tests |
+| **Shared mutable state**      | Check for module-level singletons, global variables, or test state leaking       |
+| **External service**          | Add retry logging; confirm the external service is stable                        |
+| **Order dependency in tests** | Run tests in reverse or random order to find the dependency                      |
+
+## Logging strategy
+
+When the root cause isn't obvious from reading the code:
+
+1. **Log at the entry point** — log the inputs to the function where you think the bug is
+2. **Log at the decision point** — log the value of the condition in every important `if` statement
+3. **Log at the exit point** — log what the function returns
+4. **Compare** — where does the actual data diverge from what you expected?
+
+**Good log message:**
+
+```
+[getUser] id=42, cache_hit=false, db_result=null (user does not exist)
+```
+
+**Bad log message:**
+
+```
+error occurred
+```
+
+## Fix strategy
+
+After identifying the root cause:
+
+1. **Fix the root cause** — not a workaround (removing the symptom without fixing the cause creates a new bug later)
+2. **Fix the minimum code** — a large fix for a small bug often introduces new bugs
+3. **Preserve the contract** — if fixing a function, don't change its inputs/outputs unless necessary
+4. **Update tests** — if a test failed, update it to reflect correct behaviour; if no test caught the bug, add one
+
+## Self-improving learnings
+
+`_learnings/debug.yaml` records:
+
+- Recurring bug patterns in this codebase
+- Common root causes for this project's tech stack
+- False leads to avoid (previous hypotheses that turned out to be wrong)
+- Areas of the codebase that are frequently buggy
+
+## Integration
+
+- **agileflow-research** — when stuck after 2 attempts with different fixes, escalate to research before trying a third approach; craft a 200+ line prompt with full context
+- **agileflow-test-writer** — after finding and fixing the root cause, spawn test-writer to add a regression test that prevents the bug from returning
+- **agileflow-refactor** — if the bug was caused by structural problems (deep coupling, unclear state), suggest a refactor pass after the fix lands
+- **agileflow-pr-reviewer** — invoke before merging a bug fix, especially for security-related bugs or fixes touching multiple modules
+- **agileflow-engineering** — delegate the implementation of the fix when it's non-trivial or spans multiple domains
+- **agileflow-audit** — if a bug suggests a systemic problem (repeated null checks, race conditions in multiple files), run an audit to find related issues
+- **agileflow-performance** — hand off if the bug is a performance regression: slow queries, memory leaks, or blocking operations
+- **agileflow-adr** — if fixing the bug requires a design change, document the new approach as an ADR before implementing it
+
+## References
+
+| File                                 | When to load                                                                         |
+| ------------------------------------ | ------------------------------------------------------------------------------------ |
+| `references/debugging-strategies.md` | Choosing the right debugging strategy for the error type and environment             |
+| `references/common-patterns.md`      | Common bug patterns by category — race conditions, null handling, async errors, etc. |
+
+## Workflows
+
+| File                         | When to follow                                                                 |
+| ---------------------------- | ------------------------------------------------------------------------------ |
+| `workflows/debug-issue.md`   | Systematic debugging flow from "something's wrong" to root cause and fix       |
+| `workflows/reproduce-bug.md` | Creating a minimal reproduction case for a hard-to-isolate or intermittent bug |