@cubis/foundry 0.3.10 → 0.3.12

package/Ai Agent Workflow/skills/database-skills/skills/vitess/SKILL.md

@@ -1,15 +1,35 @@
 ---
 name: vitess
-description: Vitess and sharded MySQL planning, VSchema, routing, and online schema migration concerns.
+description: Vitess sharding strategy, VSchema/vindex design, routing, and scale operations.
 ---
 
 # Vitess
 
-Load references as needed:
+## Primary focus
+
+- Choose a strong primary vindex.
+- Keep hot queries shard-local.
+- Minimize cross-shard joins and transactions.
+
+## Optimization workflow
+
+1. Identify dominant access patterns and routing keys.
+2. Design VSchema/vindex for locality and balanced distribution.
+3. Validate routing/fan-out behavior on critical queries.
+4. Use seek pagination anchored on routing key and tie-breaker.
+5. Stage topology/resharding changes with rollback checkpoints.
+
+## Pagination techniques
+
+- Prefer shard-aware keyset pagination.
+- Avoid broad fan-out + offset combinations.
+
+## Operational guardrails
+
+- Treat resharding as staged production operation.
+- Monitor fan-out, replica lag, and failover behavior during topology changes.
+
+## References
+
 - `references/sharding-routing.md`
 - `references/operational-safety.md`
-
-Key rules:
-- Model entity ownership before keyspace/shard strategy.
-- Avoid cross-shard patterns unless explicitly required.
-- Plan online schema changes with observability checkpoints.

package/Ai Agent Workflow/skills/database-skills/skills/vitess/references/operational-safety.md

@@ -1,5 +1,105 @@
-# Vitess Operational Safety
+# Vitess — Online DDL and Operational Safety
 
-- Stage schema changes and monitor replication lag.
-- Include fallback plan for routing or migration regressions.
-- Verify SLA metrics after each rollout phase.
+## Online DDL strategies
+
+Vitess manages schema changes as tracked, non-blocking, revertible migrations. This is the **only recommended approach** for production schema changes.
+
+Set DDL strategy per session or globally:
+```sql
+SET @@ddl_strategy = 'vitess';
+```
+
+| Strategy | Description |
+| --- | --- |
+| `vitess` (**recommended**) | VReplication-based. Non-blocking, revertible, failover-safe. |
+| `online` | Alias for `vitess`. |
+| `mysql` | Native MySQL DDL managed by Vitess scheduler. Blocking depends on the DDL. |
+| `direct` | Unmanaged — direct DDL to MySQL. Not trackable. Avoid in production. |
+
+## Running a migration
+
+```sql
+SET @@ddl_strategy = 'vitess';
+ALTER TABLE orders ADD COLUMN notes TEXT;   -- returns migration UUID immediately
+```
+
+```bash
+vtctldclient ApplySchema --ddl-strategy "vitess" \
+  --sql "ALTER TABLE orders ADD COLUMN notes TEXT" commerce
+```
+
+## Migration lifecycle
+
+```
+queued → ready → running → complete
+                         ↘ failed
+         ↘ cancelled
+```
+
+## Monitor and control migrations
+
+```sql
+SHOW VITESS_MIGRATIONS;                                         -- all migrations
+SHOW VITESS_MIGRATIONS LIKE '<uuid>';                           -- specific migration
+```
+
+Key columns to watch: `migration_status`, `progress`, `started_timestamp`, `completed_timestamp`, `message`.
+
+```sql
+ALTER VITESS_MIGRATION '<uuid>' CANCEL;    -- cancel pending
+ALTER VITESS_MIGRATION '<uuid>' RETRY;     -- retry failed
+ALTER VITESS_MIGRATION '<uuid>' COMPLETE;  -- complete a postponed migration
+ALTER VITESS_MIGRATION '<uuid>' LAUNCH;    -- launch a postponed migration
+REVERT VITESS_MIGRATION '<uuid>';          -- revert a completed migration (non-destructive)
+```
+
+## Key DDL strategy flags
+
+Append flags to strategy string:
+```sql
+SET @@ddl_strategy = 'vitess --postpone-completion --allow-concurrent';
+```
+
+| Flag | Effect |
+| --- | --- |
+| `--postpone-launch` | Queue migration but don't start automatically |
+| `--postpone-completion` | Run migration but don't cut over — you control timing |
+| `--allow-concurrent` | Allow multiple migrations to run simultaneously |
+| `--declarative` | Provide desired `CREATE TABLE`; Vitess computes the ALTER |
+| `--prefer-instant-ddl` | Use MySQL INSTANT DDL when possible |
+| `--singleton` | Only one migration on this table at a time |
+
+## Declarative migrations
+
+Supply the desired schema; Vitess computes the diff and runs the minimal ALTER:
+```sql
+SET @@ddl_strategy = 'vitess --declarative';
+CREATE TABLE demo (id BIGINT UNSIGNED NOT NULL, status VARCHAR(32), PRIMARY KEY (id));
+```
+
+## Throttling and failover safety
+
+- The **tablet throttler** automatically slows migrations when replication lag is high.
+  Enable: `vtctldclient UpdateThrottlerConfig --enable <keyspace>`
+- VReplication-based migrations **auto-resume** after primary reparenting (new primary must come up within 10 min).
+
+## Operational guardrails
+
+- **Stage topology and resharding changes** in maintenance windows; keep blast radius small per operation.
+- **Watch fan-out and replication lag** as release gates — do not proceed if lag is elevated.
+- **Prepare explicit rollback procedures per shard move**: `MoveTables SwitchTraffic --reverse`, then `MoveTables Complete` on the old keyspace.
+- Validate with `VDiff` before completing table moves: `vtctldclient VDiff <keyspace> <workflow>`.
+- Test resharding end-to-end on staging with production-like data volume before production.
+
+## Best practices
+
+1. Always use `vitess` strategy in production — never `direct`.
+2. Use `--postpone-completion` for critical tables to control cut-over timing precisely.
+3. Enable the tablet throttler to prevent replication lag buildup.
+4. Use declarative migrations for desired-state schema management.
+5. Monitor all running migrations with `SHOW VITESS_MIGRATIONS` before and after deployments.
+
+## Sources
+- Online DDL guide: https://vitess.io/docs/user-guides/schema-changes/
+- VReplication reference: https://vitess.io/docs/reference/vreplication/
+- Release notes / lifecycle: https://vitess.io/docs/releases/

package/Ai Agent Workflow/skills/database-skills/skills/vitess/references/sharding-routing.md

@@ -1,5 +1,125 @@
-# Vitess Sharding and Routing
+# Vitess — Sharding and VSchema Design
 
-- Define shard key from access patterns and growth model.
-- Keep hot paths shard-local when possible.
-- Validate query routing behavior before launch.
+## Core concept
+
+The **VSchema** tells VTGate how to route queries. It defines which tables belong to which keyspace, which column determines shard placement (the **primary vindex**), and how tables relate across shards.
+
+## Shard routing types
+
+| Routing | Condition | Performance |
+| --- | --- | --- |
+| **Single-shard** | `WHERE` on primary vindex with `=` | Best |
+| **Multi-shard (targeted)** | `WHERE` with `IN` on primary vindex | Good |
+| **Scatter** | No primary vindex filter | Expensive — hits all shards |
+
+Always include the primary vindex column in `WHERE` clauses to avoid scatter queries.
+
+## Choosing a primary vindex column
+
+Pick the column that:
+1. Appears in your highest-QPS `WHERE` clauses.
+2. Enables join co-location — tables frequently joined should shard on the same column.
+3. Keeps transactions within a single shard.
+4. Has high cardinality for even distribution.
+5. Is immutable — changing it after insert requires a data migration.
+
+Common choices: `user_id`, `tenant_id`, `org_id`, `account_id`.
+
+## Vindex types
+
+| Type | When to use |
+| --- | --- |
+| `xxhash` | Any column type — most common |
+| `unicode_loose_xxhash` | Text columns needing case-insensitive hashing |
+| `binary_md5` | MD5-based alternative for any column type |
+
+## VSchema structure (sharded keyspace)
+
+```json
+{
+  "sharded": true,
+  "vindexes": {
+    "xxhash": { "type": "xxhash" }
+  },
+  "tables": {
+    "orders": {
+      "column_vindexes": [
+        { "column": "customer_id", "name": "xxhash" }
+      ],
+      "auto_increment": {
+        "column": "id",
+        "sequence": "unsharded_keyspace.orders_seq"
+      }
+    }
+  }
+}
+```
+
+## Sequences — replacing AUTO_INCREMENT
+
+Per-shard `AUTO_INCREMENT` produces duplicate IDs across shards. Use Vitess Sequences instead.
+
+1. Create in an **unsharded** keyspace:
+   ```sql
+   CREATE TABLE orders_seq (id BIGINT, next_id BIGINT, cache BIGINT, PRIMARY KEY(id))
+     COMMENT 'vitess_sequence';
+   INSERT INTO orders_seq (id, next_id, cache) VALUES (0, 1, 1000);
+   ```
+2. Register in unsharded VSchema: `{ "orders_seq": { "type": "sequence" } }`
+3. Link to sharded table via `auto_increment` in sharded VSchema (see above).
+
+Sequence gaps from caching/restarts are expected and harmless.
+
+## Lookup vindexes (secondary routing)
+
+Lookup vindexes let you route by non-primary-vindex columns without scatter. They are backed by a separate mapping table. **They are expensive** — avoid unless scatter on the column is genuinely a problem.
+
+```json
+"customer_email_lookup": {
+  "type": "consistent_lookup",
+  "params": {
+    "table": "product.customer_email_lookup",
+    "from": "email",
+    "to": "keyspace_id"
+  },
+  "owner": "customer"
+}
+```
+
+Use `consistent_lookup_unique` only when uniqueness across shards must be enforced at the DB level (this is a scalability anti-pattern — prefer app-level enforcement).
+
+## Discover existing VSchema
+
+```bash
+vtctldclient GetVSchema <keyspace>                     # full JSON
+vtctldclient GetVSchema <keyspace> | jq '.vindexes'   # just vindexes
+```
+
+```sql
+SHOW VSCHEMA TABLES;        -- tables known to VTGate
+SHOW VSCHEMA VINDEXES;      -- vindexes and types
+```
+
+## Troubleshoot scatter queries
+
+```sql
+VEXPLAIN PLAN SELECT * FROM orders WHERE customer_id = 42;
+-- Look for Route variant: EqualUnique (single-shard) vs Scatter (all shards)
+
+VEXPLAIN ALL SELECT ...;    -- includes MySQL plans from each tablet
+VEXPLAIN TRACE SELECT ...;  -- shows row counts passed between query parts
+```
+
+If a query scatters:
+1. Is `WHERE` filtering on the primary vindex column? If not, add it.
+2. Is a lookup vindex configured for that column? If needed, add one.
+3. Primary vindex column updates are blocked — use `MoveTables` to re-shard if needed.
+
+## Reference tables
+
+Small, rarely-changing lookup data (countries, currencies, enums) can use `"type": "reference"` — they are replicated to all shards and never scatter.
+
+## Sources
+- VSchema guide: https://vitess.io/docs/user-guides/vschema-guide/
+- Vindexes reference: https://vitess.io/docs/reference/features/vschema/
+- Sharding key selection: https://vitess.io/docs/faq/advanced-configuration/vschema/how-do-you-select-your-sharding-key-for-vitess/