@cubis/foundry 0.3.10 → 0.3.11

package/Ai Agent Workflow/skills/database-skills/skills/postgres/references/schema-indexing.md

@@ -1,5 +1,79 @@
-# Postgres Schema and Indexing
+# Postgres — Schema Design and Indexing
 
-- Use explicit constraints for invariants.
-- Prefer composite indexes for equality + range access patterns.
-- Validate index usage via plans and statistics.
+## Schema design
+
+- Declare `NOT NULL` on every column that should never be null; reduces planner uncertainty and storage overhead.
+- Use `CHECK` constraints for domain rules (`amount > 0`, `status IN (...)`) — they are enforced transactionally and inform the planner.
+- Use `FOREIGN KEY` constraints for referential integrity; add indexes on FK columns to prevent sequential scans during cascades and joins.
+- Prefer `BIGINT` generated identity columns or UUIDs (v7 for sortability) as primary keys. Avoid random UUIDs as clustered keys on write-heavy tables — they fragment B-tree pages.
+- Use `TEXT` over `VARCHAR(n)` unless you need the length constraint enforced at the DB layer.
+- Prefer `TIMESTAMPTZ` over `TIMESTAMP` for all time columns.
+
+## Index types and when to use them
+
+| Type | When to use |
+| --- | --- |
+| **B-tree** (default) | Equality, range, `ORDER BY`, `LIKE 'prefix%'` |
+| **GIN** | JSONB containment (`@>`), full-text search, array overlap (`&&`) |
+| **GiST** | Geometric types, range type overlap, nearest-neighbor search |
+| **BRIN** | Append-mostly tables (time-series, logs) with high physical correlation |
+| **Hash** | Pure equality lookups only (`=`), smaller than B-tree |
+
+## Multicolumn (composite) indexes
+
+- Column order matters: place equality predicates first, then range/sort columns.
+- The planner can use a composite index for any leading prefix of its columns.
+- Example: `(status, created_at)` supports `WHERE status = 'open' ORDER BY created_at` but NOT `WHERE created_at > ...` alone without a full scan.
+
+```sql
+CREATE INDEX idx_orders_status_created ON orders (status, created_at DESC);
+```
+
+## Partial indexes
+
+- Index only the rows that queries actually need — dramatically smaller, faster to update.
+- Use when a hot query always includes a constant predicate.
+
+```sql
+-- Only index unpaid invoices, avoiding index bloat from millions of paid rows
+CREATE INDEX idx_invoices_unpaid ON invoices (due_date) WHERE paid = false;
+```
+
+## Covering indexes (INCLUDE)
+
+- Add non-predicate columns to the index to allow index-only scans (no heap fetch).
+- Put predicate columns in the key; put projection-only columns in `INCLUDE`.
+
+```sql
+CREATE INDEX idx_orders_user_covering ON orders (user_id, status)
+INCLUDE (total_amount, created_at);
+```
+
+## BRIN for append-mostly tables
+
+- Works by storing min/max per block range. Tiny index size, fast sequential insans.
+- Best when table rows are physically written in correlation with the indexed column (e.g., `created_at` on an append-only event table).
+- Useless for randomly ordered data.
+
+```sql
+CREATE INDEX idx_events_created_brin ON events USING BRIN (created_at);
+```
+
+## Index maintenance rules
+
+- Run `ANALYZE` after large data loads to refresh planner statistics.
+- Detect unused indexes:
+  ```sql
+  SELECT schemaname, relname, indexrelname, idx_scan
+  FROM pg_stat_user_indexes
+  WHERE idx_scan = 0;
+  ```
+- Drop unused indexes — they slow writes and increase vacuum cost.
+- Use `CREATE INDEX CONCURRENTLY` in production to avoid table locks.
+
+## Sources
+- PostgreSQL docs: Indexes — https://www.postgresql.org/docs/current/indexes.html
+- Multicolumn: https://www.postgresql.org/docs/current/indexes-multicolumn.html
+- Partial: https://www.postgresql.org/docs/current/indexes-partial.html
+- Covering (INCLUDE): https://www.postgresql.org/docs/current/sql-createindex.html
+- BRIN: https://www.postgresql.org/docs/current/brin.html

package/Ai Agent Workflow/skills/database-skills/skills/redis/SKILL.md

@@ -1,15 +1,36 @@
 ---
 name: redis
-description: Redis data modeling, caching strategy, latency tuning, and operational safety.
+description: Redis data modeling, caching strategy, throughput/latency optimization, and operational safety.
 ---
 
 # Redis
 
-Load references as needed:
+## Optimization workflow
+
+1. Define key schema and TTL policy first.
+2. Reduce round-trips with pipelining and batching.
+3. Tune memory footprint by data structure choice.
+4. Diagnose latency with server + system context.
+5. Validate hot-key and cluster-slot distribution for scale.
+
+## Indexing-style patterns in Redis
+
+- Redis is key-based; design key schema as your primary access index.
+- Use sorted sets and secondary lookup structures for query-like access.
+- Use `SCAN`-family commands for incremental traversal; avoid `KEYS` in production.
+
+## Pagination techniques
+
+- For ordered feeds/leaderboards, paginate with sorted set score/member boundaries.
+- For keyspace traversal, cursor-based `SCAN` pagination only.
+
+## Performance guardrails
+
+- Keep value payloads bounded; avoid giant hot keys.
+- Monitor expiry storms and eviction behavior.
+- Use realistic load tests for pipeline and memory tuning.
+
+## References
+
 - `references/cache-patterns.md`
 - `references/operations.md`
-
-Key rules:
-- Treat Redis as a data structure server, not generic storage.
-- Define TTL, invalidation, and consistency strategy upfront.
-- Monitor memory, eviction policy, and command latency.

package/Ai Agent Workflow/skills/database-skills/skills/redis/references/cache-patterns.md

@@ -1,5 +1,154 @@
-# Redis Cache Patterns
+# Redis — Cache and Throughput Patterns
 
-- Prefer explicit key design and namespacing.
-- Use write-through/write-behind intentionally.
-- Prevent cache stampede with locking or jittered TTL.
+## Choosing the right data structure
+
+This is the most important Redis decision. Using the wrong type causes unnecessary memory use and complexity.
+
+| Use case | Data structure | Key pattern |
+| --- | --- | --- |
+| Single value / counter | `String` | `user:{id}:session` |
+| Object with fields | `Hash` | `user:{id}` |
+| Ordered leaderboard / timeline | `Sorted Set` (ZSET) | `leaderboard:global` |
+| Unique membership / deduplication | `Set` | `online_users` |
+| Message queue / feed | `List` (LPUSH/RPOP) or `Stream` | `queue:emails` |
+| Feature flags per user | `Hash` or `Bitmap` | `flags:{user_id}` |
+| Rate limiting | `String` + `INCR` + `EXPIRE` | `rate:{user_id}:{window}` |
+| Time-series / event log | `Stream` | `events:{service}` |
+| Pub/Sub fan-out | `Pub/Sub` channel | `channel:notifications` |
+
+**Never** store serialized JSON in a `String` if you only need individual fields — use `Hash` and `HGET`/`HSET` to avoid deserializing the whole blob.
+
+## Key naming conventions
+
+Consistent naming prevents key collisions and makes `SCAN`-based debugging possible.
+
+```
+<namespace>:<entity>:<id>[:<field>]
+
+user:42:session        → session token for user 42
+product:99:views       → view counter for product 99
+rate:192.168.1.1:1706  → rate limit bucket for IP + minute window
+leaderboard:global     → global ZSET leaderboard
+queue:emails           → email send queue (List)
+```
+
+Rules:
+- Use `:` as separator, not `.` or `/`.
+- Keep keys short — key names count toward memory.
+- Avoid dynamic segments that produce unbounded key space without TTL.
+
+## TTL strategy
+
+Always set TTLs on cache keys. Omitting TTL = memory leak.
+
+```redis
+SET user:42:session "token" EX 3600         # expires in 1 hour
+SETEX user:42:profile 300 "{...}"           # 5 minutes
+EXPIRE user:42:temp 60                       # set TTL on existing key
+
+# Atomic set + expire (preferred over SETEX)
+SET key value EX 300 NX                     # set only if not exists + TTL
+```
+
+Check TTL on a key:
+```redis
+TTL user:42:session     # -1 = no TTL (danger!), -2 = key gone, N = seconds left
+```
+
+## Pipelining — batch commands to reduce round-trips
+
+Each Redis command is a network round-trip. Pipeline batches commands into a single trip.
+
+```ts
+// Node.js (ioredis) example
+const pipeline = redis.pipeline();
+pipeline.get('user:1:session');
+pipeline.incr('user:1:views');
+pipeline.expire('user:1:views', 3600);
+const results = await pipeline.exec();
+```
+
+Use pipelining for any hot path that issues 3+ Redis commands in sequence.
+
+## SCAN instead of KEYS in production
+
+`KEYS pattern` blocks the Redis event loop for its entire duration — it will freeze Redis under load.
+
+```redis
+# NEVER in production
+KEYS user:*
+
+# CORRECT: iterative cursor scan
+SCAN 0 MATCH user:* COUNT 100
+# Use returned cursor until it returns 0
+```
+
+In code:
+```ts
+let cursor = '0';
+do {
+  const [newCursor, keys] = await redis.scan(cursor, 'MATCH', 'user:*', 'COUNT', 100);
+  cursor = newCursor;
+  // process keys
+} while (cursor !== '0');
+```
+
+## Rate limiting pattern
+
+```ts
+// Sliding window rate limit: max 100 requests per minute per user
+async function isRateLimited(userId: string): Promise<boolean> {
+  const key = `rate:${userId}:${Math.floor(Date.now() / 60000)}`;
+  const count = await redis.incr(key);
+  if (count === 1) await redis.expire(key, 120); // 2 min window safety margin
+  return count > 100;
+}
+```
+
+## Sorted sets for leaderboards and pagination
+
+```redis
+# Add score
+ZADD leaderboard:global 1500 "user:42"
+
+# Top 10
+ZREVRANGEBYSCORE leaderboard:global +inf -inf WITHSCORES LIMIT 0 10
+
+# Rank of a user (0-indexed)
+ZREVRANK leaderboard:global "user:42"
+```
+
+Cursor-based pagination with sorted sets:
+```redis
+# Page 1: get top 20
+ZREVRANGE leaderboard:global 0 19 WITHSCORES
+
+# Page 2: skip 20
+ZREVRANGE leaderboard:global 20 39 WITHSCORES
+```
+
+## Cache invalidation patterns
+
+| Pattern | When to use |
+| --- | --- |
+| **TTL expiry** | Acceptable staleness (most cases) |
+| **Write-through** | Update cache on every write — consistent but coupled |
+| **Write-behind** | Write to cache; async flush to DB — fast writes, risk of loss |
+| **Cache-aside** | App reads cache → miss → read DB → populate cache |
+| **Tag-based invalidation** | Delete multiple related keys via a shared tag key |
+
+Tag-based with a Set:
+```ts
+// On write: register the key under its tag
+await redis.sadd('tag:user:42', 'user:42:profile', 'user:42:orders');
+
+// On invalidation: delete all tagged keys
+const keys = await redis.smembers('tag:user:42');
+await redis.del(...keys, 'tag:user:42');
+```
+
+## Sources
+- Pipelining: https://redis.io/docs/latest/develop/using-commands/pipelining/
+- SCAN command: https://redis.io/docs/latest/commands/scan/
+- Sorted sets: https://redis.io/docs/latest/develop/data-types/sorted-sets/
+- Data types overview: https://redis.io/docs/latest/develop/data-types/

package/Ai Agent Workflow/skills/database-skills/skills/redis/references/data-modeling.md

@@ -0,0 +1,152 @@
+# Redis — Data Modeling and Key Design
+
+## The golden rule
+
+In Redis, the **data structure determines your access pattern** — pick the structure based on what operations you need, not what the data looks like.
+
+## Data structure decision guide
+
+### String — simplest value
+
+Use for: counters, single scalar values, serialized blobs, feature flags.
+
+```redis
+SET user:42:points 1500
+INCR user:42:points              # atomic increment
+INCRBY user:42:points 50
+GET user:42:points
+
+# Serialized JSON (avoid if you only need individual fields — use Hash instead)
+SET product:99 '{"name":"Widget","price":9.99}'
+```
+
+### Hash — object with fields
+
+Use for: user profiles, session data, config objects with named fields. Avoids deserializing a full JSON blob to update one field.
+
+```redis
+HSET user:42 name "Alice" email "alice@example.com" plan "pro"
+HGET user:42 email
+HMGET user:42 name plan          # fetch specific fields
+HINCRBY user:42 login_count 1    # atomic field increment
+HDEL user:42 temp_field           # remove one field
+HGETALL user:42                   # get all fields (watch size)
+```
+
+Memory note: Hashes with ≤128 fields and short values use a compact ziplist encoding — much cheaper than separate String keys.
+
+### List — ordered sequence / queue
+
+Use for: message queues, activity feeds (append + trim), task lists.
+
+```redis
+LPUSH queue:emails '{"to":"a@b.com"}'   # push to head
+RPUSH queue:emails '{"to":"c@d.com"}'   # push to tail
+LPOP queue:emails                        # dequeue from head (FIFO)
+RPOP queue:emails                        # dequeue from tail (LIFO)
+LLEN queue:emails                        # current length
+LTRIM feed:user:42 0 99                  # keep only last 100 items
+```
+
+For producer/consumer queues in production prefer **Streams** (more features, consumer groups, acknowledgment).
+
+### Set — unordered unique values
+
+Use for: membership checks, unique visitors, tagging, deduplication.
+
+```redis
+SADD online_users user:42 user:99
+SISMEMBER online_users user:42   # O(1) membership check
+SMEMBERS online_users            # all members (avoid on large sets)
+SCARD online_users               # count
+SREM online_users user:42        # remove
+
+# Set operations (great for permission/feature checks)
+SUNION editors:org:1 editors:org:2          # union
+SINTER subscribers premium_users           # intersection
+```
+
+### Sorted Set (ZSET) — ordered by score
+
+Use for: leaderboards, rate limiting, scheduled jobs, range queries, time-series indexes.
+
+```redis
+ZADD leaderboard:global 1500 "user:42"    # score = 1500
+ZINCRBY leaderboard:global 100 "user:42"  # atomic score increment
+ZREVRANK leaderboard:global "user:42"      # rank (0-indexed, desc)
+ZREVRANGE leaderboard:global 0 9 WITHSCORES  # top 10 with scores
+
+# Range by score (e.g., all jobs due in next 60s)
+ZRANGEBYSCORE scheduled_jobs 0 1706000060 WITHSCORES LIMIT 0 10
+ZREM scheduled_jobs "job:123"             # remove after processing
+```
+
+### Stream — append-only log with consumer groups
+
+Use for: event sourcing, reliable message delivery, audit logs, notifications.
+
+```redis
+XADD events:orders '*' type order_created orderId 99 userId 42  # auto-ID
+XLEN events:orders
+XRANGE events:orders - +                 # all events
+XRANGE events:orders 1706000000000-0 +  # from timestamp
+
+# Consumer group for reliable delivery
+XGROUP CREATE events:orders workers $ MKSTREAM
+XREADGROUP GROUP workers consumer1 COUNT 10 STREAMS events:orders >  # read new
+XACK events:orders workers <message-id>  # acknowledge after processing
+```
+
+## Key naming
+
+```
+<entity>:<id>:<field>
+
+user:42:points          String — points counter
+user:42                 Hash   — user profile object
+leaderboard:global      ZSET   — global score board
+queue:emails            List   — email send queue
+online_users            Set    — currently online user IDs
+events:orders           Stream — order event log
+rate:user:42:202501     String — rate limit bucket
+session:abc123          String/Hash — session data
+```
+
+Rules:
+- Use `:` as separator.
+- Keep namespaces consistent across the codebase.
+- Include an entity type so `SCAN type:*` works for debugging.
+- Include a time bucket in rate limit keys so they auto-expire when the window passes.
+
+## TTL strategy
+
+| Key type | TTL strategy |
+| --- | --- |
+| Sessions | Explicit TTL on set, slide it on read (`EXPIRE key 3600`) |
+| Cache keys | Fixed TTL matching acceptable staleness |
+| Rate limit buckets | 2× the window duration |
+| Leaderboards | No TTL (explicit reset with `DEL`) |
+| Streams | Use `MAXLEN` to cap length, or `EXPIRE` on the whole stream |
+
+```redis
+# Sliding session TTL
+SET session:abc123 "{...}" EX 3600
+# On each read, reset the TTL:
+EXPIRE session:abc123 3600
+```
+
+## Avoid these patterns
+
+| Antipattern | Problem | Fix |
+| --- | --- | --- |
+| Storing JSON in String when you need individual fields | Full deser on every access | Use Hash |
+| Unbounded List/Set with no trim | Memory growth without bound | `LTRIM`, `SREM`, or `MAXLEN` |
+| `KEYS *` in production | Blocks event loop | Use `SCAN` |
+| No TTL on cache keys | Memory leak | Always set TTL |
+| Huge Hash (`HGETALL`) on hot path | Transfers more data than needed | Project with `HMGET` |
+| Sequences as String with `INCR` shared globally | Bottleneck at high QPS | Shard or batch |
+
+## Sources
+- Redis data types: https://redis.io/docs/latest/develop/data-types/
+- Streams: https://redis.io/docs/latest/develop/data-types/streams/
+- Sorted sets: https://redis.io/docs/latest/develop/data-types/sorted-sets/

package/Ai Agent Workflow/skills/database-skills/skills/redis/references/operations.md

@@ -1,5 +1,144 @@
-# Redis Operations
+# Redis — Operations, Memory, and Latency
 
-- Set maxmemory and eviction policy intentionally.
-- Monitor keyspace size and slowlog.
-- Validate persistence/replication settings for data risk profile.
+## Memory management
+
+Redis stores all data in memory. Running out of memory is the #1 operational failure mode.
+
+### Monitor memory
+
+```redis
+INFO memory
+# Key fields:
+# used_memory_human        — actual data memory
+# used_memory_rss_human    — RSS from OS (includes fragmentation)
+# mem_fragmentation_ratio  — rss / used. >1.5 = high fragmentation
+# maxmemory                — configured limit (0 = unlimited — dangerous)
+# maxmemory_human          — human readable limit
+```
+
+Always set `maxmemory`:
+```redis
+CONFIG SET maxmemory 2gb
+```
+
+### Eviction policies
+
+When `maxmemory` is reached, Redis uses the configured eviction policy:
+
+| Policy | Behavior |
+| --- | --- |
+| `noeviction` | Returns error on writes — safe for primary data stores |
+| `allkeys-lru` | Evicts least recently used keys — good for caches |
+| `volatile-lru` | Evicts LRU keys that have a TTL set |
+| `allkeys-lfu` | Evicts least frequently used — better than LRU for skewed access |
+| `volatile-ttl` | Evicts keys with shortest remaining TTL first |
+| `allkeys-random` | Random eviction — rarely useful |
+
+For pure caches: `allkeys-lru` or `allkeys-lfu`.
+For mixed primary + cache data: `volatile-lru` (only evicts keys with TTL).
+
+```redis
+CONFIG SET maxmemory-policy allkeys-lru
+```
+
+### Handle fragmentation
+
+When `mem_fragmentation_ratio > 1.5`, trigger active defragmentation (Redis 4.0+):
+```redis
+CONFIG SET activedefrag yes
+CONFIG SET active-defrag-ignore-bytes 100mb
+CONFIG SET active-defrag-threshold-lower 10
+```
+
+## Latency diagnostics
+
+Redis should respond in microseconds. If you're seeing millisecond latency, investigate:
+
+```redis
+# Check for slow commands (logged automatically)
+SLOWLOG GET 10
+SLOWLOG LEN
+SLOWLOG RESET
+
+# Default slow threshold is 10ms — lower for tighter monitoring:
+CONFIG SET slowlog-log-slower-than 1000   # microseconds = 1ms
+
+# Latency history (built-in latency monitoring)
+LATENCY LATEST
+LATENCY HISTORY event_name
+LATENCY RESET
+```
+
+### Common latency causes
+
+| Cause | Symptom | Fix |
+| --- | --- | --- |
+| `KEYS *` in production | Periodic spikes | Replace with `SCAN` |
+| Large key values (MB-range) | Slow reads/writes | Chunk data or use a different store |
+| AOF fsync on every write | Consistent high latency | Switch to `fsync everysec` |
+| Memory pressure / eviction | Increasing latency trend | Add memory or tune eviction |
+| Blocking commands (`BLPOP`, `BRPOP`) | Thread holds connection | Use timeouts |
+| Fork for RDB/AOF rewrite | Latency spike every N minutes | Schedule rewrites in off-peak windows |
+
+## Persistence configuration
+
+Choose based on durability requirements:
+
+| Config | Durability | Performance |
+| --- | --- | --- |
+| No persistence | None (cache only) | Fastest |
+| RDB snapshots only | Up to snapshot interval | Fast |
+| AOF `everysec` | Up to ~1s of data loss | Good |
+| AOF `always` | No data loss | Slowest — 1 fsync per write |
+| RDB + AOF | Best of both | Moderate |
+
+For caches where losing data on restart is acceptable, disable persistence:
+```redis
+CONFIG SET save ""         # disable RDB
+CONFIG SET appendonly no   # disable AOF
+```
+
+## Monitor key space
+
+```redis
+# Count keys per database
+INFO keyspace
+
+# Sample key patterns (does not block)
+redis-cli --scan --pattern 'user:*' | head -20
+
+# Key size distribution — pick a sample
+redis-cli --bigkeys       # finds largest keys by type
+```
+
+## Connection limits
+
+```redis
+INFO clients
+# connected_clients       — current connections
+# maxclients              — configured limit (default 10000)
+```
+
+Redis is single-threaded for command execution. High connection counts add overhead via:
+- Select/poll overhead on the socket set.
+- Memory per connection (~20KB).
+
+Use a connection pool in your application. Most Redis clients (ioredis, redis-py) pool by default. Check pool size matches your concurrency model.
+
+## Key expiry and eviction monitoring
+
+```redis
+INFO stats
+# expired_keys     — keys removed by TTL expiry since start
+# evicted_keys     — keys removed by maxmemory policy
+# keyspace_hits    — cache hit count
+# keyspace_misses  — cache miss count
+
+# Hit rate = hits / (hits + misses). Below 80% suggests key design issues.
+```
+
+## Sources
+- Memory optimization: https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/memory-optimization/
+- Latency optimization: https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency/
+- Persistence options: https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/
+- SLOWLOG: https://redis.io/docs/latest/commands/slowlog/

package/Ai Agent Workflow/skills/database-skills/skills/sqlite/SKILL.md

@@ -1,15 +1,36 @@
 ---
 name: sqlite
-description: SQLite local/edge data strategy, schema design, indexing, and WAL tuning.
+description: SQLite local/edge data strategy, schema/index design, query planning, and WAL tuning.
 ---
 
 # SQLite
 
-Load references as needed:
+## Optimization workflow
+
+1. Inspect plan with `EXPLAIN QUERY PLAN`.
+2. Add multicolumn or covering indexes for hot read paths.
+3. Use keyset-style pagination for large datasets.
+4. Tune WAL/checkpoint behavior for write-heavy workloads.
+5. Re-check plans after schema/data-distribution changes.
+
+## Indexing techniques
+
+- Composite indexes for combined filter+sort paths.
+- Covering indexes when read latency is critical.
+- Keep indexes minimal on write-heavy local stores.
+
+## Pagination techniques
+
+- Offset is acceptable for small tables and shallow lists.
+- For large traversals, use deterministic keyset pagination.
+
+## Operational guardrails
+
+- Batch writes in explicit transactions.
+- Use WAL mode for mixed read/write concurrency.
+- Validate checkpoint strategy for mobile/edge IO limits.
+
+## References
+
 - `references/local-first.md`
 - `references/performance.md`
-
-Key rules:
-- Use WAL mode for mixed read/write workloads.
-- Keep write transactions short.
-- Plan upgrade path if concurrency demands exceed SQLite limits.