shmakk 1.1.0 → 1.2.0

package/skills/database-durable-objects.md

@@ -0,0 +1,187 @@
+---
+name: durable-objects
+description: Create and review Cloudflare Durable Objects. Use when building stateful coordination (chat rooms, multiplayer games, booking systems), implementing RPC methods, SQLite storage, alarms, WebSockets, or reviewing DO code for best practices. Covers Workers integration, wrangler config, and testing with Vitest. Biases towards retrieval from Cloudflare docs over pre-trained knowledge.
+category: database
+---
+
+# Durable Objects
+
+Build stateful, coordinated applications on Cloudflare's edge using Durable Objects.
+
+## Retrieval Sources
+
+Your knowledge of Durable Objects APIs and configuration may be outdated. **Prefer retrieval over pre-training** for any Durable Objects task.
+
+| Resource | URL |
+|----------|-----|
+| Docs | https://developers.cloudflare.com/durable-objects/ |
+| API Reference | https://developers.cloudflare.com/durable-objects/api/ |
+| Best Practices | https://developers.cloudflare.com/durable-objects/best-practices/ |
+| Examples | https://developers.cloudflare.com/durable-objects/examples/ |
+
+Fetch the relevant doc page when implementing features.
+
+## When to Use
+
+- Creating new Durable Object classes for stateful coordination
+- Implementing RPC methods, alarms, or WebSocket handlers
+- Reviewing existing DO code for best practices
+- Configuring wrangler.jsonc/toml for DO bindings and migrations
+- Writing tests with `@cloudflare/vitest-pool-workers`
+- Designing sharding strategies and parent-child relationships
+
+## Reference Documentation
+
+- `./references/rules.md` - Core rules, storage, concurrency, RPC, alarms
+- `./references/testing.md` - Vitest setup, unit/integration tests, alarm testing
+- `./references/workers.md` - Workers handlers, types, wrangler config, observability
+
+Search: `blockConcurrencyWhile`, `idFromName`, `getByName`, `setAlarm`, `sql.exec`
+
+## Core Principles
+
+### Use Durable Objects For
+
+| Need | Example |
+|------|---------|
+| Coordination | Chat rooms, multiplayer games, collaborative docs |
+| Strong consistency | Inventory, booking systems, turn-based games |
+| Per-entity storage | Multi-tenant SaaS, per-user data |
+| Persistent connections | WebSockets, real-time notifications |
+| Scheduled work per entity | Subscription renewals, game timeouts |
+
+### Do NOT Use For
+
+- Stateless request handling (use plain Workers)
+- Maximum global distribution needs
+- High fan-out independent requests
+
+## Quick Reference
+
+### Wrangler Configuration
+
+```jsonc
+// wrangler.jsonc
+{
+  "durable_objects": {
+    "bindings": [{ "name": "MY_DO", "class_name": "MyDurableObject" }]
+  },
+  "migrations": [{ "tag": "v1", "new_sqlite_classes": ["MyDurableObject"] }]
+}
+```
+
+### Basic Durable Object Pattern
+
+```typescript
+import { DurableObject } from "cloudflare:workers";
+
+export interface Env {
+  MY_DO: DurableObjectNamespace<MyDurableObject>;
+}
+
+export class MyDurableObject extends DurableObject<Env> {
+  constructor(ctx: DurableObjectState, env: Env) {
+    super(ctx, env);
+    ctx.blockConcurrencyWhile(async () => {
+      this.ctx.storage.sql.exec(`
+        CREATE TABLE IF NOT EXISTS items (
+          id INTEGER PRIMARY KEY AUTOINCREMENT,
+          data TEXT NOT NULL
+        )
+      `);
+    });
+  }
+
+  async addItem(data: string): Promise<number> {
+    const result = this.ctx.storage.sql.exec<{ id: number }>(
+      "INSERT INTO items (data) VALUES (?) RETURNING id",
+      data
+    );
+    return result.one().id;
+  }
+}
+
+export default {
+  async fetch(request: Request, env: Env): Promise<Response> {
+    const stub = env.MY_DO.getByName("my-instance");
+    const id = await stub.addItem("hello");
+    return Response.json({ id });
+  },
+};
+```
+
+## Critical Rules
+
+1. **Model around coordination atoms** - One DO per chat room/game/user, not one global DO
+2. **Use `getByName()` for deterministic routing** - Same input = same DO instance
+3. **Use SQLite storage** - Configure `new_sqlite_classes` in migrations
+4. **Initialize in constructor** - Use `blockConcurrencyWhile()` for schema setup only
+5. **Use RPC methods** - Not fetch() handler (compatibility date >= 2024-04-03)
+6. **Persist first, cache second** - Always write to storage before updating in-memory state
+7. **One alarm per DO** - `setAlarm()` replaces any existing alarm
+
+## Anti-Patterns (NEVER)
+
+- Single global DO handling all requests (bottleneck)
+- Using `blockConcurrencyWhile()` on every request (kills throughput)
+- Storing critical state only in memory (lost on eviction/crash)
+- Using `await` between related storage writes (breaks atomicity)
+- Holding `blockConcurrencyWhile()` across `fetch()` or external I/O
+
+## Stub Creation
+
+```typescript
+// Deterministic - preferred for most cases
+const stub = env.MY_DO.getByName("room-123");
+
+// From existing ID string
+const id = env.MY_DO.idFromString(storedIdString);
+const stub = env.MY_DO.get(id);
+
+// New unique ID - store mapping externally
+const id = env.MY_DO.newUniqueId();
+const stub = env.MY_DO.get(id);
+```
+
+## Storage Operations
+
+```typescript
+// SQL (synchronous, recommended)
+this.ctx.storage.sql.exec("INSERT INTO t (c) VALUES (?)", value);
+const rows = this.ctx.storage.sql.exec<Row>("SELECT * FROM t").toArray();
+
+// KV (async)
+await this.ctx.storage.put("key", value);
+const val = await this.ctx.storage.get<Type>("key");
+```
+
+## Alarms
+
+```typescript
+// Schedule (replaces existing)
+await this.ctx.storage.setAlarm(Date.now() + 60_000);
+
+// Handler
+async alarm(): Promise<void> {
+  // Process scheduled work
+  // Optionally reschedule: await this.ctx.storage.setAlarm(...)
+}
+
+// Cancel
+await this.ctx.storage.deleteAlarm();
+```
+
+## Testing Quick Start
+
+```typescript
+import { env } from "cloudflare:test";
+import { describe, it, expect } from "vitest";
+
+describe("MyDO", () => {
+  it("should work", async () => {
+    const stub = env.MY_DO.getByName("test");
+    const result = await stub.addItem("test");
+    expect(result).toBe(1);
+  });
+});
+```

package/skills/database-ef-core.md

@@ -0,0 +1,76 @@
+---
+name: ef-core
+description: 'Get best practices for Entity Framework Core'
+category: database
+---
+
+# Entity Framework Core Best Practices
+
+Your goal is to help me follow best practices when working with Entity Framework Core.
+
+## Data Context Design
+
+- Keep DbContext classes focused and cohesive
+- Use constructor injection for configuration options
+- Override OnModelCreating for fluent API configuration
+- Separate entity configurations using IEntityTypeConfiguration
+- Consider using DbContextFactory pattern for console apps or tests
+
+## Entity Design
+
+- Use meaningful primary keys (consider natural vs surrogate keys)
+- Implement proper relationships (one-to-one, one-to-many, many-to-many)
+- Use data annotations or fluent API for constraints and validations
+- Implement appropriate navigational properties
+- Consider using owned entity types for value objects
+
+## Performance
+
+- Use AsNoTracking() for read-only queries
+- Implement pagination for large result sets with Skip() and Take()
+- Use Include() to eager load related entities when needed
+- Consider projection (Select) to retrieve only required fields
+- Use compiled queries for frequently executed queries
+- Avoid N+1 query problems by properly including related data
+
+## Migrations
+
+- Create small, focused migrations
+- Name migrations descriptively
+- Verify migration SQL scripts before applying to production
+- Consider using migration bundles for deployment
+- Add data seeding through migrations when appropriate
+
+## Querying
+
+- Use IQueryable judiciously and understand when queries execute
+- Prefer strongly-typed LINQ queries over raw SQL
+- Use appropriate query operators (Where, OrderBy, GroupBy)
+- Consider database functions for complex operations
+- Implement specifications pattern for reusable queries
+
+## Change Tracking & Saving
+
+- Use appropriate change tracking strategies
+- Batch your SaveChanges() calls
+- Implement concurrency control for multi-user scenarios
+- Consider using transactions for multiple operations
+- Use appropriate DbContext lifetimes (scoped for web apps)
+
+## Security
+
+- Avoid SQL injection by using parameterized queries
+- Implement appropriate data access permissions
+- Be careful with raw SQL queries
+- Consider data encryption for sensitive information
+- Use migrations to manage database user permissions
+
+## Testing
+
+- Use in-memory database provider for unit tests
+- Create separate testing contexts with SQLite for integration tests
+- Mock DbContext and DbSet for pure unit tests
+- Test migrations in isolated environments
+- Consider snapshot testing for model changes
+
+When reviewing my EF Core code, identify issues and suggest improvements that follow these best practices.

package/skills/database-fabric-lakehouse.md

@@ -0,0 +1,107 @@
+---
+name: fabric-lakehouse
+description: 'Use this skill to get context about Fabric Lakehouse and its features for software systems and AI-powered functions. It offers descriptions of Lakehouse data components, organization with schemas and shortcuts, access control, and code examples. This skill supports users in designing, building, and optimizing Lakehouse solutions using best practices.'
+metadata:
+  author: tedvilutis
+  version: "1.0"
+category: database
+---
+
+# When to Use This Skill
+
+Use this skill when you need to:
+- Generate a document or explanation that includes definition and context about Fabric Lakehouse and its capabilities.
+- Design, build, and optimize Lakehouse solutions using best practices.
+- Understand the core concepts and components of a Lakehouse in Microsoft Fabric.
+- Learn how to manage tabular and non-tabular data within a Lakehouse.
+
+# Fabric Lakehouse
+
+## Core Concepts
+
+### What is a Lakehouse?
+
+Lakehouse in Microsoft Fabric is an item that gives users a place to store their tabular data (like tables) and non-tabular data (like files). It combines the flexibility of a data lake with the management capabilities of a data warehouse. It provides:
+
+- **Unified storage** in OneLake for structured and unstructured data
+- **Delta Lake format** for ACID transactions, versioning, and time travel
+- **SQL analytics endpoint** for T-SQL queries
+- **Semantic model** for Power BI integration
+- Support for other table formats like CSV, Parquet
+- Support for any file formats
+- Tools for table optimization and data management
+
+### Key Components
+
+- **Delta Tables**: Managed tables with ACID compliance and schema enforcement
+- **Files**: Unstructured/semi-structured data in the Files section
+- **SQL Endpoint**: Auto-generated read-only SQL interface for querying
+- **Shortcuts**: Virtual links to external/internal data without copying
+- **Fabric Materialized Views**: Pre-computed tables for fast query performance
+
+### Tabular data in a Lakehouse
+
+Tabular data in a form of tables are stored under "Tables" folder. Main format for tables in Lakehouse is Delta. Lakehouse can store tabular data in other formats like CSV or Parquet, these formats are only available for Spark querying.
+Tables can be internal, when data is stored under "Tables" folder, or external, when only reference to a table is stored under "Tables" folder but the data itself is stored in a referenced location. Tables are referenced through Shortcuts, which can be internal (pointing to another location in Fabric) or external (pointing to data stored outside of Fabric).
+
+### Schemas for tables in a Lakehouse
+
+When creating a lakehouse, users can choose to enable schemas. Schemas are used to organize Lakehouse tables. Schemas are implemented as folders under the "Tables" folder and store tables inside of those folders. The default schema is "dbo" and it can't be deleted or renamed. All other schemas are optional and can be created, renamed, or deleted. Users can reference a schema located in another lakehouse using a Schema Shortcut, thereby referencing all tables in the destination schema with a single shortcut.
+
+### Files in a Lakehouse
+
+Files are stored under "Files" folder. Users can create folders and subfolders to organize their files. Any file format can be stored in Lakehouse.
+
+### Fabric Materialized Views
+
+Set of pre-computed tables that are automatically updated based on a schedule. They provide fast query performance for complex aggregations and joins. Materialized views are defined using PySpark or Spark SQL and stored in an associated Notebook.
+
+### Spark Views
+
+Logical tables defined by a SQL query. They do not store data but provide a virtual layer for querying. Views are defined using Spark SQL and stored in Lakehouse next to Tables.
+
+## Security
+
+### Item access or control plane security
+
+Users can have workspace roles (Admin, Member, Contributor, Viewer) that provide different levels of access to Lakehouse and its contents. Users can also get access permission using sharing capabilities of Lakehouse.
+
+### Data access or OneLake Security
+
+For data access use OneLake security model, which is based on Microsoft Entra ID (formerly Azure Active Directory) and role-based access control (RBAC). Lakehouse data is stored in OneLake, so access to data is controlled through OneLake permissions. In addition to object-level permissions, Lakehouse also supports column-level and row-level security for tables, allowing fine-grained control over who can see specific columns or rows in a table.
+
+
+## Lakehouse Shortcuts
+
+Shortcuts create virtual links to data without copying:
+
+### Types of Shortcuts
+
+- **Internal**: Link to other Fabric Lakehouses/tables, cross-workspace data sharing 
+- **ADLS Gen2**: Link to ADLS Gen2 containers in Azure
+- **Amazon S3**: AWS S3 buckets, cross-cloud data access
+- **Dataverse**: Microsoft Dataverse, business application data
+- **Google Cloud Storage**: GCS buckets, cross-cloud data access
+
+## Performance Optimization
+
+### V-Order Optimization
+
+For faster data read with semantic model enable V-Order optimization on Delta tables. This presorts data in a way that improves query performance for common access patterns.
+
+### Table Optimization
+
+Tables can also be optimized using the OPTIMIZE command, which compacts small files into larger ones and can also apply Z-ordering to improve query performance on specific columns. Regular optimization helps maintain performance as data is ingested and updated over time. The Vacuum command can be used to clean up old files and free up storage space, especially after updates and deletes.
+
+## Lineage
+
+The Lakehouse item supports lineage, which allows users to track the origin and transformations of data. Lineage information is automatically captured for tables and files in Lakehouse, showing how data flows from source to destination. This helps with debugging, auditing, and understanding data dependencies.
+
+## PySpark Code Examples
+
+See [PySpark code](references/pyspark.md) for details.
+
+## Getting data into Lakehouse
+
+See [Get data](references/getdata.md) for details.
+

package/skills/database-neon-postgres-egress-optimizer.md

@@ -0,0 +1,213 @@
+---
+name: neon-postgres-egress-optimizer
+description: >-
+  Diagnose and fix excessive Postgres egress (network data transfer) in a codebase.
+  Use when a user mentions high database bills, unexpected data transfer costs,
+  network transfer charges, egress spikes, "why is my Neon bill so high",
+  "database costs jumped", SELECT * optimization, query overfetching,
+  reduce Neon costs, optimize database usage, or wants to reduce data sent
+  from their database to their application. Also use when reviewing query
+  patterns for cost efficiency, even if the user doesn't explicitly mention
+  egress or data transfer.
+category: database
+---
+
+# Postgres Egress Optimizer
+
+Guide the user through diagnosing and fixing application-side query patterns that cause excessive data transfer (egress) from their Postgres database. Most high egress bills come from the application fetching more data than it uses.
+
+## Step 1: Diagnose
+
+Identify which queries transfer the most data. The primary tool is the `pg_stat_statements` extension.
+
+### Check if pg_stat_statements is available
+
+```sql
+SELECT 1 FROM pg_stat_statements LIMIT 1;
+```
+
+If this errors, the extension needs to be created:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
+```
+
+On Neon, it is available by default but may need this CREATE EXTENSION step.
+
+### Handle empty stats
+
+Stats are cleared when a Neon compute scales to zero and restarts. If the stats are empty or the compute recently woke up:
+
+1. Reset the stats to start a clean measurement window: `SELECT pg_stat_statements_reset();`
+2. Let the application run under representative traffic for at least an hour.
+3. Return and run the diagnostic queries below.
+
+If the user has stats from a production database, use those. If they have no access to production stats, proceed to Step 2 and analyze the codebase directly — code-level patterns are often sufficient to identify the worst offenders.
+
+### Diagnostic queries
+
+Run these to identify the top egress contributors. Focus on queries that return many rows, return wide rows (JSONB, TEXT, BYTEA columns), or are called very frequently.
+
+**Queries returning the most total rows:**
+
+```sql
+SELECT query, calls, rows AS total_rows, rows / calls AS avg_rows_per_call
+FROM pg_stat_statements
+WHERE calls > 0
+ORDER BY rows DESC
+LIMIT 10;
+```
+
+**Queries returning the most rows per execution** (poorly scoped SELECTs, missing pagination):
+
+```sql
+SELECT query, calls, rows AS total_rows, rows / calls AS avg_rows_per_call
+FROM pg_stat_statements
+WHERE calls > 0
+ORDER BY avg_rows_per_call DESC
+LIMIT 10;
+```
+
+**Most frequently called queries** (candidates for caching):
+
+```sql
+SELECT query, calls, rows AS total_rows, rows / calls AS avg_rows_per_call
+FROM pg_stat_statements
+WHERE calls > 0
+ORDER BY calls DESC
+LIMIT 10;
+```
+
+**Longest running queries** (not a direct egress measure, but helps identify problem queries during a spike):
+
+```sql
+SELECT query, calls, rows AS total_rows,
+  round(total_exec_time::numeric, 2) AS total_exec_time_ms
+FROM pg_stat_statements
+WHERE calls > 0
+ORDER BY total_exec_time DESC
+LIMIT 10;
+```
+
+### Interpret the results
+
+Rank findings by estimated egress impact:
+
+- **High row count + wide rows** = biggest egress. A query returning 1,000 rows where each row includes a 50KB JSONB column transfers ~50MB per call.
+- **Extreme call frequency** on even small queries adds up. A query called 50,000 times/day returning 10 rows each = 500,000 rows/day.
+- **Cross-reference with the schema** to identify which columns are wide. Look for JSONB, TEXT, BYTEA, and large VARCHAR columns.
+
+## Step 2: Analyze codebase
+
+For each query identified in Step 1, or for each database query in the codebase if no stats are available, check:
+
+- Does it select only the columns the response needs?
+- Does it return a bounded number of rows (LIMIT/pagination)?
+- Is it called frequently enough to benefit from caching?
+- Does it fetch raw data that gets aggregated in application code?
+- Does it use a JOIN that duplicates parent data across child rows?
+
+## Step 3: Fix
+
+Apply the appropriate fix for each problem found. Below are the most common egress anti-patterns and how to fix them.
+
+### Unused columns (SELECT \*)
+
+**Problem:** The query fetches all columns but the application only uses a few. Large columns (JSONB blobs, TEXT fields) get transferred over the wire and discarded.
+
+**Before:**
+
+```sql
+SELECT * FROM products;
+```
+
+**After:**
+
+```sql
+SELECT id, name, price, image_urls FROM products;
+```
+
+### Missing pagination
+
+**Problem:** A list endpoint returns all rows with no LIMIT. This is an unbounded egress risk — every new row in the table increases data transfer on every request. Flag this regardless of current table size.
+
+This is easy to miss because the application may work fine with small datasets. But at scale, an unpaginated endpoint returning 10,000 rows with even moderate column widths can transfer hundreds of megabytes per day.
+
+**Before:**
+
+```sql
+SELECT id, name, price FROM products;
+```
+
+**After:**
+
+```sql
+SELECT id, name, price FROM products
+ORDER BY id
+LIMIT 50 OFFSET 0;
+```
+
+When adding pagination, check whether the consuming client already supports paginated responses. If not, pick sensible defaults and document the pagination parameters in the API.
+
+### High-frequency queries on static data
+
+**Problem:** A query is called thousands of times per day but returns data that rarely changes. Every call transfers the same rows from the database. This pattern is only visible from `pg_stat_statements` — the code itself looks normal.
+
+Look for queries with extremely high call counts relative to other queries. Common examples: configuration tables, category lists, feature flags, user role definitions.
+
+**Fix:** Add a caching layer between the application and the database so it avoids hitting the database on every request.
+
+### Application-side aggregation
+
+**Problem:** The application fetches all rows from a table and then computes aggregates (averages, counts, sums, groupings) in application code. The full dataset transfers over the wire even though the result is a small summary.
+
+**Fix:** Push the aggregation into SQL.
+
+**Before:** The application fetches entire tables and aggregates in code with loops or `.reduce()`.
+
+**After:**
+
+```sql
+SELECT p.category_id,
+       AVG(r.rating) AS avg_rating,
+       COUNT(r.id) AS review_count
+FROM reviews r
+INNER JOIN products p ON r.product_id = p.id
+GROUP BY p.category_id;
+```
+
+### JOIN duplication
+
+**Problem:** A JOIN between a wide parent table and a child table duplicates all parent columns across every child row. If a product has 200 reviews and the product row includes a 50KB JSONB column, the join sends that 50KB x 200 = ~10MB for a single request.
+
+This is distinct from the SELECT \* problem. Even if you select only needed columns, a JOIN still repeats the parent data for every child row. The fix is structural: avoid the join entirely.
+
+**Before:**
+
+```sql
+SELECT * FROM products
+LEFT JOIN reviews ON reviews.product_id = products.id
+WHERE products.id = 1;
+```
+
+**After (two separate queries):**
+
+```sql
+SELECT id, name, price, description, image_urls FROM products WHERE id = 1;
+SELECT id, user_name, rating, body FROM reviews WHERE product_id = 1;
+```
+
+Two queries instead of one JOIN. The product data is fetched once. The reviews are fetched once. No duplication.
+
+## Step 4: Verify
+
+After applying fixes:
+
+1. **Run existing tests** to confirm nothing broke.
+2. **Check the responses** — make sure the API still returns the same data shape. Column selection and pagination changes can break clients that depend on specific fields or full result sets.
+3. **Measure the improvement** — if pg_stat_statements data is available, reset it (`SELECT pg_stat_statements_reset();`), let traffic run, then re-run the diagnostic queries to compare before and after.
+
+## Further reading
+
+- https://neon.com/docs/introduction/network-transfer.md
+- https://neon.com/docs/introduction/cost-optimization.md