cokit-cli 1.0.0 → 1.0.2

package/skills/ck-databases/SKILL.md

@@ -0,0 +1,231 @@
+---
+name: 'ck-databases'
+description: 'Work with MongoDB (document database, BSON documents, aggregation pipelines, Atlas cloud) and PostgreSQL (relational database, SQL queries, psql CLI, pgAdmin). Use when designing database schemas, writing queries and aggregations, optimizing indexes for performance, performing database migrations, configuring replication and sharding, implementing backup and restore strategies, managing database users and permissions, analyzing query performance, or administering production databases.'
+---
+
+# Databases Skill
+
+Unified guide for working with MongoDB (document-oriented) and PostgreSQL (relational) databases. Choose the right database for your use case and master both systems.
+
+## When to Use This Skill
+
+Use when:
+- Designing database schemas and data models
+- Writing queries (SQL or MongoDB query language)
+- Building aggregation pipelines or complex joins
+- Optimizing indexes and query performance
+- Implementing database migrations
+- Setting up replication, sharding, or clustering
+- Configuring backups and disaster recovery
+- Managing database users and permissions
+- Analyzing slow queries and performance issues
+- Administering production database deployments
+
+## Database Selection Guide
+
+### Choose MongoDB When:
+- Schema flexibility: frequent structure changes, heterogeneous data
+- Document-centric: natural JSON/BSON data model
+- Horizontal scaling: need to shard across multiple servers
+- High write throughput: IoT, logging, real-time analytics
+- Nested/hierarchical data: embedded documents preferred
+- Rapid prototyping: schema evolution without migrations
+
+**Best for:** Content management, catalogs, IoT time series, real-time analytics, mobile apps, user profiles
+
+### Choose PostgreSQL When:
+- Strong consistency: ACID transactions critical
+- Complex relationships: many-to-many joins, referential integrity
+- SQL requirement: team expertise, reporting tools, BI systems
+- Data integrity: strict schema validation, constraints
+- Mature ecosystem: extensive tooling, extensions
+- Complex queries: window functions, CTEs, analytical workloads
+
+**Best for:** Financial systems, e-commerce transactions, ERP, CRM, data warehousing, analytics
+
+### Both Support:
+- JSON/JSONB storage and querying
+- Full-text search capabilities
+- Geospatial queries and indexing
+- Replication and high availability
+- ACID transactions (MongoDB 4.0+)
+- Strong security features
+
+## Quick Start
+
+### MongoDB Setup
+
+```bash
+# Atlas (Cloud) - Recommended
+# 1. Sign up at mongodb.com/atlas
+# 2. Create M0 free cluster
+# 3. Get connection string
+
+# Connection
+mongodb+srv://user:pass@cluster.mongodb.net/db
+
+# Shell
+mongosh "mongodb+srv://cluster.mongodb.net/mydb"
+
+# Basic operations
+db.users.insertOne({ name: "Alice", age: 30 })
+db.users.find({ age: { $gte: 18 } })
+db.users.updateOne({ name: "Alice" }, { $set: { age: 31 } })
+db.users.deleteOne({ name: "Alice" })
+```
+
+### PostgreSQL Setup
+
+```bash
+# Ubuntu/Debian
+sudo apt-get install postgresql postgresql-contrib
+
+# Start service
+sudo systemctl start postgresql
+
+# Connect
+psql -U postgres -d mydb
+
+# Basic operations
+CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT, age INT);
+INSERT INTO users (name, age) VALUES ('Alice', 30);
+SELECT * FROM users WHERE age >= 18;
+UPDATE users SET age = 31 WHERE name = 'Alice';
+DELETE FROM users WHERE name = 'Alice';
+```
+
+## Common Operations
+
+### Create/Insert
+```javascript
+// MongoDB
+db.users.insertOne({ name: "Bob", email: "bob@example.com" })
+db.users.insertMany([{ name: "Alice" }, { name: "Charlie" }])
+```
+
+```sql
+-- PostgreSQL
+INSERT INTO users (name, email) VALUES ('Bob', 'bob@example.com');
+INSERT INTO users (name, email) VALUES ('Alice', NULL), ('Charlie', NULL);
+```
+
+### Read/Query
+```javascript
+// MongoDB
+db.users.find({ age: { $gte: 18 } })
+db.users.findOne({ email: "bob@example.com" })
+```
+
+```sql
+-- PostgreSQL
+SELECT * FROM users WHERE age >= 18;
+SELECT * FROM users WHERE email = 'bob@example.com' LIMIT 1;
+```
+
+### Update
+```javascript
+// MongoDB
+db.users.updateOne({ name: "Bob" }, { $set: { age: 25 } })
+db.users.updateMany({ status: "pending" }, { $set: { status: "active" } })
+```
+
+```sql
+-- PostgreSQL
+UPDATE users SET age = 25 WHERE name = 'Bob';
+UPDATE users SET status = 'active' WHERE status = 'pending';
+```
+
+### Delete
+```javascript
+// MongoDB
+db.users.deleteOne({ name: "Bob" })
+db.users.deleteMany({ status: "deleted" })
+```
+
+```sql
+-- PostgreSQL
+DELETE FROM users WHERE name = 'Bob';
+DELETE FROM users WHERE status = 'deleted';
+```
+
+### Indexing
+```javascript
+// MongoDB
+db.users.createIndex({ email: 1 })
+db.users.createIndex({ status: 1, createdAt: -1 })
+```
+
+```sql
+-- PostgreSQL
+CREATE INDEX idx_users_email ON users(email);
+CREATE INDEX idx_users_status_created ON users(status, created_at DESC);
+```
+
+## Reference Navigation
+
+### MongoDB References
+- **[mongodb-crud.md](references/mongodb-crud.md)** - CRUD operations, query operators, atomic updates
+- **[mongodb-aggregation.md](references/mongodb-aggregation.md)** - Aggregation pipeline, stages, operators, patterns
+- **[mongodb-indexing.md](references/mongodb-indexing.md)** - Index types, compound indexes, performance optimization
+- **[mongodb-atlas.md](references/mongodb-atlas.md)** - Atlas cloud setup, clusters, monitoring, search
+
+### PostgreSQL References
+- **[postgresql-queries.md](references/postgresql-queries.md)** - SELECT, JOINs, subqueries, CTEs, window functions
+- **[postgresql-psql-cli.md](references/postgresql-psql-cli.md)** - psql commands, meta-commands, scripting
+- **[postgresql-performance.md](references/postgresql-performance.md)** - EXPLAIN, query optimization, vacuum, indexes
+- **[postgresql-administration.md](references/postgresql-administration.md)** - User management, backups, replication, maintenance
+
+## Python Utilities
+
+Database utility scripts in `scripts/`:
+- **db_migrate.py** - Generate and apply migrations for both databases
+- **db_backup.py** - Backup and restore MongoDB and PostgreSQL
+- **db_performance_check.py** - Analyze slow queries and recommend indexes
+
+```bash
+# Generate migration
+python scripts/db_migrate.py --db mongodb --generate "add_user_index"
+
+# Run backup
+python scripts/db_backup.py --db postgres --output /backups/
+
+# Check performance
+python scripts/db_performance_check.py --db mongodb --threshold 100ms
+```
+
+## Key Differences Summary
+
+| Feature | MongoDB | PostgreSQL |
+|---------|---------|------------|
+| Data Model | Document (JSON/BSON) | Relational (Tables/Rows) |
+| Schema | Flexible, dynamic | Strict, predefined |
+| Query Language | MongoDB Query Language | SQL |
+| Joins | $lookup (limited) | Native, optimized |
+| Transactions | Multi-document (4.0+) | Native ACID |
+| Scaling | Horizontal (sharding) | Vertical (primary), Horizontal (extensions) |
+| Indexes | Single, compound, text, geo, etc | B-tree, hash, GiST, GIN, etc |
+
+## Best Practices
+
+**MongoDB:**
+- Use embedded documents for 1-to-few relationships
+- Reference documents for 1-to-many or many-to-many
+- Index frequently queried fields
+- Use aggregation pipeline for complex transformations
+- Enable authentication and TLS in production
+- Use Atlas for managed hosting
+
+**PostgreSQL:**
+- Normalize schema to 3NF, denormalize for performance
+- Use foreign keys for referential integrity
+- Index foreign keys and frequently filtered columns
+- Use EXPLAIN ANALYZE to optimize queries
+- Regular VACUUM and ANALYZE maintenance
+- Connection pooling (pgBouncer) for web apps
+
+## Resources
+
+- MongoDB: https://www.mongodb.com/docs/
+- PostgreSQL: https://www.postgresql.org/docs/
+- MongoDB University: https://learn.mongodb.com/
+- PostgreSQL Tutorial: https://www.postgresqltutorial.com/

package/skills/ck-databases/references/mongodb-aggregation.md

@@ -0,0 +1,447 @@
+# MongoDB Aggregation Pipeline
+
+Aggregation pipeline for complex data transformations, analytics, and multi-stage processing.
+
+## Pipeline Concept
+
+Aggregation processes documents through multiple stages. Each stage transforms documents and passes results to next stage.
+
+```javascript
+db.collection.aggregate([
+  { /* Stage 1 */ },
+  { /* Stage 2 */ },
+  { /* Stage 3 */ }
+])
+```
+
+## Core Pipeline Stages
+
+### $match (Filter Documents)
+```javascript
+// Filter early in pipeline for efficiency
+db.orders.aggregate([
+  { $match: { status: "completed", total: { $gte: 100 } } },
+  // Subsequent stages process only matched documents
+])
+
+// Multiple conditions
+db.orders.aggregate([
+  { $match: {
+    $and: [
+      { orderDate: { $gte: startDate } },
+      { status: { $in: ["completed", "shipped"] } }
+    ]
+  }}
+])
+```
+
+### $project (Reshape Documents)
+```javascript
+// Select and reshape fields
+db.orders.aggregate([
+  { $project: {
+    orderNumber: 1,
+    total: 1,
+    customerName: "$customer.name",
+    year: { $year: "$orderDate" },
+    _id: 0  // Exclude _id
+  }}
+])
+
+// Computed fields
+db.orders.aggregate([
+  { $project: {
+    total: 1,
+    tax: { $multiply: ["$total", 0.1] },
+    grandTotal: { $add: ["$total", { $multiply: ["$total", 0.1] }] }
+  }}
+])
+```
+
+### $group (Aggregate Data)
+```javascript
+// Group and count
+db.orders.aggregate([
+  { $group: {
+    _id: "$status",
+    count: { $sum: 1 }
+  }}
+])
+
+// Multiple aggregations
+db.orders.aggregate([
+  { $group: {
+    _id: "$customerId",
+    totalSpent: { $sum: "$total" },
+    orderCount: { $sum: 1 },
+    avgOrderValue: { $avg: "$total" },
+    maxOrder: { $max: "$total" },
+    minOrder: { $min: "$total" }
+  }}
+])
+
+// Group by multiple fields
+db.sales.aggregate([
+  { $group: {
+    _id: {
+      year: { $year: "$date" },
+      month: { $month: "$date" },
+      product: "$productId"
+    },
+    revenue: { $sum: "$amount" }
+  }}
+])
+```
+
+### $sort (Order Results)
+```javascript
+// Sort by field
+db.orders.aggregate([
+  { $sort: { total: -1 } }  // -1: descending, 1: ascending
+])
+
+// Sort by multiple fields
+db.orders.aggregate([
+  { $sort: { status: 1, orderDate: -1 } }
+])
+```
+
+### $limit / $skip (Pagination)
+```javascript
+// Limit results
+db.orders.aggregate([
+  { $sort: { orderDate: -1 } },
+  { $limit: 10 }
+])
+
+// Pagination
+const page = 2;
+const pageSize = 20;
+db.orders.aggregate([
+  { $sort: { orderDate: -1 } },
+  { $skip: (page - 1) * pageSize },
+  { $limit: pageSize }
+])
+```
+
+### $lookup (Join Collections)
+```javascript
+// Simple join
+db.orders.aggregate([
+  { $lookup: {
+    from: "customers",
+    localField: "customerId",
+    foreignField: "_id",
+    as: "customer"
+  }},
+  { $unwind: "$customer" }  // Convert array to object
+])
+
+// Pipeline join (more powerful)
+db.orders.aggregate([
+  { $lookup: {
+    from: "products",
+    let: { items: "$items" },
+    pipeline: [
+      { $match: { $expr: { $in: ["$_id", "$$items.productId"] } } },
+      { $project: { name: 1, price: 1 } }
+    ],
+    as: "productDetails"
+  }}
+])
+```
+
+### $unwind (Deconstruct Arrays)
+```javascript
+// Unwind array field
+db.orders.aggregate([
+  { $unwind: "$items" }
+])
+
+// Preserve null/empty arrays
+db.orders.aggregate([
+  { $unwind: {
+    path: "$items",
+    preserveNullAndEmptyArrays: true
+  }}
+])
+
+// Include array index
+db.orders.aggregate([
+  { $unwind: {
+    path: "$items",
+    includeArrayIndex: "itemIndex"
+  }}
+])
+```
+
+### $addFields (Add New Fields)
+```javascript
+// Add computed fields
+db.orders.aggregate([
+  { $addFields: {
+    totalWithTax: { $multiply: ["$total", 1.1] },
+    year: { $year: "$orderDate" }
+  }}
+])
+```
+
+### $replaceRoot (Replace Document Root)
+```javascript
+// Promote subdocument to root
+db.orders.aggregate([
+  { $replaceRoot: { newRoot: "$customer" } }
+])
+
+// Merge fields
+db.orders.aggregate([
+  { $replaceRoot: {
+    newRoot: { $mergeObjects: ["$customer", { orderId: "$_id" }] }
+  }}
+])
+```
+
+## Aggregation Operators
+
+### Arithmetic Operators
+```javascript
+// Basic math
+db.products.aggregate([
+  { $project: {
+    name: 1,
+    profit: { $subtract: ["$price", "$cost"] },
+    margin: { $multiply: [
+      { $divide: [
+        { $subtract: ["$price", "$cost"] },
+        "$price"
+      ]},
+      100
+    ]}
+  }}
+])
+
+// Other operators: $add, $multiply, $divide, $mod, $abs, $ceil, $floor, $round
+```
+
+### String Operators
+```javascript
+// String manipulation
+db.users.aggregate([
+  { $project: {
+    fullName: { $concat: ["$firstName", " ", "$lastName"] },
+    email: { $toLower: "$email" },
+    initials: { $concat: [
+      { $substr: ["$firstName", 0, 1] },
+      { $substr: ["$lastName", 0, 1] }
+    ]}
+  }}
+])
+
+// Other: $toUpper, $trim, $split, $substr, $regexMatch
+```
+
+### Date Operators
+```javascript
+// Date extraction
+db.events.aggregate([
+  { $project: {
+    event: 1,
+    year: { $year: "$timestamp" },
+    month: { $month: "$timestamp" },
+    day: { $dayOfMonth: "$timestamp" },
+    hour: { $hour: "$timestamp" },
+    dayOfWeek: { $dayOfWeek: "$timestamp" }
+  }}
+])
+
+// Date math
+db.events.aggregate([
+  { $project: {
+    event: 1,
+    expiresAt: { $add: ["$createdAt", 1000 * 60 * 60 * 24 * 30] }, // +30 days
+    ageInDays: { $divide: [
+      { $subtract: [new Date(), "$createdAt"] },
+      1000 * 60 * 60 * 24
+    ]}
+  }}
+])
+```
+
+### Array Operators
+```javascript
+// Array operations
+db.posts.aggregate([
+  { $project: {
+    title: 1,
+    tagCount: { $size: "$tags" },
+    firstTag: { $arrayElemAt: ["$tags", 0] },
+    lastTag: { $arrayElemAt: ["$tags", -1] },
+    hasMongoDBTag: { $in: ["mongodb", "$tags"] }
+  }}
+])
+
+// Array filtering
+db.posts.aggregate([
+  { $project: {
+    title: 1,
+    activeTags: {
+      $filter: {
+        input: "$tags",
+        as: "tag",
+        cond: { $ne: ["$$tag.status", "deprecated"] }
+      }
+    }
+  }}
+])
+```
+
+### Conditional Operators
+```javascript
+// $cond (ternary)
+db.products.aggregate([
+  { $project: {
+    name: 1,
+    status: {
+      $cond: {
+        if: { $gte: ["$stock", 10] },
+        then: "In Stock",
+        else: "Low Stock"
+      }
+    }
+  }}
+])
+
+// $switch (multiple conditions)
+db.orders.aggregate([
+  { $project: {
+    status: 1,
+    priority: {
+      $switch: {
+        branches: [
+          { case: { $gte: ["$total", 1000] }, then: "High" },
+          { case: { $gte: ["$total", 100] }, then: "Medium" }
+        ],
+        default: "Low"
+      }
+    }
+  }}
+])
+```
+
+## Advanced Patterns
+
+### Time-Based Aggregation
+```javascript
+// Daily sales
+db.orders.aggregate([
+  { $match: { orderDate: { $gte: startDate } } },
+  { $group: {
+    _id: {
+      year: { $year: "$orderDate" },
+      month: { $month: "$orderDate" },
+      day: { $dayOfMonth: "$orderDate" }
+    },
+    revenue: { $sum: "$total" },
+    orderCount: { $sum: 1 }
+  }},
+  { $sort: { "_id.year": 1, "_id.month": 1, "_id.day": 1 } }
+])
+```
+
+### Faceted Search
+```javascript
+// Multiple aggregations in one query
+db.products.aggregate([
+  { $match: { category: "electronics" } },
+  { $facet: {
+    priceRanges: [
+      { $bucket: {
+        groupBy: "$price",
+        boundaries: [0, 100, 500, 1000, 5000],
+        default: "5000+",
+        output: { count: { $sum: 1 } }
+      }}
+    ],
+    topBrands: [
+      { $group: { _id: "$brand", count: { $sum: 1 } } },
+      { $sort: { count: -1 } },
+      { $limit: 5 }
+    ],
+    avgPrice: [
+      { $group: { _id: null, avg: { $avg: "$price" } } }
+    ]
+  }}
+])
+```
+
+### Window Functions
+```javascript
+// Running totals and moving averages
+db.sales.aggregate([
+  { $setWindowFields: {
+    partitionBy: "$region",
+    sortBy: { date: 1 },
+    output: {
+      runningTotal: {
+        $sum: "$amount",
+        window: { documents: ["unbounded", "current"] }
+      },
+      movingAvg: {
+        $avg: "$amount",
+        window: { documents: [-7, 0] }  // Last 7 days
+      }
+    }
+  }}
+])
+```
+
+### Text Search with Aggregation
+```javascript
+// Full-text search (requires text index)
+db.articles.aggregate([
+  { $match: { $text: { $search: "mongodb database" } } },
+  { $addFields: { score: { $meta: "textScore" } } },
+  { $sort: { score: -1 } },
+  { $limit: 10 }
+])
+```
+
+### Geospatial Aggregation
+```javascript
+// Find nearby locations
+db.places.aggregate([
+  { $geoNear: {
+    near: { type: "Point", coordinates: [lon, lat] },
+    distanceField: "distance",
+    maxDistance: 5000,
+    spherical: true
+  }},
+  { $limit: 10 }
+])
+```
+
+## Performance Tips
+
+1. **$match early** - Filter documents before other stages
+2. **$project early** - Reduce document size
+3. **Index usage** - $match and $sort can use indexes (only at start)
+4. **$limit after $sort** - Reduce memory usage
+5. **Avoid $lookup** - Prefer embedded documents when possible
+6. **Use $facet sparingly** - Can be memory intensive
+7. **allowDiskUse** - Enable for large datasets
+```javascript
+db.collection.aggregate(pipeline, { allowDiskUse: true })
+```
+
+## Best Practices
+
+1. **Order stages efficiently** - $match → $project → $group → $sort → $limit
+2. **Use $expr carefully** - Can prevent index usage
+3. **Monitor memory** - Default limit: 100MB per stage
+4. **Test with explain** - Analyze pipeline performance
+```javascript
+db.collection.explain("executionStats").aggregate(pipeline)
+```
+5. **Break complex pipelines** - Use $out/$merge for intermediate results
+6. **Use $sample** - For random document selection
+7. **Leverage $addFields** - Cleaner than $project for adding fields