@heyai-rules/pilo-masterkit 1.2.2 → 3.1.0

package/.agent/skills/clickhouse-io/SKILL.md

@@ -0,0 +1,439 @@
+---
+name: clickhouse-io
+description: ClickHouse database patterns, query optimization, analytics, and data engineering best practices for high-performance analytical workloads.
+origin: ECC
+---
+
+# ClickHouse Analytics Patterns
+
+ClickHouse-specific patterns for high-performance analytics and data engineering.
+
+## When to Activate
+
+- Designing ClickHouse table schemas (MergeTree engine selection)
+- Writing analytical queries (aggregations, window functions, joins)
+- Optimizing query performance (partition pruning, projections, materialized views)
+- Ingesting large volumes of data (batch inserts, Kafka integration)
+- Migrating from PostgreSQL/MySQL to ClickHouse for analytics
+- Implementing real-time dashboards or time-series analytics
+
+## Overview
+
+ClickHouse is a column-oriented database management system (DBMS) for online analytical processing (OLAP). It's optimized for fast analytical queries on large datasets.
+
+**Key Features:**
+- Column-oriented storage
+- Data compression
+- Parallel query execution
+- Distributed queries
+- Real-time analytics
+
+## Table Design Patterns
+
+### MergeTree Engine (Most Common)
+
+```sql
+CREATE TABLE markets_analytics (
+    date Date,
+    market_id String,
+    market_name String,
+    volume UInt64,
+    trades UInt32,
+    unique_traders UInt32,
+    avg_trade_size Float64,
+    created_at DateTime
+) ENGINE = MergeTree()
+PARTITION BY toYYYYMM(date)
+ORDER BY (date, market_id)
+SETTINGS index_granularity = 8192;
+```
+
+### ReplacingMergeTree (Deduplication)
+
+```sql
+-- For data that may have duplicates (e.g., from multiple sources)
+CREATE TABLE user_events (
+    event_id String,
+    user_id String,
+    event_type String,
+    timestamp DateTime,
+    properties String
+) ENGINE = ReplacingMergeTree()
+PARTITION BY toYYYYMM(timestamp)
+ORDER BY (user_id, event_id, timestamp)
+PRIMARY KEY (user_id, event_id);
+```
+
+### AggregatingMergeTree (Pre-aggregation)
+
+```sql
+-- For maintaining aggregated metrics
+CREATE TABLE market_stats_hourly (
+    hour DateTime,
+    market_id String,
+    total_volume AggregateFunction(sum, UInt64),
+    total_trades AggregateFunction(count, UInt32),
+    unique_users AggregateFunction(uniq, String)
+) ENGINE = AggregatingMergeTree()
+PARTITION BY toYYYYMM(hour)
+ORDER BY (hour, market_id);
+
+-- Query aggregated data
+SELECT
+    hour,
+    market_id,
+    sumMerge(total_volume) AS volume,
+    countMerge(total_trades) AS trades,
+    uniqMerge(unique_users) AS users
+FROM market_stats_hourly
+WHERE hour >= toStartOfHour(now() - INTERVAL 24 HOUR)
+GROUP BY hour, market_id
+ORDER BY hour DESC;
+```
+
+## Query Optimization Patterns
+
+### Efficient Filtering
+
+```sql
+-- PASS: GOOD: Use indexed columns first
+SELECT *
+FROM markets_analytics
+WHERE date >= '2025-01-01'
+  AND market_id = 'market-123'
+  AND volume > 1000
+ORDER BY date DESC
+LIMIT 100;
+
+-- FAIL: BAD: Filter on non-indexed columns first
+SELECT *
+FROM markets_analytics
+WHERE volume > 1000
+  AND market_name LIKE '%election%'
+  AND date >= '2025-01-01';
+```
+
+### Aggregations
+
+```sql
+-- PASS: GOOD: Use ClickHouse-specific aggregation functions
+SELECT
+    toStartOfDay(created_at) AS day,
+    market_id,
+    sum(volume) AS total_volume,
+    count() AS total_trades,
+    uniq(trader_id) AS unique_traders,
+    avg(trade_size) AS avg_size
+FROM trades
+WHERE created_at >= today() - INTERVAL 7 DAY
+GROUP BY day, market_id
+ORDER BY day DESC, total_volume DESC;
+
+-- PASS: Use quantile for percentiles (more efficient than percentile)
+SELECT
+    quantile(0.50)(trade_size) AS median,
+    quantile(0.95)(trade_size) AS p95,
+    quantile(0.99)(trade_size) AS p99
+FROM trades
+WHERE created_at >= now() - INTERVAL 1 HOUR;
+```
+
+### Window Functions
+
+```sql
+-- Calculate running totals
+SELECT
+    date,
+    market_id,
+    volume,
+    sum(volume) OVER (
+        PARTITION BY market_id
+        ORDER BY date
+        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
+    ) AS cumulative_volume
+FROM markets_analytics
+WHERE date >= today() - INTERVAL 30 DAY
+ORDER BY market_id, date;
+```
+
+## Data Insertion Patterns
+
+### Bulk Insert (Recommended)
+
+```typescript
+import { ClickHouse } from 'clickhouse'
+
+const clickhouse = new ClickHouse({
+  url: process.env.CLICKHOUSE_URL,
+  port: 8123,
+  basicAuth: {
+    username: process.env.CLICKHOUSE_USER,
+    password: process.env.CLICKHOUSE_PASSWORD
+  }
+})
+
+// PASS: Batch insert (efficient)
+async function bulkInsertTrades(trades: Trade[]) {
+  const values = trades.map(trade => `(
+    '${trade.id}',
+    '${trade.market_id}',
+    '${trade.user_id}',
+    ${trade.amount},
+    '${trade.timestamp.toISOString()}'
+  )`).join(',')
+
+  await clickhouse.query(`
+    INSERT INTO trades (id, market_id, user_id, amount, timestamp)
+    VALUES ${values}
+  `).toPromise()
+}
+
+// FAIL: Individual inserts (slow)
+async function insertTrade(trade: Trade) {
+  // Don't do this in a loop!
+  await clickhouse.query(`
+    INSERT INTO trades VALUES ('${trade.id}', ...)
+  `).toPromise()
+}
+```
+
+### Streaming Insert
+
+```typescript
+// For continuous data ingestion
+import { createWriteStream } from 'fs'
+import { pipeline } from 'stream/promises'
+
+async function streamInserts() {
+  const stream = clickhouse.insert('trades').stream()
+
+  for await (const batch of dataSource) {
+    stream.write(batch)
+  }
+
+  await stream.end()
+}
+```
+
+## Materialized Views
+
+### Real-time Aggregations
+
+```sql
+-- Create materialized view for hourly stats
+CREATE MATERIALIZED VIEW market_stats_hourly_mv
+TO market_stats_hourly
+AS SELECT
+    toStartOfHour(timestamp) AS hour,
+    market_id,
+    sumState(amount) AS total_volume,
+    countState() AS total_trades,
+    uniqState(user_id) AS unique_users
+FROM trades
+GROUP BY hour, market_id;
+
+-- Query the materialized view
+SELECT
+    hour,
+    market_id,
+    sumMerge(total_volume) AS volume,
+    countMerge(total_trades) AS trades,
+    uniqMerge(unique_users) AS users
+FROM market_stats_hourly
+WHERE hour >= now() - INTERVAL 24 HOUR
+GROUP BY hour, market_id;
+```
+
+## Performance Monitoring
+
+### Query Performance
+
+```sql
+-- Check slow queries
+SELECT
+    query_id,
+    user,
+    query,
+    query_duration_ms,
+    read_rows,
+    read_bytes,
+    memory_usage
+FROM system.query_log
+WHERE type = 'QueryFinish'
+  AND query_duration_ms > 1000
+  AND event_time >= now() - INTERVAL 1 HOUR
+ORDER BY query_duration_ms DESC
+LIMIT 10;
+```
+
+### Table Statistics
+
+```sql
+-- Check table sizes
+SELECT
+    database,
+    table,
+    formatReadableSize(sum(bytes)) AS size,
+    sum(rows) AS rows,
+    max(modification_time) AS latest_modification
+FROM system.parts
+WHERE active
+GROUP BY database, table
+ORDER BY sum(bytes) DESC;
+```
+
+## Common Analytics Queries
+
+### Time Series Analysis
+
+```sql
+-- Daily active users
+SELECT
+    toDate(timestamp) AS date,
+    uniq(user_id) AS daily_active_users
+FROM events
+WHERE timestamp >= today() - INTERVAL 30 DAY
+GROUP BY date
+ORDER BY date;
+
+-- Retention analysis
+SELECT
+    signup_date,
+    countIf(days_since_signup = 0) AS day_0,
+    countIf(days_since_signup = 1) AS day_1,
+    countIf(days_since_signup = 7) AS day_7,
+    countIf(days_since_signup = 30) AS day_30
+FROM (
+    SELECT
+        user_id,
+        min(toDate(timestamp)) AS signup_date,
+        toDate(timestamp) AS activity_date,
+        dateDiff('day', signup_date, activity_date) AS days_since_signup
+    FROM events
+    GROUP BY user_id, activity_date
+)
+GROUP BY signup_date
+ORDER BY signup_date DESC;
+```
+
+### Funnel Analysis
+
+```sql
+-- Conversion funnel
+SELECT
+    countIf(step = 'viewed_market') AS viewed,
+    countIf(step = 'clicked_trade') AS clicked,
+    countIf(step = 'completed_trade') AS completed,
+    round(clicked / viewed * 100, 2) AS view_to_click_rate,
+    round(completed / clicked * 100, 2) AS click_to_completion_rate
+FROM (
+    SELECT
+        user_id,
+        session_id,
+        event_type AS step
+    FROM events
+    WHERE event_date = today()
+)
+GROUP BY session_id;
+```
+
+### Cohort Analysis
+
+```sql
+-- User cohorts by signup month
+SELECT
+    toStartOfMonth(signup_date) AS cohort,
+    toStartOfMonth(activity_date) AS month,
+    dateDiff('month', cohort, month) AS months_since_signup,
+    count(DISTINCT user_id) AS active_users
+FROM (
+    SELECT
+        user_id,
+        min(toDate(timestamp)) OVER (PARTITION BY user_id) AS signup_date,
+        toDate(timestamp) AS activity_date
+    FROM events
+)
+GROUP BY cohort, month, months_since_signup
+ORDER BY cohort, months_since_signup;
+```
+
+## Data Pipeline Patterns
+
+### ETL Pattern
+
+```typescript
+// Extract, Transform, Load
+async function etlPipeline() {
+  // 1. Extract from source
+  const rawData = await extractFromPostgres()
+
+  // 2. Transform
+  const transformed = rawData.map(row => ({
+    date: new Date(row.created_at).toISOString().split('T')[0],
+    market_id: row.market_slug,
+    volume: parseFloat(row.total_volume),
+    trades: parseInt(row.trade_count)
+  }))
+
+  // 3. Load to ClickHouse
+  await bulkInsertToClickHouse(transformed)
+}
+
+// Run periodically
+setInterval(etlPipeline, 60 * 60 * 1000)  // Every hour
+```
+
+### Change Data Capture (CDC)
+
+```typescript
+// Listen to PostgreSQL changes and sync to ClickHouse
+import { Client } from 'pg'
+
+const pgClient = new Client({ connectionString: process.env.DATABASE_URL })
+
+pgClient.query('LISTEN market_updates')
+
+pgClient.on('notification', async (msg) => {
+  const update = JSON.parse(msg.payload)
+
+  await clickhouse.insert('market_updates', [
+    {
+      market_id: update.id,
+      event_type: update.operation,  // INSERT, UPDATE, DELETE
+      timestamp: new Date(),
+      data: JSON.stringify(update.new_data)
+    }
+  ])
+})
+```
+
+## Best Practices
+
+### 1. Partitioning Strategy
+- Partition by time (usually month or day)
+- Avoid too many partitions (performance impact)
+- Use DATE type for partition key
+
+### 2. Ordering Key
+- Put most frequently filtered columns first
+- Consider cardinality (high cardinality first)
+- Order impacts compression
+
+### 3. Data Types
+- Use smallest appropriate type (UInt32 vs UInt64)
+- Use LowCardinality for repeated strings
+- Use Enum for categorical data
+
+### 4. Avoid
+- SELECT * (specify columns)
+- FINAL (merge data before query instead)
+- Too many JOINs (denormalize for analytics)
+- Small frequent inserts (batch instead)
+
+### 5. Monitoring
+- Track query performance
+- Monitor disk usage
+- Check merge operations
+- Review slow query log
+
+**Remember**: ClickHouse excels at analytical workloads. Design tables for your query patterns, batch inserts, and leverage materialized views for real-time aggregations.

package/.agent/skills/codebase-onboarding/SKILL.md

@@ -0,0 +1,233 @@
+---
+name: codebase-onboarding
+description: Analyze an unfamiliar codebase and generate a structured onboarding guide with architecture map, key entry points, conventions, and a starter CLAUDE.md. Use when joining a new project or setting up Claude Code for the first time in a repo.
+origin: ECC
+---
+
+# Codebase Onboarding
+
+Systematically analyze an unfamiliar codebase and produce a structured onboarding guide. Designed for developers joining a new project or setting up Claude Code in an existing repo for the first time.
+
+## When to Use
+
+- First time opening a project with Claude Code
+- Joining a new team or repository
+- User asks "help me understand this codebase"
+- User asks to generate a CLAUDE.md for a project
+- User says "onboard me" or "walk me through this repo"
+
+## How It Works
+
+### Phase 1: Reconnaissance
+
+Gather raw signals about the project without reading every file. Run these checks in parallel:
+
+```
+1. Package manifest detection
+   → package.json, go.mod, Cargo.toml, pyproject.toml, pom.xml, build.gradle,
+     Gemfile, composer.json, mix.exs, pubspec.yaml
+
+2. Framework fingerprinting
+   → next.config.*, nuxt.config.*, angular.json, vite.config.*,
+     django settings, flask app factory, fastapi main, rails config
+
+3. Entry point identification
+   → main.*, index.*, app.*, server.*, cmd/, src/main/
+
+4. Directory structure snapshot
+   → Top 2 levels of the directory tree, ignoring node_modules, vendor,
+     .git, dist, build, __pycache__, .next
+
+5. Config and tooling detection
+   → .eslintrc*, .prettierrc*, tsconfig.json, Makefile, Dockerfile,
+     docker-compose*, .github/workflows/, .env.example, CI configs
+
+6. Test structure detection
+   → tests/, test/, __tests__/, *_test.go, *.spec.ts, *.test.js,
+     pytest.ini, jest.config.*, vitest.config.*
+```
+
+### Phase 2: Architecture Mapping
+
+From the reconnaissance data, identify:
+
+**Tech Stack**
+- Language(s) and version constraints
+- Framework(s) and major libraries
+- Database(s) and ORMs
+- Build tools and bundlers
+- CI/CD platform
+
+**Architecture Pattern**
+- Monolith, monorepo, microservices, or serverless
+- Frontend/backend split or full-stack
+- API style: REST, GraphQL, gRPC, tRPC
+
+**Key Directories**
+Map the top-level directories to their purpose:
+
+<!-- Example for a React project — replace with detected directories -->
+```
+src/components/  → React UI components
+src/api/         → API route handlers
+src/lib/         → Shared utilities
+src/db/          → Database models and migrations
+tests/           → Test suites
+scripts/         → Build and deployment scripts
+```
+
+**Data Flow**
+Trace one request from entry to response:
+- Where does a request enter? (router, handler, controller)
+- How is it validated? (middleware, schemas, guards)
+- Where is business logic? (services, models, use cases)
+- How does it reach the database? (ORM, raw queries, repositories)
+
+### Phase 3: Convention Detection
+
+Identify patterns the codebase already follows:
+
+**Naming Conventions**
+- File naming: kebab-case, camelCase, PascalCase, snake_case
+- Component/class naming patterns
+- Test file naming: `*.test.ts`, `*.spec.ts`, `*_test.go`
+
+**Code Patterns**
+- Error handling style: try/catch, Result types, error codes
+- Dependency injection or direct imports
+- State management approach
+- Async patterns: callbacks, promises, async/await, channels
+
+**Git Conventions**
+- Branch naming from recent branches
+- Commit message style from recent commits
+- PR workflow (squash, merge, rebase)
+- If the repo has no commits yet or only a shallow history (e.g. `git clone --depth 1`), skip this section and note "Git history unavailable or too shallow to detect conventions"
+
+### Phase 4: Generate Onboarding Artifacts
+
+Produce two outputs:
+
+#### Output 1: Onboarding Guide
+
+```markdown
+# Onboarding Guide: [Project Name]
+
+## Overview
+[2-3 sentences: what this project does and who it serves]
+
+## Tech Stack
+<!-- Example for a Next.js project — replace with detected stack -->
+| Layer | Technology | Version |
+|-------|-----------|---------|
+| Language | TypeScript | 5.x |
+| Framework | Next.js | 14.x |
+| Database | PostgreSQL | 16 |
+| ORM | Prisma | 5.x |
+| Testing | Jest + Playwright | - |
+
+## Architecture
+[Diagram or description of how components connect]
+
+## Key Entry Points
+<!-- Example for a Next.js project — replace with detected paths -->
+- **API routes**: `src/app/api/` — Next.js route handlers
+- **UI pages**: `src/app/(dashboard)/` — authenticated pages
+- **Database**: `prisma/schema.prisma` — data model source of truth
+- **Config**: `next.config.ts` — build and runtime config
+
+## Directory Map
+[Top-level directory → purpose mapping]
+
+## Request Lifecycle
+[Trace one API request from entry to response]
+
+## Conventions
+- [File naming pattern]
+- [Error handling approach]
+- [Testing patterns]
+- [Git workflow]
+
+## Common Tasks
+<!-- Example for a Node.js project — replace with detected commands -->
+- **Run dev server**: `npm run dev`
+- **Run tests**: `npm test`
+- **Run linter**: `npm run lint`
+- **Database migrations**: `npx prisma migrate dev`
+- **Build for production**: `npm run build`
+
+## Where to Look
+<!-- Example for a Next.js project — replace with detected paths -->
+| I want to... | Look at... |
+|--------------|-----------|
+| Add an API endpoint | `src/app/api/` |
+| Add a UI page | `src/app/(dashboard)/` |
+| Add a database table | `prisma/schema.prisma` |
+| Add a test | `tests/` matching the source path |
+| Change build config | `next.config.ts` |
+```
+
+#### Output 2: Starter CLAUDE.md
+
+Generate or update a project-specific CLAUDE.md based on detected conventions. If `CLAUDE.md` already exists, read it first and enhance it — preserve existing project-specific instructions and clearly call out what was added or changed.
+
+```markdown
+# Project Instructions
+
+## Tech Stack
+[Detected stack summary]
+
+## Code Style
+- [Detected naming conventions]
+- [Detected patterns to follow]
+
+## Testing
+- Run tests: `[detected test command]`
+- Test pattern: [detected test file convention]
+- Coverage: [if configured, the coverage command]
+
+## Build & Run
+- Dev: `[detected dev command]`
+- Build: `[detected build command]`
+- Lint: `[detected lint command]`
+
+## Project Structure
+[Key directory → purpose map]
+
+## Conventions
+- [Commit style if detectable]
+- [PR workflow if detectable]
+- [Error handling patterns]
+```
+
+## Best Practices
+
+1. **Don't read everything** — reconnaissance should use Glob and Grep, not Read on every file. Read selectively only for ambiguous signals.
+2. **Verify, don't guess** — if a framework is detected from config but the actual code uses something different, trust the code.
+3. **Respect existing CLAUDE.md** — if one already exists, enhance it rather than replacing it. Call out what's new vs existing.
+4. **Stay concise** — the onboarding guide should be scannable in 2 minutes. Details belong in the code, not the guide.
+5. **Flag unknowns** — if a convention can't be confidently detected, say so rather than guessing. "Could not determine test runner" is better than a wrong answer.
+
+## Anti-Patterns to Avoid
+
+- Generating a CLAUDE.md that's longer than 100 lines — keep it focused
+- Listing every dependency — highlight only the ones that shape how you write code
+- Describing obvious directory names — `src/` doesn't need an explanation
+- Copying the README — the onboarding guide adds structural insight the README lacks
+
+## Examples
+
+### Example 1: First time in a new repo
+**User**: "Onboard me to this codebase"
+**Action**: Run full 4-phase workflow → produce Onboarding Guide + Starter CLAUDE.md
+**Output**: Onboarding Guide printed directly to the conversation, plus a `CLAUDE.md` written to the project root
+
+### Example 2: Generate CLAUDE.md for existing project
+**User**: "Generate a CLAUDE.md for this project"
+**Action**: Run Phases 1-3, skip Onboarding Guide, produce only CLAUDE.md
+**Output**: Project-specific `CLAUDE.md` with detected conventions
+
+### Example 3: Enhance existing CLAUDE.md
+**User**: "Update the CLAUDE.md with current project conventions"
+**Action**: Read existing CLAUDE.md, run Phases 1-3, merge new findings
+**Output**: Updated `CLAUDE.md` with additions clearly marked