@pzy560117/codex-harness 0.1.4 → 0.1.6

package/package-source/docs/codex-harness-engineering/templates/package-assets/skills/clickhouse-io/SKILL.md

@@ -0,0 +1,429 @@
+---
+name: clickhouse-io
+description: ClickHouse database patterns, query optimization, analytics, and data engineering best practices for high-performance analytical workloads.
+---
+
+# ClickHouse Analytics Patterns
+
+ClickHouse-specific patterns for high-performance analytics and data engineering.
+
+## Overview
+
+ClickHouse is a column-oriented database management system (DBMS) for online analytical processing (OLAP). It's optimized for fast analytical queries on large datasets.
+
+**Key Features:**
+- Column-oriented storage
+- Data compression
+- Parallel query execution
+- Distributed queries
+- Real-time analytics
+
+## Table Design Patterns
+
+### MergeTree Engine (Most Common)
+
+```sql
+CREATE TABLE markets_analytics (
+    date Date,
+    market_id String,
+    market_name String,
+    volume UInt64,
+    trades UInt32,
+    unique_traders UInt32,
+    avg_trade_size Float64,
+    created_at DateTime
+) ENGINE = MergeTree()
+PARTITION BY toYYYYMM(date)
+ORDER BY (date, market_id)
+SETTINGS index_granularity = 8192;
+```
+
+### ReplacingMergeTree (Deduplication)
+
+```sql
+-- For data that may have duplicates (e.g., from multiple sources)
+CREATE TABLE user_events (
+    event_id String,
+    user_id String,
+    event_type String,
+    timestamp DateTime,
+    properties String
+) ENGINE = ReplacingMergeTree()
+PARTITION BY toYYYYMM(timestamp)
+ORDER BY (user_id, event_id, timestamp)
+PRIMARY KEY (user_id, event_id);
+```
+
+### AggregatingMergeTree (Pre-aggregation)
+
+```sql
+-- For maintaining aggregated metrics
+CREATE TABLE market_stats_hourly (
+    hour DateTime,
+    market_id String,
+    total_volume AggregateFunction(sum, UInt64),
+    total_trades AggregateFunction(count, UInt32),
+    unique_users AggregateFunction(uniq, String)
+) ENGINE = AggregatingMergeTree()
+PARTITION BY toYYYYMM(hour)
+ORDER BY (hour, market_id);
+
+-- Query aggregated data
+SELECT
+    hour,
+    market_id,
+    sumMerge(total_volume) AS volume,
+    countMerge(total_trades) AS trades,
+    uniqMerge(unique_users) AS users
+FROM market_stats_hourly
+WHERE hour >= toStartOfHour(now() - INTERVAL 24 HOUR)
+GROUP BY hour, market_id
+ORDER BY hour DESC;
+```
+
+## Query Optimization Patterns
+
+### Efficient Filtering
+
+```sql
+-- ✅ GOOD: Use indexed columns first
+SELECT *
+FROM markets_analytics
+WHERE date >= '2025-01-01'
+  AND market_id = 'market-123'
+  AND volume > 1000
+ORDER BY date DESC
+LIMIT 100;
+
+-- ❌ BAD: Filter on non-indexed columns first
+SELECT *
+FROM markets_analytics
+WHERE volume > 1000
+  AND market_name LIKE '%election%'
+  AND date >= '2025-01-01';
+```
+
+### Aggregations
+
+```sql
+-- ✅ GOOD: Use ClickHouse-specific aggregation functions
+SELECT
+    toStartOfDay(created_at) AS day,
+    market_id,
+    sum(volume) AS total_volume,
+    count() AS total_trades,
+    uniq(trader_id) AS unique_traders,
+    avg(trade_size) AS avg_size
+FROM trades
+WHERE created_at >= today() - INTERVAL 7 DAY
+GROUP BY day, market_id
+ORDER BY day DESC, total_volume DESC;
+
+-- ✅ Use quantile for percentiles (more efficient than percentile)
+SELECT
+    quantile(0.50)(trade_size) AS median,
+    quantile(0.95)(trade_size) AS p95,
+    quantile(0.99)(trade_size) AS p99
+FROM trades
+WHERE created_at >= now() - INTERVAL 1 HOUR;
+```
+
+### Window Functions
+
+```sql
+-- Calculate running totals
+SELECT
+    date,
+    market_id,
+    volume,
+    sum(volume) OVER (
+        PARTITION BY market_id
+        ORDER BY date
+        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
+    ) AS cumulative_volume
+FROM markets_analytics
+WHERE date >= today() - INTERVAL 30 DAY
+ORDER BY market_id, date;
+```
+
+## Data Insertion Patterns
+
+### Bulk Insert (Recommended)
+
+```typescript
+import { ClickHouse } from 'clickhouse'
+
+const clickhouse = new ClickHouse({
+  url: process.env.CLICKHOUSE_URL,
+  port: 8123,
+  basicAuth: {
+    username: process.env.CLICKHOUSE_USER,
+    password: process.env.CLICKHOUSE_PASSWORD
+  }
+})
+
+// ✅ Batch insert (efficient)
+async function bulkInsertTrades(trades: Trade[]) {
+  const values = trades.map(trade => `(
+    '${trade.id}',
+    '${trade.market_id}',
+    '${trade.user_id}',
+    ${trade.amount},
+    '${trade.timestamp.toISOString()}'
+  )`).join(',')
+
+  await clickhouse.query(`
+    INSERT INTO trades (id, market_id, user_id, amount, timestamp)
+    VALUES ${values}
+  `).toPromise()
+}
+
+// ❌ Individual inserts (slow)
+async function insertTrade(trade: Trade) {
+  // Don't do this in a loop!
+  await clickhouse.query(`
+    INSERT INTO trades VALUES ('${trade.id}', ...)
+  `).toPromise()
+}
+```
+
+### Streaming Insert
+
+```typescript
+// For continuous data ingestion
+import { createWriteStream } from 'fs'
+import { pipeline } from 'stream/promises'
+
+async function streamInserts() {
+  const stream = clickhouse.insert('trades').stream()
+
+  for await (const batch of dataSource) {
+    stream.write(batch)
+  }
+
+  await stream.end()
+}
+```
+
+## Materialized Views
+
+### Real-time Aggregations
+
+```sql
+-- Create materialized view for hourly stats
+CREATE MATERIALIZED VIEW market_stats_hourly_mv
+TO market_stats_hourly
+AS SELECT
+    toStartOfHour(timestamp) AS hour,
+    market_id,
+    sumState(amount) AS total_volume,
+    countState() AS total_trades,
+    uniqState(user_id) AS unique_users
+FROM trades
+GROUP BY hour, market_id;
+
+-- Query the materialized view
+SELECT
+    hour,
+    market_id,
+    sumMerge(total_volume) AS volume,
+    countMerge(total_trades) AS trades,
+    uniqMerge(unique_users) AS users
+FROM market_stats_hourly
+WHERE hour >= now() - INTERVAL 24 HOUR
+GROUP BY hour, market_id;
+```
+
+## Performance Monitoring
+
+### Query Performance
+
+```sql
+-- Check slow queries
+SELECT
+    query_id,
+    user,
+    query,
+    query_duration_ms,
+    read_rows,
+    read_bytes,
+    memory_usage
+FROM system.query_log
+WHERE type = 'QueryFinish'
+  AND query_duration_ms > 1000
+  AND event_time >= now() - INTERVAL 1 HOUR
+ORDER BY query_duration_ms DESC
+LIMIT 10;
+```
+
+### Table Statistics
+
+```sql
+-- Check table sizes
+SELECT
+    database,
+    table,
+    formatReadableSize(sum(bytes)) AS size,
+    sum(rows) AS rows,
+    max(modification_time) AS latest_modification
+FROM system.parts
+WHERE active
+GROUP BY database, table
+ORDER BY sum(bytes) DESC;
+```
+
+## Common Analytics Queries
+
+### Time Series Analysis
+
+```sql
+-- Daily active users
+SELECT
+    toDate(timestamp) AS date,
+    uniq(user_id) AS daily_active_users
+FROM events
+WHERE timestamp >= today() - INTERVAL 30 DAY
+GROUP BY date
+ORDER BY date;
+
+-- Retention analysis
+SELECT
+    signup_date,
+    countIf(days_since_signup = 0) AS day_0,
+    countIf(days_since_signup = 1) AS day_1,
+    countIf(days_since_signup = 7) AS day_7,
+    countIf(days_since_signup = 30) AS day_30
+FROM (
+    SELECT
+        user_id,
+        min(toDate(timestamp)) AS signup_date,
+        toDate(timestamp) AS activity_date,
+        dateDiff('day', signup_date, activity_date) AS days_since_signup
+    FROM events
+    GROUP BY user_id, activity_date
+)
+GROUP BY signup_date
+ORDER BY signup_date DESC;
+```
+
+### Funnel Analysis
+
+```sql
+-- Conversion funnel
+SELECT
+    countIf(step = 'viewed_market') AS viewed,
+    countIf(step = 'clicked_trade') AS clicked,
+    countIf(step = 'completed_trade') AS completed,
+    round(clicked / viewed * 100, 2) AS view_to_click_rate,
+    round(completed / clicked * 100, 2) AS click_to_completion_rate
+FROM (
+    SELECT
+        user_id,
+        session_id,
+        event_type AS step
+    FROM events
+    WHERE event_date = today()
+)
+GROUP BY session_id;
+```
+
+### Cohort Analysis
+
+```sql
+-- User cohorts by signup month
+SELECT
+    toStartOfMonth(signup_date) AS cohort,
+    toStartOfMonth(activity_date) AS month,
+    dateDiff('month', cohort, month) AS months_since_signup,
+    count(DISTINCT user_id) AS active_users
+FROM (
+    SELECT
+        user_id,
+        min(toDate(timestamp)) OVER (PARTITION BY user_id) AS signup_date,
+        toDate(timestamp) AS activity_date
+    FROM events
+)
+GROUP BY cohort, month, months_since_signup
+ORDER BY cohort, months_since_signup;
+```
+
+## Data Pipeline Patterns
+
+### ETL Pattern
+
+```typescript
+// Extract, Transform, Load
+async function etlPipeline() {
+  // 1. Extract from source
+  const rawData = await extractFromPostgres()
+
+  // 2. Transform
+  const transformed = rawData.map(row => ({
+    date: new Date(row.created_at).toISOString().split('T')[0],
+    market_id: row.market_slug,
+    volume: parseFloat(row.total_volume),
+    trades: parseInt(row.trade_count)
+  }))
+
+  // 3. Load to ClickHouse
+  await bulkInsertToClickHouse(transformed)
+}
+
+// Run periodically
+setInterval(etlPipeline, 60 * 60 * 1000)  // Every hour
+```
+
+### Change Data Capture (CDC)
+
+```typescript
+// Listen to PostgreSQL changes and sync to ClickHouse
+import { Client } from 'pg'
+
+const pgClient = new Client({ connectionString: process.env.DATABASE_URL })
+
+pgClient.query('LISTEN market_updates')
+
+pgClient.on('notification', async (msg) => {
+  const update = JSON.parse(msg.payload)
+
+  await clickhouse.insert('market_updates', [
+    {
+      market_id: update.id,
+      event_type: update.operation,  // INSERT, UPDATE, DELETE
+      timestamp: new Date(),
+      data: JSON.stringify(update.new_data)
+    }
+  ])
+})
+```
+
+## Best Practices
+
+### 1. Partitioning Strategy
+- Partition by time (usually month or day)
+- Avoid too many partitions (performance impact)
+- Use DATE type for partition key
+
+### 2. Ordering Key
+- Put most frequently filtered columns first
+- Consider cardinality (high cardinality first)
+- Order impacts compression
+
+### 3. Data Types
+- Use smallest appropriate type (UInt32 vs UInt64)
+- Use LowCardinality for repeated strings
+- Use Enum for categorical data
+
+### 4. Avoid
+- SELECT * (specify columns)
+- FINAL (merge data before query instead)
+- Too many JOINs (denormalize for analytics)
+- Small frequent inserts (batch instead)
+
+### 5. Monitoring
+- Track query performance
+- Monitor disk usage
+- Check merge operations
+- Review slow query log
+
+**Remember**: ClickHouse excels at analytical workloads. Design tables for your query patterns, batch inserts, and leverage materialized views for real-time aggregations.

package/package-source/docs/codex-harness-engineering/templates/package-assets/skills/code-review/SKILL.md

@@ -0,0 +1,171 @@
+---
+name: code-review
+description: Code Review Skill
+---
+
+# Code Review Skill
+
+代码审查清单和 PR 规范。
+
+## PR 提交规范
+
+### PR 标题格式
+```
+<type>(<scope>): <description>
+
+示例:
+feat(auth): 添加 OAuth2.0 登录功能
+fix(api): 修复用户列表分页问题
+refactor(db): 优化查询性能
+```
+
+### PR 描述模板
+```markdown
+## 变更说明
+简要描述本次变更的内容和目的
+
+## 变更类型
+- [ ] 新功能
+- [ ] Bug 修复
+- [ ] 重构
+- [ ] 性能优化
+- [ ] 文档更新
+
+## 关联 Issue
+Closes #123
+
+## 测试说明
+- [ ] 单元测试通过
+- [ ] 集成测试通过
+- [ ] 手动测试通过
+
+## 截图（如适用）
+UI 变更请附截图
+
+## 检查清单
+- [ ] 代码符合项目规范
+- [ ] 已添加必要的测试
+- [ ] 已更新相关文档
+- [ ] 无敏感信息泄露
+```
+
+## 代码审查清单
+
+### 功能正确性
+- [ ] 代码实现了预期功能
+- [ ] 边界条件处理正确
+- [ ] 错误处理完善
+- [ ] 无明显的逻辑漏洞
+
+### 代码质量
+- [ ] 命名清晰有意义
+- [ ] 函数职责单一
+- [ ] 无重复代码
+- [ ] 复杂逻辑有注释
+- [ ] 无硬编码的魔法数字
+
+### 安全性
+- [ ] 输入验证完整
+- [ ] 无 SQL 注入风险
+- [ ] 无 XSS 风险
+- [ ] 敏感信息已脱敏
+- [ ] 权限检查到位
+
+### 性能
+- [ ] 无 N+1 查询
+- [ ] 无不必要的循环
+- [ ] 大数据量有分页
+- [ ] 耗时操作有缓存或异步
+
+### 可维护性
+- [ ] 代码结构清晰
+- [ ] 依赖关系合理
+- [ ] 易于测试
+- [ ] 易于扩展
+
+## 审查反馈规范
+
+### 评论分类
+```
+[必须] 必须修改才能合并
+[建议] 建议修改，可讨论
+[疑问] 需要解释或讨论
+[赞] 写得好的地方
+```
+
+### 反馈示例
+```
+[必须] 这里存在 SQL 注入风险，请使用参数化查询
+
+[建议] 这个函数超过 50 行，建议拆分为更小的函数
+
+[疑问] 为什么这里用 setTimeout 而不是 debounce？
+
+[赞] 这个抽象设计得很好，复用性强
+```
+
+## 审查流程
+
+```
+┌─────────────┐
+│  提交 PR    │
+└──────┬──────┘
+       ▼
+┌─────────────┐
+│  CI 检查    │ ← 自动化测试、lint、构建
+└──────┬──────┘
+       ▼
+┌─────────────┐
+│  代码审查   │ ← 至少 1 人 approve
+└──────┬──────┘
+       ▼
+┌─────────────┐
+│  修改反馈   │ ← 根据评论修改
+└──────┬──────┘
+       ▼
+┌─────────────┐
+│  合并代码   │ ← Squash merge
+└─────────────┘
+```
+
+## 审查原则
+
+### 审查者
+- 及时响应（24小时内）
+- 提供建设性反馈
+- 尊重作者的设计决策
+- 区分必须修改和建议修改
+
+### 作者
+- PR 尽量小（<400行）
+- 提供充分的上下文
+- 积极回应反馈
+- 不要 defensive
+
+## 自动化检查
+
+```yaml
+# GitHub Actions 示例
+name: PR Check
+
+on: [pull_request]
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Lint
+        run: npm run lint
+        
+      - name: Type Check
+        run: npm run type-check
+        
+      - name: Test
+        run: npm run test
+        
+      - name: Build
+        run: npm run build
+```
+

package/package-source/docs/codex-harness-engineering/templates/package-assets/skills/code-reviewer/SKILL.md

@@ -0,0 +1,104 @@
+---
+name: code-reviewer
+description: Expert code review specialist. Proactively reviews code for quality, security, and maintainability. Use immediately after writing or modifying code. MUST BE USED for all code changes.
+tools: Read, Grep, Glob, Bash
+model: opus
+---
+
+You are a senior code reviewer ensuring high standards of code quality and security.
+
+When invoked:
+1. Run git diff to see recent changes
+2. Focus on modified files
+3. Begin review immediately
+
+Review checklist:
+- Code is simple and readable
+- Functions and variables are well-named
+- No duplicated code
+- Proper error handling
+- No exposed secrets or API keys
+- Input validation implemented
+- Good test coverage
+- Performance considerations addressed
+- Time complexity of algorithms analyzed
+- Licenses of integrated libraries checked
+
+Provide feedback organized by priority:
+- Critical issues (must fix)
+- Warnings (should fix)
+- Suggestions (consider improving)
+
+Include specific examples of how to fix issues.
+
+## Security Checks (CRITICAL)
+
+- Hardcoded credentials (API keys, passwords, tokens)
+- SQL injection risks (string concatenation in queries)
+- XSS vulnerabilities (unescaped user input)
+- Missing input validation
+- Insecure dependencies (outdated, vulnerable)
+- Path traversal risks (user-controlled file paths)
+- CSRF vulnerabilities
+- Authentication bypasses
+
+## Code Quality (HIGH)
+
+- Large functions (>50 lines)
+- Large files (>800 lines)
+- Deep nesting (>4 levels)
+- Missing error handling (try/catch)
+- console.log statements
+- Mutation patterns
+- Missing tests for new code
+
+## Performance (MEDIUM)
+
+- Inefficient algorithms (O(n²) when O(n log n) possible)
+- Unnecessary re-renders in React
+- Missing memoization
+- Large bundle sizes
+- Unoptimized images
+- Missing caching
+- N+1 queries
+
+## Best Practices (MEDIUM)
+
+- Emoji usage in code/comments
+- TODO/FIXME without tickets
+- Missing JSDoc for public APIs
+- Accessibility issues (missing ARIA labels, poor contrast)
+- Poor variable naming (x, tmp, data)
+- Magic numbers without explanation
+- Inconsistent formatting
+
+## Review Output Format
+
+For each issue:
+```
+[CRITICAL] Hardcoded API key
+File: src/api/client.ts:42
+Issue: API key exposed in source code
+Fix: Move to environment variable
+
+const apiKey = "sk-abc123";  // ❌ Bad
+const apiKey = process.env.API_KEY;  // ✓ Good
+```
+
+## Approval Criteria
+
+- ✅ Approve: No CRITICAL or HIGH issues
+- ⚠️ Warning: MEDIUM issues only (can merge with caution)
+- ❌ Block: CRITICAL or HIGH issues found
+
+## Project-Specific Guidelines (Example)
+
+Add your project-specific checks here. Examples:
+- Follow MANY SMALL FILES principle (200-400 lines typical)
+- No emojis in codebase
+- Use immutability patterns (spread operator)
+- Verify database RLS policies
+- Check AI integration error handling
+- Validate cache fallback behavior
+
+Customize based on your project's `CLAUDE.md` or skill files.