aigroup-workflow 2.1.1 → 2.2.0

package/skills/security-reviewer/references/report-template.md

@@ -1,170 +0,0 @@
-# Security Report Template
-
-## Full Report Template
-
-```markdown
-# Security Review Report
-
-## Executive Summary
-
-| Field | Value |
-|-------|-------|
-| **Application** | [Application Name] |
-| **Review Date** | [YYYY-MM-DD] |
-| **Reviewer** | [Name] |
-| **Scope** | [Files/modules reviewed] |
-| **Overall Risk Level** | [Critical/High/Medium/Low] |
-
-### Key Findings
-- X Critical vulnerabilities requiring immediate attention
-- Y High-severity issues to address before deployment
-- Z Medium/Low issues for future consideration
-
-## Findings Summary
-
-| Severity | Count | Status |
-|----------|-------|--------|
-| Critical | X | Requires immediate fix |
-| High | X | Fix before deployment |
-| Medium | X | Fix in next sprint |
-| Low | X | Backlog |
-
-## Detailed Findings
-
-### [CRITICAL] SQL Injection in User Search
-
-| Field | Value |
-|-------|-------|
-| **ID** | SEC-001 |
-| **Location** | `src/api/users.ts:45` |
-| **CWE** | CWE-89 |
-| **CVSS** | 9.8 (Critical) |
-
-**Description**
-User input directly concatenated into SQL query without sanitization.
-
-**Vulnerable Code**
-```typescript
-const query = `SELECT * FROM users WHERE name LIKE '%${searchTerm}%'`;
-```
-
-**Proof of Concept**
-```
-GET /api/users?search=' OR '1'='1
-```
-
-**Impact**
-- Full database access
-- Data exfiltration
-- Data modification/deletion
-- Potential RCE via SQL features
-
-**Remediation**
-Use parameterized queries:
-```typescript
-const query = 'SELECT * FROM users WHERE name LIKE $1';
-db.query(query, [`%${searchTerm}%`]);
-```
-
-**Effort**: 1 hour
-**Priority**: Immediate
-
----
-
-### [HIGH] Weak Password Requirements
-
-| Field | Value |
-|-------|-------|
-| **ID** | SEC-002 |
-| **Location** | `src/auth/validation.ts:12` |
-| **CWE** | CWE-521 |
-| **CVSS** | 7.5 (High) |
-
-**Description**
-Password policy requires only 6 characters with no complexity requirements.
-
-**Current Policy**
-```typescript
-const isValid = password.length >= 6;
-```
-
-**Impact**
-- Susceptible to brute force attacks
-- Dictionary attack vulnerability
-
-**Remediation**
-Implement stronger requirements:
-```typescript
-const isValid =
-  password.length >= 12 &&
-  /[A-Z]/.test(password) &&
-  /[a-z]/.test(password) &&
-  /[0-9]/.test(password) &&
-  /[^A-Za-z0-9]/.test(password);
-```
-
-**Effort**: 30 minutes
-**Priority**: Before deployment
-
-## Automated Scan Results
-
-### Dependency Vulnerabilities
-| Package | Severity | CVE | Fix |
-|---------|----------|-----|-----|
-| lodash | High | CVE-2021-xxxx | Upgrade to 4.17.21 |
-
-### SAST Findings
-| Tool | Critical | High | Medium | Low |
-|------|----------|------|--------|-----|
-| Semgrep | 1 | 3 | 5 | 8 |
-| npm audit | 0 | 2 | 4 | 10 |
-
-## Recommendations
-
-### Immediate (This Sprint)
-1. Fix SQL injection vulnerability (SEC-001)
-2. Implement parameterized queries globally
-3. Update vulnerable dependencies
-
-### Short-term (Next Sprint)
-1. Strengthen password policy (SEC-002)
-2. Add input validation middleware
-3. Enable security headers
-
-### Long-term
-1. Implement SAST in CI/CD pipeline
-2. Schedule regular security reviews
-3. Security training for developers
-
-## Appendix
-
-### Tools Used
-- Semgrep v1.x
-- npm audit
-- Gitleaks v8.x
-- Manual code review
-
-### References
-- OWASP Top 10 2021
-- CWE Database
-- CVSS Calculator
-```
-
-## Severity Definitions
-
-| Severity | CVSS Score | Response Time |
-|----------|------------|---------------|
-| Critical | 9.0 - 10.0 | Immediate |
-| High | 7.0 - 8.9 | 24-48 hours |
-| Medium | 4.0 - 6.9 | 1-2 weeks |
-| Low | 0.1 - 3.9 | Next release |
-
-## Quick Reference
-
-| Section | Purpose |
-|---------|---------|
-| Executive Summary | Management overview |
-| Findings Summary | Quick count by severity |
-| Detailed Findings | Technical details |
-| Scan Results | Automated tool output |
-| Recommendations | Prioritized action items |

package/skills/security-reviewer/references/sast-tools.md

@@ -1,117 +0,0 @@
-# SAST Tools
-
-## JavaScript/TypeScript
-
-```bash
-# Dependency vulnerabilities
-npm audit
-npm audit --json > npm-audit.json
-
-# ESLint security plugin
-npm install eslint-plugin-security --save-dev
-npx eslint --ext .js,.ts . --plugin security
-
-# Snyk
-npx snyk test
-npx snyk code test
-```
-
-## Python
-
-```bash
-# Bandit - Python SAST
-pip install bandit
-bandit -r . -f json -o bandit-report.json
-bandit -r . -ll  # Only high severity
-
-# Safety - Dependency check
-pip install safety
-safety check
-safety check -r requirements.txt --json > safety-report.json
-
-# Pyup Safety
-pip install pyupio-safety
-pyupio-safety check
-```
-
-## Go
-
-```bash
-# GoSec - Go security checker
-go install github.com/securego/gosec/v2/cmd/gosec@latest
-gosec ./...
-gosec -fmt=json -out=gosec-report.json ./...
-
-# Go vulnerability database
-go install golang.org/x/vuln/cmd/govulncheck@latest
-govulncheck ./...
-```
-
-## Multi-Language Tools
-
-```bash
-# Semgrep - Universal SAST
-pip install semgrep
-semgrep --config=auto .
-semgrep --config=p/security-audit .
-semgrep --config=p/owasp-top-ten .
-
-# Trivy - Comprehensive scanner
-brew install trivy
-trivy fs .
-trivy fs --security-checks vuln,secret,config .
-
-# SonarQube (requires server)
-sonar-scanner -Dsonar.projectKey=myproject
-```
-
-## CI/CD Integration
-
-### GitHub Actions
-
-```yaml
-- name: Run Semgrep
-  uses: returntocorp/semgrep-action@v1
-  with:
-    config: p/security-audit
-
-- name: Run npm audit
-  run: npm audit --audit-level=high
-
-- name: Run Trivy
-  uses: aquasecurity/trivy-action@master
-  with:
-    scan-type: 'fs'
-    severity: 'CRITICAL,HIGH'
-```
-
-### GitLab CI
-
-```yaml
-security-scan:
-  image: returntocorp/semgrep
-  script:
-    - semgrep --config=auto --json -o semgrep.json .
-  artifacts:
-    reports:
-      sast: semgrep.json
-```
-
-## Quick Reference
-
-| Language | Primary Tool | Dependency Check |
-|----------|--------------|------------------|
-| JavaScript | ESLint + security | npm audit |
-| TypeScript | ESLint + security | npm audit |
-| Python | Bandit | Safety |
-| Go | GoSec | govulncheck |
-| Java | SpotBugs | OWASP Dependency-Check |
-| Ruby | Brakeman | bundler-audit |
-
-| Tool | Strengths | Best For |
-|------|-----------|----------|
-| Semgrep | Multi-language, custom rules | General SAST |
-| Trivy | Container + code + secrets | Comprehensive |
-| Bandit | Python-specific | Python projects |
-| GoSec | Go-specific | Go projects |
-| npm audit | Built-in, fast | Node.js deps |

package/skills/security-reviewer/references/secret-scanning.md

@@ -1,125 +0,0 @@
-# Secret Scanning
-
-## Gitleaks
-
-```bash
-# Install
-brew install gitleaks
-
-# Scan current directory
-gitleaks detect --source . --verbose
-
-# Scan with report
-gitleaks detect --source . -f json -r gitleaks-report.json
-
-# Scan git history
-gitleaks detect --source . --log-opts="--all"
-
-# Use baseline (ignore known)
-gitleaks detect --baseline-path .gitleaks-baseline.json
-```
-
-## TruffleHog
-
-```bash
-# Install
-pip install trufflehog
-
-# Scan filesystem
-trufflehog filesystem .
-
-# Scan git repo
-trufflehog git file://. --since-commit HEAD~100
-
-# Scan with JSON output
-trufflehog filesystem . --json > trufflehog-report.json
-```
-
-## Manual Grep Patterns
-
-```bash
-# Common secret patterns
-grep -rn "api_key\|apikey\|api-key" --include="*.{ts,js,py}" .
-grep -rn "secret\|password\|passwd" --include="*.{ts,js,py}" .
-grep -rn "private_key\|privatekey" --include="*.{ts,js,py}" .
-grep -rn "access_token\|accesstoken" --include="*.{ts,js,py}" .
-
-# AWS credentials
-grep -rn "AKIA[0-9A-Z]{16}" .
-grep -rn "aws_secret_access_key" .
-
-# Base64 encoded (potential secrets)
-grep -rn "[A-Za-z0-9+/]{40,}=" .
-
-# JWT tokens
-grep -rn "eyJ[A-Za-z0-9_-]*\.eyJ[A-Za-z0-9_-]*\." .
-```
-
-## Common Secret Patterns
-
-| Type | Pattern | Example |
-|------|---------|---------|
-| AWS Access Key | `AKIA[0-9A-Z]{16}` | AKIAIOSFODNN7EXAMPLE |
-| AWS Secret Key | 40 char base64 | wJalrXUtnFEMI/K7MDENG... |
-| GitHub Token | `ghp_[A-Za-z0-9]{36}` | ghp_xxxxxxxxxxxx |
-| Slack Token | `xox[baprs]-` | xoxb-xxx-xxx |
-| Stripe Key | `sk_live_[A-Za-z0-9]{24}` | sk_live_xxxx |
-| Private Key | `-----BEGIN.*PRIVATE KEY-----` | RSA/EC keys |
-| JWT | `eyJ[A-Za-z0-9_-]*\.eyJ` | Encoded tokens |
-
-## Pre-commit Hook
-
-```yaml
-# .pre-commit-config.yaml
-repos:
-  - repo: https://github.com/gitleaks/gitleaks
-    rev: v8.18.0
-    hooks:
-      - id: gitleaks
-```
-
-## CI/CD Integration
-
-```yaml
-# GitHub Actions
-- name: Gitleaks
-  uses: gitleaks/gitleaks-action@v2
-  env:
-    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
-# GitLab CI
-secret_detection:
-  image: zricethezav/gitleaks
-  script:
-    - gitleaks detect --source . -f sarif -r gl-secret-detection-report.sarif
-  artifacts:
-    reports:
-      secret_detection: gl-secret-detection-report.sarif
-```
-
-## Remediation Steps
-
-1. **Rotate immediately** - Consider secret compromised
-2. **Remove from history** - Use git filter-branch or BFG
-3. **Add to .gitignore** - Prevent future commits
-4. **Use env variables** - Move to environment
-5. **Use secret manager** - AWS Secrets Manager, Vault
-
-```bash
-# Remove from git history (BFG)
-bfg --replace-text passwords.txt repo.git
-
-# Or git filter-branch
-git filter-branch --force --index-filter \
-  "git rm --cached --ignore-unmatch path/to/secret" \
-  --prune-empty --tag-name-filter cat -- --all
-```
-
-## Quick Reference
-
-| Tool | Best For | Speed |
-|------|----------|-------|
-| Gitleaks | Git history | Fast |
-| TruffleHog | Deep scanning | Medium |
-| grep | Quick checks | Fast |
-| GitHub Secret Scanning | GitHub repos | Auto |

package/skills/security-reviewer/references/vulnerability-patterns.md

@@ -1,152 +0,0 @@
-# Vulnerability Patterns
-
-## SQL Injection
-
-```typescript
-// VULNERABLE
-const query = `SELECT * FROM users WHERE id = ${userId}`;
-const query = `SELECT * FROM users WHERE name = '${name}'`;
-
-// SECURE - Parameterized queries
-const query = 'SELECT * FROM users WHERE id = $1';
-db.query(query, [userId]);
-
-// SECURE - ORM
-const user = await User.findOne({ where: { id: userId } });
-```
-
-## XSS (Cross-Site Scripting)
-
-```typescript
-// VULNERABLE - Direct HTML injection
-element.innerHTML = userInput;
-document.write(userInput);
-
-// SECURE - Text content
-element.textContent = userInput;
-
-// SECURE - Sanitization
-import DOMPurify from 'dompurify';
-element.innerHTML = DOMPurify.sanitize(userInput);
-
-// SECURE - React (auto-escaped)
-return <div>{userInput}</div>;
-
-// VULNERABLE - React dangerouslySetInnerHTML
-return <div dangerouslySetInnerHTML={{ __html: userInput }} />;
-```
-
-## Path Traversal
-
-```typescript
-// VULNERABLE
-const file = path.join(uploadDir, req.query.filename);
-res.sendFile(file);
-
-// SECURE - Validate and normalize
-const filename = path.basename(req.query.filename);
-const file = path.resolve(uploadDir, filename);
-
-if (!file.startsWith(path.resolve(uploadDir))) {
-  throw new Error('Invalid path');
-}
-res.sendFile(file);
-```
-
-## Command Injection
-
-```typescript
-// VULNERABLE
-exec(`ls ${userInput}`);
-exec('git clone ' + repoUrl);
-
-// SECURE - Use arrays, avoid shell
-execFile('ls', [userInput]);
-spawn('git', ['clone', repoUrl]);
-
-// SECURE - Validation
-const allowedCommands = ['status', 'log'];
-if (!allowedCommands.includes(cmd)) throw new Error('Invalid');
-```
-
-## IDOR (Insecure Direct Object Reference)
-
-```typescript
-// VULNERABLE - No authorization check
-app.get('/documents/:id', async (req, res) => {
-  const doc = await Document.findById(req.params.id);
-  res.json(doc);
-});
-
-// SECURE - Verify ownership
-app.get('/documents/:id', async (req, res) => {
-  const doc = await Document.findOne({
-    _id: req.params.id,
-    userId: req.user.id  // Ensure user owns document
-  });
-  if (!doc) return res.status(404).json({ error: 'Not found' });
-  res.json(doc);
-});
-```
-
-## Insecure Deserialization
-
-```typescript
-// VULNERABLE - Python pickle
-import pickle
-data = pickle.loads(user_input)
-
-// SECURE - Use JSON
-import json
-data = json.loads(user_input)
-
-// VULNERABLE - Node.js
-const obj = eval('(' + userInput + ')');
-
-// SECURE
-const obj = JSON.parse(userInput);
-```
-
-## Sensitive Data Exposure
-
-```typescript
-// VULNERABLE - Logging sensitive data
-logger.info('User login', { email, password });
-console.log('Token:', authToken);
-
-// SECURE - Redact sensitive fields
-logger.info('User login', { email, password: '[REDACTED]' });
-
-// VULNERABLE - Error response exposes internals
-res.status(500).json({ error: err.stack });
-
-// SECURE - Generic error
-res.status(500).json({ error: 'Internal server error' });
-```
-
-## Quick Reference
-
-| Vulnerability | Input Vector | Prevention |
-|---------------|--------------|------------|
-| SQL Injection | Query params | Parameterized queries |
-| XSS | User content | Output encoding |
-| Path Traversal | File paths | path.basename + validation |
-| Command Injection | Shell args | execFile, no shell |
-| IDOR | Resource IDs | Authorization checks |
-| Deserialization | Serialized data | JSON only |
-| Data Exposure | Logs, errors | Redaction, generic errors |
-
-## OWASP Top 10 Mapping
-
-| OWASP | Vulnerabilities |
-|-------|-----------------|
-| A01 Broken Access Control | IDOR, path traversal |
-| A02 Cryptographic Failures | Weak encryption, plaintext |
-| A03 Injection | SQL, XSS, command |
-| A04 Insecure Design | Missing auth, IDOR |
-| A05 Security Misconfiguration | Debug mode, default creds |
-| A06 Vulnerable Components | Outdated dependencies |
-| A07 Auth Failures | Weak passwords, session issues |
-| A08 Data Integrity | Insecure deserialization |
-| A09 Logging Failures | Missing logs, exposed data |
-| A10 SSRF | Unvalidated URLs |

package/skills/spark-engineer/SKILL.md

@@ -1,148 +0,0 @@
----
-name: spark-engineer
-description: Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
-license: MIT
-metadata:
-  author: https://github.com/Jeffallan
-  version: "1.1.0"
-  domain: data-ml
-  triggers: Apache Spark, PySpark, Spark SQL, distributed computing, big data, DataFrame API, RDD, Spark Streaming, structured streaming, data partitioning, Spark performance, cluster computing, data processing pipeline
-  role: expert
-  scope: implementation
-  output-format: code
-  related-skills: python-pro, sql-pro, devops-engineer
----
-
-# Spark Engineer
-
-Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.
-
-## Core Workflow
-
-1. **Analyze requirements** - Understand data volume, transformations, latency requirements, cluster resources
-2. **Design pipeline** - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
-3. **Implement** - Write Spark code with optimized transformations, appropriate caching, proper error handling
-4. **Optimize** - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
-5. **Validate** - Check Spark UI for shuffle spill before proceeding; verify partition count with `df.rdd.getNumPartitions()`; if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets
-
-## Reference Guide
-
-Load detailed guidance based on context:
-
-| Topic | Reference | Load When |
-|-------|-----------|-----------|
-| Spark SQL & DataFrames | `references/spark-sql-dataframes.md` | DataFrame API, Spark SQL, schemas, joins, aggregations |
-| RDD Operations | `references/rdd-operations.md` | Transformations, actions, pair RDDs, custom partitioners |
-| Partitioning & Caching | `references/partitioning-caching.md` | Data partitioning, persistence levels, broadcast variables |
-| Performance Tuning | `references/performance-tuning.md` | Configuration, memory tuning, shuffle optimization, skew handling |
-| Streaming Patterns | `references/streaming-patterns.md` | Structured Streaming, watermarks, stateful operations, sinks |
-
-## Code Examples
-
-### Quick-Start Mini-Pipeline (PySpark)
-
-```python
-from pyspark.sql import SparkSession
-from pyspark.sql import functions as F
-from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType
-
-spark = SparkSession.builder \
-    .appName("example-pipeline") \
-    .config("spark.sql.shuffle.partitions", "400") \
-    .config("spark.sql.adaptive.enabled", "true") \
-    .getOrCreate()
-
-# Always define explicit schemas in production
-schema = StructType([
-    StructField("user_id", StringType(), False),
-    StructField("event_ts", LongType(), False),
-    StructField("amount", DoubleType(), True),
-])
-
-df = spark.read.schema(schema).parquet("s3://bucket/events/")
-
-result = df \
-    .filter(F.col("amount").isNotNull()) \
-    .groupBy("user_id") \
-    .agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))
-
-# Verify partition count before writing
-print(f"Partition count: {result.rdd.getNumPartitions()}")
-
-result.write.mode("overwrite").parquet("s3://bucket/output/")
-```
-
-### Broadcast Join (small dimension table < 200 MB)
-
-```python
-from pyspark.sql.functions import broadcast
-
-# Spark will automatically broadcast dim_table; hint makes intent explicit
-enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")
-```
-
-### Handling Data Skew with Salting
-
-```python
-import pyspark.sql.functions as F
-
-SALT_BUCKETS = 50
-
-# Add salt to the skewed key on both sides
-skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \
-    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
-
-other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \
-    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
-
-result = skewed_df.join(other_df, on="salted_key", how="inner") \
-    .drop("salt", "salted_key")
-```
-
-### Correct Caching Pattern
-
-```python
-# Cache ONLY when the DataFrame is reused multiple times
-df_cleaned = df.filter(...).withColumn(...).cache()
-df_cleaned.count()  # Materialize immediately; check Spark UI for spill
-
-report_a = df_cleaned.groupBy("region").agg(...)
-report_b = df_cleaned.groupBy("product").agg(...)
-
-df_cleaned.unpersist()  # Release when done
-```
-
-## Constraints
-
-### MUST DO
-- Use DataFrame API over RDD for structured data processing
-- Define explicit schemas for production pipelines
-- Partition data appropriately (200-1000 partitions per executor core)
-- Cache intermediate results only when reused multiple times
-- Use broadcast joins for small dimension tables (<200MB)
-- Handle data skew with salting or custom partitioning
-- Monitor Spark UI for shuffle, spill, and GC metrics
-- Test with production-scale data volumes
-
-### MUST NOT DO
-- Use collect() on large datasets (causes OOM)
-- Skip schema definition and rely on inference in production
-- Cache every DataFrame without measuring benefit
-- Ignore shuffle partition tuning (default 200 often wrong)
-- Use UDFs when built-in functions available (10-100x slower)
-- Process small files without coalescing (small file problem)
-- Run transformations without understanding lazy evaluation
-- Ignore data skew warnings in Spark UI
-
-## Output Templates
-
-When implementing Spark solutions, provide:
-1. Complete Spark code (PySpark or Scala) with type hints/types
-2. Configuration recommendations (executors, memory, shuffle partitions)
-3. Partitioning strategy explanation
-4. Performance analysis (expected shuffle size, memory usage)
-5. Monitoring recommendations (key Spark UI metrics to watch)
-
-## Knowledge Reference
-
-Spark DataFrame API, Spark SQL, RDD transformations/actions, catalyst optimizer, tungsten execution engine, partitioning strategies, broadcast variables, accumulators, structured streaming, watermarks, checkpointing, Spark UI analysis, memory management, shuffle optimization