everything-gemini 1.2.0 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/ISSUE_TEMPLATE/bug_report.md +29 -0
- package/.github/ISSUE_TEMPLATE/feature_request.md +27 -0
- package/.github/ISSUE_TEMPLATE/subagent_proposal.md +29 -0
- package/.github/workflows/ci.yml +72 -1
- package/.github/workflows/release.yml +17 -2
- package/CHANGELOG.md +155 -0
- package/CONTRIBUTING.md +85 -4
- package/README.md +134 -19
- package/action.yml +35 -0
- package/agents/browser-qa.md +1 -1
- package/agents/database-migrator.md +26 -0
- package/agents/documentation-writer.md +27 -0
- package/agents/incident-responder.md +34 -0
- package/agents/performance-tuner.md +27 -0
- package/agents/release-engineer.md +34 -0
- package/bin/everything-gemini +1517 -122
- package/docs/COMMUNITY_SUBMISSIONS.md +88 -0
- package/docs/LAUNCH_KIT.md +138 -13
- package/docs/architecture/progressive-disclosure.md +3 -3
- package/docs/index.html +401 -40
- package/docs/launch-kit.md +163 -0
- package/examples/cli-workflows/demo_workflow.sh +15 -7
- package/examples/python-sdk/README.md +28 -5
- package/examples/python-sdk/documentation_audit_pipeline.py +83 -0
- package/examples/python-sdk/incident_triage_pipeline.py +170 -0
- package/examples/python-sdk/multi_agent_pipeline.py +95 -0
- package/examples/python-sdk/release_readiness_audit.py +144 -0
- package/hooks/secret-scrubber/hooks.json +16 -0
- package/hooks/secret-scrubber/scripts/scrub-secrets.sh +62 -0
- package/mcp/README.md +20 -0
- package/mcp/mcp_config.example.json +35 -0
- package/package.json +3 -1
- package/plugins/core/rules/core-practices.md +63 -4
- package/plugins/python-ai/rules/ai-agent-patterns.md +67 -4
- package/plugins/security-shield/rules/zero-trust.md +60 -4
- package/plugins/web-fullstack/rules/frontend-standards.md +74 -4
- package/rules/paradigms/security-hardened.md +53 -10
- package/rules/paradigms/tdd.md +65 -8
- package/scripts/install.ps1 +14 -0
- package/scripts/install.sh +15 -1
- package/scripts/validate.sh +103 -12
- package/skills/apache-kafka-streams/SKILL.md +152 -0
- package/skills/argo-workflows/SKILL.md +151 -0
- package/skills/axum-web-service/SKILL.md +159 -0
- package/skills/cilium-ebpf-networking/SKILL.md +145 -0
- package/skills/clickhouse-analytics/SKILL.md +128 -0
- package/skills/dbt-transformation/SKILL.md +179 -0
- package/skills/debezium-cdc/SKILL.md +166 -0
- package/skills/duckdb-local-olap/SKILL.md +134 -0
- package/skills/fastapi-production/SKILL.md +157 -0
- package/skills/grpc-connect-rpc/SKILL.md +176 -0
- package/skills/langgraph-orchestrator/SKILL.md +144 -0
- package/skills/litellm-proxy-gateway/SKILL.md +154 -0
- package/skills/nats-jetstream/SKILL.md +122 -0
- package/skills/open-telemetry-tracing/SKILL.md +151 -0
- package/skills/playwright-visual-regression/SKILL.md +124 -0
- package/skills/redis-cache-patterns/SKILL.md +159 -0
- package/skills/svelte-kit-architect/SKILL.md +190 -0
- package/skills/temporal-workflows/SKILL.md +158 -0
- package/skills/trpc-react-query/SKILL.md +169 -0
- package/skills/turborepo-monorepo/SKILL.md +199 -0
- package/skills/vllm-inference/SKILL.md +144 -0
- package/skills/wasm-rust-optimizer/SKILL.md +130 -0
- package/templates/full-power/hooks.json +14 -0
- package/templates/full-power/mcp_config.json +43 -0
- package/templates/recommended/hooks.json +14 -0
- package/tests/cli.test.js +383 -0
|
@@ -1,17 +1,60 @@
|
|
|
1
1
|
# Security & Hardening Protocol
|
|
2
2
|
|
|
3
|
-
Security rules enforced for all agent code generation and execution.
|
|
3
|
+
Security rules and defensive engineering standards enforced for all agent code generation and execution across projects.
|
|
4
|
+
|
|
5
|
+
---
|
|
4
6
|
|
|
5
7
|
## 1. Zero Secret Leakage
|
|
6
|
-
- Never
|
|
7
|
-
-
|
|
8
|
-
-
|
|
8
|
+
- **Never Commit Secrets**:
|
|
9
|
+
- Never print, log, or commit API keys, personal access tokens, private keys, or passwords.
|
|
10
|
+
- Always ingest sensitive configuration via environment variables (`process.env`, `os.environ`).
|
|
11
|
+
- **Git & Artifact Sanitation**:
|
|
12
|
+
- Ensure `.env`, `.env.local`, `.pem`, and certificate files are explicitly listed in `.gitignore`.
|
|
13
|
+
- When creating demonstration snippets or documentation, use standardized redaction placeholders (e.g. `GEMINI_API_KEY="YOUR_GEMINI_API_KEY"`).
|
|
14
|
+
|
|
15
|
+
---
|
|
9
16
|
|
|
10
17
|
## 2. Command & Injection Defense
|
|
11
|
-
-
|
|
12
|
-
-
|
|
13
|
-
-
|
|
18
|
+
- **Safe Process Spawning**:
|
|
19
|
+
- Never concatenate unvalidated user strings into shell commands.
|
|
20
|
+
- Avoid `shell: true` in Node.js `child_process.spawn`. Pass binary names and arguments as separate arrays:
|
|
21
|
+
```javascript
|
|
22
|
+
// INSECURE: exec(`curl ${userInput}`)
|
|
23
|
+
// SECURE:
|
|
24
|
+
const { execFile } = require('child_process');
|
|
25
|
+
execFile('curl', ['--silent', sanitizedUrl]);
|
|
26
|
+
```
|
|
27
|
+
- **SQL & Query Injection**:
|
|
28
|
+
- Always use parameterized queries or type-safe ORM query builders (Prisma, Drizzle, SQLAlchemy). Never interpolate raw string variables into SQL statements.
|
|
29
|
+
- **Path Traversal Guardrails**:
|
|
30
|
+
- Sanitize all file paths with `path.resolve` and verify that the target path begins with the authorized root directory:
|
|
31
|
+
```javascript
|
|
32
|
+
const resolvedPath = path.resolve(authorizedDir, userFilename);
|
|
33
|
+
if (!resolvedPath.startsWith(path.resolve(authorizedDir))) {
|
|
34
|
+
throw new Error('Access denied: Path traversal attempt detected.');
|
|
35
|
+
}
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## 3. Cryptographic Hygiene
|
|
41
|
+
- **Constant-Time Comparisons**:
|
|
42
|
+
- When comparing webhook signatures, API tokens, or password hashes, always use constant-time comparison utilities (`crypto.timingSafeEqual`) to eliminate timing attack vectors.
|
|
43
|
+
- **Modern Cryptographic Primitives**:
|
|
44
|
+
- Use cryptographically secure pseudorandom number generators (`crypto.randomBytes` or `secrets.token_hex`). Never use `Math.random()` for security-critical values.
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## 4. Prototype Pollution & Input Validation
|
|
49
|
+
- **Deep Object Cloning**:
|
|
50
|
+
- Prevent prototype pollution by sanitizing keys against `__proto__`, `constructor`, and `prototype`.
|
|
51
|
+
- **Strict Schema Parsing**:
|
|
52
|
+
- Validate all JSON payloads at HTTP route boundaries using schema validators (Zod, Pydantic, or TypeBox). Strip unknown properties before processing.
|
|
53
|
+
|
|
54
|
+
---
|
|
14
55
|
|
|
15
|
-
##
|
|
16
|
-
-
|
|
17
|
-
- Run
|
|
56
|
+
## 5. Dependency Audit & Supply Chain Security
|
|
57
|
+
- **Automated CVE Scans**:
|
|
58
|
+
- Run vulnerability audits (`npm audit`, `pip-audit`, `govulncheck`) on third-party dependencies before introducing new packages.
|
|
59
|
+
- **Lockfile Enforcement**:
|
|
60
|
+
- Commit deterministic lockfiles (`package-lock.json`, `poetry.lock`) to ensure repeatable, tamper-evident builds.
|
package/rules/paradigms/tdd.md
CHANGED
|
@@ -1,19 +1,76 @@
|
|
|
1
1
|
# Test-Driven Development (TDD) Protocol
|
|
2
2
|
|
|
3
|
-
Enforce this workflow
|
|
3
|
+
Enforce this workflow whenever developing features, fixing bugs, or refactoring existing logic across the repository.
|
|
4
|
+
|
|
5
|
+
---
|
|
4
6
|
|
|
5
7
|
## The TDD Iron Law
|
|
6
8
|
|
|
7
|
-
> **Never write production code unless an existing test is currently failing because that code does not exist.**
|
|
9
|
+
> **Never write production code unless an existing test is currently failing because that code does not exist or does not yet behave as expected.**
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## 1. The Three Disciplined Phases
|
|
8
14
|
|
|
9
15
|
### Phase 1: RED (Reproduce & Specify)
|
|
10
|
-
1. Write
|
|
11
|
-
|
|
16
|
+
1. **Write the Failing Test First**:
|
|
17
|
+
- Write a minimal, focused unit or integration test describing the desired behavior or reproducing the reported bug.
|
|
18
|
+
2. **Execute to Confirm Failure**:
|
|
19
|
+
- Run the test suite and verify that the test fails for the expected reason (e.g. `AssertionError: expected false to be true` or `Function not defined`, not due to unrelated compilation or import syntax errors).
|
|
12
20
|
|
|
13
21
|
### Phase 2: GREEN (Minimal Implementation)
|
|
14
|
-
1.
|
|
15
|
-
|
|
22
|
+
1. **Surgical Implementation**:
|
|
23
|
+
- Write the absolute minimum production code required to make the failing test pass.
|
|
24
|
+
2. **Resist Premature Generalization**:
|
|
25
|
+
- Do not write speculative helper functions, premature optimizations, or unrelated features during the Green phase. Get to green as quickly and simply as possible.
|
|
26
|
+
3. **Verify All Tests Pass**:
|
|
27
|
+
- Confirm that the new test passes AND all existing regression tests remain green.
|
|
16
28
|
|
|
17
29
|
### Phase 3: REFACTOR (Clean & Harden)
|
|
18
|
-
1.
|
|
19
|
-
|
|
30
|
+
1. **Eliminate Duplication**:
|
|
31
|
+
- Clean up repeated logic, extract reusable helpers, and clarify variable naming.
|
|
32
|
+
2. **Enforce Type Safety**:
|
|
33
|
+
- Tighten type annotations, interfaces, and parameter docstrings.
|
|
34
|
+
3. **Zero Regression Guarantee**:
|
|
35
|
+
- Re-run the automated test suite after every refactoring edit. Never finalize a refactor while any test is failing.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## 2. Standardized Test Structure: AAA Pattern
|
|
40
|
+
|
|
41
|
+
Every test case must follow the **Arrange-Act-Assert (AAA)** pattern:
|
|
42
|
+
```typescript
|
|
43
|
+
describe('TokenCompactor', () => {
|
|
44
|
+
it('should truncate oldest turns when token budget is exceeded', () => {
|
|
45
|
+
// Arrange: Set up initial state and test inputs
|
|
46
|
+
const budget = 1000;
|
|
47
|
+
const history = createTranscriptFixture({ totalTokens: 1500 });
|
|
48
|
+
|
|
49
|
+
// Act: Invoke the unit under test
|
|
50
|
+
const compacted = compactHistory(history, budget);
|
|
51
|
+
|
|
52
|
+
// Assert: Verify expected output and invariants
|
|
53
|
+
expect(compacted.totalTokens).toBeLessThanOrEqual(budget);
|
|
54
|
+
expect(compacted.turns[0].isSummary).toBe(true);
|
|
55
|
+
});
|
|
56
|
+
});
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## 3. Mandatory Edge-Case Checklist
|
|
62
|
+
|
|
63
|
+
Every feature or bugfix test suite must assert against these boundary conditions:
|
|
64
|
+
- [ ] **Empty / Nil Inputs**: Null, undefined, empty strings `""`, empty arrays `[]`, empty objects `{}`.
|
|
65
|
+
- [ ] **Boundary Extremes**: Zero, negative integers, max integer bounds (`Number.MAX_SAFE_INTEGER`), very large strings (>1MB).
|
|
66
|
+
- [ ] **Malformed Payloads**: Invalid JSON, broken YAML frontmatter, truncated streams.
|
|
67
|
+
- [ ] **Network & Error Boundaries**: Simulated HTTP 429 (rate limits), 503 timeouts, socket hangups.
|
|
68
|
+
- [ ] **Idempotent Retries**: Running the same operation twice returns identical deterministic results without side effects.
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## 4. Test Isolation & Determinism
|
|
73
|
+
- **No Shared Mutable State**:
|
|
74
|
+
- Each test must run independently. Clean up databases, mocks, and temporary directories in `afterEach` hooks.
|
|
75
|
+
- **Deterministic Clocks**:
|
|
76
|
+
- Avoid `setTimeout` loops in tests. Use mock timers (`vi.useFakeTimers()` or `jest.useFakeTimers()`) to eliminate flaky tests and speed up test execution.
|
package/scripts/install.ps1
CHANGED
|
@@ -8,9 +8,18 @@ param (
|
|
|
8
8
|
Write-Host "🪐 Everything Gemini (everything-gemini) Installer" -ForegroundColor Cyan
|
|
9
9
|
Write-Host "The premier agent harness & customization suite for Google Antigravity`n" -ForegroundColor Gray
|
|
10
10
|
|
|
11
|
+
$origDir = Get-Location
|
|
12
|
+
$tempDir = ""
|
|
13
|
+
$cleanupTemp = $false
|
|
14
|
+
|
|
15
|
+
if (-not $Global -and $Workspace -eq "") {
|
|
16
|
+
$Workspace = Join-Path $origDir ".agents"
|
|
17
|
+
}
|
|
18
|
+
|
|
11
19
|
if (-not (Test-Path "bin\everything-gemini")) {
|
|
12
20
|
Write-Host "Downloading latest everything-gemini release..." -ForegroundColor Yellow
|
|
13
21
|
$tempDir = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), [System.IO.Path]::GetRandomFileName())
|
|
22
|
+
$cleanupTemp = $true
|
|
14
23
|
git clone --depth 1 https://github.com/salmandev/everything-gemini.git $tempDir
|
|
15
24
|
Set-Location $tempDir
|
|
16
25
|
}
|
|
@@ -24,3 +33,8 @@ if ($Workspace -ne "") {
|
|
|
24
33
|
}
|
|
25
34
|
|
|
26
35
|
node bin\everything-gemini @cliArgs
|
|
36
|
+
|
|
37
|
+
if ($cleanupTemp -and (Test-Path $tempDir)) {
|
|
38
|
+
Set-Location $origDir
|
|
39
|
+
Remove-Item -Recurse -Force $tempDir -ErrorAction SilentlyContinue
|
|
40
|
+
}
|
package/scripts/install.sh
CHANGED
|
@@ -15,9 +15,12 @@ echo " 🪐 Everything Gemini (everything-gemini) Installer"
|
|
|
15
15
|
echo " The premier agent harness & customization suite for Google Antigravity"
|
|
16
16
|
echo -e "${RESET}"
|
|
17
17
|
|
|
18
|
+
ORIG_DIR="$(pwd)"
|
|
18
19
|
PROFILE="developer"
|
|
19
20
|
INSTALL_GLOBAL=false
|
|
20
21
|
TARGET_DIR=""
|
|
22
|
+
TMP_DIR=""
|
|
23
|
+
CLEANUP_TMP=false
|
|
21
24
|
|
|
22
25
|
while [[ "$#" -gt 0 ]]; do
|
|
23
26
|
case $1 in
|
|
@@ -29,10 +32,15 @@ while [[ "$#" -gt 0 ]]; do
|
|
|
29
32
|
shift
|
|
30
33
|
done
|
|
31
34
|
|
|
32
|
-
|
|
35
|
+
if [ "$INSTALL_GLOBAL" = false ] && [ -z "$TARGET_DIR" ]; then
|
|
36
|
+
TARGET_DIR="$ORIG_DIR/.agents"
|
|
37
|
+
fi
|
|
38
|
+
|
|
39
|
+
# If run directly via curl pipe, clone into temporary directory
|
|
33
40
|
if [ ! -f "bin/everything-gemini" ]; then
|
|
34
41
|
echo -e "${YELLOW}Downloading latest everything-gemini release...${RESET}"
|
|
35
42
|
TMP_DIR=$(mktemp -d)
|
|
43
|
+
CLEANUP_TMP=true
|
|
36
44
|
git clone --depth 1 https://github.com/salmandev/everything-gemini.git "$TMP_DIR/everything-gemini" 2>/dev/null || {
|
|
37
45
|
echo "Failed to clone repository. Please ensure git is installed."
|
|
38
46
|
exit 1
|
|
@@ -50,3 +58,9 @@ if [ -n "$TARGET_DIR" ]; then
|
|
|
50
58
|
fi
|
|
51
59
|
|
|
52
60
|
./bin/everything-gemini $CLI_ARGS
|
|
61
|
+
|
|
62
|
+
# Cleanup temp clone if created
|
|
63
|
+
if [ "$CLEANUP_TMP" = true ] && [ -n "$TMP_DIR" ] && [ -d "$TMP_DIR" ]; then
|
|
64
|
+
cd "$ORIG_DIR"
|
|
65
|
+
node -e "try { require('fs').rmSync(process.argv[1], {recursive: true, force: true}); } catch(e){}" "$TMP_DIR" 2>/dev/null || true
|
|
66
|
+
fi
|
package/scripts/validate.sh
CHANGED
|
@@ -7,7 +7,7 @@ ERRORS=0
|
|
|
7
7
|
echo "🔍 Validating Everything Gemini (everything-gemini) repository..."
|
|
8
8
|
|
|
9
9
|
# 1. Validate SKILL.md Frontmatter
|
|
10
|
-
echo " [1/
|
|
10
|
+
echo " [1/6] Checking Skills & YAML frontmatter..."
|
|
11
11
|
SKILL_FILES=$(find "$REPO_ROOT/skills" "$REPO_ROOT/plugins" -name "SKILL.md" 2>/dev/null || true)
|
|
12
12
|
|
|
13
13
|
for skill_file in $SKILL_FILES; do
|
|
@@ -35,8 +35,29 @@ for skill_file in $SKILL_FILES; do
|
|
|
35
35
|
echo " ✔ $rel_path"
|
|
36
36
|
done
|
|
37
37
|
|
|
38
|
-
# 2. Validate
|
|
39
|
-
echo " [2/
|
|
38
|
+
# 2. Validate Scoped Subagents
|
|
39
|
+
echo " [2/6] Checking Subagents specifications..."
|
|
40
|
+
REQUIRED_AGENTS=("planner.md" "code-reviewer.md" "browser-qa.md" "build-repair.md" "security-auditor.md" "architect.md" "database-migrator.md" "performance-tuner.md" "documentation-writer.md" "release-engineer.md" "incident-responder.md")
|
|
41
|
+
for ag in "${REQUIRED_AGENTS[@]}"; do
|
|
42
|
+
if [ ! -f "$REPO_ROOT/agents/$ag" ]; then
|
|
43
|
+
echo " ❌ ERROR: Required subagent specification 'agents/$ag' is missing!"
|
|
44
|
+
ERRORS=$((ERRORS + 1))
|
|
45
|
+
fi
|
|
46
|
+
done
|
|
47
|
+
|
|
48
|
+
AGENT_FILES=$(find "$REPO_ROOT/agents" -name "*.md" 2>/dev/null || true)
|
|
49
|
+
for agent_file in $AGENT_FILES; do
|
|
50
|
+
rel_path="${agent_file#$REPO_ROOT/}"
|
|
51
|
+
if ! grep -q "^# " "$agent_file"; then
|
|
52
|
+
echo " ❌ ERROR: $rel_path is missing top-level '# ' title"
|
|
53
|
+
ERRORS=$((ERRORS + 1))
|
|
54
|
+
else
|
|
55
|
+
echo " ✔ $rel_path"
|
|
56
|
+
fi
|
|
57
|
+
done
|
|
58
|
+
|
|
59
|
+
# 3. Validate JSON & YAML Files
|
|
60
|
+
echo " [3/6] Validating JSON files & Action manifest..."
|
|
40
61
|
JSON_FILES=$(find "$REPO_ROOT" -name "*.json" ! -path "*/node_modules/*" 2>/dev/null || true)
|
|
41
62
|
|
|
42
63
|
for json_file in $JSON_FILES; do
|
|
@@ -49,21 +70,91 @@ for json_file in $JSON_FILES; do
|
|
|
49
70
|
fi
|
|
50
71
|
done
|
|
51
72
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
73
|
+
if [ -f "$REPO_ROOT/action.yml" ]; then
|
|
74
|
+
if grep -q "name:" "$REPO_ROOT/action.yml" && grep -q "runs:" "$REPO_ROOT/action.yml"; then
|
|
75
|
+
echo " ✔ action.yml (composite GitHub action valid)"
|
|
76
|
+
else
|
|
77
|
+
echo " ❌ ERROR: action.yml is missing required composite action fields"
|
|
78
|
+
ERRORS=$((ERRORS + 1))
|
|
79
|
+
fi
|
|
80
|
+
fi
|
|
81
|
+
|
|
82
|
+
# Validate Relative Documentation Links
|
|
83
|
+
if ! node -e "
|
|
84
|
+
const fs = require('fs');
|
|
85
|
+
const path = require('path');
|
|
86
|
+
const mdFiles = [
|
|
87
|
+
'README.md', 'CONTRIBUTING.md', 'SECURITY.md', 'CHANGELOG.md',
|
|
88
|
+
...fs.readdirSync('skills').map(s => 'skills/' + s + '/SKILL.md'),
|
|
89
|
+
...fs.readdirSync('agents').map(a => 'agents/' + a)
|
|
90
|
+
];
|
|
91
|
+
let broken = 0;
|
|
92
|
+
for (const file of mdFiles) {
|
|
93
|
+
if (!fs.existsSync(file)) continue;
|
|
94
|
+
let content = fs.readFileSync(file, 'utf8');
|
|
95
|
+
content = content.replace(/\`\`\`[\s\S]*?\`\`\`/g, '');
|
|
96
|
+
content = content.replace(/\`[^\`]*?\`/g, '');
|
|
97
|
+
const matches = content.matchAll(/\[([^\]]+)\]\(([^)]+)\)/g);
|
|
98
|
+
for (const m of matches) {
|
|
99
|
+
const link = m[2];
|
|
100
|
+
if (link.startsWith('http://') || link.startsWith('https://') || link.startsWith('#') || link.startsWith('mailto:') || link.startsWith('file://')) continue;
|
|
101
|
+
const cleanLink = link.split('#')[0];
|
|
102
|
+
if (!cleanLink) continue;
|
|
103
|
+
const resolved = path.resolve(path.dirname(file), cleanLink);
|
|
104
|
+
if (!fs.existsSync(resolved)) {
|
|
105
|
+
console.error('Broken link in ' + file + ' -> ' + link);
|
|
106
|
+
broken++;
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
if (broken > 0) process.exit(1);
|
|
111
|
+
" 2>/dev/null; then
|
|
112
|
+
echo " ❌ ERROR: Broken relative links detected in documentation"
|
|
113
|
+
ERRORS=$((ERRORS + 1))
|
|
114
|
+
else
|
|
115
|
+
echo " ✔ Documentation relative links verified"
|
|
116
|
+
fi
|
|
117
|
+
|
|
118
|
+
# 4. Validate Shell Scripts & CLI Executability
|
|
119
|
+
echo " [4/6] Verifying script permissions & code syntax..."
|
|
120
|
+
EXEC_FILES=$(find "$REPO_ROOT/scripts" "$REPO_ROOT/hooks" "$REPO_ROOT/skills" "$REPO_ROOT/examples" "$REPO_ROOT/bin" -type f \( -name "*.sh" -o -name "everything-gemini" \) 2>/dev/null || true)
|
|
55
121
|
|
|
56
|
-
for
|
|
57
|
-
rel_path="${
|
|
58
|
-
if [ ! -x "$
|
|
122
|
+
for exec_file in $EXEC_FILES; do
|
|
123
|
+
rel_path="${exec_file#$REPO_ROOT/}"
|
|
124
|
+
if [ ! -x "$exec_file" ]; then
|
|
59
125
|
echo " ⚠️ Fixing executable bit on: $rel_path"
|
|
60
|
-
chmod +x "$
|
|
126
|
+
chmod +x "$exec_file"
|
|
61
127
|
fi
|
|
62
128
|
echo " ✔ $rel_path (executable)"
|
|
63
129
|
done
|
|
64
130
|
|
|
65
|
-
#
|
|
66
|
-
|
|
131
|
+
# Validate Python examples syntax
|
|
132
|
+
if command -v python3 >/dev/null 2>&1; then
|
|
133
|
+
PY_FILES=$(find "$REPO_ROOT/examples" -name "*.py" 2>/dev/null || true)
|
|
134
|
+
for py_file in $PY_FILES; do
|
|
135
|
+
rel_path="${py_file#$REPO_ROOT/}"
|
|
136
|
+
if ! python3 -m py_compile "$py_file" 2>/dev/null; then
|
|
137
|
+
echo " ❌ ERROR: Syntax error in $rel_path"
|
|
138
|
+
ERRORS=$((ERRORS + 1))
|
|
139
|
+
else
|
|
140
|
+
echo " ✔ $rel_path (valid Python syntax)"
|
|
141
|
+
fi
|
|
142
|
+
done
|
|
143
|
+
fi
|
|
144
|
+
|
|
145
|
+
# 5. CLI E2E Integration Suite
|
|
146
|
+
echo " [5/6] Running CLI E2E Integration Suite..."
|
|
147
|
+
if [ -f "$REPO_ROOT/tests/cli.test.js" ]; then
|
|
148
|
+
if ! node "$REPO_ROOT/tests/cli.test.js"; then
|
|
149
|
+
echo " ❌ ERROR: CLI test suite failed."
|
|
150
|
+
ERRORS=$((ERRORS + 1))
|
|
151
|
+
else
|
|
152
|
+
echo " ✔ CLI test suite passed (26 assertions)"
|
|
153
|
+
fi
|
|
154
|
+
fi
|
|
155
|
+
|
|
156
|
+
# 6. Summary
|
|
157
|
+
echo " [6/6] Validation Summary"
|
|
67
158
|
if [ "$ERRORS" -gt 0 ]; then
|
|
68
159
|
echo "❌ Validation failed with $ERRORS error(s)."
|
|
69
160
|
exit 1
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: apache-kafka-streams
|
|
3
|
+
description: >-
|
|
4
|
+
Architects, deploys, and optimizes Apache Kafka event streaming architectures, stateful Kafka Streams / Faust topologies, exactly-once semantics (EOS), consumer group rebalancing, and Schema Registry serialization.
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Apache Kafka Event Streaming & Stream Processing
|
|
8
|
+
|
|
9
|
+
This skill guides engineering resilient, high-throughput distributed event streaming systems with Apache Kafka, covering partition key strategies, transactional exactly-once processing (EOS), stateful sliding/tumbling window joins, and Schema Registry Avro/Protobuf governance.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## ⚡ Core Patterns
|
|
14
|
+
|
|
15
|
+
### 1. Production Producer with Transactional Exactly-Once Semantics (EOS)
|
|
16
|
+
|
|
17
|
+
Configure idempotency, reliable acknowledgments, and transactional boundaries:
|
|
18
|
+
|
|
19
|
+
```typescript
|
|
20
|
+
import { Kafka, CompressionTypes, logLevel } from 'kafkajs';
|
|
21
|
+
|
|
22
|
+
const kafka = new Kafka({
|
|
23
|
+
clientId: 'order-dispatch-service',
|
|
24
|
+
brokers: ['kafka-1.internal:9092', 'kafka-2.internal:9092', 'kafka-3.internal:9092'],
|
|
25
|
+
ssl: true,
|
|
26
|
+
logLevel: logLevel.WARN,
|
|
27
|
+
});
|
|
28
|
+
|
|
29
|
+
const producer = kafka.producer({
|
|
30
|
+
// Idempotent producer guarantees partition deduplication and in-flight order preservation
|
|
31
|
+
idempotent: true,
|
|
32
|
+
maxInFlightRequests: 1,
|
|
33
|
+
transactionalId: 'order-tx-producer-instance-01',
|
|
34
|
+
});
|
|
35
|
+
|
|
36
|
+
async function publishOrderEvent(order: { id: string; customerId: string; amount: number }) {
|
|
37
|
+
await producer.connect();
|
|
38
|
+
const transaction = await producer.transaction();
|
|
39
|
+
|
|
40
|
+
try {
|
|
41
|
+
await transaction.send({
|
|
42
|
+
topic: 'enterprise.commerce.orders.v1',
|
|
43
|
+
compression: CompressionTypes.GZIP,
|
|
44
|
+
messages: [
|
|
45
|
+
{
|
|
46
|
+
key: order.customerId, // Ensures hash-partition ordering per customer
|
|
47
|
+
value: JSON.stringify(order),
|
|
48
|
+
headers: {
|
|
49
|
+
'correlation-id': crypto.randomUUID(),
|
|
50
|
+
'event-type': 'OrderPlaced',
|
|
51
|
+
'schema-version': '1.0.0',
|
|
52
|
+
},
|
|
53
|
+
},
|
|
54
|
+
],
|
|
55
|
+
});
|
|
56
|
+
|
|
57
|
+
await transaction.commit();
|
|
58
|
+
} catch (err) {
|
|
59
|
+
await transaction.abort();
|
|
60
|
+
throw err;
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
### 2. High-Throughput Consumer Group with Cooperative Sticky Rebalancing
|
|
68
|
+
|
|
69
|
+
Prevent stop-the-world partition revocations during scale-up or rolling deployments:
|
|
70
|
+
|
|
71
|
+
```typescript
|
|
72
|
+
const consumer = kafka.consumer({
|
|
73
|
+
groupId: 'order-fulfillment-workers',
|
|
74
|
+
sessionTimeout: 30000,
|
|
75
|
+
heartbeatInterval: 3000,
|
|
76
|
+
rebalanceTimeout: 60000,
|
|
77
|
+
// CooperativeSticky assignor migrates only transferred partitions without halting healthy consumers
|
|
78
|
+
partitionAssigners: [
|
|
79
|
+
(config) => new (require('kafkajs').PartitionAssigners.cooperativeSticky)(config),
|
|
80
|
+
],
|
|
81
|
+
});
|
|
82
|
+
|
|
83
|
+
await consumer.connect();
|
|
84
|
+
await consumer.subscribe({ topic: 'enterprise.commerce.orders.v1', fromBeginning: false });
|
|
85
|
+
|
|
86
|
+
await consumer.run({
|
|
87
|
+
autoCommit: false,
|
|
88
|
+
eachBatchAutoResolve: false,
|
|
89
|
+
eachBatch: async ({ batch, resolveOffset, heartbeat, isRunning, isStale }) => {
|
|
90
|
+
for (const message of batch.messages) {
|
|
91
|
+
if (!isRunning() || isStale()) break;
|
|
92
|
+
|
|
93
|
+
try {
|
|
94
|
+
const payload = JSON.parse(message.value.toString());
|
|
95
|
+
await processOrderFulfillment(payload);
|
|
96
|
+
|
|
97
|
+
// Mark individual offset for manual deterministic commit
|
|
98
|
+
resolveOffset(message.offset);
|
|
99
|
+
await heartbeat();
|
|
100
|
+
} catch (err) {
|
|
101
|
+
// Route unrecoverable poison pill to Dead-Letter Queue (DLQ)
|
|
102
|
+
await publishToDLQ('enterprise.commerce.orders.dlq', message, err);
|
|
103
|
+
resolveOffset(message.offset);
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
},
|
|
107
|
+
});
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
### 3. Stateful Stream Processing (Kafka Streams / Java Topology)
|
|
113
|
+
|
|
114
|
+
Perform stateful 5-minute tumbling window aggregations with RocksDB state stores:
|
|
115
|
+
|
|
116
|
+
```java
|
|
117
|
+
StreamsBuilder builder = new StreamsBuilder();
|
|
118
|
+
KStream<String, OrderEvent> orderStream = builder.stream(
|
|
119
|
+
"enterprise.commerce.orders.v1",
|
|
120
|
+
Consumed.with(Serdes.String(), new JsonSerde<>(OrderEvent.class))
|
|
121
|
+
);
|
|
122
|
+
|
|
123
|
+
KTable<Windowed<String>, Long> orderCountsPerStore = orderStream
|
|
124
|
+
.selectKey((key, order) -> order.getStoreId())
|
|
125
|
+
.groupByKey(Grouped.with(Serdes.String(), new JsonSerde<>(OrderEvent.class)))
|
|
126
|
+
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
|
|
127
|
+
.count(
|
|
128
|
+
Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("store-order-counts-store")
|
|
129
|
+
.withRetention(Duration.ofDays(1))
|
|
130
|
+
);
|
|
131
|
+
|
|
132
|
+
orderCountsPerStore.toStream()
|
|
133
|
+
.map((windowedKey, count) -> new KeyValue<>(windowedKey.key(), new StoreOrderSummary(windowedKey.window().start(), count)))
|
|
134
|
+
.to("enterprise.commerce.store-aggregates.v1", Produced.with(Serdes.String(), new JsonSerde<>(StoreOrderSummary.class)));
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## 🎯 Verification & Diagnostics
|
|
140
|
+
|
|
141
|
+
1. **Check Consumer Group Lag & Partition Distribution**:
|
|
142
|
+
```bash
|
|
143
|
+
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-fulfillment-workers
|
|
144
|
+
```
|
|
145
|
+
2. **Inspect Topic Partition Watermarks**:
|
|
146
|
+
```bash
|
|
147
|
+
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic enterprise.commerce.orders.v1 --time -1
|
|
148
|
+
```
|
|
149
|
+
3. **Tail Stream Events with Header Decoding**:
|
|
150
|
+
```bash
|
|
151
|
+
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic enterprise.commerce.orders.v1 --from-beginning --property print.key=true --property print.headers=true
|
|
152
|
+
```
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: argo-workflows
|
|
3
|
+
description: >-
|
|
4
|
+
Architects, defines, and executes Kubernetes-native DAG workflows, artifact passing, container steps, retry strategies, and CronWorkflows using Argo Workflows.
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Argo Workflows & Kubernetes-Native DAG Orchestration
|
|
8
|
+
|
|
9
|
+
This skill provides operational patterns for designing, testing, and deploying Kubernetes-native workflow orchestration via Argo Workflows, powering parallel data pipelines, machine learning training graphs, and automated batch processing.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## ⚡ Core Architecture & CRDs
|
|
14
|
+
|
|
15
|
+
1. **`Workflow`**: Ephemeral, single-run execution specification.
|
|
16
|
+
2. **`WorkflowTemplate`**: Reusable workflow definitions stored inside the Kubernetes cluster.
|
|
17
|
+
3. **`CronWorkflow`**: Time-triggered recurring workflows (cron schedule).
|
|
18
|
+
4. **DAGs vs Steps**: DAGs declare declarative dependency graphs (`dependencies: [taskA, taskB]`), whereas Steps declare sequential matrix execution.
|
|
19
|
+
5. **Artifacts**: Input/output binary blobs, model weights, and logs backed by S3, GCS, or MinIO.
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## 🏗️ Production DAG Workflow Specification (`workflow-dag.yaml`)
|
|
24
|
+
|
|
25
|
+
```yaml
|
|
26
|
+
apiVersion: argoproj.io/v1alpha1
|
|
27
|
+
kind: Workflow
|
|
28
|
+
metadata:
|
|
29
|
+
generateName: ml-pipeline-dag-
|
|
30
|
+
namespace: argo
|
|
31
|
+
spec:
|
|
32
|
+
entrypoint: ml-pipeline
|
|
33
|
+
serviceAccountName: argo-workflow-executor
|
|
34
|
+
shutdown: "Stop" # Graceful termination support
|
|
35
|
+
onExit: pipeline-cleanup-handler
|
|
36
|
+
|
|
37
|
+
templates:
|
|
38
|
+
- name: ml-pipeline
|
|
39
|
+
dag:
|
|
40
|
+
tasks:
|
|
41
|
+
- name: ingest-data
|
|
42
|
+
template: data-ingestion-step
|
|
43
|
+
arguments:
|
|
44
|
+
parameters:
|
|
45
|
+
- name: dataset-version
|
|
46
|
+
value: "v2026.10"
|
|
47
|
+
|
|
48
|
+
- name: validate-schema
|
|
49
|
+
dependencies: [ingest-data]
|
|
50
|
+
template: schema-validator-step
|
|
51
|
+
arguments:
|
|
52
|
+
artifacts:
|
|
53
|
+
- name: raw-dataset
|
|
54
|
+
from: "{{tasks.ingest-data.outputs.artifacts.dataset}}"
|
|
55
|
+
|
|
56
|
+
- name: train-model
|
|
57
|
+
dependencies: [validate-schema]
|
|
58
|
+
template: model-training-step
|
|
59
|
+
arguments:
|
|
60
|
+
artifacts:
|
|
61
|
+
- name: validated-data
|
|
62
|
+
from: "{{tasks.validate-schema.outputs.artifacts.clean-dataset}}"
|
|
63
|
+
|
|
64
|
+
- name: data-ingestion-step
|
|
65
|
+
inputs:
|
|
66
|
+
parameters:
|
|
67
|
+
- name: dataset-version
|
|
68
|
+
container:
|
|
69
|
+
image: python:3.11-slim
|
|
70
|
+
command: [python, -c]
|
|
71
|
+
args: ["import json; open('/tmp/dataset.csv', 'w').write('id,score\\n1,0.95\\n')"]
|
|
72
|
+
outputs:
|
|
73
|
+
artifacts:
|
|
74
|
+
- name: dataset
|
|
75
|
+
path: /tmp/dataset.csv
|
|
76
|
+
|
|
77
|
+
- name: schema-validator-step
|
|
78
|
+
inputs:
|
|
79
|
+
artifacts:
|
|
80
|
+
- name: raw-dataset
|
|
81
|
+
path: /inputs/dataset.csv
|
|
82
|
+
container:
|
|
83
|
+
image: python:3.11-slim
|
|
84
|
+
command: [python, -c]
|
|
85
|
+
args: ["import shutil; shutil.copy('/inputs/dataset.csv', '/outputs/clean.csv')"]
|
|
86
|
+
outputs:
|
|
87
|
+
artifacts:
|
|
88
|
+
- name: clean-dataset
|
|
89
|
+
path: /outputs/clean.csv
|
|
90
|
+
|
|
91
|
+
- name: model-training-step
|
|
92
|
+
inputs:
|
|
93
|
+
artifacts:
|
|
94
|
+
- name: validated-data
|
|
95
|
+
path: /data/clean.csv
|
|
96
|
+
retryStrategy:
|
|
97
|
+
limit: "3"
|
|
98
|
+
retryPolicy: "Always"
|
|
99
|
+
backoff:
|
|
100
|
+
duration: "10s"
|
|
101
|
+
factor: 2
|
|
102
|
+
container:
|
|
103
|
+
image: python:3.11-slim
|
|
104
|
+
resources:
|
|
105
|
+
requests:
|
|
106
|
+
cpu: "2"
|
|
107
|
+
memory: "4Gi"
|
|
108
|
+
limits:
|
|
109
|
+
cpu: "4"
|
|
110
|
+
memory: "8Gi"
|
|
111
|
+
command: [python, -c]
|
|
112
|
+
args: ["print('Model trained successfully')"]
|
|
113
|
+
|
|
114
|
+
- name: pipeline-cleanup-handler
|
|
115
|
+
container:
|
|
116
|
+
image: alpine:latest
|
|
117
|
+
command: [sh, -c]
|
|
118
|
+
args: ["echo Pipeline finished with status: {{workflow.status}}"]
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## 🛠️ CLI Operations & Debugging Runbook
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
# Submit a workflow and follow real-time execution
|
|
127
|
+
argo submit --watch workflow-dag.yaml -n argo
|
|
128
|
+
|
|
129
|
+
# List active and completed workflows
|
|
130
|
+
argo list -n argo
|
|
131
|
+
|
|
132
|
+
# Stream logs of a specific DAG task step
|
|
133
|
+
argo logs @latest -c main -n argo
|
|
134
|
+
|
|
135
|
+
# Resubmit a failed workflow with memoized step caching
|
|
136
|
+
argo resubmit --memoized @latest -n argo
|
|
137
|
+
|
|
138
|
+
# Terminate or suspend a runaway workflow
|
|
139
|
+
argo terminate @latest -n argo
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
## 🛡️ Best Practices & Anti-Patterns
|
|
145
|
+
|
|
146
|
+
| Category | ❌ Anti-Pattern | ✅ Production Best Practice |
|
|
147
|
+
| :--- | :--- | :--- |
|
|
148
|
+
| **Artifacts** | Storing giant datasets directly inside pod disk | Use external artifact repositories (S3 / GCS) with lifecycle policies |
|
|
149
|
+
| **Retries** | Unbounded retries without exponential backoff | Set explicit `retryStrategy.limit` and exponential `backoff.factor` |
|
|
150
|
+
| **Resources** | Omitting CPU/Memory resource constraints | Always specify `requests` and `limits` to prevent node OOM crashes |
|
|
151
|
+
| **Secrets** | Hardcoding API keys or database URLs in YAML | Reference Kubernetes Secrets via `env.valueFrom.secretKeyRef` |
|