tribunal-kit 1.0.0 → 2.4.0

package/.agent/skills/lint-and-validate/SKILL.md

@@ -1,45 +1,247 @@
 ---
 name: lint-and-validate
-description: Automatic quality control, linting, and static analysis procedures. Use after every code modification to ensure syntax correctness and project standards. Triggers onKeywords: lint, format, check, validate, types, static analysis.
-allowed-tools: Read, Glob, Grep, Bash
+description: Linting and validation principles for code quality enforcement.
+allowed-tools: Read, Write, Edit, Glob, Grep
+version: 1.0.0
+last-updated: 2026-03-12
+applies-to-model: gemini-2.5-pro, claude-3-7-sonnet
 ---
 
-# Lint and Validate Skill
+# Linting & Validation
 
-> **MANDATORY:** Run appropriate validation tools after EVERY code change. Do not finish a task until the code is error-free.
+> A linter is an automated code reviewer that never gets tired, never gets distracted,
+> and catches the same class of problems every single time.
 
-### Procedures by Ecosystem
+---
+
+## Why Linting Matters
+
+Linting catches problems that code review misses:
+- Unused variables left in after refactoring
+- Missing `await` on async functions (silently returns a Promise instead of the value)
+- Inconsistent code style that makes diffs hard to read
+- Known dangerous patterns (e.g., `==` instead of `===` in JS)
+
+Run linting in CI. Every PR that merges should pass lint. A lint check that doesn't block the build is decoration.
+
+---
+
+## JavaScript / TypeScript (ESLint + Prettier)
+
+```bash
+# Install
+npm install -D eslint @typescript-eslint/eslint-plugin @typescript-eslint/parser prettier
+
+# Run
+npx eslint . --ext .ts,.tsx
+npx prettier --check .
+
+# Fix auto-fixable issues
+npx eslint . --ext .ts,.tsx --fix
+npx prettier --write .
+```
+
+**Recommended rules to enforce:**
+
+```json
+// .eslintrc.json
+{
+  "extends": [
+    "eslint:recommended",
+    "plugin:@typescript-eslint/recommended",
+    "plugin:@typescript-eslint/recommended-requiring-type-checking"
+  ],
+  "rules": {
+    "@typescript-eslint/no-explicit-any": "error",
+    "@typescript-eslint/no-floating-promises": "error",
+    "@typescript-eslint/await-thenable": "error",
+    "no-console": ["warn", { "allow": ["warn", "error"] }],
+    "eqeqeq": ["error", "always"]
+  }
+}
+```
+
+**Key rules explained:**
+
+| Rule | Why It Matters |
+|---|---|
+| `no-floating-promises` | Missing `await` on async call = silent bug |
+| `no-explicit-any` | `any` disables TypeScript's only protection |
+| `eqeqeq` | `==` has coercion surprises; `===` is always explicit |
+| `await-thenable` | Prevents `await`-ing non-async functions (always a mistake) |
 
-#### Node.js / TypeScript
-1. **Lint/Fix:** `npm run lint` or `npx eslint "path" --fix`
-2. **Types:** `npx tsc --noEmit`
-3. **Security:** `npm audit --audit-level=high`
+---
+
+## Python (Ruff)
+
+Ruff replaces flake8, black, isort, and pyupgrade in one fast tool:
+
+```bash
+# Install
+pip install ruff
+
+# Check
+ruff check .
+
+# Fix auto-fixable
+ruff check . --fix
+
+# Format (replaces black)
+ruff format .
 
-#### Python
-1. **Linter (Ruff):** `ruff check "path" --fix` (Fast & Modern)
-2. **Security (Bandit):** `bandit -r "path" -ll`
-3. **Types (MyPy):** `mypy "path"`
+# Pre-commit config
+# .pre-commit-config.yaml
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  hooks:
+    - id: ruff
+      args: [--fix]
+    - id: ruff-format
+```
 
-## The Quality Loop
-1. **Write/Edit Code**
-2. **Run Audit:** `npm run lint && npx tsc --noEmit`
-3. **Analyze Report:** Check the "FINAL AUDIT REPORT" section.
-4. **Fix & Repeat:** Submitting code with "FINAL AUDIT" failures is NOT allowed.
+```toml
+# pyproject.toml
+[tool.ruff]
+line-length = 100
+target-version = "py311"
 
-## Error Handling
-- If `lint` fails: Fix the style or syntax issues immediately.
-- If `tsc` fails: Correct type mismatches before proceeding.
-- If no tool is configured: Check the project root for `.eslintrc`, `tsconfig.json`, `pyproject.toml` and suggest creating one.
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "UP", "B", "SIM", "ANN"]
+# E: pycodestyle, F: pyflakes, I: isort, N: naming, UP: pyupgrade
+# B: bugbear (common bugs), SIM: simplify, ANN: annotations
+```
 
 ---
-**Strict Rule:** No code should be committed or reported as "done" without passing these checks.
+
+## Type Checking
+
+Linting and type checking catch different things. Run both.
+
+**TypeScript:**
+```bash
+npx tsc --noEmit   # type check without emitting files
+```
+
+**Python:**
+```bash
+mypy src/ --ignore-missing-imports
+# or
+pyright src/
+```
+
+**Required compiler options (TypeScript):**
+```json
+{
+  "compilerOptions": {
+    "strict": true,           // enables all strict checks
+    "noImplicitAny": true,   
+    "noUncheckedIndexedAccess": true,  // index access can be undefined
+    "exactOptionalPropertyTypes": true
+  }
+}
+```
+
+---
+
+## Pre-commit Integration
+
+Run linting automatically before every commit:
+
+```yaml
+# .pre-commit-config.yaml
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    hooks:
+      - id: check-merge-conflict
+      - id: check-added-large-files
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+
+  - repo: local
+    hooks:
+      - id: eslint
+        name: ESLint
+        language: node
+        entry: npx eslint --ext .ts,.tsx
+        types: [javascript, ts]
+
+      - id: tsc
+        name: TypeScript
+        language: node
+        entry: npx tsc --noEmit
+        pass_filenames: false
+```
 
 ---
 
 ## Scripts
 
-| Script | Purpose | Command |
-|--------|---------|---------|
-| `scripts/lint_runner.py` | Unified lint check | `python scripts/lint_runner.py <project_path>` |
-| `scripts/type_coverage.py` | Type coverage analysis | `python scripts/type_coverage.py <project_path>` |
+| Script | Purpose | Run With |
+|---|---|---|
+| `scripts/lint_runner.py` | Runs project linting and reports findings | `python scripts/lint_runner.py <project_path>` |
+| `scripts/type_coverage.py` | Measures TypeScript type coverage | `python scripts/type_coverage.py <project_path>` |
+
+---
+
+## Output Format
+
+When this skill produces or reviews code, structure your output as follows:
+
+```
+━━━ Lint And Validate Report ━━━━━━━━━━━━━━━━━━━━━━━━
+Skill:       Lint And Validate
+Language:    [detected language / framework]
+Scope:       [N files · N functions]
+─────────────────────────────────────────────────
+✅ Passed:   [checks that passed, or "All clean"]
+⚠️  Warnings: [non-blocking issues, or "None"]
+❌ Blocked:  [blocking issues requiring fix, or "None"]
+─────────────────────────────────────────────────
+VBC status:  PENDING → VERIFIED
+Evidence:    [test output / lint pass / compile success]
+```
+
+**VBC (Verification-Before-Completion) is mandatory.**
+Do not mark status as VERIFIED until concrete terminal evidence is provided.
+
+
+
+---
+
+## 🤖 LLM-Specific Traps
+
+AI coding assistants often fall into specific bad habits when dealing with this domain. These are strictly forbidden:
+
+1. **Over-engineering:** Proposing complex abstractions or distributed systems when a simpler approach suffices.
+2. **Hallucinated Libraries/Methods:** Using non-existent methods or packages. Always `// VERIFY` or check `package.json` / `requirements.txt`.
+3. **Skipping Edge Cases:** Writing the "happy path" and ignoring error handling, timeouts, or data validation.
+4. **Context Amnesia:** Forgetting the user's constraints and offering generic advice instead of tailored solutions.
+5. **Silent Degradation:** Catching and suppressing errors without logging or re-raising.
+
+---
+
+## 🏛️ Tribunal Integration (Anti-Hallucination)
+
+**Slash command: `/review` or `/tribunal-full`**
+**Active reviewers: `logic-reviewer` · `security-auditor`**
+
+### ❌ Forbidden AI Tropes
+
+1. **Blind Assumptions:** Never make an assumption without documenting it clearly with `// VERIFY: [reason]`.
+2. **Silent Degradation:** Catching and suppressing errors without logging or handling.
+3. **Context Amnesia:** Forgetting the user's constraints and offering generic advice instead of tailored solutions.
+
+### ✅ Pre-Flight Self-Audit
+
+Review these questions before confirming output:
+```
+✅ Did I rely ONLY on real, verified tools and methods?
+✅ Is this solution appropriately scoped to the user's constraints?
+✅ Did I handle potential failure modes and edge cases?
+✅ Have I avoided generic boilerplate that doesn't add value?
+```
+
+### 🛑 Verification-Before-Completion (VBC) Protocol
 
+**CRITICAL:** You must follow a strict "evidence-based closeout" state machine.
+- ❌ **Forbidden:** Declaring a task complete because the output "looks correct."
+- ✅ **Required:** You are explicitly forbidden from finalizing any task without providing **concrete evidence** (terminal output, passing tests, compile success, or equivalent proof) that your output works as intended.

package/.agent/skills/llm-engineering/SKILL.md

@@ -0,0 +1,258 @@
+---
+name: llm-engineering
+description: LLM engineering principles for production AI systems. RAG pipeline design, vector store selection, prompt engineering, evals, and LLMOps. Use when building AI features, chat interfaces, semantic search, or any system calling an LLM API.
+allowed-tools: Read, Write, Edit, Glob, Grep
+version: 1.0.0
+last-updated: 2026-03-12
+applies-to-model: gemini-2.5-pro, claude-3-7-sonnet
+---
+
+# LLM Engineering Principles
+
+> An LLM is a probabilistic function, not a deterministic API.
+> Design your system to be correct despite that — not because you got lucky.
+
+---
+
+## When This Skill Activates
+
+- Adding AI chat, completion, or summarization to an app
+- Building a RAG (Retrieval-Augmented Generation) pipeline
+- Integrating with OpenAI, Anthropic, Google Gemini, or local models
+- Designing semantic search
+- Setting up AI evals or monitoring
+
+---
+
+## Core Architecture Decision: What Pattern?
+
+| Pattern | Use When | Avoid When |
+|---|---|---|
+| **Simple prompt** | Single-turn, no user docs | Needs accuracy on user data |
+| **RAG** | Answers must cite user/company docs | Data changes every second |
+| **Fine-tuning** | Consistent tone/style at scale | You have < 1000 examples |
+| **Agent loop** | Multi-step tasks, tool use | Single-answer questions |
+| **Hybrid** | RAG + agent (most production apps) | Over-engineering simple use case |
+
+---
+
+## RAG Pipeline Design
+
+The core pattern for grounding LLMs in real data:
+
+```
+INGEST                    RETRIEVE                  GENERATE
+─────────                 ─────────                 ─────────
+Documents                 User query                Retrieved chunks
+    │                         │                         │
+    ▼                         ▼                         ▼
+Chunk (512 tokens)    Embed query vector     Rerank by relevance
+    │                         │                         │
+    ▼                         ▼                         ▼
+Embed chunks          ANN search in          Build prompt:
+    │                 vector store           [system] + [chunks] + [query]
+    ▼                         │                         │
+Store in vector DB    Top-K results          Call LLM → stream response
+```
+
+### Chunking Strategy
+
+```ts
+// ❌ Fixed-size chunks break semantic units
+chunk(document, { size: 512 });  // Splits mid-sentence
+
+// ✅ Semantic chunking — split at natural boundaries
+chunk(document, {
+  strategy: 'markdown-headers',   // Or 'sentence', 'paragraph'
+  maxTokens: 512,
+  overlap: 64,                    // Overlap to preserve context at boundaries
+});
+```
+
+### Embedding Model Selection
+
+| Scale | Model | Dimensions | Notes |
+|---|---|---|---|
+| General English | `text-embedding-3-small` | 1536 | Best quality/cost ratio |
+| Multilingual | `multilingual-e5-large` | 1024 | Open source, self-hostable |
+| Code | `text-embedding-3-large` | 3072 | Higher cost, better code retrieval |
+| Local/private | `nomic-embed-text` | 768 | Runs on CPU via Ollama |
+
+---
+
+## Vector Store Selection
+
+| Need | Choose | Why |
+|---|---|---|
+| Already on PostgreSQL | `pgvector` | Zero infra, SQL joins with metadata |
+| Managed, billion-scale | Pinecone | Hosted ANN, hybrid search built-in |
+| Open source, self-hosted | Qdrant | Rust-native, fast, rich filtering |
+| Already on Weaviate | Weaviate | GraphQL API, multimodal support |
+| Embedded/local | ChromaDB | Zero infra, great for prototyping |
+
+```ts
+// pgvector — stays inside your existing PostgreSQL
+import { pgvector } from '@pgvector/pg';
+
+// Store
+await db.query(
+  'INSERT INTO documents (content, embedding) VALUES ($1, $2)',
+  [text, JSON.stringify(embedding)]  // embedding is float[]
+);
+
+// Query — cosine similarity
+await db.query(
+  'SELECT content FROM documents ORDER BY embedding <=> $1 LIMIT 5',
+  [JSON.stringify(queryEmbedding)]
+);
+```
+
+---
+
+## Prompt Engineering Principles
+
+### Message Structure
+
+```ts
+const messages = [
+  {
+    role: 'system',
+    content: `You are a helpful assistant for [Company].
+You ONLY answer questions based on the provided context.
+If the answer is not in the context, say "I don't have that information."
+Do NOT make up information.`,
+  },
+  {
+    // Retrieved chunks injected here — NOT into system prompt
+    role: 'user',
+    content: `Context:\n${retrievedChunks.join('\n\n')}\n\nQuestion: ${userQuery}`,
+  },
+];
+```
+
+### Few-Shot Examples
+
+```ts
+// ❌ Zero-shot on complex tasks — model guesses the format
+"Extract entities from: John called Mary at 5pm"
+
+// ✅ Few-shot — show the expected output format
+`Extract entities. Output as JSON array.
+
+Example:
+Input: "Alice met Bob in London"
+Output: [{"name":"Alice","type":"person"},{"name":"Bob","type":"person"},{"name":"London","type":"location"}]
+
+Input: "${userText}"
+Output:`
+```
+
+---
+
+## Evals: How to Know If It's Working
+
+```
+Deterministic evals:   Output matches expected exactly → code comparison
+LLM-as-judge evals:    Another LLM grades the output (1-5 scale)
+Human evals:           Gold standard, expensive, for calibration
+A/B testing:           Compare model/prompt versions on live traffic
+```
+
+### Eval Categories
+
+| Category | What It Measures | Tooling |
+|---|---|---|
+| **Faithfulness** | Does answer match sources? | Ragas, ARES |
+| **Relevance** | Does answer address the question? | LLM-as-judge |
+| **Completeness** | Missing important info? | Human + LLM |
+| **Groundedness** | Hallucination rate | Ragas |
+| **Latency** | p50/p95 response time | OpenTelemetry |
+
+---
+
+## LLMOps: Production Concerns
+
+### Cost Control
+
+```ts
+// Track tokens per request
+const response = await openai.chat.completions.create({ ... });
+const { prompt_tokens, completion_tokens } = response.usage;
+logger.info({ prompt_tokens, completion_tokens, model: 'gpt-4o', cost_usd: calcCost() });
+
+// Cache identical prompts — LLMs are deterministic at temp=0
+const cacheKey = hash(systemPrompt + userQuery);
+const cached = await cache.get(cacheKey);
+if (cached) return cached;
+```
+
+### Retry with Exponential Backoff
+
+```ts
+async function callWithRetry(fn: () => Promise<any>, maxRetries = 3) {
+  for (let attempt = 0; attempt < maxRetries; attempt++) {
+    try {
+      return await fn();
+    } catch (err: any) {
+      if (err.status === 429 || err.status === 503) {
+        const delay = Math.pow(2, attempt) * 1000 + Math.random() * 500;
+        await sleep(delay);
+        continue;
+      }
+      throw err;  // Non-retryable errors bubble up immediately
+    }
+  }
+  throw new Error('Max retries exceeded');
+}
+```
+
+---
+
+## Output Format
+
+When this skill produces or reviews code, structure your output as follows:
+
+```
+━━━ Llm Engineering Report ━━━━━━━━━━━━━━━━━━━━━━━━
+Skill:       Llm Engineering
+Language:    [detected language / framework]
+Scope:       [N files · N functions]
+─────────────────────────────────────────────────
+✅ Passed:   [checks that passed, or "All clean"]
+⚠️  Warnings: [non-blocking issues, or "None"]
+❌ Blocked:  [blocking issues requiring fix, or "None"]
+─────────────────────────────────────────────────
+VBC status:  PENDING → VERIFIED
+Evidence:    [test output / lint pass / compile success]
+```
+
+**VBC (Verification-Before-Completion) is mandatory.**
+Do not mark status as VERIFIED until concrete terminal evidence is provided.
+
+
+---
+
+## 🏛️ Tribunal Integration (Anti-Hallucination)
+
+**Slash command: `/review-ai`**
+**Active reviewers: `logic` · `security` · `ai-code-reviewer`**
+
+### ❌ Forbidden AI Tropes in LLM Engineering
+
+1. **Hallucinated model names** — `gpt-5`, `claude-4`, `gemini-ultra-3` — verify against current provider docs.
+2. **Prompt injection via concatenation** — never `systemPrompt + userInput`. Use separate message roles.
+3. **No eval strategy** — shipping LLM features with zero eval coverage is shipping blind.
+4. **Ignoring token limits** — context exceeding `max_tokens` silently fails or truncates unpredictably.
+5. **No cost tracking** — LLM costs compound at scale — always instrument from day one.
+6. **Synchronous LLM calls** — all LLM API calls are async. Never block the event loop waiting for them.
+
+### ✅ Pre-Flight Self-Audit
+
+```
+✅ Are all model names verified against current provider documentation?
+✅ Is user input isolated in role:"user" messages, never concatenated into system prompt?
+✅ Is there retry logic with backoff for 429 / 503 errors?
+✅ Is token usage logged per request for cost tracking?
+✅ Is there an eval strategy (even minimal) to detect regressions?
+✅ Are context windows respected — chunked or summarized if approaching limits?
+```