@oriro/orirocli 0.1.0

package/skills/craft/become-an-ai-engineer-26/SKILL.md

@@ -0,0 +1,667 @@
+---
+watermark: ORIRO
+name: become-an-ai-engineer-26
+description: >
+  Complete AI Engineering skill for 2026. Use this skill whenever someone wants
+  to become an AI engineer, learn agent systems, build production agents, understand
+  harness engineering, context engineering, eval design, or needs a learning roadmap
+  for AI engineering. Triggers on: "become an AI engineer", "learn AI engineering",
+  "build an agent", "agent loop", "harness engineering", "context engineering",
+  "eval harness", "LangGraph agent", "production agent", "AI engineer roadmap",
+  "how to build AI systems", "agent architecture", "tool dispatch", "sub-agents",
+  "LLM evals", "regression harness", "build AI without CS degree", or any question
+  about professional AI engineering practice. Always use this skill when the goal
+  is building systems around AI models — not just calling APIs.
+provider: ORIRO.ai
+copyright: Copyright (c) 2026 ORIRO.ai
+version: "1.0"
+year: "2026"
+license: MIT
+---
+
+# How To Become An AI Engineer in 2026
+
+> Built from production experience. No CS degree required.
+> No bootcamp required. No prior AI knowledge required.
+
+---
+
+## CORE INSIGHT FIRST
+
+The companies hiring right now don't need people who understand the math.
+They need people who **build systems that survive production.**
+
+```
+Same model. Two harnesses. 36-point benchmark gap.
+
+Claude Code harness  →  78% on CORE benchmark
+Smolagents harness   →  42% on CORE benchmark
+
+The harness is the job.
+```
+
+**The replaceable:** building thin API wrappers
+**The unfireable:** shipping autonomous systems with evals and durability
+
+---
+
+## WHAT AN AI ENGINEER ACTUALLY DOES
+
+Not writing prompts. Not picking models.
+An AI engineer builds and operates the **system around the model.**
+
+- Designs the agent loop and tool dispatch
+- Engineers context — what tokens appear in front of the model at every step
+- Writes tools the model picks correctly
+- Adds memory, durability, and sandboxing for production traffic
+- Wires evals and CI regression gates so "better" becomes measurable
+- Ships agents that survive real users and real cost
+
+### The 4 Context Primitives
+
+| Primitive    | What It Means                                            |
+| ------------ | -------------------------------------------------------- |
+| **Write**    | Scratchpads and memory files the agent reads and updates |
+| **Select**   | Retrieval at the point of use — not upfront dumping      |
+| **Compress** | Summarisation at 85–95% of the context window            |
+| **Isolate**  | Sub-agents with their own separate context windows       |
+
+> **Context engineering replaced prompt engineering.** Prompt engineering is dead
+> as a standalone skill. The four primitives above are the new fundamentals.
+
+---
+
+## THE 17-WEEK ROADMAP
+
+17 weeks full-time. 40 weeks moonlighting.
+Every phase ships something concrete. No exceptions.
+
+```
+Phase 0  Weeks 1-2   Mental Models         → 2-page concept doc
+Phase 1  Weeks 3-5   First Agent           → Agent shipped with Skill + hook
+Phase 2  Weeks 6-9   Real Architecture     → LangGraph agent + LangSmith trace
+Phase 3  Weeks 10-13 Build the Harness     → 1,500-line mini-harness
+Phase 4  Weeks 14-17 Evals & CI            → Regression harness in GitHub Actions
+Phase 5  Forever     Production Hardening  → Never ends
+```
+
+---
+
+## PHASE 0 — BUILD CORRECT MENTAL MODELS (Weeks 1-2)
+
+**Do not write a single line of agent code yet.**
+
+### 1. Workflow vs Agent
+
+```
+Workflow:  fixed control flow YOU wrote     A → B → C → Done
+Agent:     dynamic control flow MODEL decides  Think → Act → Observe → Repeat
+```
+
+Building an agent when you need a workflow costs 10× more and breaks twice as often.
+
+### 2. The 5 Workflow Patterns (from Anthropic)
+
+```python
+# 1. Prompt chaining — pass output from one call to the next
+result_1 = llm(prompt_1)
+result_2 = llm(prompt_2 + result_1)
+
+# 2. Routing — different models for different tasks
+model = route(task)  # haiku / sonnet / opus
+
+# 3. Parallelisation — run multiple tasks at the same time
+results = await asyncio.gather(*[llm(p) for p in prompts])
+
+# 4. Orchestrator-worker — one brain, many hands
+plan = orchestrator.plan(task)
+results = [worker.execute(step) for step in plan]
+
+# 5. Evaluator-optimiser — generate → judge → improve
+while not judge(output): output = generate(feedback)
+```
+
+### 3. The Harness Stack
+
+```
+┌─────────────────────────────┐
+│         Your Agent          │  ← your code + skills
+├─────────────────────────────┤
+│       Harness (OS)          │  ← loop, tools, context, memory
+├─────────────────────────────┤
+│       Model API (CPU)       │  ← raw inference
+└─────────────────────────────┘
+
+The OS determines what the CPU can do.
+The harness determines what the model can do.
+```
+
+**Phase 0 project:** Write a 2-page doc in your own words defining:
+workflow vs agent · the 5 patterns · the 4 context primitives · orchestrator-worker.
+If you can't write it without looking, you haven't understood it yet.
+
+---
+
+## PHASE 1 — BUILD YOUR FIRST AGENT FROM SCRATCH (Weeks 3-5)
+
+Write the agent twice. Raw API first. Agent SDK second. Feel the difference.
+
+### Build #1 — The Raw Loop (~100 lines of Python)
+
+```python
+messages = [{"role": "user", "content": user_input}]
+stop_reason = None
+
+while stop_reason != "end_turn":
+    response = client.messages.create(
+        model="claude-sonnet-4-5",
+        messages=messages,
+        tools=tools
+    )
+    stop_reason = response.stop_reason
+
+    if stop_reason == "tool_use":
+        tool_results = []
+        for block in response.content:
+            if block.type == "tool_use":
+                result = execute_tool(block.name, block.input)
+                tool_results.append({
+                    "type": "tool_result",
+                    "tool_use_id": block.id,
+                    "content": result
+                })
+        messages.append({"role": "assistant", "content": response.content})
+        messages.append({"role": "user", "content": tool_results})
+```
+
+Give it 3 tools: `web_search` · `read_file` · `write_file`
+Run on a research task. Read every step of the trace.
+
+### Build #2 — The Same Agent on Claude Agent SDK
+
+```
+Add to your agent:
+  CLAUDE.md          → project conventions the agent reads
+  skills/research/   → SKILL.md defining output format
+  hooks/post_tool.py → PostToolUse: auto-format every written file
+  Task tool          → spawn a sub-agent for a subtask
+```
+
+**Phase 1 project:** Daily briefing agent.
+Reads your Markdown notes + RSS feeds.
+Writes a summarised briefing to disk every morning.
+Run it for a week. Watch it fail. Fix it.
+
+---
+
+## PHASE 2 — REAL ARCHITECTURE (Weeks 6-9)
+
+Build on **LangGraph + Deep Agents.** This is the production stack.
+
+### LangGraph Gives You
+
+```python
+from langgraph.graph import StateGraph
+from langgraph.checkpoint.postgres import PostgresSaver
+
+# State machine with durable checkpointing
+graph = StateGraph(AgentState)
+graph.add_node("plan", plan_node)
+graph.add_node("execute", execute_node)
+graph.add_edge("plan", "execute")
+
+# Survive any process kill — resume exactly where you left off
+checkpointer = PostgresSaver.from_conn_string(POSTGRES_URL)
+app = graph.compile(checkpointer=checkpointer)
+```
+
+### Middleware Hooks
+
+```python
+# Customise without forking the packaged harness
+@agent.before_agent
+def inject_context(state): ...        # runs before the loop starts
+
+@agent.wrap_model_call
+def add_system_prompt(call): ...      # wraps every LLM call
+
+@agent.before_tools
+def validate_tool_input(tool): ...    # runs before any tool executes
+
+@agent.after_tools
+def log_tool_result(result): ...      # runs after any tool executes
+```
+
+### Sub-Agents with Isolated Context
+
+```
+Lead Agent (plans, writes TODO list)
+    │
+    ├── Search Sub-Agent 1  ← isolated context
+    ├── Search Sub-Agent 2  ← isolated context
+    └── Search Sub-Agent 3  ← isolated context
+                │
+            Files on Disk
+                │
+         Writer Agent
+                │
+         Final Report
+```
+
+> Sub-agents = isolated context, not just parallel processes.
+> Each sub-agent has NO access to the lead agent's memory.
+> They communicate via files and compressed summaries.
+
+**Phase 2 project:** Research Analyst Agent
+→ Lead agent plans → spawns 3 search sub-agents in parallel
+→ Sub-agents write results to files → citation sub-agent verifies
+→ Writer agent produces Markdown with inline citations
+→ State persists via PostgresSaver (kill → resume)
+→ Human-in-the-loop interrupt before exceeding $1 in tokens
+→ Ship a LangSmith trace URL with your README
+
+---
+
+## PHASE 3 — BUILD THE HARNESS YOURSELF (Weeks 10-13)
+
+**This is the highest-leverage phase in the roadmap.**
+You will never make the right trade-offs in production until you've built one.
+
+### The 10 Components
+
+```python
+class MiniHarness:
+    # 1. Loop control
+    def run(self, task: str) -> str:
+        while self.stop_reason != "end_turn":
+            self._step()
+
+    # 2. Tool dispatch
+    @tool(schema={"type": "function", ...})
+    def web_search(self, query: str) -> str: ...
+
+    # 3. Context management
+    def _maybe_compact(self):
+        if self._token_count() > 0.85 * self.context_limit:
+            self._summarise_and_compress()
+
+    # 4. Persistence
+    def _checkpoint(self, state: dict):
+        self.db.execute(
+            "INSERT INTO runs VALUES (?,?,?)",
+            [self.run_id, self.step, json.dumps(state)]
+        )
+
+    # 5. Sub-agent orchestration
+    def spawn(self, task: str) -> "MiniHarness":
+        child = MiniHarness(context=[])  # isolated context
+        return child
+
+    # 6. Skills / progressive disclosure
+    def _load_skill(self, name: str) -> str:
+        return Path(f"skills/{name}/SKILL.md").read_text()
+
+    # 7. Hooks
+    def pre_tool(self, tool_name: str, args: dict): ...
+    def post_tool(self, tool_name: str, result: str): ...
+
+    # 8. Observability
+    @tracer.start_as_current_span("tool_call")
+    def _execute_tool(self, name: str, args: dict): ...
+
+    # 9. Sandboxing
+    def execute_code(self, code: str) -> str:
+        return modal.run_sandbox(code)  # never exec() directly
+
+    # 10. Auth brokering
+    def call_api(self, service: str, **kwargs):
+        creds = vault.get(service)  # model never sees the key
+        return requests.post(endpoint, headers={"Authorization": creds})
+```
+
+**Phase 3 project:** 1,500-line mini-harness in Python.
+Must include: tool registry · CLAUDE.md loader · SKILL.md progressive disclosure
+· sub-agent spawn primitive · filesystem offload (>20K tokens → disk)
+· auto-compaction at 85% · pluggable hooks · OpenTelemetry · SQLite resume.
+
+Deliverable: a 1,000-word post-mortem comparing your harness to
+Claude Agent SDK and Deep Agents. What you got right. What you cut.
+What you'd do differently.
+
+---
+
+## PHASE 4 — EVALS & REGRESSION HARNESS (Weeks 14-17)
+
+> Without this, every "improvement" is vibes.
+
+### The 4 Eval Types
+
+```
+1. Single-turn evals
+   Input → Output → deterministic check
+   "Did it produce valid JSON?"  "Is the answer factually correct?"
+   Cheapest. Run constantly. Block CI on failure.
+
+2. Trajectory evals
+   Input → [tool_1, tool_2, tool_3] → expected sequence
+   "Did it search before answering?"
+   "Did it call calc before making a financial claim?"
+
+3. LLM-as-judge
+   Response → LLM grader → score on rubric
+   Use for: research reports, code review, explanations
+   Calibrate weekly against human-graded examples.
+
+4. End-state evals
+   Before state → agent runs → After state → compare
+   "Did the correct rows get written to the database?"
+   "Did the right files get created?"
+```
+
+### Regression Harness in GitHub Actions
+
+```yaml
+# .github/workflows/eval.yml
+name: Agent Regression
+on: [pull_request]
+
+jobs:
+  eval:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run golden dataset
+        run: python evals/run_golden.py
+      - name: Check pass rate
+        run: |
+          PASS_RATE=$(cat results/pass_rate.json | jq .rate)
+          BASELINE=$(cat evals/baseline.json | jq .rate)
+          python -c "
+          pass_rate = $PASS_RATE
+          baseline = $BASELINE
+          if pass_rate < baseline - 0.03:
+              print(f'FAIL: {pass_rate:.1%} < baseline {baseline:.1%} - 3pp')
+              exit(1)
+          print(f'PASS: {pass_rate:.1%} (baseline {baseline:.1%})')
+          "
+```
+
+**Phase 4 project:**
+→ Golden dataset: 30-50 hand-graded questions (3 difficulty levels)
+→ Deterministic graders for factual queries
+→ LLM-as-judge with 5-criterion rubric
+→ Trajectory eval: did the agent plan, spawn sub-agents, cite, stay under budget?
+→ Wire into GitHub Actions: block merge if pass rate drops 3+ points
+→ Production sampling: 1% of live traces auto-graded nightly
+
+---
+
+## PHASE 5 — PRODUCTION HARDENING (Forever)
+
+### 1. Cost Discipline
+
+```python
+# Cache system prompt, CLAUDE.md, tool definitions — up to 90% savings
+client.messages.create(
+    system=[{"type": "text", "text": system_prompt,
+             "cache_control": {"type": "ephemeral"}}],
+    ...
+)
+
+# Route by difficulty
+model = (
+    "claude-haiku-4-5" if complexity < 0.3 else
+    "claude-sonnet-4-5" if complexity < 0.8 else
+    "claude-opus-4-6"
+)
+
+# Batch API: 50% off for non-real-time work
+```
+
+### 2. Latency
+
+```python
+# Parallel tool calls — ALWAYS
+# Anthropic's own research agent system prompt:
+# "you MUST use parallel tool calls"
+results = await asyncio.gather(
+    search_tool("query_1"),
+    search_tool("query_2"),
+    read_file("context.md")
+)
+
+# Sub-agent fan-out
+# 60-step sequential → 10-step lead + 5 parallel 10-step sub-agents
+```
+
+### 3. Safety and Sandboxing
+
+```python
+# NEVER do this:
+exec(model_output)            # ← model output in main process
+
+# ALWAYS do this:
+result = modal.run_sandbox(   # ← isolated container
+    code=model_output,
+    timeout=30,
+    network_access=False
+)
+
+# NEVER pass credentials to model context:
+# BAD:  "Use API key sk-abc123 to call the service"
+# GOOD: creds = vault.get("service"); call_api(creds=creds)
+```
+
+### 4. Monitoring and Drift
+
+```python
+# Alert thresholds — set these on day one
+ALERTS = {
+    "token_cost_per_request_usd": 0.50,    # alert above $0.50
+    "tool_call_failure_rate":      0.05,    # alert above 5%
+    "llm_judge_score":             0.70,    # alert below 70%
+    "p95_latency_seconds":        10.0,    # alert above 10s
+}
+
+# Re-baseline after EVERY model upgrade
+# Harnesses encode assumptions about model limitations.
+# Those assumptions go stale when the model changes.
+```
+
+### 5. Resilience
+
+```python
+# Durable execution for any agent running > 60 seconds
+from langgraph.checkpoint.postgres import PostgresSaver
+
+# Checkpoint after every node — rewind and fork always possible
+@graph.node
+def expensive_step(state):
+    result = do_expensive_thing(state)
+    return {**state, "result": result}
+    # LangGraph auto-checkpoints after each node
+```
+
+---
+
+## THE 5 PORTFOLIO PROJECTS
+
+Pick one. Build it this weekend.
+
+```
+Level       Project                         Proves
+──────────────────────────────────────────────────────────────
+Beginner    Mobile App + SLM               Edge AI, resource constraints
+Intermediate Self-Improving Coding Agent   Agentic loops, debugging
+Advanced    Video Editor Agent             Multimodal AI, tool integration
+Expert      Personal Life OS Agent         Context management, privacy
+Master      Enterprise Workflow Agent      Production orchestration
+```
+
+### Project 1 — Mobile App + SLM (Beginner)
+
+Offline-first mobile app using a small language model. Zero API costs. Complete privacy.
+
+- Lazy-load models on demand, unload under memory pressure
+- Sliding context window with semantic chunking
+- 4-bit quantisation for older devices, 8-bit for newer
+- Batch inference to reduce battery wake cycles
+
+### Project 2 — Self-Improving Coding Agent (Intermediate)
+
+Agent writes code → runs tests → learns from failures → loops until functional.
+
+- Plan → Execute → Test → Reflect loop with max iteration limit
+- Isolated execution per task with resource limits
+- Memory: short-term (last 5) + long-term (successes) + failure memory (error signatures)
+- Static analysis before execution — detect dangerous operations
+
+### Project 3 — Video Editor Agent (Advanced)
+
+Fork open-source editor. User says "make this cinematic." Agent handles cuts, transitions, colour.
+
+- Vision model analyses frames + audio model analyses dialogue
+- Intent translation: "cinematic" → concrete parameters
+- Scene detection via frame-difference analysis
+- Incremental preview — only re-render affected sections
+
+### Project 4 — Personal Life OS Agent (Expert)
+
+Manages calendar, finances, and health. Plans months ahead. Detects burnout.
+
+- Real-time ingestion: calendar, finance, health, communications
+- Personal knowledge graph of entities and relationships
+- Background thread every 6 hours checking anomalies
+- Value alignment: user states priorities — recommendations validated against them
+- All data encrypted at rest with user-controlled keys
+
+### Project 5 — Enterprise Workflow Agent (Master)
+
+Monitors Slack/Jira → plans execution → delegates → reports with audit logs.
+
+- Event-driven: Slack, Jira, email, monitoring systems
+- Multi-agent delegation: orchestrator → communication, data, analysis, documentation agents
+- Self-healing: exponential backoff, circuit breakers, retry decisions
+- Immutable audit log: every action, authorisation, outcome
+- Human-in-the-loop: agent proposes plan before executing on critical workflows
+
+---
+
+## THE STACK
+
+```
+Framework:     LangGraph 1.0 + Deep Agents
+               (NOT CrewAI for prod / NOT OpenAI Swarm / NOT Assistants API)
+
+Harness ref:   Claude Agent SDK
+               Same harness as Claude Code.
+               CLAUDE.md + Skills + sub-agents + hooks + filesystem-as-memory.
+
+Observability: LangSmith       (if using LangGraph)
+               Braintrust       (framework-agnostic, $249/mo CI gating)
+               Arize Phoenix    (open-source, OTEL-native, free)
+
+Skip in 2026:  OpenAI Swarm            (not production-ready)
+               OpenAI Assistants API   (sunsetting mid-2026)
+               No-code agent platforms (throwaway only)
+               Custom vector store     (measure recall first)
+```
+
+---
+
+## BENCHMARK NUMBERS (May 2026)
+
+```
+SWE-bench Verified (coding):
+  Claude Opus 4.7   ~87.6%
+  GPT-5.5           ~88.7%
+
+GAIA (general agent tasks):
+  Claude Sonnet 4.5   74.6%   ← leads
+
+τ-bench (customer service):
+  Claude Mythos Preview   89.2%   ← leads
+
+Key stat: same benchmark, different harness = swing of 10–36 points.
+The model matters less than the harness.
+```
+
+---
+
+## THE ADVANCED SCIENCE (Context for Engineers)
+
+### Attention and Context Windows
+
+AI models use attention to weight which tokens matter most at each step.
+Attention cost grows quadratically with input length — double the input, 4× the compute.
+This is why the **Compress** context primitive is not optional: it's survival.
+
+**YaRN (Yet another RoPE extensioN)** — ICLR 2024
+Extends context from 4K to 128K tokens using 10× fewer training steps.
+Works by treating high-frequency and low-frequency position components differently.
+Models with YaRN can read 500 pages in one context window.
+
+### Training Stability at Scale (mHC)
+
+**mHC — Manifold-Constrained Hyper-Connections** (DeepSeek, arxiv:2512.24880, Dec 2025)
+
+Standard residual connections preserve signal by adding input to output.
+Hyper-Connections (HC) added 4 parallel streams for better performance —
+but broke identity mapping, causing loss spikes at step 12,000.
+
+mHC fixes this by projecting the stream-mixing matrix onto a Sinkhorn manifold
+(a mathematical surface where every row and column sums to 1).
+Identity mapping is automatically preserved. Result: 6.7% overhead, zero loss spikes,
+evaluated at 3B/9B/27B parameter scales.
+
+**Harness parallel:** The same principle applies to harness design.
+Constrain the system at the right point (context compaction at 85%, credential brokering,
+sandboxed execution) and the whole system becomes stable under real production load.
+
+---
+
+## QUICK REFERENCE
+
+| Term                | Definition                                                     |
+| ------------------- | -------------------------------------------------------------- |
+| Agent               | AI that makes its own control-flow decisions in a loop         |
+| Workflow            | Fixed control flow you wrote — use unless you need an agent    |
+| Harness             | Everything between you and the model API                       |
+| Context window      | The model's RAM — all it can see at once                       |
+| Context engineering | Deciding what goes into that RAM at every step                 |
+| Tool                | A function the model can call                                  |
+| Sub-agent           | Agent spawned with isolated context                            |
+| Eval                | Automated test measuring if the agent got better or worse      |
+| LLM-as-judge        | Using an AI to grade another AI's output                       |
+| YaRN                | Extends context to 128K tokens — 10× efficient                 |
+| mHC                 | Stable training architecture — 6.7% overhead, zero spikes      |
+| Checkpoint          | Saved state for resume after process kill                      |
+| Sandboxing          | Code runs in isolated container — model never has server creds |
+| OTEL                | OpenTelemetry — standard for tracing every call                |
+
+---
+
+## THE CLOSING ARGUMENT
+
+> 57% of teams now have agents in production.
+> 89% of those have observability wired.
+> Quality is the #1 barrier (32% of teams cite it).
+>
+> The entire field is bottlenecked on engineers who can build evals and harnesses.
+> Not on engineers who can call an LLM API.
+> That is the job opening.
+
+**Next:**
+
+1. Pick one project
+2. Build it this weekend
+3. Document everything — architecture decisions, failures, recoveries
+4. Build in public
+
+**Expertise is the only job security left.
+Production systems are the only portfolio that matters.**
+
+---
+
+_AI Engineering by <user> · Oriro-Labs Learning Series · 2026 · MIT License_
+_Public domain knowledge — share freely, build openly, teach everyone._

package/skills/craft/become-an-ai-engineer-26/community/BUILDS.md

@@ -0,0 +1,13 @@
+# Community Builds
+
+Shipped something using this roadmap? Add it here via PR. One line each:
+
+```
+- [Project name](link) — what it does · which phase/project it proves · @yourhandle
+```
+
+See [CONTRIBUTING.md](../CONTRIBUTING.md).
+
+---
+
+<!-- Add your build below this line -->

package/skills/craft/become-an-ai-engineer-26/community/DISCUSSIONS.md

@@ -0,0 +1,8 @@
+# Discussions
+
+Open questions, research threads, and ongoing conversations. Use GitHub Issues
+for anything actionable; use this file to log durable findings worth keeping.
+
+## Open threads
+
+- _(none yet — start one)_

package/skills/craft/become-an-ai-engineer-26/phases/phase-0-mental-models/README.md

@@ -0,0 +1,14 @@
+# Phase 0 — Build Correct Mental Models (Weeks 1-2)
+
+**Do not write a single line of agent code yet.**
+
+Read `SKILL.md` → _Phase 0_. By the end you should be able to explain, in your
+own words and without looking:
+
+- Workflow vs Agent (and why building an agent when you need a workflow costs 10×)
+- The 5 workflow patterns (chaining, routing, parallelisation, orchestrator-worker, evaluator-optimiser)
+- The 4 context primitives (Write, Select, Compress, Isolate)
+- The harness stack (model = CPU, harness = OS)
+
+**Deliverable:** the 2-page concept doc. Use [`project/TEMPLATE.md`](project/TEMPLATE.md).
+If you can't write it without looking, you haven't understood it yet.