dojo.md 0.1.0 → 0.2.0

package/courses/github-actions-cicd/scenarios/level-5/cicd-consulting-engagement.yaml

@@ -0,0 +1,61 @@
+meta:
+  id: cicd-consulting-engagement
+  level: 5
+  course: github-actions-cicd
+  type: output
+  description: "Lead a CI/CD consulting engagement — diagnose and transform a client's broken CI/CD infrastructure as an external consultant"
+  tags: [github, actions, consulting, transformation, engagement, master]
+
+state: {}
+
+trigger: |
+  You're a senior DevOps consultant hired for a 12-week, $200K
+  engagement to transform CI/CD at a 300-engineer healthcare company.
+  They're 18 months into a failed GitHub Actions migration from Jenkins
+  and are about to lose their largest customer ($40M contract) due to
+  compliance gaps.
+
+  Client diagnostic findings (Week 1):
+  - 200 repos: 80 still on Jenkins, 100 on GitHub Actions (inconsistent
+    config), 20 with no CI at all
+  - Average deployment: 2 per week (target: 10 per day)
+  - Failed deployments: 30% (target: < 5%)
+  - Mean time to recovery: 6 hours (target: < 30 minutes)
+  - No automated security scanning (HIPAA violation risk)
+  - Self-hosted runners run on unpatched VMs with shared credentials
+  - The Jenkins→Actions migration was led by one person who left
+  - No documentation of the current CI/CD architecture
+  - The compliance team manually generates audit reports (2 weeks/quarter)
+  - Developer satisfaction with CI/CD: 18% (industry benchmark: 65%)
+
+  Client stakeholders:
+  - CTO: "We need to pass the HIPAA audit in 90 days or we lose the
+    customer. Fix this."
+  - VP Engineering: "My teams are demoralized. They've been promised a
+    'better CI/CD' for 18 months and things got worse."
+  - CISO: "I have no visibility into what's deployed or how. Our
+    security posture is unacceptable."
+  - Engineering Manager: "The last consultant just wrote a document.
+    We need someone who will actually help us fix things."
+
+  Task: Design the complete consulting engagement. Write: the client
+  diagnostic report (executive summary with quantified risk), the
+  12-week transformation roadmap (phased with weekly deliverables),
+  the quick wins for Weeks 1-3 (building credibility and addressing
+  the HIPAA audit deadline), the technical implementation plan (complete
+  the Jenkins migration, standardize Actions, fix security), and the
+  handoff package (documentation, training, maintenance runbooks).
+
+assertions:
+  - type: llm_judge
+    criteria: "Diagnostic report quantifies risk — connects CI/CD failures to business impact ($40M customer at risk), identifies the root cause of the failed migration (single person, no documentation, no plan), and presents findings without blame"
+    weight: 0.35
+    description: "Risk-quantifying diagnostic"
+  - type: llm_judge
+    criteria: "Roadmap addresses the HIPAA deadline — Weeks 1-3 focus on the most critical compliance gaps (security scanning, audit trails, runner security), Weeks 4-8 complete the migration, Weeks 9-12 establish sustainability, and the 90-day HIPAA audit timeline is achievable"
+    weight: 0.35
+    description: "HIPAA-focused roadmap"
+  - type: llm_judge
+    criteria: "Handoff package ensures sustainability — includes documentation of all CI/CD architecture, runbooks for common operations, training curriculum for the team, and a monitoring dashboard so the client can maintain the system after the consultant leaves"
+    weight: 0.30
+    description: "Sustainability-ensuring handoff"

package/courses/github-actions-cicd/scenarios/level-5/cicd-industry-benchmarks.yaml

@@ -0,0 +1,63 @@
+meta:
+  id: cicd-industry-benchmarks
+  level: 5
+  course: github-actions-cicd
+  type: output
+  description: "Publish CI/CD benchmarks — create the definitive 'State of CI/CD' report with cross-industry analysis and DORA insights"
+  tags: [github, actions, benchmarks, DORA, research, industry-analysis, master]
+
+state: {}
+
+trigger: |
+  You're the Head of Research at a developer tools company publishing
+  the annual "State of CI/CD 2026" report. This report is read by
+  60,000+ engineering leaders. You have data from 3,000 organizations.
+
+  Raw data highlights:
+  - DORA metrics by company size:
+    | Metric              | Startup | Mid-market | Enterprise |
+    |---------------------|---------|-----------|------------|
+    | Deploy frequency    | 12/day  | 3/day     | 2/week     |
+    | Lead time           | 1 hr    | 8 hrs     | 5 days     |
+    | MTTR                | 15 min  | 1 hr      | 8 hrs      |
+    | Change failure rate | 5%      | 8%        | 15%        |
+
+  - CI/CD platform market share:
+    GitHub Actions: 62% | GitLab CI: 18% | Jenkins: 12% |
+    CircleCI: 4% | Other: 4%
+
+  - Key correlations:
+    Teams with CI under 10 min: 3x more deploys, 50% lower failure rate
+    Teams using OIDC: 75% fewer credential incidents
+    Teams with automated rollback: 80% lower MTTR
+    Monorepo teams: 2x CI cost but 1.5x deployment frequency
+
+  - Emerging trends:
+    AI in CI/CD: 28% using AI for test selection, 15% for failure
+    prediction, 8% for automated remediation
+    Platform engineering: 45% have dedicated platform teams (up from 12%)
+    Supply chain security: 52% pin actions by SHA (up from 8%)
+
+  - Industry breakdown (deployment frequency):
+    SaaS: 15/day | E-commerce: 8/day | Fintech: 3/day |
+    Healthcare: 1/day | Government: 2/month | Gaming: 20/day
+
+  Task: Write the "State of CI/CD 2026" report. Include: the executive
+  summary, the benchmarking methodology, the DORA metrics analysis by
+  company size and industry, the emerging trends analysis (AI, platform
+  engineering, supply chain), the strategic recommendations, and the
+  predictions for 2027-2028.
+
+assertions:
+  - type: llm_judge
+    criteria: "Methodology is credible — describes sample selection, survey design, data validation, limitations (survivorship bias, self-reporting), and statistical significance. Acknowledges correlation ≠ causation for the key findings"
+    weight: 0.35
+    description: "Credible methodology"
+  - type: llm_judge
+    criteria: "Analysis reveals actionable insights — explains why enterprise DORA metrics lag (compliance overhead, organizational complexity), identifies the 10-minute CI threshold as a tipping point, and provides specific recommendations by company size"
+    weight: 0.35
+    description: "Actionable insight analysis"
+  - type: llm_judge
+    criteria: "Predictions are specific and bold — makes concrete predictions about AI adoption rates, platform engineering growth, GitHub Actions market share evolution, and the decline of Jenkins, with reasoning grounded in the trend data"
+    weight: 0.30
+    description: "Specific bold predictions"

package/courses/github-actions-cicd/scenarios/level-5/cicd-ma-integration.yaml

@@ -0,0 +1,73 @@
+meta:
+  id: cicd-ma-integration
+  level: 5
+  course: github-actions-cicd
+  type: output
+  description: "M&A CI/CD integration — consolidate CI/CD platforms and practices across acquired companies"
+  tags: [github, actions, M&A, integration, consolidation, migration, master]
+
+state: {}
+
+trigger: |
+  You're the CTO overseeing CI/CD integration for 3 recently acquired
+  companies. Each uses a different CI/CD platform and has different
+  engineering practices. The board expects full integration within 18
+  months.
+
+  Parent company (1,000 engineers):
+  - GitHub Actions with sophisticated platform team
+  - DORA Elite: 50 deploys/day, 2hr lead time, 15min MTTR, 2% CFR
+  - Fully automated, OIDC auth, compliance automation
+  - Cost: $2M/year for CI/CD infrastructure
+
+  Acquisition A — SaaS startup (200 engineers):
+  - CircleCI with custom orbs
+  - Fast but fragile: 30 deploys/day but 15% failure rate
+  - No security scanning, no compliance controls
+  - 80 repos, all on CircleCI
+  - Cost: $400K/year
+
+  Acquisition B — Enterprise software (300 engineers):
+  - Jenkins on-premise (50+ plugins, custom Groovy pipelines)
+  - Slow but stable: 2 deploys/week, 3% failure rate
+  - Heavy compliance (SOX, HIPAA), manual audit processes
+  - 120 repos on Bitbucket + Jenkins
+  - Cost: $800K/year (hardware + maintenance)
+
+  Acquisition C — AI/ML company (150 engineers):
+  - Mix of GitHub Actions (for apps) and custom Python scripts (for
+    ML pipelines)
+  - ML model training uses separate GPU pipeline outside CI/CD
+  - Jupyter notebooks deployed via manual process
+  - 40 repos on GitHub + internal GitLab for ML experiments
+  - Cost: $300K/year
+
+  Integration constraints:
+  - Must consolidate to GitHub Actions within 18 months
+  - Cannot disrupt any acquisition's shipping velocity during migration
+  - Must achieve SOX + HIPAA compliance for all entities
+  - Must retain 90% of acquired engineering talent
+  - Budget: $3M for migration (separate from ongoing costs)
+  - ML pipeline integration is the most technically complex
+
+  Task: Design the M&A CI/CD integration strategy. Write: the assessment
+  of each acquisition's CI/CD maturity, the migration plan for each
+  (CircleCI → Actions, Jenkins → Actions, custom scripts → Actions),
+  the compliance harmonization plan (bringing all entities to SOX + HIPAA),
+  the ML pipeline integration strategy, and the talent retention plan.
+  Include: the 18-month timeline with quarterly milestones and the $3M
+  budget allocation.
+
+assertions:
+  - type: llm_judge
+    criteria: "Migration plans are tailored — CircleCI migration is relatively straightforward (similar paradigm), Jenkins migration requires complete rethinking (Groovy → YAML), ML pipeline migration needs custom approach (GPU runners, notebook deployment), and each has a specific timeline"
+    weight: 0.35
+    description: "Tailored migration plans"
+  - type: llm_judge
+    criteria: "Compliance harmonization is realistic — addresses the gap between startup's no-compliance and enterprise's heavy compliance, proposes a unified compliance framework that works for all entities, and the timeline achieves SOX + HIPAA compliance within the 18-month window"
+    weight: 0.35
+    description: "Realistic compliance harmonization"
+  - type: llm_judge
+    criteria: "Budget allocation and timeline are justified — the $3M is distributed across the 3 acquisitions proportionally to migration complexity, the timeline accounts for parallel workstreams, and the talent retention plan addresses flight risk during migration disruption"
+    weight: 0.30
+    description: "Justified budget and timeline"

package/courses/github-actions-cicd/scenarios/level-5/cicd-product-development.yaml

@@ -0,0 +1,68 @@
+meta:
+  id: cicd-product-development
+  level: 5
+  course: github-actions-cicd
+  type: output
+  description: "Build a CI/CD product — design and launch a CI/CD SaaS startup competing in the developer tools market"
+  tags: [github, actions, product, startup, SaaS, go-to-market, master]
+
+state: {}
+
+trigger: |
+  You're the co-founder/CTO of a startup building "PipelineHQ" — a
+  next-generation CI/CD platform. You've raised $8M in seed funding
+  and have 18 months of runway. Your thesis: GitHub Actions is powerful
+  but enterprise teams need better observability, cost management, and
+  governance on top of it.
+
+  Market analysis:
+  - TAM: $12B (CI/CD and DevOps tools market)
+  - SAM: $3B (CI/CD platforms and add-ons)
+  - SOM: $300M (GitHub Actions ecosystem tools)
+  - Competitors: Harness ($2B valuation), Buildkite ($200M), Earthly
+    ($20M), Depot ($5M), Trunk ($30M)
+  - Gap: No single tool combines Actions analytics + cost management +
+    governance in a developer-friendly package
+
+  Product vision:
+  1. Cost Intelligence: Real-time per-team cost tracking, optimization
+     recommendations, budget enforcement
+  2. Pipeline Analytics: DORA metrics, failure patterns, bottleneck
+     detection, workflow performance trending
+  3. Governance: Policy-as-code for Actions, action allowlisting,
+     compliance automation, audit reporting
+  4. Developer Experience: Smart caching, predictive failures, auto-
+     retry with root cause analysis
+
+  Technical constraints:
+  - Must work with GitHub Actions (not replace it)
+  - GitHub App integration (webhook-based, no code injection)
+  - Customer workflow data stays in their GitHub (privacy)
+  - Must handle 10,000+ workflow runs/day per customer
+
+  Go-to-market:
+  - 18-month runway ($8M, burn rate $450K/month)
+  - 10-person engineering team
+  - Need $2M ARR to raise Series A
+  - PLG + enterprise sales motion
+
+  Task: Design the complete product strategy. Write: the MVP roadmap
+  (3-month scope for first paying customer), the technical architecture
+  (GitHub App, webhook processing, analytics engine), the pricing
+  strategy (free/team/enterprise tiers), the GTM plan (PLG adoption
+  loop, enterprise sales playbook), and the competitive positioning.
+  Include the financial projection and Series A pitch narrative.
+
+assertions:
+  - type: llm_judge
+    criteria: "MVP scope is achievable in 3 months by 10 engineers — focuses on one killer feature (likely cost intelligence or analytics, not all 4), has a clear first customer profile, and the scope decision is justified with competitive analysis"
+    weight: 0.35
+    description: "Achievable MVP scope"
+  - type: llm_judge
+    criteria: "Technical architecture is sound — GitHub App with webhook processing handles the scale, respects data privacy constraints, the analytics engine can process 10K+ runs/day, and the architecture supports the product roadmap beyond MVP"
+    weight: 0.35
+    description: "Sound technical architecture"
+  - type: llm_judge
+    criteria: "Financial projection reaches $2M ARR in 18 months — PLG funnel conversion assumptions are realistic, pricing tiers incentivize upgrade, enterprise sales pipeline is modeled, and the burn rate / runway math is consistent"
+    weight: 0.30
+    description: "Realistic financial projection"

package/courses/github-actions-cicd/scenarios/level-5/cicd-regulatory-landscape.yaml

@@ -0,0 +1,72 @@
+meta:
+  id: cicd-regulatory-landscape
+  level: 5
+  course: github-actions-cicd
+  type: output
+  description: "Navigate CI/CD regulatory landscape — map how global regulations shape CI/CD requirements and prepare for emerging frameworks"
+  tags: [github, actions, regulatory, compliance, global, supply-chain, master]
+
+state: {}
+
+trigger: |
+  You're the Chief Compliance Officer of a global software company
+  operating in 30 countries. Regulations increasingly require
+  demonstrable CI/CD controls, and you need to map the landscape.
+
+  Current regulatory requirements affecting CI/CD:
+
+  1. EU Cyber Resilience Act (CRA, effective 2027):
+  - Requires documented secure development lifecycle for all software
+    sold in the EU
+  - CI/CD must produce verifiable build provenance (SLSA Level 3)
+  - Vulnerability management must be automated in the pipeline
+  - Penalty: up to 15M EUR or 2.5% of global turnover
+
+  2. US Executive Order 14028 (Software Supply Chain):
+  - Software Bill of Materials (SBOM) required for government contracts
+  - CI/CD must generate and sign SBOMs automatically
+  - Build environment integrity must be verifiable
+  - Applies to your $200M government contract portfolio
+
+  3. SEC Cybersecurity Rules:
+  - Material cybersecurity incidents must be disclosed in 4 days
+  - CI/CD audit trails are part of "reasonable controls"
+  - Board must oversee cybersecurity risk management
+
+  4. India CERT-In Directive:
+  - 6-hour incident reporting requirement
+  - CI/CD logs must be retained for 180 days
+  - Your Bangalore development center is affected
+
+  5. Proposed EU AI Act CI/CD implications:
+  - High-risk AI systems must have documented CI/CD processes
+  - Model training pipelines need audit trails
+  - Continuous monitoring must be built into deployment
+
+  Emerging trends:
+  - SLSA framework adoption for build provenance
+  - Sigstore/cosign for artifact signing
+  - OpenSSF Scorecard for repository security
+  - NIST SSDF (Secure Software Development Framework)
+  - Software Liability Act proposals (US + EU)
+
+  Task: Map the regulatory landscape. Write: the regulatory matrix
+  (which regulations apply to which CI/CD components), the compliance
+  gap analysis, the unified CI/CD compliance framework (one pipeline
+  that satisfies all regulations), the SLSA Level 3 implementation
+  plan, and the future-proofing strategy. Include the board brief on
+  regulatory risk.
+
+assertions:
+  - type: llm_judge
+    criteria: "Regulatory matrix is comprehensive — maps each regulation to specific CI/CD controls (build provenance, SBOM generation, audit trail retention, artifact signing), identifies overlapping requirements, and handles jurisdictional complexity"
+    weight: 0.35
+    description: "Comprehensive regulatory matrix"
+  - type: llm_judge
+    criteria: "SLSA Level 3 implementation is technically correct — includes hermetic builds, build provenance generation, artifact signing with Sigstore, SBOM generation (CycloneDX or SPDX), and the GitHub Actions workflow modifications needed"
+    weight: 0.35
+    description: "Correct SLSA Level 3 implementation"
+  - type: llm_judge
+    criteria: "Future-proofing strategy anticipates emerging regulations — prepares for Software Liability Act, EU AI Act CI/CD requirements, and NIST SSDF compliance before they take effect, with a regulatory monitoring process"
+    weight: 0.30
+    description: "Anticipatory future-proofing"

package/courses/github-actions-cicd/scenarios/level-5/comprehensive-cicd-system.yaml

@@ -0,0 +1,66 @@
+meta:
+  id: comprehensive-cicd-system
+  level: 5
+  course: github-actions-cicd
+  type: output
+  description: "Design a comprehensive CI/CD system — architect a 5-layer CI/CD platform for a Fortune 500 company"
+  tags: [github, actions, comprehensive, enterprise, architecture, Fortune-500, master]
+
+state: {}
+
+trigger: |
+  You're the CTO of a Fortune 500 company ($10B revenue, 4,000
+  engineers) designing a unified CI/CD system. The company has 8
+  business units acquired over 15 years, each with different platforms,
+  processes, and cultures. The board mandated unification after a
+  regulatory finding.
+
+  Current state:
+  - BU1 Cloud (800 eng): GitHub Actions, mature, 50 deploys/day
+  - BU2 Consumer (600 eng): GitLab CI, fast, no compliance
+  - BU3 Enterprise (500 eng): Jenkins, slow, SOX/SOC 2 compliant
+  - BU4 Data (400 eng): Airflow + custom scripts, no standard CI
+  - BU5 Mobile (300 eng): Bitrise + Fastlane, iOS/Android specific
+  - BU6 IoT (200 eng): Custom embedded CI, hardware-in-loop testing
+  - BU7 Fintech (150 eng): CircleCI, PCI DSS compliant
+  - BU8 AI Lab (50 eng): No CI, Jupyter notebooks + manual deploys
+
+  Design the CI/CD system in 5 layers:
+
+  Layer 1 — Governance: Universal policies, compliance mapping (SOX,
+  SOC 2, PCI DSS, HIPAA, FedRAMP), action allowlisting, audit trails
+  Layer 2 — Platform: GitHub Actions standardization, runner fleet,
+  reusable workflows, secret management (Vault)
+  Layer 3 — Delivery: Deployment pipelines per BU (web, mobile, IoT,
+  ML, data pipelines), environment management, release automation
+  Layer 4 — Intelligence: DORA metrics, cost optimization, failure
+  prediction, anomaly detection, compliance reporting
+  Layer 5 — Experience: Developer portal, workflow catalog, self-
+  service onboarding, documentation, training
+
+  Constraints:
+  - $20M budget over 3 years
+  - Cannot disrupt any BU's current shipping velocity
+  - Must handle specialized needs (IoT hardware testing, ML GPU
+    training, mobile signing)
+  - Must satisfy 5 compliance frameworks simultaneously
+  - Must handle 10,000+ pipeline runs per day
+
+  Task: Design all 5 layers. For each, write the detailed design,
+  migration plan, interactions with other layers, and success metrics.
+  Then write: the 3-year phased roadmap, the $20M budget allocation,
+  and the board presentation for approval.
+
+assertions:
+  - type: llm_judge
+    criteria: "5-layer design is coherent — each layer builds on the one below, there are clear interfaces, and the design handles the extreme diversity of 8 BUs without over-simplifying. Specialized needs (IoT, ML, mobile) are accommodated, not forced into a single template"
+    weight: 0.35
+    description: "Coherent 5-layer design"
+  - type: llm_judge
+    criteria: "Migration plan handles complexity — phases the 8 BUs based on difficulty and business criticality, doesn't force simultaneous migration, provides BU-specific migration paths (Jenkins→Actions, GitLab→Actions, custom→Actions), and has rollback plans"
+    weight: 0.35
+    description: "Complexity-handling migration plan"
+  - type: llm_judge
+    criteria: "Budget allocation is justified — the $20M is distributed across layers, BUs, and years with clear rationale, build-vs-buy decisions for each component, and the ROI projection shows when the investment pays back"
+    weight: 0.30
+    description: "Justified budget allocation"

package/courses/github-actions-cicd/scenarios/level-5/master-cicd-shift.yaml

@@ -0,0 +1,76 @@
+meta:
+  id: master-cicd-shift
+  level: 5
+  course: github-actions-cicd
+  type: output
+  description: "Master CI/CD shift — navigate existential threats to the CI/CD platform while maintaining organizational trust and delivery velocity"
+  tags: [github, actions, shift-simulation, crisis, existential, master]
+
+state: {}
+
+trigger: |
+  You're the CTO facing a perfect storm of CI/CD-related crises that
+  threaten the company's delivery capability, compliance posture, and
+  competitive position simultaneously.
+
+  Crisis 1 — Supply chain attack:
+  A critical supply chain attack was discovered in a widely-used GitHub
+  Action (200K+ installations). The malicious version was active for
+  48 hours before detection. Your organization used this action in 85
+  repositories. The malicious code exfiltrated GITHUB_TOKEN and any
+  secrets passed to the action. GitHub has revoked the compromised
+  action but the damage may be done.
+  Questions: How many of your secrets are compromised? Was production
+  code tampered with? Can you trust any deployment from the last 48
+  hours?
+
+  Crisis 2 — Regulatory enforcement:
+  The SEC issued an enforcement action against your company for
+  "inadequate change management controls." They cite: deployments
+  without code review (12% via workflow_dispatch), missing audit trails
+  for 3 months of workflow runs (GitHub data retention expired), and
+  inability to prove who approved what change. The hearing is in 45
+  days. Legal estimates potential fines of $5-10M.
+
+  Crisis 3 — Platform vendor risk:
+  GitHub announced they're deprecating key features you depend on:
+  (a) self-hosted runner auto-scaling API changes in 60 days,
+  (b) organization-level required workflows moving to a new system,
+  (c) pricing restructure eliminating the large runner discount.
+  The changes require 3 months of engineering work with your current
+  (depleted) team.
+
+  Crisis 4 — Engineering exodus:
+  The "State of CI/CD" blog post by your competitor (the one in Crisis
+  2's news) went viral. 8 senior engineers cited it in exit interviews:
+  "Our CI/CD is years behind. I want to work somewhere with modern
+  infrastructure." Your platform team lost 5 of 15 engineers in the
+  last quarter.
+
+  Crisis 5 — Board emergency meeting:
+  The board called an emergency session in 48 hours. They want a
+  unified briefing on all 4 crises above and a strategic response.
+  Two board members are pushing to "abandon GitHub Actions entirely."
+  One board member (former CISO) is pushing to "freeze all deployments
+  until the supply chain attack is fully investigated."
+
+  Task: Navigate all 5 crises. Write: the 48-hour action plan, the
+  supply chain attack response (forensics, containment, communication),
+  the SEC response strategy (legal coordination, evidence preservation,
+  remediation), the vendor risk mitigation plan, the talent retention
+  strategy, and the board presentation that unifies everything into a
+  coherent strategic response.
+
+assertions:
+  - type: llm_judge
+    criteria: "Board presentation unifies all crises — shows interconnections (supply chain attack worsens SEC case, talent loss impacts remediation capacity), doesn't present 5 separate plans but a coherent strategy, and addresses both board extremes (abandon GitHub vs freeze deployments)"
+    weight: 0.35
+    description: "Unified crisis board presentation"
+  - type: llm_judge
+    criteria: "Supply chain response is forensically sound — identifies all 85 affected repos, traces which secrets were exfiltrated, verifies deployment integrity for the 48-hour window, rotates all compromised credentials, and implements permanent supply chain protections (SHA pinning, action allowlist)"
+    weight: 0.35
+    description: "Forensically sound supply chain response"
+  - type: llm_judge
+    criteria: "SEC response balances legal and technical — coordinates with legal counsel, preserves evidence (does not delete any data), presents a proactive remediation plan, and the corrective actions address the specific findings (workflow_dispatch controls, audit trail retention, approval documentation)"
+    weight: 0.30
+    description: "Legal-technical SEC response"

package/courses/github-pr-review/scenarios/level-2/api-change-review.yaml

@@ -0,0 +1,82 @@
+meta:
+  id: api-change-review
+  level: 2
+  course: github-pr-review
+  type: output
+  description: "Review API changes — evaluate backward compatibility, versioning strategy, and migration paths in API-modifying PRs"
+  tags: [github, pr-review, API, backward-compatibility, versioning, intermediate]
+
+state: {}
+
+trigger: |
+  You're reviewing a PR titled "Update user API — v2 endpoints." The PR
+  modifies the public REST API consumed by 3 mobile apps and 12 third-
+  party integrations. The changes include:
+
+  Change 1 — Response format change:
+  ```
+  // Before (v1)
+  GET /api/users/123
+  { "id": 123, "name": "Jane Doe", "email": "jane@example.com",
+    "created": "2024-01-15" }
+
+  // After (v2 in this PR)
+  GET /api/users/123
+  { "id": "usr_123", "fullName": "Jane Doe",
+    "contact": { "email": "jane@example.com" },
+    "metadata": { "createdAt": "2024-01-15T00:00:00Z" } }
+  ```
+
+  Change 2 — Endpoint restructuring:
+  ```
+  // Before
+  GET /api/users/:id/orders
+  POST /api/users/:id/orders
+
+  // After
+  GET /api/v2/users/:id/orders?include=items,shipping
+  POST /api/v2/orders  (userId in request body instead of URL)
+  ```
+
+  Change 3 — Authentication change:
+  ```
+  // Before: API key in query parameter
+  GET /api/users?api_key=abc123
+
+  // After: Bearer token in header
+  GET /api/v2/users
+  Authorization: Bearer <token>
+  ```
+
+  Change 4 — New pagination:
+  ```
+  // Before: offset-based
+  GET /api/users?page=2&per_page=20
+
+  // After: cursor-based
+  GET /api/v2/users?cursor=eyJpZCI6MTIzfQ&limit=20
+  ```
+
+  The PR has no migration guide, no deprecation timeline, and the old
+  endpoints are removed (not deprecated).
+
+  Task: Review the API changes. Write: the backward compatibility
+  assessment (what breaks and for whom), the versioning strategy
+  recommendation (how v1 and v2 should coexist), the migration plan for
+  consumers (timeline, communication, tooling), the review comments for
+  each change (what to keep, what to change, what to add), and the
+  checklist for future API-breaking PRs.
+
+assertions:
+  - type: llm_judge
+    criteria: "Breaking changes are comprehensively identified — field renames (name→fullName), nested restructuring (email→contact.email), ID format change (123→usr_123), endpoint moves, auth mechanism change, and pagination model change are all flagged as breaking"
+    weight: 0.35
+    description: "Comprehensive breaking change identification"
+  - type: llm_judge
+    criteria: "Versioning strategy is sound — recommends keeping v1 endpoints active with deprecation timeline (not immediate removal), API versioning approach (URL path vs header), and coexistence strategy during migration period"
+    weight: 0.35
+    description: "Sound versioning strategy"
+  - type: llm_judge
+    criteria: "Migration plan is actionable — includes consumer communication (changelog, migration guide), timeline with milestones, tooling suggestions (compatibility layer, SDK updates), and a checklist that would prevent this situation in future API PRs"
+    weight: 0.30
+    description: "Actionable migration plan"

package/courses/github-pr-review/scenarios/level-2/automated-review-tooling.yaml

@@ -0,0 +1,53 @@
+meta:
+  id: automated-review-tooling
+  level: 2
+  course: github-pr-review
+  type: output
+  description: "Set up automated review tools — configure linters, CodeQL, and automated checks to catch issues before human review"
+  tags: [github, pr-review, automation, linting, CodeQL, CI, intermediate]
+
+state: {}
+
+trigger: |
+  Your team spends 40% of review time on issues that could be caught
+  automatically: formatting inconsistencies, unused imports, type errors,
+  known vulnerability patterns, and missing test coverage. You've been
+  tasked with setting up automated review tools to reduce this burden.
+
+  Current state:
+  - TypeScript/React monorepo with Node.js backend
+  - GitHub Actions CI runs build and tests only
+  - No linting in CI (only local, inconsistently used)
+  - No security scanning
+  - No automated review comments
+  - PRs average 3 review rounds before merge (1st round is mostly
+    formatting and lint issues)
+
+  Available tools to evaluate and configure:
+  1. ESLint + Prettier (formatting/linting)
+  2. GitHub CodeQL (security analysis)
+  3. Codecov/Coveralls (test coverage reporting)
+  4. danger.js (custom PR checks: size, description, labels)
+  5. GitHub branch protection rules
+
+  Task: Design the automated review pipeline. Write: the GitHub Actions
+  workflow file(s) for the automated checks, the configuration for each
+  tool (ESLint rules focused on bugs not style, Prettier config, CodeQL
+  queries, danger.js rules for PR hygiene), the branch protection rules
+  that require these checks to pass, and the team rollout plan (how to
+  introduce without disrupting existing workflows). Include the expected
+  impact on review time and number of review rounds.
+
+assertions:
+  - type: llm_judge
+    criteria: "GitHub Actions workflow is correct and complete — runs lint, type-check, CodeQL, coverage, and danger.js as separate jobs or steps, uses proper triggers (pull_request), caches dependencies, and handles monorepo paths correctly"
+    weight: 0.35
+    description: "Complete CI workflow"
+  - type: llm_judge
+    criteria: "Tool configurations are practical — ESLint focuses on bug-catching rules over style (since Prettier handles formatting), CodeQL uses relevant queries for JS/TS, danger.js checks PR size/description/labels, and coverage thresholds are reasonable"
+    weight: 0.35
+    description: "Practical tool configurations"
+  - type: llm_judge
+    criteria: "Rollout plan is realistic — addresses existing code that won't pass new rules (baseline approach), gradual enforcement strategy, team communication, and doesn't block all PRs on day one"
+    weight: 0.30
+    description: "Realistic rollout plan"