@salesforce/afv-skills 1.45.0 → 1.47.0

package/skills/education-cloud-course-catalog-migrate/SKILL.md

@@ -0,0 +1,321 @@
+---
+name: education-cloud-course-catalog-migrate
+description: "Use this skill to migrate course catalog data from external sources (CSV, PDF, website) and bulk-create Learning and LearningCourse records in Education Cloud. Triggers when a user wants to import courses, parse course catalog documents, or set up a course catalog for an academic term. Handles multi-source data ingestion, intelligent interpretation of course structures (lecture+lab linked courses via IsLinkedOnly flag, credit hours, course types), batch creation via sObject REST API, data quality validation, conflict resolution for existing courses, and gap identification. Explains what's missing (descriptions, types) and requests additional sources. Pauses for clarification on multi-component courses and conflicting data. DO NOT TRIGGER for creating individual Learning Course records manually, for creating Learning Programs (different workflow), or for enrollment/registration workflows."
+metadata:
+  version: "1.0"
+  domains:
+    - "Education"
+  accessCheck:
+    - type: "accessCheck"
+      value: "IndustriesEducation.userHasEducationCloudAccess"
+  cliTools:
+    - tool: ["python3"]
+      semver: ">=3.9.0"
+---
+
+# Migrating Course Catalog to Education Cloud
+
+Import course catalog data from multiple sources (CSV, PDF, website) and bulk-create Learning Course records with structure detection and data quality validation.
+
+## Scope
+
+- **In scope**: Multi-source ingestion (CSV, PDF, web), Learning + LearningCourse bulk creation via sObject REST API (Composite batches up to 200), linked course detection (lecture+lab via IsLinkedOnly), credit hour parsing (Duration/DurationUnit), course type mapping, data quality validation, conflict resolution, gap identification/remediation
+- **Out of scope**: Creating individual Learning Course records manually (delegate to metadata generation skills), creating Learning Programs (Learning.Type = 'LearningProgram' + separate child object), creating External Learning (Learning.Type = 'ExternalLearning'), course sections/offerings (CourseOffering object), enrollment/registration workflows (no dedicated skill yet — ask the user), SIS sync (separate integration workflow)
+
+---
+
+## Required Inputs
+
+Gather or infer before proceeding:
+
+- **Target org**: Active Salesforce org with Education Cloud enabled
+- **Primary data source**: CSV file, PDF document, or website URL with course catalog data
+- **Business Account**: the Academic Operations Business Account to link courses to (e.g., College of Engineering)
+- **Academic term** (optional): context only, not stored on Learning Course
+
+Additional sources (gathered during workflow):
+- Supplementary data sources for missing fields (descriptions, types, mappings)
+- Lecture-lab mapping files (if multi-component courses detected)
+- Conflict resolution decisions (if existing courses found)
+
+Defaults unless specified:
+- Course status: Active (IsActive = true on Learning)
+- Batch size: 25 records per Composite API call as an agent (API supports 200; tool-call payload limits are the binding constraint — see Step 7)
+- Data quality validation: Enabled
+- Learning.Type: 'LearningCourse' (required for course catalog entries)
+
+---
+
+## Execution Transport
+
+**Resolve the API version first — do not hardcode.** `GET /services/data/` (no version segment) returns an endpoint map already resolved to the org's max version, not a `{version, label, url}` list. Pipe it into `scripts/resolve_api_version.py` to get `{"version": "69.0"}`; use as `vXX.X` everywhere below, including the transport probe.
+
+Then probe transport health: `GET /services/data/vXX.X/limits` via headless-360 dispatch. 2xx → route all calls through it. On absent/4xx/5xx → probe other transports (other Salesforce MCP, custom MCP), use first healthy, announce it. None healthy → stop, ask user to connect one; never fabricate. (SOQL `/query` is routable on headless-360 — confirmed by live test; `/tooling/query` and `?fields=` are unconfirmed, use describe + direct record reads instead.)
+
+Never fall back to `sf` CLI or shell out for org credentials — a slow/oversized batch is a batch-size problem (shrink it, step 7), not a transport failure, and CLI shell-out trips an unbypassable approval gate on some surfaces. A version too new for the org returns `404 NOT_FOUND` on an otherwise-healthy transport — re-read `GET /services/data/`, drop to the org's max version, retry, don't switch transport. See `references/gotchas.md` for both.
+
+---
+
+## Workflow
+
+All steps are sequential. Do not skip or reorder. If blocked, stop and ask for missing context.
+
+### Phase 1 — Ingest Primary Source
+
+1. **Identify data source type** — determine if primary source is CSV, PDF, or website URL. Ask user to provide the source if not already specified.
+
+2. **Parse primary source** — extract course data:
+   - For CSV: Use `Read` tool to load file, parse columns
+   - For PDF: Use `Read` tool (supports PDF extraction), parse text. Over 20 pages, read in sequential page-range chunks and merge parsed rows before validating completeness — see `references/large-catalog-handling.md` §1
+   - For website: Use `WebFetch` tool to retrieve HTML, parse course listings
+   
+   Look for: course codes (identity — split into SubjectAbbreviation + CourseNumber, see step 8/9), course names (Name), credit hours (Duration + DurationUnit), descriptions (Description), course types (Lecture/Lab/Exam/Drill/Recitation), prerequisites, corequisites, linked course indicators (IsLinkedOnly), delivery modality (if the same course code repeats with different Duration/DurationUnit — see `references/large-catalog-handling.md` §2).
+
+3. **Validate data completeness** — pipe the parsed records into the completeness helper:
+   ```bash
+   python3 scripts/validate_completeness.py --required "CourseCode,Name,Duration" --recommended "Description,CourseType" <<'JSON'
+   [ { "CourseCode": "...", "Name": "...", ... }, ... ]
+   JSON
+   ```
+   The report gives present/missing counts per field. Optional fields (prerequisites, corequisites, linked courses) are not checked — their absence never blocks progress.
+
+   If any required field shows `missing > 0`, report error and stop. If recommended fields are missing, note the gaps and continue to Phase 2.
+
+### Phase 2 — Identify Gaps and Request Additional Sources
+
+4. **Report data quality** — print a summary table showing which fields are present and which are missing. Example:
+   ```text
+   Data Quality Report:
+   • Course codes: 120/120 OK
+   • Course names: 120/120 OK
+   • Credit hours: 120/120 OK
+   • Descriptions: 115/120 (5 missing)
+   • Course types: 118/120 (2 missing)
+   ```
+
+5. **Request supplementary sources** — if key fields are missing (descriptions, types), ask user: "Do you have additional sources — a website, department document, or mapping file?"
+   
+   - If yes: receive additional source, parse it, and merge data with primary source
+   - If no: offer to create courses with placeholder values or stop and wait for complete data
+   
+   Repeat this step until all gaps are filled or user confirms to proceed with incomplete data.
+
+   If subject-to-Business-Account mapping is ambiguous, resolve every ambiguous case in one pass (source doc + live public catalog + course level/number patterns) and present them together, not one round at a time (see `references/gotchas.md`).
+
+6. **Detect multi-component courses** — pipe the course codes into the lecture/lab pairing helper:
+   ```bash
+   python3 scripts/detect_linked_courses.py --suffixes "L,LAB" <<'JSON'
+   [ "BIO-201", "BIO-201L", ... ]
+   JSON
+   ```
+   Each `{"lecture", "lab"}` pair returned is a candidate multi-component course.
+   - If any pairs are found, pause and explain: "I found X multi-component (lecture + lab) courses. Structure them as (1) separate standalone courses, or (2) linked courses (IsLinkedOnly flag, students must take both)?"
+   
+   If user selects "linked courses", request a mapping file or confirm the naming pattern is reliable.
+
+   **Detect modality duplicates** — group parsed rows by course code. Same code, different `Duration`/`DurationUnit`/`Description` = a modality variant (e.g. Online vs. On-Campus), not a duplicate — never silently dedupe/merge. Resolve the representation once per run, not per-course. See `references/large-catalog-handling.md` §2.
+
+### Phase 3 — Batch Create Learning and LearningCourse Records
+
+7. **Prepare batch creation plan** — pipe the parsed course records into the batching helper. The Composite API accepts up to 200 records/call, but an agent's tool-call payload has a smaller practical ceiling (~56KB/150 records). Use `--batch-size 25` when executing as an agent (raise toward 200 only if the calling environment confirms it can handle larger payloads):
+   ```bash
+   python3 scripts/batch_courses.py --batch-size 25 <<'JSON'
+   [ { "Name": "...", ... }, ... ]
+   JSON
+   ```
+   Each element of the returned array is one Composite API call's worth of records. Print: "Starting course catalog import. I'll use Composite API to create records in batches. Total: X courses."
+
+   If a batch still won't inline through a healthy dispatch transport, re-run with a smaller `--batch-size` (e.g. 10) instead of switching transport. Track dispatch vs. fallback counts and report both in the Phase 5 summary.
+
+   **Over 200 courses (8+ batches)**: disclose total batch count upfront, print the Batch N complete line after every batch (not just at the end), and — only if the surface exposes a parallel-task/subagent tool — follow the fan-out rules in `references/large-catalog-handling.md` §3-4. Never assume that capability exists; default to sequential batches.
+
+8. **Check for existing courses** — before creating Batch 1, scope the check to this batch's own course codes, never a global fetch (an unrelated catalog entry already in the org is not a conflict).
+
+   Split this batch's course codes first (`scripts/split_course_code.py`) — key the check on those identity fields, never `CipCode` (many orgs leave it null — see `references/gotchas.md`).
+
+   > **Headless-360 note**: SOQL `/query` is routable via the headless-360 dispatch tool (confirmed by live test) — query `LearningCourse`, matching each split `{SubjectAbbreviation, CourseNumber}` pair exactly (SOQL has no tuple `IN`, so OR the pairs): `SELECT Id, LearningId, Name, SubjectAbbreviation, CourseNumber FROM LearningCourse WHERE (SubjectAbbreviation = 'BIO' AND CourseNumber = '201') OR ...`. If Ids are unknown and a SOQL lookup isn't available, ask the user to supply them (or a prior import manifest). Never do this via Bash/CLI — see `references/gotchas.md`.
+
+   If any rows come back (i.e. a SubjectAbbreviation+CourseNumber pair in this batch already exists in the org), pause and explain conflict:
+   - Show existing course details vs. new import data side-by-side
+   - Check if existing courses have dependencies (course offerings, enrollments)
+   - Offer options: (1) Update existing courses, (2) Skip existing courses, (3) Cancel and review manually
+   
+   Wait for user decision before proceeding.
+
+9. **Create Batch 1** — two-step process for each course:
+
+   > **Field placement (verified via describe)**: `CipCode`, `Duration`, `DurationUnit`, `ProviderId` are createable on **Learning ONLY** — `ProviderId` on the child fails with `INVALID_FIELD_FOR_INSERT_UPDATE`. LearningCourse takes `Name`, `LearningId`, `CourseType`, `IsLinkedOnly`, `SubjectAbbreviation`, `CourseNumber`, `Description`, `CourseLevelDescription`, `FieldOfStudy` — it inherits the account link via `LearningId`. Confirm with describe before building the body.
+
+   > **CipCode is opt-in.** Federal CIP taxonomy field, not the course's identity — leave unset unless the source supplies a real CIP code and the user confirms.
+
+   **Step 1: Create Learning (parent) records via Composite API:**
+   ```http
+   POST /services/data/vXX.X/composite/sobjects
+   {
+     "allOrNone": false,
+     "records": [
+       {
+         "attributes": {"type": "Learning", "referenceId": "learning1"},
+         "Name": "<course-name>",
+         "Type": "LearningCourse",
+         "Description": "<description>",
+         "ProviderId": "<account-id>",
+         "Duration": <hours>,
+         "DurationUnit": "Credit Hours",
+         "IsActive": true
+       },
+       ...
+     ]
+   }
+   ```
+
+   Capture each returned Id (order matches the `records` array) to use as `LearningId` in Step 2.
+
+   **Step 2: Create LearningCourse (child) records via Composite API:**
+   ```http
+   POST /services/data/vXX.X/composite/sobjects
+   {
+     "allOrNone": false,
+     "records": [
+       {
+         "attributes": {"type": "LearningCourse"},
+         "Name": "<course-record-name>",
+         "LearningId": "<learning-id-from-step-1>",
+         "SubjectAbbreviation": "<subject-prefix>",
+         "CourseNumber": "<course-number>",
+         "CourseType": "<parsed-course-type>",
+         "IsLinkedOnly": false
+       },
+       ...
+     ]
+   }
+   ```
+
+   Set `CourseType` from the parsed/mapped course type for that record (see the Gotchas table for value mapping) — do not hard-code it. A lab record must get `CourseType: "Lab"`, not `"Lecture"`.
+
+   > **CourseType** picklist values (verified): `Lecture`, `Lab`, `Exam`, `Drill`, `Recitation`.
+
+   > **`restrictedPicklist: true` means only today's values are valid to submit, not that the set is closed.** If a source value (e.g. `DurationUnit` = "Program Points") isn't in describe's list, don't silently drop/remap it — offer adding the picklist value via metadata alongside remapping or a placeholder (see `references/gotchas.md`).
+
+   Print progress: `Batch 1 complete — X Learning + X LearningCourse records created`.
+
+10. **Handle multi-component courses** — if user selected "linked courses" in step 6:
+    - Create lecture courses with `CourseType: "Lecture"`, `IsLinkedOnly = false`
+    - Create lab courses with `CourseType: "Lab"`, `IsLinkedOnly = true`
+    - Use the `detect_linked_courses.py` pairs (or a user-supplied mapping file) to determine which labs are linked-only
+    - Print: `X linked lab courses created (IsLinkedOnly = true)`
+
+11. **Repeat for remaining batches** — continue steps 8-10 for Batch 2, Batch 3, etc. until all courses created.
+
+### Phase 4 — Data Quality Validation and Conflict Resolution
+
+12. **Scan for incomplete records** — after all batches complete, read back each newly created record by the Id captured from its create response:
+    ```http
+    GET /services/data/vXX.X/sobjects/Learning/<id>
+    GET /services/data/vXX.X/sobjects/LearningCourse/<id>
+    ```
+
+    > **Headless-360 note**: SOQL `/query` is routable via the headless-360 dispatch tool (confirmed by live test) — a `CreatedDate = TODAY` query works as an alternative to iterating captured Ids with direct record GETs.
+
+    Run the completeness helper **separately per sObject** — `CourseType` exists only on `LearningCourse`, `Description` only on `Learning`; combining both shapes into one call false-positives every parent as missing `CourseType`:
+    ```bash
+    python3 scripts/validate_completeness.py --required "Name" --recommended "Description" <<'JSON'
+    [ { "Name": "...", "Description": "...", ... }, ... ]   # Learning read-backs only
+    JSON
+    ```
+    Repeat with `--recommended "CourseType"` against LearningCourse read-backs only. Identify courses with missing descriptions (Learning report) or missing types (LearningCourse report). Print data quality report (same format as step 4).
+
+13. **Detect conflicting data** — if multiple sources provided different values for the same course, flag conflicts, e.g. "CHEM-301: website says X, PDF says Y — which should I use?" Wait for user decision, then update the conflicting courses.
+
+14. **Remediate gaps** — if user provides additional sources to fill gaps, parse them and update the incomplete courses:
+    ```http
+    PATCH /services/data/vXX.X/sobjects/Learning/<id>
+    {"Description": "..."}
+    
+    PATCH /services/data/vXX.X/sobjects/LearningCourse/<id>
+    {"CourseType": "<parsed-course-type>"}
+    ```
+
+### Phase 5 — Summary and Verification
+
+15. **Generate import summary** — print final report:
+    ```text
+    Course catalog import finished!
+    Summary:
+    • X Learning + X LearningCourse records created
+    • Y standalone, Z linked lab courses (IsLinkedOnly)
+    • All courses linked to <Business Account name>, set Active
+    Data sources used: 1. <primary> 2. <supplementary 1> 3. <supplementary 2>
+    ```
+
+    **If any gaps were remediated from a supplementary source (step 5/14 ran)**, the final report MUST also include, in this order:
+    - **Data Quality Report** — two tables, "Initial data (from `<primary source>`)" and "After supplementary source (`<supplementary source>`)", each listing every field from step 4 with its present/total count. This is the before/after pair, not just the final counts.
+    - **Gap Remediation** — grouped by field (e.g. "Courses with missing descriptions (filled from `<source>`)"), a numbered list of every remediated course with the value it received. One list per remediated field.
+
+16. **Offer verification** — ask user: "Would you like me to query specific courses to verify the data? For example, I can show you a course with its LearningCourse child record, or check for any remaining data quality issues."
+
+17. **Check the running user can actually see what was just created** — don't just flag this, check it (see `references/gotchas.md` for the resolve-running-user + FLS/RecordTypeVisibility pattern), against every object touched (`Learning`, `LearningCourse`, `Account` if Person Accounts came up). Name any gap and its profile, offer the specific fix, and wait for confirmation before writing `Profile`/`PermissionSet` metadata. Default-Private OWD/sharing is org-wide, not per-user — flag it, don't offer to change it.
+
+---
+
+## Rules / Constraints
+
+| Constraint | Rationale |
+|-----------|-----------|
+| Use Composite API, batched at 25 records/call for agent execution (API supports up to 200) | sObject REST standard in H360 MCP; a 150-record payload (~56KB) exceeds typical tool-call limits |
+| Create Learning first, then LearningCourse | LearningCourse.LearningId requires parent to exist first |
+| Always set Learning.Type = 'LearningCourse' | Learning is polymorphic — Type distinguishes courses from programs |
+| Always query for existing courses before creating | Prevents duplicates; lets user decide on conflicts |
+| Pause for clarification on multi-component courses | IsLinkedOnly flag affects enrollment behavior — user must confirm structure |
+| Report data quality gaps before proceeding | Lets user provide complete data before committing to org |
+| Wait for user decision on conflicting data | Agent cannot determine which source is authoritative |
+| Do not invent missing data | Placeholder values must be explicitly approved by user |
+| Set ProviderId (Business Account) on the parent Learning ONLY | Createable on Learning, not LearningCourse — see step 9 field-placement note |
+| Set all courses to Active status by default (IsActive = true) | Must be Active to appear in catalog search/registration |
+| Use sObject REST API, not CLI commands for org calls | H360 MCP skills have no execution environment for reaching the org. The `scripts/` helpers are local data transforms (no org call) — if no Bash tool is available, perform the equivalent check inline instead of skipping it |
+| Use tool-agnostic language | Skill ships to multiple AI surfaces beyond Claude Code |
+| Over 200 courses or 20+ source pages: follow `references/large-catalog-handling.md`, not ad hoc chunking | Improvised chunking risks split-page data loss or duplicate creates |
+
+---
+
+## Gotchas
+
+See `references/gotchas.md` for known failure modes and resolutions (parsing quirks, missing Business Account, picklist mapping, partial Composite failures, and more).
+
+---
+
+## Output Expectations
+
+Deliverables:
+- X Learning records created (Type = 'LearningCourse')
+- X LearningCourse child records created (LearningId lookups)
+- Data quality report showing completeness per field
+- Import summary listing all data sources used
+- Optional: course verification via direct record reads or SOQL `/query` (both routable via headless-360)
+
+No files are produced — all output is org records created via sObject REST API.
+
+---
+
+## Cross-Skill Integration
+
+| Need | Delegate to |
+|------|-------------|
+| Create individual Learning Course manually | Platform metadata generation skills |
+| Create Learning Programs | No dedicated skill yet — ask the user |
+| Set up course sections/offerings | No dedicated skill yet — ask the user |
+| Configure enrollment eligibility rules | No dedicated skill yet — ask the user |
+
+---
+
+## Reference File Index
+
+- `references/gotchas.md` — known failure modes and their resolutions
+- `references/gotchas-detail.md` — lower-frequency gotchas, linked from gotchas.md
+- `references/large-catalog-handling.md` — rules for >200 courses or >20-page source docs: paginated reads, modality-duplicate detection, checkpointing, gated parallel fan-out (steps 2, 6, 7)
+- `scripts/resolve_api_version.py` — extracts org's max API version from the headless-360 endpoint map (Execution Transport)
+- `scripts/validate_completeness.py` — required/recommended field presence check (steps 3, 12)
+- `scripts/detect_linked_courses.py` — lecture/lab course-code pairing (step 6)
+- `scripts/detect_modality_variants.py` — flags CourseCode rows differing by Duration/DurationUnit/Description, for `large-catalog-handling.md` §2
+- `scripts/batch_courses.py` — Composite API batch splitting (step 7)
+- `scripts/split_course_code.py` — SubjectAbbreviation/CourseNumber splitting (step 9)

package/skills/education-cloud-course-catalog-migrate/references/gotchas-detail.md

@@ -0,0 +1,16 @@
+# Gotchas — edge cases
+
+Lower-frequency gotchas moved out of the core `gotchas.md` table to keep it scannable. Same
+resolution format, just less common in a typical run.
+
+| Issue | Resolution |
+|-------|------------|
+| Linked course naming pattern is not reliable | Always request a mapping file if user selects "linked courses" option |
+| Existing courses have enrollments | Warn user that updating courses with active enrollments may impact students — offer to skip instead |
+| User provides incomplete data and wants to proceed | Explicitly confirm with user: "5 courses will be created without descriptions. Proceed?" |
+| Bulk import creates thousands of records under default Private sharing, or Person Accounts is enabled without the running profile having record-type visibility | Not this skill's job to change sharing/security settings, but flag it once at the end of the run: recommend the user (or an admin) review OWD/sharing rules and record-type visibility on the objects just populated, since a large private-shared import can be invisible to other admins/agents |
+| describe pulls the full object schema (tens of KB) just to check one field's picklist values | Call describe once per object per run and cache the result (picklist values, createable fields) for reuse across every subsequent lookup on that object instead of re-describing per field/record |
+| A subject code's department/account mapping is ambiguous (shared prefixes, courses left in the catalog from a retired program) | Before guessing or asking the user to correct it round-by-round, cross-check every ambiguous mapping in one pass: the source document, the institution's live public catalog if one exists, and course level/number patterns (e.g. HRM vs. general Business) — surfacing all ambiguous cases together with your proposed resolution up front costs one round of user review instead of several |
+| Running user can't see records/values just created (recurring pattern — reported across multiple test sessions, not unique to this skill) | Metadata creates (new field, object, record type, or a picklist value on a record-type-scoped business process) don't auto-grant visibility to any profile. Resolve the running user via `GET /services/data/vXX.X/` → parse the `identity` URL's trailing `/<userId>` → `SELECT ProfileId, Profile.Name FROM User WHERE Id = '<userId>'`. Then check the relevant grant for what changed: a new/changed field → `FieldPermissions`; a record type touched (e.g. Person Account enabled in a prior session) → `RecordTypeVisibility`; a new object → `ObjectPermissions`. Name the exact gap and profile, offer the specific fix, and wait for confirmation before writing `Profile`/`PermissionSet` metadata — it's profile-wide, not scoped to this run |
+| Adding a picklist value fixes the *global* value set but not necessarily every record type's available values | If the picklist field is record-type-scoped (a business process / per-record-type value set), a newly added value may still be unavailable on record types that existed before the value was added. Check record-type-specific availability, not just the field's global `restrictedPicklist` value list, before declaring the fix complete |
+| Agent stalls indefinitely (no error, no progress) on ANY shelled-out org call — an ad hoc existing-record check (e.g. "check which CipCodes already exist"), or fetching org credentials via "sf org display" to hand-roll direct REST/Composite calls | Some agent surfaces show a manual "approval needed even in bypass" gate for any shell command containing `$(...)`, backticks, brace expansion, pipes into `python3`/`jq`, or eval — a gate that cannot be auto-approved and has no timeout of its own. Never shell out (CLI + command substitution, or CLI piped into a parser) for org lookups or credentials, even as a "faster" alternative to the MCP dispatch tool; use the headless-360 dispatch tool's SOQL `/query` or a direct REST read instead — same result, no approval gate, no CLI dependency |

package/skills/education-cloud-course-catalog-migrate/references/gotchas.md

@@ -0,0 +1,16 @@
+# Gotchas
+
+| Issue | Resolution |
+|-------|------------|
+| CSV column headers don't match expected names | Infer column mapping from context (e.g., "Code" → course code, split via `scripts/split_course_code.py` into SubjectAbbreviation/CourseNumber — not CipCode, see step 8/9; "Title" → Name, "Credits" → Duration) |
+| Credit hours stored as "3.0 credits" instead of numeric | Parse and extract numeric value, set DurationUnit = "Credit Hours" |
+| Course codes have inconsistent formatting | Normalize before creating (e.g., "BIO 201" → "BIO-201", "bio-201" → "BIO-201") |
+| PDF has multi-column layout | Text extraction may be out of order — validate course sequence makes sense |
+| Website has pagination | Fetch all pages before parsing (detect "next page" links and loop) |
+| Missing Business Account ID | Ask the user to supply the Account Id (ProviderId), read a known Account with `GET /services/data/vXX.X/sobjects/Account/<id>`, or look it up by name with a SOQL `/query` (confirmed routable via headless-360) |
+| Course type values don't match Education Cloud picklist | Map common values (e.g., "Lec" → "Lecture", "Laboratory" → "Lab", "Drill" → "Drill", "Recitation" → "Recitation") |
+| Composite API returns partial success | Check response for each record — some may fail while others succeed (allOrNone = false) |
+| Two-object create pattern complexity | Create Learning records first (store IDs), then create LearningCourse records with LearningId lookups |
+| Source value not in describe's picklist list (e.g. DurationUnit = "Program Points") | `restrictedPicklist: true` means only active-today values are valid, not that the set is closed — offer the user the option to add the new picklist value via metadata, in addition to remapping or placeholder options. Do not assume it's a dead end |
+
+More edge-case gotchas (linked-course mapping, enrollment conflicts, sharing defaults, describe caching, ambiguous subject codes, post-write visibility, record-type-scoped picklists, shell-out stalls): see `references/gotchas-detail.md`.

package/skills/education-cloud-course-catalog-migrate/references/large-catalog-handling.md

@@ -0,0 +1,42 @@
+# Large Catalog Handling
+
+Applies whenever the parsed course count exceeds **200 courses** (8+ batches at the standard batch size of 25) OR the primary source document exceeds **20 pages**. Below these thresholds, follow the standard Workflow in `SKILL.md` as written — nothing here overrides it.
+
+## 1. Large source documents (PDF/website)
+
+- The `Read` tool caps large PDFs at **~20 pages per call** and requires an explicit page range above 10 pages. Never attempt a single `Read` on a 100+ page catalog PDF.
+- Read in sequential page-range chunks (e.g. `pages: "1-20"`, `"21-40"`, ...). Merge parsed course rows across chunks before running the completeness check (step 3) — do not validate completeness per-chunk, a course description that spans a page boundary will false-positive as missing.
+- For a paginated website catalog, the existing "fetch all pages before parsing" rule (`references/gotchas.md`) already covers this — no change, just note it applies here too.
+- State the total page/chunk count to the user before starting the read pass: `"Source document is 640 pages — I'll read it in 32 chunks of 20 pages."` Silent chunking with no upfront count leaves the user unable to judge how long ingestion will take.
+
+## 2. Modality / delivery-method duplicates
+
+Real catalogs sometimes list the same course twice under different delivery modalities (e.g. Online vs. On-Campus), each with its own `Duration`/`DurationUnit`. This is a **source-data shape issue**, detected during Phase 2 parsing — distinct from the step 8 org-conflict check, which only looks at records *already in the org*.
+
+- After parsing, run `scripts/detect_modality_variants.py` on the parsed rows (JSON array with `CourseCode`, `Duration`, `DurationUnit`, `Description` per row). It groups by `CourseCode` and flags any group whose rows differ in those fields. Use its flagged output for the representation decision below — never silently dedupe (drop one) or silently merge (average/pick one) a flagged group.
+- Before assuming there's no dedicated field for the distinction, run `GET /services/data/vXX.X/sobjects/Learning/describe` and check for a delivery-method/modality field. Education Cloud orgs vary — do not assume absence without checking.
+- If no such field exists on the describe result, pause and ask the user how to distinguish the variants once (not per-course): typically either (a) separate `Learning` records with the modality appended to `Name` (e.g. `"BIO-201 (Online)"`), or (b) a single record if the org treats modality as an offering-level (not course-level) attribute. Apply the chosen pattern to every flagged variant in this run, and state the pattern in the Phase 5 summary.
+
+## 3. Batch-count disclosure and checkpointing
+
+- Before Phase 3 begins, tell the user the total course count and batch count: `"1,438 courses parsed. This will run as 58 batches of 25 (116 Composite API calls). I'll report progress after every batch."`
+- Print the existing `Batch N complete — X Learning + X LearningCourse records created` line (step 9) after **every** batch, not just at the end — this is the checkpoint. The step 8 existing-course check is what makes a checkpoint useful: if this run is interrupted and re-invoked, step 8 will find the already-created courses and skip them, so print it as an explicit resume contract to the user rather than only as a conflict check: `"Resuming — found 425 of 1,438 courses already created from a prior run, continuing from course 426."`
+- Do not build a separate progress-manifest file. The org itself (queryable via step 8) is the source of truth for what's done — a local manifest can drift from it after an interruption and is one more thing to keep in sync.
+
+## 4. Parallel batch fan-out (optional acceleration)
+
+Only attempt this when the current surface exposes a capability to run independent units of work concurrently (e.g. Claude Code's `Agent`/Task tool). **Never assume this capability exists** — Agentforce, plain MCP surfaces, and the ADK eval harness's `vibes-cli` runtime do not expose it. If no such capability is available, run batches sequentially per the standard Workflow — this is the default, not a degraded fallback.
+
+When available, follow these rules exactly:
+
+1. **Fan-out unit is one whole batch**, never a half-batch. A batch's Learning-create-then-LearningCourse-create sequence (step 9) is strictly ordered — the child create needs the parent's returned Id. Splitting a batch's two steps across workers is not allowed.
+2. **Partition before fan-out, by index, not content.** Divide the full course list into disjoint batches up front (course 1-25 → batch 1, 26-50 → batch 2, ...). Two workers must never receive overlapping course records.
+3. **Run step 8 (existing-course check) once, globally, before fan-out** — never per-worker. A per-worker check races: two workers can both see "no conflict" for the same course code in the same instant and both create it.
+4. **Cap concurrency at 4 workers.** Do not fan out all batches at once — Salesforce enforces per-org concurrent long-running-request limits; unbounded fan-out risks `REQUEST_LIMIT_EXCEEDED` that a sequential run would never hit. Launch the next batch only when a worker slot frees up.
+5. **Each worker's prompt must be fully self-contained.** A freshly spawned worker does not see this conversation. Include in its prompt: the exact course-record slice (not a range to look up), the resolved API version, the `ProviderId`, the field-placement rule that `ProviderId`/`Duration`/`DurationUnit` go on `Learning` only, never on `LearningCourse` (and that `CipCode` is opt-in — leave unset unless the source supplies a real CIP code), and that the existing-course conflict check keys on `LearningCourse.SubjectAbbreviation`+`CourseNumber`, never `Learning.CipCode` — restating the gotcha per-worker is cheaper than a worker rediscovering it via a failed create.
+6. **Aggregate before Phase 4/5 — this is a hard barrier.** Collect every worker's created-Ids-and-errors list before running the Phase 4 data quality scan or the Phase 5 summary. Do not run quality validation on a partial result set.
+7. **Isolate failures per batch.** A failed worker must not abort sibling workers. Track outcome per batch (`succeeded` / `failed` / `partial`) and report each in the Phase 5 summary — do not collapse to a single aggregate count when any batch failed or partially failed.
+
+### Verification status
+
+This fan-out path is **not covered by the current ADK eval harness** — the eval runtime (`vibes-cli` surface) has no subagent/parallel-task tool, the same class of gap as headless-360's absence there. Verification for this path is manual, against real large catalogs (e.g. a 1,438-course, two-modality catalog), not automated eval — until that manual pass is done and recorded, treat this as unverified best-effort, and say so plainly if asked whether it's covered by CI.

package/skills/education-cloud-course-catalog-migrate/scripts/batch_courses.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""Split a list of course records into fixed-size batches.
+
+Reads a JSON array of course record objects from stdin. Prints a JSON array
+of batches (each a list of records), each capped at --batch-size records —
+the Composite API's per-call record limit.
+
+Usage:
+    python3 scripts/batch_courses.py --batch-size 200
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+
+def batch(records: list[dict], size: int) -> list[list[dict]]:
+    return [records[i : i + size] for i in range(0, len(records), size)]
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--batch-size", type=int, default=200)
+    args = parser.parse_args()
+
+    records = json.load(sys.stdin)
+    batches = batch(records, args.batch_size)
+    json.dump(batches, sys.stdout, indent=2)
+    print()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/skills/education-cloud-course-catalog-migrate/scripts/detect_linked_courses.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+"""Pair lecture/lab course codes by suffix.
+
+Reads a JSON array of course code strings from stdin. A code is treated as a
+lab section of another code if it equals that code plus one of the configured
+suffixes (default: L, LAB). Prints a JSON array of {"lecture", "lab"} pairs
+for every match found; codes with no match are omitted.
+
+Usage:
+    python3 scripts/detect_linked_courses.py --suffixes L,LAB,-L
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+
+def normalize(code: str) -> str:
+    return code.strip().upper()
+
+
+def detect(codes: list[str], suffixes: list[str]) -> list[dict]:
+    normalized = {normalize(c): c for c in codes}
+    pairs = []
+    for norm_code, original in normalized.items():
+        for suffix in suffixes:
+            if norm_code.endswith(suffix) and len(norm_code) > len(suffix):
+                base = norm_code[: -len(suffix)]
+                if base in normalized:
+                    pairs.append({"lecture": normalized[base], "lab": original})
+    return pairs
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--suffixes", default="L,LAB", help="Comma-separated lab-code suffixes")
+    args = parser.parse_args()
+
+    codes = json.load(sys.stdin)
+    suffixes = [s.strip().upper() for s in args.suffixes.split(",") if s.strip()]
+
+    pairs = detect(codes, suffixes)
+    json.dump(pairs, sys.stdout, indent=2)
+    print()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/skills/education-cloud-course-catalog-migrate/scripts/detect_modality_variants.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+"""Flag rows that share a CourseCode but differ in modality-bearing fields.
+
+Reads a JSON array of course row objects from stdin, each with at least
+"CourseCode" and optionally "Duration", "DurationUnit", "Description". Groups
+rows by CourseCode; any group with 2+ rows that differ in Duration,
+DurationUnit, or Description is a modality variant (e.g. Online vs.
+On-Campus). Prints a JSON array of {"CourseCode", "rows"} for every flagged
+group; groups with no variance are omitted.
+
+Usage:
+    python3 scripts/detect_modality_variants.py < courses.json
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from collections import defaultdict
+
+VARIANT_FIELDS = ("Duration", "DurationUnit", "Description")
+
+
+def detect(rows: list[dict]) -> list[dict]:
+    grouped: dict[str, list[dict]] = defaultdict(list)
+    for row in rows:
+        grouped[row["CourseCode"]].append(row)
+
+    flagged = []
+    for code, group in grouped.items():
+        if len(group) < 2:
+            continue
+        distinct = {tuple(row.get(f) for f in VARIANT_FIELDS) for row in group}
+        if len(distinct) > 1:
+            flagged.append({"CourseCode": code, "rows": group})
+    return flagged
+
+
+def main() -> int:
+    rows = json.load(sys.stdin)
+    flagged = detect(rows)
+    json.dump(flagged, sys.stdout, indent=2)
+    print()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/skills/education-cloud-course-catalog-migrate/scripts/resolve_api_version.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""Extract the org's current max REST API version from a headless-360 endpoint map.
+
+Reads the JSON object returned by `GET /services/data/` (no version segment) from
+stdin — a map of endpoint name to path, e.g. {"limits": "/services/data/v69.0/limits",
+...}, not a `{version, label, url}` list. Scans every value for a `vNN.N` segment and
+prints the highest one found as {"version": "69.0"}.
+
+Usage:
+    python3 scripts/resolve_api_version.py < endpoint_map.json
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+
+VERSION_RE = re.compile(r"v(\d+\.\d+)")
+
+
+def resolve(endpoint_map: dict) -> str:
+    versions = set()
+    for value in endpoint_map.values():
+        if isinstance(value, str):
+            match = VERSION_RE.search(value)
+            if match:
+                versions.add(match.group(1))
+    if not versions:
+        raise ValueError("No vNN.N version found in any endpoint path")
+    return max(versions, key=lambda v: tuple(int(p) for p in v.split(".")))
+
+
+def main() -> int:
+    endpoint_map = json.load(sys.stdin)
+    version = resolve(endpoint_map)
+    json.dump({"version": version}, sys.stdout)
+    print()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/skills/education-cloud-course-catalog-migrate/scripts/split_course_code.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""Split course codes into SubjectAbbreviation + CourseNumber.
+
+Reads a JSON array of course code strings from stdin (e.g. "BIO-201",
+"bio 201l"). Prints a JSON array of {"code", "SubjectAbbreviation",
+"CourseNumber"} objects — LearningCourse has no single CipCode field, so
+these two fields must be populated separately.
+
+Usage:
+    python3 scripts/split_course_code.py
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+
+PATTERN = re.compile(r"^\s*([A-Za-z]+)[\s-]*([0-9]+[A-Za-z]*)\s*$")
+
+
+def split(code: str) -> dict:
+    match = PATTERN.match(code)
+    if not match:
+        return {"code": code, "SubjectAbbreviation": None, "CourseNumber": None}
+    subject, number = match.groups()
+    return {"code": code, "SubjectAbbreviation": subject.upper(), "CourseNumber": number.upper()}
+
+
+def main() -> int:
+    codes = json.load(sys.stdin)
+    results = [split(c) for c in codes]
+    json.dump(results, sys.stdout, indent=2)
+    print()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())