@learnpack/learnpack 5.0.352 → 5.0.354

package/src/scripts/README.md

@@ -0,0 +1,244 @@
+# Descriptions scripts (`src/scripts/`)
+
+Ops scripts for the **step descriptions** of published courses. They are dev/ops
+entry points, not oclif commands: they never appear under `learnpack <something>`
+and are run with `node` against the compiled output.
+
+| Script                    | What it does                                                            | Calls the LLM |
+| ------------------------- | ----------------------------------------------------------------------- | ------------- |
+| `descriptionsS3Backfill`  | Generates descriptions for **published** courses (S3)                   | Yes           |
+| `descriptionsGcsBackfill` | Copies existing descriptions from S3 into the **draft** (GCS)           | No            |
+| `descriptionsSweep`       | Finishes publications that stalled mid-flight; runs on Heroku Scheduler | Yes           |
+
+All three delegate the per-course work to the same service,
+[`generateCourseDescriptions`](../utils/descriptions/generateCourseDescriptions.ts),
+which is also what the post-publish stage uses. They differ only in **which**
+courses they run it on and **what else** they do around it.
+
+## Build and run
+
+```bash
+npm run tsc
+node lib/scripts/<script>.js [flags]
+```
+
+They run against `lib/`, not `src/`, so recompile after editing. There is no
+`--help`: the flags are documented here and in each script's header.
+
+## Environment
+
+Copy from [`.env.example`](../../.env.example) into `learnpack-cli/.env`. These
+scripts read `process.env` directly — they do **not** load the `.env` themselves,
+so export it (`set -a; . ./.env; set +a`) or pass the variables inline.
+
+| Variable                                   | Needed by                                                                       |
+| ------------------------------------------ | ------------------------------------------------------------------------------- |
+| `RIGOBOT_SYSTEM_TOKEN`                     | S3 backfill, sweep (generation)                                                 |
+| `BREATHECODE_SYSTEM_TOKEN`                 | `--emit-events`, sweep (pending events)                                         |
+| `AWS_ACCESS_KEY_ID` / `_SECRET_ACCESS_KEY` | anything reading the published bucket                                           |
+| `AWS_REGION`                               | optional, default `us-east-1`                                                   |
+| `S3_PACKAGES_BUCKET`                       | optional in the backfills (default `learnpack-paquetes`), required by the sweep |
+| `CLOUDFRONT_DISTRIBUTION_ID`               | optional; without it manifests are re-projected but not invalidated             |
+| `GCP_CREDENTIALS_JSON`                     | GCS backfill, sweep, `--mirror-draft`                                           |
+| `GCP_BUCKET_NAME`                          | same; no default on purpose                                                     |
+
+Descriptions are always generated with the Rigobot **service** account, never
+with a user token: they are platform enrichment, not something a creator is
+billed for.
+
+---
+
+## Mode 1 — targeted run (a few slugs, on demand)
+
+The everyday case: a course was published before descriptions existed, or one
+needs remediation. Three flags make the S3 backfill reproduce per course
+everything a publication does, in one process:
+
+```bash
+node lib/scripts/descriptionsS3Backfill.js --slug my-course \
+  --reproject-manifest --mirror-draft --emit-events --concurrency 2
+```
+
+| Flag                   | Without it                                                                              |
+| ---------------------- | --------------------------------------------------------------------------------------- |
+| `--reproject-manifest` | `package-manifest.json` keeps the old descriptions until a later pass                   |
+| `--mirror-draft`       | the draft has no fingerprints, so the **next publication regenerates the whole course** |
+| `--emit-events`        | breathecode is never told the manifest changed                                          |
+
+All three are off by default, so the catalogue-wide run below is unaffected.
+
+Run it once per slug. Do a `--dry-run` first if you want to see the scope —
+it reports what it would generate and mirror without writing anything.
+
+**During high-traffic hours**, lower `--concurrency` (default 5 completions in
+flight per course). A single completion takes ~15s.
+
+Notes:
+
+- `--emit-events` **requires** `--reproject-manifest`. The event reports a
+  manifest update, so emitting it without projecting would state something
+  untrue. It is required rather than implied on purpose: writing to S3 and the
+  CDN as a side effect of another flag is the kind of surprise a backfill should
+  not have.
+- `--emit-events` is rejected together with `--dry-run`: there is no dry run of
+  an event breathecode already received.
+- `--mirror-draft` is independent of the other two — it touches the draft, not
+  the manifest.
+- Prefer `--slug` over `--limit 4` when you care about _which_ courses are
+  touched: `--limit` counts **processed** courses (already-settled ones are
+  skipped without counting) and walks the catalogue by recency.
+
+### What each course does, in order
+
+1. Generate the missing descriptions and write them into the published
+   `initialSyllabus.json` (S3).
+2. `--reproject-manifest`: rebuild `package-manifest.json` from that syllabus
+   (the same `processPackage` the manifest backfill runs) and invalidate its
+   CloudFront path.
+3. `--mirror-draft`: copy the descriptions back into the GCS draft, guarded by
+   the README content hash — anything the teacher edited since publishing is
+   counted as `missed` and waits for the next publication.
+4. `--emit-events`: send `package_manifest_updated` to breathecode.
+
+A failure in step 3 does not change the event status: the published package and
+its manifest are already correct, only the draft lags behind. A failure in step 2
+**does** — see below.
+
+---
+
+## Mode 2 — catalogue backfill (fases 0a / 0b)
+
+The one-off sweep over the ~400 published courses and ~800 drafts. Here the
+defaults are the right ones: manifests are cheaper to project in a single later
+pass than course by course, and the draft is seeded by copy rather than by
+re-generating.
+
+```bash
+# Fase 0a — descriptions for the published catalogue, 50 courses per run.
+# Re-run until it stops reporting processed courses.
+node lib/scripts/descriptionsS3Backfill.js --limit 50
+
+# Then project every manifest in one pass (cheaper than per course).
+npm run backfill:package-manifest
+
+# Fase 0b — seed the drafts by copying from S3. Never calls the LLM.
+node lib/scripts/descriptionsGcsBackfill.js --limit 200
+```
+
+Order matters: 0b reads the **published** syllabus, so 0a must have written it
+first. Whatever 0b cannot reuse (drafts edited since their last publication) is
+left for the next publication of that course, or for a single-slug 0a run.
+
+No events are emitted in this mode. If breathecode needs to be told about the
+catalogue, that is a separate decision — adding `--emit-events` to a 400-course
+run would deliver 400 events.
+
+---
+
+## Flag reference
+
+### `descriptionsS3Backfill`
+
+| Flag                   | Default              | Description                                                       |
+| ---------------------- | -------------------- | ----------------------------------------------------------------- |
+| `--slug <slug>`        | —                    | single course (remediation / on-demand)                           |
+| `--limit <n>`          | 50                   | courses **processed** per run, ordered by syllabus recency        |
+| `--s3-bucket <name>`   | `learnpack-paquetes` | published bucket                                                  |
+| `--gcs-bucket <name>`  | `GCP_BUCKET_NAME`    | draft bucket, for `--mirror-draft`                                |
+| `--region <region>`    | `us-east-1`          | AWS region                                                        |
+| `--concurrency <n>`    | 5                    | completions in flight per course                                  |
+| `--target-words <n>`   | 25                   | words per description                                             |
+| `--dry-run`            | off                  | preview; no writes, no completions                                |
+| `--force`              | off                  | regenerate courses already settled at the current prompt version  |
+| `--no-reconcile`       | off                  | skip the additive syllabus reconciliation                         |
+| `--reproject-manifest` | off                  | rebuild + invalidate `package-manifest.json` per course           |
+| `--mirror-draft`       | off                  | copy the descriptions into the GCS draft                          |
+| `--emit-events`        | off                  | send `package_manifest_updated` (requires `--reproject-manifest`) |
+
+### `descriptionsGcsBackfill`
+
+| Flag                  | Default              | Description                      |
+| --------------------- | -------------------- | -------------------------------- |
+| `--slug <slug>`       | —                    | single course                    |
+| `--limit <n>`         | 50                   | courses per run                  |
+| `--s3-bucket <name>`  | `learnpack-paquetes` | source (published) bucket        |
+| `--gcs-bucket <name>` | `GCP_BUCKET_NAME`    | target (draft) bucket            |
+| `--region <region>`   | `us-east-1`          | AWS region                       |
+| `--dry-run`           | off                  | preview without writing          |
+| `--no-reconcile`      | off                  | skip the syllabus reconciliation |
+
+### `descriptionsSweep`
+
+No flags — it is a scheduled job, configured by env:
+
+```bash
+node lib/scripts/descriptionsSweep.js
+```
+
+Two sources of work:
+
+1. **Publish journals** (`publish-journal/` in GCS), the primary one. Every
+   publication that stalled is resumed: missing descriptions generated, draft
+   mirrored, and the owed `package_manifest_updated` emitted. Journals are
+   dropped once complete and abandoned after a few attempts.
+2. **A deep scan** of published courses (`DESCRIPTIONS_SWEEP_FULL=1`), for
+   packages that never got a journal at all. Expensive — it reads every course —
+   and off by default. It re-projects manifests but does **not** emit events.
+
+Tuning: `DESCRIPTIONS_SWEEP_LIMIT` (courses per deep scan, default 25),
+`DESCRIPTION_STALE_AFTER_MS` (how long a publication may look stalled before it
+is resumed, default 15 min).
+
+---
+
+## The `package_manifest_updated` event
+
+Emitted by `--emit-events`, and by the sweep for the publications it finishes.
+Built in [`publishEvents.ts`](../utils/publishEvents.ts), sent to the breathecode
+telemetry endpoint with `BREATHECODE_SYSTEM_TOKEN`.
+
+Breathecode processes it **independently of `package_published`**, so a backfill
+event with no preceding publication is fine.
+
+**`publish_id` in a backfill** is `backfill-{runId}-{slug}`:
+
+- the `backfill-` prefix makes the origin readable in the webhook log and
+  discriminable in code (`isBackfillPublishId`); real publish ids are bare
+  uuids, so the two spaces cannot collide;
+- `{slug}` keeps it unique per event;
+- `{runId}` is one uuid per **invocation**, shared by every course it touches, so
+  "every event from the run I fired at 15:40" is a single substring query. It is
+  printed at start and in the closing summary.
+
+Note that running four separate `--slug` commands produces four run ids. Use
+`--limit` if you want one id for the batch — at the cost of not choosing the
+slugs.
+
+**Status is tied to the real outcome**, not to the script finishing:
+
+| Outcome                                  | `status`  |
+| ---------------------------------------- | --------- |
+| Descriptions written, manifest projected | `success` |
+| Generation failed outright               | `failed`  |
+| Manifest projection failed               | `failed`  |
+| Course unchanged (`skipped`)             | no event  |
+| Course threw                             | no event  |
+
+A failed manifest projection is not a failed run — the syllabus, which is the
+source of truth, is already saved — but this event reports the manifest, so it
+cannot claim success over one that was never rewritten. That is what
+`GenerateCourseDescriptionsResult.manifestProjected` exists for.
+
+The last row is deliberate: unlike a publication, a backfill never _announced_ an
+event, so it owes none. The course is simply re-run.
+
+---
+
+## Related
+
+- [`../utils/descriptions/`](../utils/descriptions/) — the shared service, the
+  mirror, the post-publish stage and the resume logic
+- [`../../scripts/README.md`](../../scripts/README.md) — the package-manifest S3
+  backfill (`npm run backfill:package-manifest`)
+- [`../utils/publishJournal.ts`](../utils/publishJournal.ts) — why the sweep can
+  know an event is owed

package/src/scripts/descriptionsGcsBackfill.ts

@@ -0,0 +1,193 @@
+/* eslint-disable no-await-in-loop */
+/**
+ * Fase 0b — GCS descriptions backfill (reuse from S3, no LLM).
+ *
+ * Seeds the GCS (draft) syllabus with descriptions already generated for the
+ * published S3 syllabus (Fase 0a) — a pure COPY, guarded by the content hash,
+ * performed by the shared mirror (`mirrorDescriptionsToDraft`). It NEVER calls
+ * the LLM: the ~800 GCS drafts are mostly unused, so anything not reusable from
+ * S3 is left for the next publication of that course (or a single-slug 0a run).
+ * Run AFTER 0a has completed.
+ *
+ * Batched like 0a: --limit courses per run (default 50), ordered by
+ * initialSyllabus modification time (newest first).
+ *
+ * Secrets/infra via env: GCP_CREDENTIALS_JSON (required), plus
+ * S3_PACKAGES_BUCKET / GCP_BUCKET_NAME / AWS_REGION as fallbacks for the
+ * corresponding flags.
+ *
+ * Per-run flags:
+ *   --s3-bucket <name>   published S3 bucket to reuse from (or env S3_PACKAGES_BUCKET)
+ *   --gcs-bucket <name>  GCS bucket (or env GCP_BUCKET_NAME, which is required)
+ *   --region <region>    AWS region (default us-east-1, or env AWS_REGION)
+ *   --slug <slug>        process a single course
+ *   --limit <n>          courses per run (default 50)
+ *   --dry-run            preview without writing
+ *   --no-reconcile       skip the additive syllabus reconciliation
+ *
+ * Examples:
+ *   node lib/scripts/descriptionsGcsBackfill.js --limit 200
+ *   node lib/scripts/descriptionsGcsBackfill.js --slug my-course
+ */
+import { parseArgs } from "node:util"
+import { Bucket, Storage } from "@google-cloud/storage"
+import { S3Client } from "@aws-sdk/client-s3"
+import { Syllabus } from "../models/creator"
+import { synchronizeSyllabusWithBucket } from "../utils/syllabusSync"
+import {
+  AwsClient,
+  fetchJsonObject,
+} from "../utils/s3/packageManifestBackfill"
+import { requireGcsBucketName } from "../utils/gcsBucketName"
+import { createGcsDescriptionsStorage } from "../utils/descriptions/gcsStorage"
+import { mirrorDescriptionsToDraft } from "../utils/descriptions/mirrorDescriptions"
+import { syllabusKey as s3SyllabusKey } from "../utils/descriptions/s3Storage"
+
+const GCS_SYLLABUS_PATTERN =
+  /^courses\/([^/]+)\/\.learn\/initialSyllabus\.json$/
+
+/** Courses (by slug) ordered by initialSyllabus.json update time, newest first. */
+async function listCoursesByRecency(bucket: Bucket): Promise<string[]> {
+  const [files] = await bucket.getFiles({ prefix: "courses/" })
+  const mtimes = new Map<string, number>()
+  for (const file of files) {
+    const match = file.name.match(GCS_SYLLABUS_PATTERN)
+    if (match) {
+      const updated = (file.metadata as { updated?: string } | undefined)
+        ?.updated
+      mtimes.set(match[1], updated ? new Date(updated).getTime() : 0)
+    }
+  }
+
+  return [...mtimes.entries()]
+    .sort((a, b) => b[1] - a[1])
+    .map(entry => entry[0])
+}
+
+async function main(): Promise<void> {
+  const { values } = parseArgs({
+    options: {
+      "s3-bucket": { type: "string" },
+      "gcs-bucket": { type: "string" },
+      region: { type: "string" },
+      slug: { type: "string" },
+      limit: { type: "string" },
+      "dry-run": { type: "boolean", default: false },
+      "no-reconcile": { type: "boolean", default: false },
+    },
+  })
+
+  const s3Bucket =
+    values["s3-bucket"] ||
+    process.env.S3_PACKAGES_BUCKET ||
+    "learnpack-paquetes"
+
+  const credentialsEnv = process.env.GCP_CREDENTIALS_JSON
+  if (!credentialsEnv) {
+    console.error("[gcs-backfill] GCP_CREDENTIALS_JSON (env) is required")
+    process.exit(1)
+  }
+
+  const bucket = new Storage({
+    credentials: JSON.parse(credentialsEnv),
+  }).bucket(values["gcs-bucket"] || requireGcsBucketName())
+
+  const s3 = new S3Client({
+    region: values.region || process.env.AWS_REGION || "us-east-1",
+  }) as unknown as AwsClient
+
+  const dryRun = values["dry-run"] ?? false
+  const reconcile = !(values["no-reconcile"] ?? false)
+  const limit = Number.parseInt(values.limit || "50", 10) || 50
+  const storage = createGcsDescriptionsStorage(bucket)
+
+  const singleSlug = values.slug
+  const slugs = singleSlug ? [singleSlug] : await listCoursesByRecency(bucket)
+
+  console.log(
+    `[gcs-backfill] Starting${
+      singleSlug ? ` (slug=${singleSlug})` : ` (limit=${limit})`
+    }${dryRun ? " (dry-run)" : ""} over ${slugs.length} candidate course(s)`
+  )
+
+  let processed = 0
+  let copied = 0
+  let missed = 0
+
+  for (const slug of slugs) {
+    if (!singleSlug && processed >= limit) {
+      break
+    }
+
+    try {
+      const publishedSyllabus = await fetchJsonObject<Syllabus>(
+        s3,
+        s3Bucket,
+        s3SyllabusKey(slug)
+      )
+      if (!publishedSyllabus) {
+        continue
+      }
+
+      if (reconcile && storage.syllabusSyncStorage) {
+        const draft = await storage.readSyllabus(slug)
+        if (draft) {
+          try {
+            const sync = await synchronizeSyllabusWithBucket(
+              storage.syllabusSyncStorage,
+              slug,
+              draft,
+              { prune: false }
+            )
+            if (sync.changed && !dryRun) {
+              await storage.writeSyllabus(slug, sync.syllabus)
+            }
+          } catch (error) {
+            console.error(
+              `[gcs-backfill] Reconciliation failed for "${slug}":`,
+              (error as Error).message
+            )
+          }
+        }
+      }
+
+      const result = await mirrorDescriptionsToDraft(
+        storage,
+        slug,
+        publishedSyllabus,
+        { dryRun }
+      )
+
+      if (result.copied === 0 && result.missed === 0) {
+        continue
+      }
+
+      processed += 1
+      copied += result.copied
+      missed += result.missed
+
+      console.log(
+        `[gcs-backfill] "${slug}": copied ${result.copied}, ${
+          result.missed
+        } miss(es)${dryRun ? " (dry-run)" : ""}`
+      )
+    } catch (error) {
+      processed += 1
+      console.error(
+        `[gcs-backfill] Failed processing "${slug}":`,
+        (error as Error).message
+      )
+    }
+  }
+
+  console.log(
+    `[gcs-backfill] Done. ${processed} course(s), ${copied} description(s) copied, ${missed} miss(es).`
+  )
+}
+
+main()
+  .then(() => process.exit(0))
+  .catch(error => {
+    console.error("[gcs-backfill] Fatal error:", error)
+    process.exit(1)
+  })