@runeditrun/rer 0.2.75 → 0.2.76

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. package/README.md +7 -2
  2. package/cli/listing.mjs +20 -0
  3. package/cli/main.mjs +1 -0
  4. package/cli/parse.mjs +2 -1
  5. package/cli/portability-migrations.mjs +46 -0
  6. package/cli/portability-worker.mjs +7 -20
  7. package/cli/portability.mjs +5 -4
  8. package/cli/server-registry.mjs +43 -18
  9. package/cli/server-resources.mjs +38 -9
  10. package/global/CHANGELOG.md +8 -0
  11. package/global/README.md +7 -2
  12. package/global/launch/state.mjs +5 -4
  13. package/global/launch/stop.mjs +30 -8
  14. package/global/main.mjs +2 -1
  15. package/modules/job-queue/command.mjs +8 -3
  16. package/modules/portability/README.md +40 -11
  17. package/modules/portability/backfills.mjs +29 -16
  18. package/modules/portability/bridge.mjs +24 -29
  19. package/modules/portability/import.mjs +1 -2
  20. package/modules/portability/index.d.mts +1 -1
  21. package/modules/portability/ledgers.mjs +22 -0
  22. package/modules/portability/migration-history.mjs +2 -2
  23. package/modules/release/README.md +135 -3
  24. package/modules/release/catalogue.mjs +6 -2
  25. package/modules/release/listing.mjs +64 -0
  26. package/modules/release/look.mjs +26 -0
  27. package/modules/release/previous-listing.mjs +44 -0
  28. package/modules/release/produce.mjs +3 -0
  29. package/modules/release/screenshots.mjs +51 -0
  30. package/modules/release/signature.mjs +3 -2
  31. package/modules/release/source.mjs +19 -6
  32. package/modules/release/statement.mjs +8 -4
  33. package/onboarding/process-cleanup.mjs +5 -5
  34. package/onboarding/process-scope.mjs +1 -1
  35. package/onboarding/process-table.mjs +6 -4
  36. package/onboarding/process-tree.mjs +4 -4
  37. package/onboarding/schema/json-schema.mjs +4 -1
  38. package/package.json +1 -1
  39. package/scripts/publish-check/README.md +14 -0
  40. package/scripts/publish-check/assignment.py +48 -0
  41. package/scripts/publish-check/credential_assignments.mjs +75 -0
  42. package/scripts/publish-check/credential_assignments.py +70 -0
  43. package/scripts/publish-check/credential_literals.mjs +66 -0
  44. package/scripts/publish-check/credential_syntax.mjs +29 -0
  45. package/scripts/publish-check/env_values.py +14 -1
  46. package/scripts/publish-check/history_scan.py +9 -2
@@ -110,6 +110,27 @@ it continues to accept serializable object policies. Malformed primary or later
110
110
  journal JSON is reported once as `archive_json` with exit 2, before local data
111
111
  migrations, without echoing archive content or parser diagnostics.
112
112
 
113
+ An installation migrated by Platform has `rer_migrations(name, sha256,
114
+ applied_at)` instead of Wrangler's ledger. Local transfers match each entry to
115
+ exactly one migration file by SHA-256 and preserve its application order. They
116
+ verify all local files are recorded and skip Wrangler; a pending file refuses
117
+ as `migration_ledger:unapplied:<file>` until Platform applies it. Unknown,
118
+ ambiguous or conflicting entries refuse by name, without replaying SQL or
119
+ creating a second ledger.
120
+
121
+ For old archives containing only Platform's migration names, the seed adapter
122
+ may declare `platformMigrations: {prefix, dataOnly}` on `localPortability`.
123
+ Read `prefix` from the seed's own deploymentSpec packaging `migrationPrefix`
124
+ for this binding's migrations directory. It declares an alias `prefix + file`
125
+ for each file; it does not strip arbitrary prefixes. `dataOnly` is an explicit
126
+ list of installation-specific Platform seed entries without schema files.
127
+ Declare only SQL that seeds data and changes no schema: these names are omitted
128
+ from schema replay. This does not authorize replacing their rows; the archive
129
+ policy still controls that. Archive SQL and names never supply these declarations.
130
+ Custom replay runners can supply each plan entry's `aliases` and pass
131
+ `{dataOnly}` as `migrationReplay`'s third argument. Verified archive ledger
132
+ hashes take precedence over name-only placement where available.
133
+
113
134
  When a policy includes operator-auth tables, local dev identity still never
114
135
  travels, and no policy option changes that: export drops `oa_sessions` rows of
115
136
  kind `dev`, every credential recorded in `oa_dev_credentials` (and that table's
@@ -322,7 +343,7 @@ literals, numbers and comments never count. Text Base cannot read to a
322
343
  statement boundary, such as an unclosed block comment, is refused. An archived table the archive holds no
323
344
  rows of restores empty, so an `UPDATE` or `DELETE` of it changes nothing and is
324
345
  allowed regardless of what it reads: no row can change or fire a trigger. An
325
- ordinary `INSERT` into it remains refused. For archives without a migration record,
346
+ `INSERT` into it must qualify for replay below. For archives without a migration record,
326
347
  every migration after the earliest prefix that gives the archive's schema is
327
348
  checked, since the archive may predate any of them.
328
349
 
@@ -337,11 +358,18 @@ destination definition, so a row rejected by its CHECK fails the whole atomic
337
358
  restore; no row is skipped or coerced. Other statements still face the usual
338
359
  data classifier, and a proven preserving block is not a competing backfill writer.
339
360
 
340
- A backfill of a catalogue table the archive does not carry (one a later
341
- migration adds, restored empty) is **replayed** instead of refused: an
342
- `INSERT ... SELECT` (or `INSERT ... VALUES`) into that table. Finalize empties
343
- the table (resetting its `AUTOINCREMENT` counter, as the live upgrade created
344
- it) and runs each of its later writes in migration order, exactly as written,
361
+ A backfill of a catalogue table is **replayed** instead of refused when it is
362
+ an eligible `INSERT ... SELECT` (or `INSERT ... VALUES`) or `REPLACE`. A table
363
+ the archive carries keeps its archived rows, including when it carries none:
364
+ the replay runs over the rows the live migration would have found there. It
365
+ must name its target columns and be the table's only later writer. For a table
366
+ with `AUTOINCREMENT`, carried-table replay is refused: the archive does not
367
+ retain `sqlite_sequence`'s high-water mark, so generated keys could differ.
368
+ Reads of a keyed table's unarchived hidden `rowid`, `_rowid_` or `oid` also
369
+ refuse, including through views: restoring its keys can change physical rowids.
370
+ For a table the archive lacks (one a later migration adds), finalize empties the table
371
+ (resetting its `AUTOINCREMENT` counter, as the live upgrade created it).
372
+ Finalize runs the eligible writes in migration order, exactly as written,
345
373
  after every archived row is inserted and while the destination's triggers are
346
374
  still dropped, in the same atomic D1 transaction as the rest of the restore. The
347
375
  table then holds what the live upgrade gave it. Time functions such as
@@ -349,8 +377,7 @@ table then holds what the live upgrade gave it. Time functions such as
349
377
  `INSERT ... SELECT` without `ORDER BY` assigns new keys in the order SQLite
350
378
  scans the restored rows. Rows the restore normalizes (import safety) are read
351
379
  as restored. The restore is refused with
352
- `schema_mismatch:migration_data:<migration>` instead when the archive carries
353
- the table (even with no rows), when a write to it is not a plain `INSERT` or
380
+ `schema_mismatch:migration_data:<migration>` instead when a write is not an `INSERT` or
354
381
  `REPLACE`, reads any table other than archived ones and itself (including through
355
382
  a view resolved from applied migration history), selects `*` or
356
383
  a secret column, when a trigger on it or a foreign key action from it reaches a
@@ -366,7 +393,8 @@ an INSERT reading a view must still resolve entirely to carried tables.
366
393
 
367
394
  The same check runs when the archive's schema identity equals the destination's:
368
395
  a later migration that only purges or backfills rows leaves the schema unchanged,
369
- yet the archive's rows never passed through it. Such an archive is refused with
396
+ yet the archive's rows never passed through it. Eligible carried-table backfills
397
+ replay here too; other rewrites are refused with
370
398
  `schema_mismatch:migration_data:<migration>` (local `rer import` exit 4, naming
371
399
  the migration) before any staging or import job; one whose later migrations only
372
400
  add indexes or touch excluded tables restores as usual. Without a replayer, an
@@ -690,11 +718,12 @@ all existing migrations and their ledger. Wire package.json:
690
718
  Before copying, capture the source checkout's actual origin with
691
719
  `git remote get-url origin` and its full revision with `git rev-parse HEAD`.
692
720
  Record this component in `edition.json`, replacing `ORIGIN` and `PIN` with those
693
- values. Keep the real source provenance in the edition's component and source
721
+ values and `MODULE_ID` with the component's existing edition identifier. Keep
722
+ the real source provenance in the edition's component and source
694
723
  lock rather than embedding a repository address in the copied module:
695
724
 
696
725
  ```json
697
- {"id":"runeditrun/portability","role":"module","source":{"kind":"local","paths":["vendor/base/modules/portability"],"origin":{"repository":"ORIGIN","revision":"PIN","paths":["modules/portability"]}}}
726
+ {"id":"MODULE_ID","role":"module","source":{"kind":"local","paths":["vendor/base/modules/portability"],"origin":{"repository":"ORIGIN","revision":"PIN","paths":["modules/portability"]}}}
698
727
  ```
699
728
 
700
729
  Follow the seed's edition-authoring source-lock workflow, then prove the copy:
@@ -11,9 +11,9 @@ import { replayViews, viewReads } from './replay-views.mjs';
11
11
  // restorable catalogue (an excluded, derived or platform table) and whose
12
12
  // writes reach no catalogue table through a trigger or foreign key action
13
13
  // (reachingTables) changes no restored row, whatever it reads: admitted.
14
- // - One whose target is a catalogue table the archive does not carry (restored
15
- // empty) is replayed at finalize when it inserts seed or backfill rows.
16
- // Every later write to that table must then be a
14
+ // - An eligible seed or backfill of a catalogue table replays at finalize.
15
+ // A carried table keeps its archived rows and admits only one later writer,
16
+ // naming its target columns. An absent table is reset first; each later write is a
17
17
  // replayable INSERT: they run in migration order, exactly as written, after
18
18
  // the archived rows are inserted and with the destination's triggers still
19
19
  // dropped, so the table holds what the live upgrade gave it. A replay may read
@@ -33,7 +33,7 @@ import { replayViews, viewReads } from './replay-views.mjs';
33
33
  export function laterData({applied,unapplied,replay,catalogue,archived,empty,tables,secrets,manifest}) {
34
34
  const split=name=>replay.statements(name).flatMap(splitStatements),fold=list=>list.map(folded);
35
35
  const statements=applied.flatMap(split);
36
- const restorable=fold(catalogue),carried=fold(archived),absent=restorable.filter(name=>!carried.includes(name));
36
+ const restorable=fold(catalogue),carried=fold(archived);
37
37
  const reached=reachingTables(statements,carried),writable=reachingTables(statements,restorable);
38
38
  const later=unapplied.flatMap(migration=>split(migration).map((sql,index)=>({migration,index,sql,data:dataStatement(sql)})));
39
39
  const {admitted,tables:rebuilt}=preservingRebuilds(later,manifest,tables,statements),views=replayViews(statements);
@@ -46,14 +46,14 @@ export function laterData({applied,unapplied,replay,catalogue,archived,empty,tab
46
46
  for(const s of later) {
47
47
  if(outside.includes(s) || admitted.has(s))continue;
48
48
  s.today=changesData(s.sql,[...reached,...tainted],empty);
49
- if(s.data && absent.includes(s.data.target)){if(!writes.has(s.data.target))writes.set(s.data.target,[]);writes.get(s.data.target).push(s);}
49
+ if(s.data && restorable.includes(s.data.target)){if(!writes.has(s.data.target))writes.set(s.data.target,[]);writes.get(s.data.target).push(s);}
50
50
  else if(s.today)refused.add(s);
51
51
  }
52
52
  const replays=[];
53
53
  for(const [table,list] of writes) {
54
54
  // Seed INSERTs into a new table replay too: the rows already present in a
55
55
  // fresh destination must be reset and derived from the restored installation.
56
- const bad=list.filter(s=>!replayable(s,{table,statements,restorable,writable,carried,tables:known,secrets,views}));
56
+ const bad=carried.includes(table) && list.length>1?list:list.filter(s=>!replayable(s,{table,statements,restorable,writable,carried,tables:known,secrets,views,manifest}));
57
57
  // Writes confined to tables created after the archive retain their
58
58
  // admission. Replay eligible seeds, but do not turn a new-table-only
59
59
  // upgrade into a refused backfill. View reads always require resolution.
@@ -66,11 +66,24 @@ export function laterData({applied,unapplied,replay,catalogue,archived,empty,tab
66
66
  const first=later.find(s=>refused.has(s));
67
67
  insist(!first,'schema_mismatch:migration_data:'+first?.migration);
68
68
  const named=Object.fromEntries(catalogue.map(name=>[folded(name),name]));
69
- return {rebuilt,replays:later.filter(s=>replays.includes(s)).map(s=>({migration:s.migration,statement:s.index,table:named[s.data.target],sql:s.sql}))};
69
+ return {rebuilt,replays:later.filter(s=>replays.includes(s)).map(s=>({migration:s.migration,statement:s.index,table:named[s.data.target],sql:s.sql,...(carried.includes(s.data.target)?{carried:true}:{})}))};
70
70
  }
71
- function replayable({data},{table,statements,restorable,writable,carried,tables,secrets,views}) {
71
+ function replayable({data},{table,statements,restorable,writable,carried,tables,secrets,views,manifest}) {
72
72
  if(!['insert','replace'].includes(data.verb) || data.star)return false;
73
- if(!viewReads(data.names,data.words,{views,tables,carried,table,secrets}))return false;
73
+ if(carried.includes(table)) {
74
+ // sqlite_sequence's high-water mark is not archived, so replay cannot
75
+ // reproduce a carried AUTOINCREMENT table's live key allocation.
76
+ if(!data.columns || autoincrement(manifest.find(t=>folded(t.name)===table).sql))return false;
77
+ // Keyed tables carry their key columns, not their hidden physical rowids.
78
+ // Treat unarchived aliases as unreadable cells, including through views.
79
+ secrets={...secrets};
80
+ for(const t of manifest)if(!t.rowidAlias) {
81
+ const hidden=['rowid','_rowid_','oid'].filter(name=>!t.columns.some(c=>folded(c.name)===name));
82
+ secrets[t.name]=[...(secrets[t.name]??[]),...hidden];
83
+ }
84
+ }
85
+ // A carried target participates in secret checks, even through a view.
86
+ if(!viewReads(data.names,data.words,{views,tables,carried,table:carried.includes(table)?undefined:table,secrets}))return false;
74
87
  if(triggerBodies(statements,table).some(name=>writable.includes(name)))return false;
75
88
  // Foreign key actions: a row the replay deletes or rekeys (REPLACE, upsert) must reach no other catalogue table.
76
89
  return !reachingTables(statements,restorable.filter(name=>name!==table)).includes(table);
@@ -85,18 +98,18 @@ function writesAnother(s,table,reach,replays) {
85
98
  return !['create table','create index','drop index','create view','drop view','trigger'].includes(shape);
86
99
  }
87
100
  // Finalize statements for the job's replays, to run after the archived rows are
88
- // inserted and before the triggers are recreated: each replayed table is
89
- // emptied (and its AUTOINCREMENT counter reset, as the live upgrade created it)
90
- // and the statements run in migration order. Each records the rows it wrote in
91
- // the job's bridge record (`replayed[i].rows`), in the same transaction.
101
+ // inserted and before the triggers are recreated. Only absent tables are emptied
102
+ // (and their AUTOINCREMENT counters reset, as the live upgrade created them).
103
+ // Carried tables keep their archived rows. Statements run in migration order,
104
+ // recording rows in the job's bridge record (`replayed[i].rows`) in the same transaction.
92
105
  export function replayStatements(db,row) {
93
106
  const replays=row.metadata.replays??[];
94
- const reset=[...new Set(replays.map(r=>r.table))].flatMap(name=>{
107
+ const reset=[...new Set(replays.filter(r=>!r.carried).map(r=>r.table))].flatMap(name=>{
95
108
  const target=row.metadata.targets.find(t=>t.name===name);
96
109
  insist(target,'schema_mismatch:replay_table:'+name);
97
- const sequenced=tokens(target.sql).some(t=>!t.quoted && folded(t.text)==='autoincrement');
98
- return [statement(db,`DELETE FROM ${quote(name)}`),...(sequenced?[statement(db,'DELETE FROM sqlite_sequence WHERE name=?',name)]:[])];
110
+ return [statement(db,`DELETE FROM ${quote(name)}`),...(autoincrement(target.sql)?[statement(db,'DELETE FROM sqlite_sequence WHERE name=?',name)]:[])];
99
111
  });
100
112
  return [...reset,...replays.flatMap((r,i)=>[db.prepare(r.sql),
101
113
  statement(db,`UPDATE rer_portability_jobs SET metadata=json_set(metadata,'$.bridge.replayed[${i}].rows',changes()) WHERE id=?`,row.id)])];
102
114
  }
115
+ const autoincrement=sql=>tokens(sql).some(t=>!t.quoted && folded(t.text)==='autoincrement');
@@ -2,7 +2,7 @@ import { all, quote, insist } from './storage.mjs';
2
2
  import { canonicalJson } from './hash.mjs';
3
3
  import { tokens, identifier, folded } from './generated.mjs';
4
4
  import { laterData } from './backfills.mjs';
5
- import { destinationLedger } from './ledgers.mjs';
5
+ import { destinationLedger, matchMigrationLedger } from './ledgers.mjs';
6
6
  import { checkRebuild } from './rebuilds.mjs';
7
7
 
8
8
  // Applied product migrations in the ledger's own order (ledgers.mjs). A
@@ -22,9 +22,15 @@ export async function appliedMigrations(db) {
22
22
  // rewrites restorable data is refused, whether or not it changes the schema,
23
23
  // unless its statement is replayed at finalize (backfills.mjs).
24
24
  async function placement(db,manifest,cat,replay,ledger,history) {
25
- const applied=ledger.map(m=>m.name);
25
+ const files=replay.names.map(name=>({name,sha256:replay.sha256?.(name),aliases:replay.aliases?.(name)}));
26
+ // An aligned replayer keeps its existing contract: hashes are optional.
27
+ // Resolve destination aliases only when its ledger uses different names.
28
+ let applied=ledger.map(m=>m.name);
29
+ if(canonicalJson(replay.names)!==canonicalJson(applied))applied=matchMigrationLedger(ledger,files,replay);
26
30
  insist(canonicalJson(replay.names)===canonicalJson(applied) && await replay.schemaAfter(applied)===cat.identity,'schema_mismatch');
27
- const recorded=manifest.migrations ?? archivedApplied(history?.(),ledger,replay);
31
+ const archived=history?.();
32
+ const historyEntries=manifest.migrations?.map(name=>archived?.find(entry=>entry.name===name) ?? {name}) ?? archived;
33
+ const recorded=historyEntries && matchMigrationLedger(historyEntries,files,replay);
28
34
  let earlier,unapplied;
29
35
  if(recorded) {
30
36
  earlier=applied.filter(name=>recorded.includes(name));
@@ -48,26 +54,12 @@ async function placement(db,manifest,cat,replay,ledger,history) {
48
54
  // restorable. Trigger DDL itself changes no row, nor does an UPDATE or DELETE
49
55
  // of a table the archive holds no rows of (migration-statements.mjs). A data
50
56
  // statement that writes only tables outside the catalogue is admitted
51
- // whatever it reads; a backfill of a table the archive lacks is replayed.
52
- const catalogue=cat.tables.map(t=>t.name),archived=catalogue.filter(name=>manifest.tables.some(t=>t.name===name));
53
- const empty=archived.filter(name=>!manifest.tables.find(t=>t.name===name).count);
57
+ // whatever it reads; eligible catalogue backfills are replayed.
58
+ const catalogue=cat.tables.map(t=>t.name),archivedTables=catalogue.filter(name=>manifest.tables.some(t=>t.name===name));
59
+ const empty=archivedTables.filter(name=>!manifest.tables.find(t=>t.name===name).count);
54
60
  const tables=(await all(db,"SELECT name FROM sqlite_master WHERE type IN ('table','view')")).map(t=>t.name);
55
- const {replays,rebuilt}=laterData({applied,unapplied,replay,catalogue,archived,empty,tables,secrets:manifest.excludedColumns??{},manifest:manifest.tables});
56
- return {earlier,replays,rebuilt};
57
- }
58
- // An archive without a migration record that carries its source's migration
59
- // ledger (an old archive from before that ledger became bookkeeping) records
60
- // exactly which migrations its rows passed through. Each must be one the
61
- // destination applied, with the same file hash where the ledger records one:
62
- // the destination ledger's, else the replayed file's. The ledger itself is
63
- // never restored (bookkeeping.mjs). history: [{name,sha256?}] from
64
- // ledgers.mjs archivedHistory, undefined when the archive carries no ledger.
65
- function archivedApplied(history,ledger,replay) {
66
- for(const m of history??[]) {
67
- const applied=ledger.find(x=>x.name===m.name);
68
- insist(applied && (m.sha256===undefined || m.sha256===(applied.sha256??replay.sha256?.(m.name))),'schema_mismatch:migration_history:'+m.name);
69
- }
70
- return history?.map(m=>m.name);
61
+ const {replays,rebuilt}=laterData({applied,unapplied,replay,catalogue,archived:archivedTables,empty,tables,secrets:manifest.excludedColumns??{},manifest:manifest.tables});
62
+ return {applied,earlier,replays,rebuilt};
71
63
  }
72
64
  // An archive with the destination's schema identity can still predate later
73
65
  // migrations that changed only data (a purge or a backfill) or added an index.
@@ -79,10 +71,14 @@ function archivedApplied(history,ledger,replay) {
79
71
  export async function sameSchemaMigrations(db,manifest,cat,migrations,history) {
80
72
  const ledger=await destinationLedger(db),applied=ledger?.map(m=>m.name);
81
73
  if(!applied)return;
82
- insist(!manifest.migrations || manifest.migrations.every(name=>applied.includes(name)),'schema_mismatch:migrations');
83
- if(!migrations){insist(!manifest.migrations || applied.every(name=>manifest.migrations.includes(name)),'schema_mismatch:migrations');return;}
84
- // Same schema: every catalogue table is archived, so nothing is replayed.
85
- await placement(db,manifest,cat,await migrations(),ledger,history);
74
+ if(!migrations){
75
+ if(manifest.migrations)matchMigrationLedger(manifest.migrations.map(name=>({name})),ledger);
76
+ insist(!manifest.migrations || applied.every(name=>manifest.migrations.includes(name)),'schema_mismatch:migrations');return;
77
+ }
78
+ // A data-only migration leaves the schema unchanged but still needs its
79
+ // carried-table replay plan retained for finalize.
80
+ const bridge=await additiveBridge(db,manifest,cat,migrations,history);
81
+ return bridge.replays?.length?bridge:undefined;
86
82
  }
87
83
  // An archive made before later migrations restores only when those migrations
88
84
  // are additive, or a proven row-preserving CHECK rebuild: new tables, new
@@ -90,9 +86,8 @@ export async function sameSchemaMigrations(db,manifest,cat,migrations,history) {
90
86
  // Anything else stays a schema_mismatch refusal.
91
87
  export async function additiveBridge(db,manifest,cat,migrations,history) {
92
88
  insist(migrations,'schema_mismatch');
93
- const replay=await migrations(),ledger=await destinationLedger(db)??[],applied=ledger.map(m=>m.name);
94
- insist(!manifest.migrations || manifest.migrations.every(name=>applied.includes(name)),'schema_mismatch:migrations');
95
- const {earlier,replays,rebuilt}=await placement(db,manifest,cat,replay,ledger,history);
89
+ const replay=await migrations(),ledger=await destinationLedger(db)??[];
90
+ const {applied,earlier,replays,rebuilt}=await placement(db,manifest,cat,replay,ledger,history);
96
91
  const columns={};
97
92
  for(const table of manifest.tables) {
98
93
  const target=cat.tables.find(t=>t.name===table.name),label=':'+table.name;
@@ -28,8 +28,7 @@ export async function createImport(context,manifest,actor='direct',legacyRecover
28
28
  const skippedTables=original.tables.filter(table=>platformTable(table.name)).map(table=>table.name);
29
29
  const cat=await catalogue(db,config),upgrade=config.acceptedSchemas[original.schema];
30
30
  const same=manifest.schema===cat.identity,history=()=>archivedHistory(original,ledgers);
31
- const {replays,...bridge}=same || upgrade?{}:await additiveBridge(db,manifest,cat,context.migrations,history);
32
- if(same && !upgrade)await sameSchemaMigrations(db,manifest,cat,context.migrations,history);
31
+ const {replays,...bridge}=upgrade?{}:same?(await sameSchemaMigrations(db,manifest,cat,context.migrations,history)??{}):await additiveBridge(db,manifest,cat,context.migrations,history);
33
32
  if(!bridge.fromSchema)insist(canonicalJson(manifest.tables.map(x=>x.name))===canonicalJson(cat.tables.map(x=>x.name)),'table_inventory');
34
33
  if(!upgrade && !bridge.fromSchema)for(const table of manifest.tables) {
35
34
  const target=cat.tables.find(t=>t.name===table.name);
@@ -38,7 +38,7 @@ export interface Status {id:string;kind:'export'|'import';phase:'building'|'stag
38
38
  /** Later additive migrations an older archive was restored across. */
39
39
  export interface Bridge {fromSchema:string;toSchema:string;migrations:string[];tables:string[];columns:Record<string,string[]>;/** Later backfills of tables the archive lacks, replayed at finalize in this order; rows is null until the apply commits. */replayed?:{migration:string;statement:number;table:string;rows:number|null}[]}
40
40
  /** A runner's replay of the destination's applied migrations: the archive schema identity after a chosen set, and each migration's statements. */
41
- export interface MigrationReplay {names:string[];statements(name:string):string[];/** The migration file's SHA-256 hex, where the runner read the file. */sha256?(name:string):string|undefined;schemaAfter(names:string[]):Promise<string>}
41
+ export interface MigrationReplay {names:string[];/** Seed-declared data-only Platform entries without migration files. */dataOnly?:string[];/** Names generated by the seed's deployment specification for this file. */aliases?(name:string):string[]|undefined;statements(name:string):string[];/** The migration file's SHA-256 hex, where the runner read the file. */sha256?(name:string):string|undefined;schemaAfter(names:string[]):Promise<string>}
42
42
  export interface PortabilityOptions {db:D1Database;buckets?:Record<string,R2Bucket>;policy:ArchivePolicy;migrations?:()=>Promise<MigrationReplay>}
43
43
  export interface Portability {
44
44
  exports: {create(actor?:string):Promise<string>;advance(id:string,options?:{expectedPart?:number;maxParts?:number}):Promise<Status>;status(id:string):Promise<Status>;download(id:string):Promise<ReadableStream<Uint8Array>>;abort(id:string):Promise<Status>};
@@ -39,6 +39,28 @@ export async function destinationLedger(db) {
39
39
  }
40
40
  const entry=row=>({name:row.name,...(typeof row.sha256==='string'?{sha256:row.sha256}:{})});
41
41
 
42
+ // Resolve ledger identities against the destination's own migration files.
43
+ // Hash-bearing entries must identify exactly one file; a coinciding name may
44
+ // confirm that file, but cannot point at another one. Hashless rows use exact
45
+ // names or seed-declared aliases. The caller keeps the destination replay order.
46
+ export function matchMigrationLedger(entries,files,{dataOnly=[]}={}) {
47
+ return entries.flatMap(m=>{
48
+ const named=files.filter(file=>file.name===m.name || file.aliases?.includes(m.name));
49
+ insist(named.length<=1,'migration_ledger:ambiguous:'+m.name,409);
50
+ if(m.sha256===undefined) {
51
+ if(!named.length && dataOnly.includes(m.name))return [];
52
+ insist(named.length,'migration_ledger:unknown:'+m.name,409);
53
+ return [named[0].name];
54
+ }
55
+ const matches=files.filter(file=>file.sha256===m.sha256);
56
+ if(!matches.length && !named.length && dataOnly.includes(m.name))return [];
57
+ insist(matches.length,'migration_ledger:unknown:'+m.name,409);
58
+ insist(matches.length===1,'migration_ledger:ambiguous:'+m.name,409);
59
+ insist(!named.length || named[0]===matches[0],'migration_ledger:conflict:'+m.name,409);
60
+ return [matches[0].name];
61
+ });
62
+ }
63
+
42
64
  /** The ledger tables an archive carries (old archives carried rer_migrations before it became bookkeeping). */
43
65
  export const archivedLedgerTables=manifest=>manifest.tables.filter(t=>ledgerNames.includes(identifier(t.name)));
44
66
 
@@ -6,9 +6,9 @@ import { policy, schemaIdentity } from './catalogue.mjs';
6
6
  // Node runners only: restore uses it to place an older archive within the
7
7
  // destination's history (see bridge.mjs). plan: [{name,statements,sha256?}],
8
8
  // sha256 being the migration file's hash where the runner read the file.
9
- export function migrationReplay(plan,input) {
9
+ export function migrationReplay(plan,input,{dataOnly=[]}={}) {
10
10
  const config=policy(input);
11
- return {names:plan.map(migration=>migration.name),statements:name=>plan.find(m=>m.name===name).statements,sha256:name=>plan.find(m=>m.name===name).sha256,async schemaAfter(names) {
11
+ return {names:plan.map(migration=>migration.name),dataOnly,aliases:name=>plan.find(m=>m.name===name).aliases,statements:name=>plan.find(m=>m.name===name).statements,sha256:name=>plan.find(m=>m.name===name).sha256,async schemaAfter(names) {
12
12
  const scratch=new DatabaseSync(':memory:');
13
13
  const db={prepare:sql=>{const query=scratch.prepare(sql);let values=[];return {bind(...args){values=args;return this;},async all(){return {results:query.all(...values)};}};}};
14
14
  try {
@@ -123,7 +123,8 @@ metadata and Git blobs are never loaded as producer code.
123
123
 
124
124
  With `--tag`, passing gates produce `public-source.tar`, containing the checked tree's
125
125
  exact ordinary files without Git history, and a hash-pinned site bundle:
126
- `site/<id>/<version>/release.json`, published documents and `site/registry.json`.
126
+ `site/<id>/<version>/release.json`, published documents, listing assets and
127
+ `site/registry.json`.
127
128
  Export attributes cannot silently omit or substitute bytes. The site metadata
128
129
  comes from committed seed and contract data; no screenshots, install guarantees
129
130
  or verified maturity claims are invented. Copy the reviewed bundle into the site
@@ -338,6 +339,135 @@ Generated source commits and annotated tags identify Run Edit Run with the
338
339
  non-contact address `releases@runeditrun.invalid`. Provenance remains bound to
339
340
  the source commit and tree; the Git author address is not a delivery destination.
340
341
 
342
+ ## Listing metadata and assets
343
+
344
+ `rer release` adds `slug`, `homepage`, `headline`, `brand`, structured `shots`
345
+ and `look` to the existing `release.json` listing. All inputs come from the
346
+ selected committed tree. Asset paths are relative to the bundle directory:
347
+ `assets/seeds/<id>/mark.svg`, optional `mark-light.svg`, and screenshot filenames.
348
+ The slug is the repository name without its `run-` prefix; the asset directory
349
+ uses `seed.json.id`, which can differ from the slug.
350
+
351
+ To produce the same listing fields and asset bytes between releases:
352
+
353
+ ```sh
354
+ rer listing --repo /path/to/seed --ref COMMIT_SHA --output /path/to/new-directory
355
+ ```
356
+
357
+ This writes `listing.json` containing `slug`, `homepage`, `headline`, `brand`,
358
+ `shots` and `look`, plus the referenced files under `assets/seeds/<id>/`.
359
+ It reads the selected ref even with a dirty working tree. The output must be a
360
+ new directory outside the source. This command runs no release gates, creates
361
+ no tags and performs no signing. It needs a committed
362
+ `rer-project.json.sourceRepository` or a credential-free GitHub `origin` to
363
+ identify the repository. Previous-shot selection can read advertised remote tags.
364
+
365
+ ### Seed author fields
366
+
367
+ | Committed input | Listing value and default |
368
+ | --- | --- |
369
+ | `seed.json.homepage` | `homepage`; optional, defaults to `https://run-<slug>.com`; credential-free HTTP(S) URL |
370
+ | `seed.json.headline` | `headline`; optional nonempty text, otherwise `null` |
371
+ | `seed.json.brand.mark` | SVG source path; defaults to `public/brand/mark.svg` |
372
+ | `seed.json.brand.markLight` | Optional SVG variant; defaults to `public/brand/mark-light.svg` when present |
373
+ | `src/branding.json` → `primaryColor` | `brand.colours.accent` |
374
+ | `public/rer-theme.css` → `--rer-text`, `--rer-page` | `brand.colours.ink`, `brand.colours.paper` |
375
+
376
+ Brand paths are repository-relative. An explicitly declared mark must exist and
377
+ contain SVG. Legacy seeds without a default mark emit `brand.mark: null`;
378
+ an absent default light variant emits `brand.markLight: null`. Missing branding
379
+ colours emit `null`. The producer copies the committed SVG bytes to the bundle's
380
+ standard mark filenames.
381
+
382
+ ### Look fingerprints
383
+
384
+ `look.home` and `look.app` fingerprint these default glob arrays:
385
+
386
+ ```json
387
+ {
388
+ "home": ["public/brand/**", "src/branding.json", "public/rer-theme.css", "public/fonts/**", "src/site/**", "!src/site/docs/**"],
389
+ "app": ["public/brand/**", "src/branding.json", "public/**/*.{css,js,mjs,html}", "src/ui/**", "src/**/*.css", "src/client/**"]
390
+ }
391
+ ```
392
+
393
+ Declare `look.home` and/or `look.app` in committed `rer-project.json` to replace
394
+ the respective default array. An omitted array keeps its default. Patterns are
395
+ repository-relative; `!` patterns exclude any matching path regardless of array
396
+ order. Mark declarations do not implicitly add files to either glob set:
397
+ include nonstandard mark paths in the appropriate overrides.
398
+
399
+ For each set, the producer sorts matching committed blob paths in JavaScript
400
+ string order and hashes UTF-8 records of `path`, NUL, Git blob object ID, LF.
401
+ The result is `sha256:<hex>`. No browser or capture runs. With the defaults,
402
+ backend-only changes outside these sets leave both fingerprints unchanged;
403
+ homepage and app changes affect the sets covering their files.
404
+
405
+ ### Release screenshots and reuse
406
+
407
+ Authors declare committed screenshots in `RELEASE.json`:
408
+
409
+ ```json
410
+ {
411
+ "screenshots": [
412
+ {
413
+ "path": ".github/assets/release/01-issues.webp",
414
+ "alt": "Issue list with grouped errors",
415
+ "caption": "Group related errors and open an issue to investigate.",
416
+ "width": 1600,
417
+ "height": 1000
418
+ }
419
+ ]
420
+ }
421
+ ```
422
+
423
+ When supplied, the array must contain 1–5 still WebP files, each with nonempty
424
+ alt text and caption, encoded dimensions exactly 1600×1000 and size at most
425
+ 300 KiB. Dimensions are checked in the image bytes as well as the declaration.
426
+ Source paths must be repository-relative, and filenames must be unique across
427
+ the set because the bundle keeps only each basename. The listing preserves
428
+ `path`, `alt`, `caption`, `width` and `height`, rewriting paths to
429
+ `assets/seeds/<id>/<filename>.webp`.
430
+
431
+ Screenshots remain optional for legacy releases. Without screenshots or reused
432
+ history, `shots` is `[]` and `look.shotsAt` is `null`. Missing screenshots and
433
+ stale screenshots are not release gates; malformed declared screenshots fail
434
+ input validation. A free-text screenshot note does not populate this array.
435
+
436
+ For reuse, the producer reads publicly advertised version tags (`vX.Y.Z`, including prereleases) from the
437
+ origin/destination remote, limited to versions at or below the selected version
438
+ and excluding tags targeting the selected commit. Local-only tags cannot supply
439
+ history. It processes prior tags in SemVer order, reading each tag's committed
440
+ `RELEASE.json` and shot bytes to preserve historical `shotsAt`:
441
+
442
+ - Unchanged `look.app` reuses the previous shots and their `shotsAt`.
443
+ - Changed `look.app` with the same source screenshot paths and bytes keeps the
444
+ previous shots and `shotsAt`, making stale shots visible as `shotsAt != app`.
445
+ Editing only alt text or captions cannot claim a fresh capture.
446
+ - A changed screenshot set with a changed app records `shotsAt = app` when
447
+ screenshots are present.
448
+
449
+ The producer does not capture images. The live homepage still is a separate sync
450
+ artifact and is not included in the signed release bundle.
451
+
452
+ ### Adopting this in seed release work
453
+
454
+ Commit the seed fields, look overrides and structured screenshots before
455
+ selecting the release SHA. Release authors capture fresh demo-data app shots
456
+ when `look.app` changes; an unchanged app reuses its gallery. Use the listing
457
+ command to inspect fingerprints and packaged assets before release.
458
+
459
+ For Spool, set `seed.json.brand.mark` to `public/theme/spool-mark.svg` and extend
460
+ the `look.home` override to cover `public/theme/landing*`, `src/ui/landing/**`
461
+ and `public/theme/spool-mark.svg` alongside the relevant default patterns.
462
+ Because an override replaces its array, retain every default pattern still
463
+ needed. Include the theme mark in `look.app` too if it appears in the app.
464
+
465
+ The B6 release-template follow-up carries the completed bundle to
466
+ `~/.orchestrator/listings/bundles/<slug>/<version>/` and posts the `RELEASED`
467
+ line to the sync lane as well as the import lane. Look landings post `LOOK`
468
+ with the seed and commit. Those template/sync actions are separate from
469
+ `rer release` and `rer listing`; neither command edits or deploys a site.
470
+
341
471
  ## Signing and checking release receipts
342
472
 
343
473
  `rer release keygen` creates a publisher key and its proof-of-possession anchor
@@ -366,8 +496,10 @@ as `rer-release.dsse.json` beside the receipt. The statement binds the public
366
496
  repository, tag, commit, tree and tag object, and hashes `public-source.tar`,
367
497
  `release-receipt.json`, `publish-check.json`, the bundle's single
368
498
  `site/<id>/<version>/release.json`, and every document under that bundle's
369
- `documents/` directory. The envelope is an artifact file; signing does not
370
- change the existing release or registry pins.
499
+ `documents/` directory, plus the listing's referenced mark, light mark and
500
+ screenshot files under `assets/seeds/<id>/`. Signing and receipt verification
501
+ check those asset hashes against the actual bytes. The envelope is an artifact
502
+ file; signing does not change the existing release or registry pins.
371
503
 
372
504
  After writing the signature, `sign` prints (but never runs) a shell-quoted
373
505
  `gh release upload` command for attaching `rer-release.dsse.json` to the
@@ -1,6 +1,7 @@
1
1
  import { createHash } from 'node:crypto';
2
2
  import { mkdirSync, writeFileSync } from 'node:fs';
3
3
  import { resolve, dirname } from 'node:path';
4
+ import { listingContent } from './listing.mjs';
4
5
  import { parseContract } from '../../foundation/runtime/seed-contract.mjs';
5
6
 
6
7
  export const sha256 = bytes => createHash('sha256').update(bytes).digest('hex');
@@ -30,13 +31,15 @@ export function catalogue(source, project, repository, gates, dryRun) {
30
31
  const docs = ['seed.json', 'CONTRACT.md', 'AGENTS.md', 'LICENSE', 'RELEASE.json',
31
32
  ...['README.md', 'BASELINE.md', 'OPERATIONS.md', 'ACCEPTANCE.md'].filter(path => source.entries.has(path))];
32
33
  const artifacts = docs.map(path => ({ path: `documents/${path}`, target: `agents/seeds/${s.id}/${path}`, sha256: sha256(source.bytes(path)) }));
34
+ const content = source.listingContent ?? listingContent(source, repository);
35
+ artifacts.push(...content.assets.map(asset => ({ path: asset.path, target: asset.path, sha256: sha256(asset.bytes) })));
33
36
  const listing = { id: s.id, name: s.name, licence: 'MIT', repository, category: s.category,
34
37
  categoryLabel: s.categoryLabel ?? s.category, version: s.version, oneLiner: s.description,
35
38
  summary: s.description, maturity: s.maturity, replaces: s.replaces ?? [], clauses: contract,
36
39
  clauseCount: contract.length, nonGoals: s.nonGoals ?? [], externals: (s.externals ?? []).map(e => ({
37
40
  name: e.name ?? e.id, reason: e.reason ?? e.why, required: e.required, data: e.data ?? [], adapters: e.adapters ?? [],
38
41
  })), extensionPoints: Array.isArray(s.extensionPoints) ? s.extensionPoints : Object.values(s.extensionPoints ?? {}).flat(),
39
- shots: [], landing: null, hasReadme: docs.includes('README.md'), hasBaseline: docs.includes('BASELINE.md'), manifestFile: 'seed.json' };
42
+ ...content.listing, landing: null, hasReadme: docs.includes('README.md'), hasBaseline: docs.includes('BASELINE.md'), manifestFile: 'seed.json' };
40
43
  for (const key of ['scope', 'spine', 'composition', 'operatingCost', 'limits', 'accessibility']) if (s[key] !== undefined) listing[key] = s[key];
41
44
  return { schemaVersion: 1, project, release: s.version, provenance: {
42
45
  kind: dryRun ? 'release-preview' : 'publisher-release', repository, revision: source.commit, tree: source.tree,
@@ -50,10 +53,11 @@ export function catalogue(source, project, repository, gates, dryRun) {
50
53
 
51
54
  export function writeCatalogue(source, output, manifest) {
52
55
  const directory = resolve(output, 'site', source.seed.id, source.seed.version);
56
+ const assets = new Map((source.listingContent ?? listingContent(source, manifest.listing.repository)).assets.map(asset => [asset.path, asset.bytes]));
53
57
  for (const artifact of manifest.artifacts) {
54
58
  const path = resolve(directory, artifact.path);
55
59
  mkdirSync(dirname(path), { recursive: true });
56
- writeFileSync(path, source.bytes(artifact.path.slice('documents/'.length)), { flag: 'wx' });
60
+ writeFileSync(path, artifact.path.startsWith('documents/') ? source.bytes(artifact.path.slice('documents/'.length)) : assets.get(artifact.path), { flag: 'wx' });
57
61
  }
58
62
  const bytes = json(manifest);
59
63
  const releasePath = `${source.seed.id}/${source.seed.version}/release.json`;
@@ -0,0 +1,64 @@
1
+ import { mkdirSync, writeFileSync, rmSync } from 'node:fs';
2
+ import { resolve, dirname } from 'node:path';
3
+ import { lookFingerprints } from './look.mjs';
4
+ import { screenshotSet } from './screenshots.mjs';
5
+ import { validRelativePath } from './signature.mjs';
6
+ import { committedSource, git, newOutput } from './source.mjs';
7
+
8
+ function projectFile(source) {
9
+ return source.entries.has('rer-project.json') ? JSON.parse(source.bytes('rer-project.json')) : {};
10
+ }
11
+
12
+ export function listingRepository(source) {
13
+ const declared = projectFile(source).sourceRepository;
14
+ if (declared) return declared;
15
+ const remote = git(source.repo, 'remote').toString().split('\n').includes('origin') ? git(source.repo, 'remote', 'get-url', 'origin').toString().trim() : '';
16
+ const match = /^(?:https:\/\/github\.com\/|git@github\.com:)([\w.-]+\/[\w.-]+?)(?:\.git)?$/.exec(remote);
17
+ if (!match) throw Error('Listing needs rer-project.json sourceRepository or a credential-free GitHub origin');
18
+ return `https://github.com/${match[1]}`;
19
+ }
20
+
21
+ export function listingContent(source, repository, previous = null) {
22
+ const seed = source.seed, slug = repository.replace(/\.git$/, '').split('/').at(-1).toLowerCase().replace(/^run-/, '');
23
+ if (!/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(slug) || !/^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(seed.id)) throw Error('Invalid listing slug or ID');
24
+ const homepage = seed.homepage ?? `https://run-${slug}.com`;
25
+ const url = new URL(homepage);
26
+ if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) throw Error('Listing homepage must be a credential-free HTTP URL');
27
+ if (seed.headline !== undefined && (typeof seed.headline !== 'string' || !seed.headline.trim())) throw Error('Listing headline must be nonempty text');
28
+ const assets = [];
29
+ const mark = (key, fallback, name) => {
30
+ const path = seed.brand?.[key] ?? fallback;
31
+ if (!validRelativePath(path)) throw Error('Brand mark must be a repository-relative path');
32
+ if (!source.entries.has(path) && seed.brand?.[key] === undefined) return null;
33
+ const bytes = source.bytes(path);
34
+ if (!path.endsWith('.svg') || !/<svg[\s/>]/.test(bytes.toString())) throw Error('Brand mark must be SVG');
35
+ const target = `assets/seeds/${seed.id}/${name}`;
36
+ assets.push({ path: target, bytes });
37
+ return target;
38
+ };
39
+ const css = source.entries.has('public/rer-theme.css') ? source.bytes('public/rer-theme.css').toString().replace(/\/\*[\s\S]*?\*\//g, '') : '';
40
+ const token = name => new RegExp(`--rer-${name}\\s*:\\s*([^;}]+)`).exec(css)?.[1].trim() ?? null;
41
+ const branding = source.entries.has('src/branding.json') ? JSON.parse(source.bytes('src/branding.json')) : {};
42
+ const brand = { mark: mark('mark', 'public/brand/mark.svg', 'mark.svg'), markLight: mark('markLight', 'public/brand/mark-light.svg', 'mark-light.svg'),
43
+ colours: { accent: branding.primaryColor ?? null, ink: token('text'), paper: token('page') } };
44
+ const look = lookFingerprints(source), current = screenshotSet(source, seed.id);
45
+ const selected = previous?.listing.shots.length && (previous.listing.look.app === look.app || (current.identity && current.identity === previous.shotIdentity)) ? previous : null;
46
+ const shots = selected ? selected.listing.shots : current.shots;
47
+ assets.push(...(selected ? selected.shotAssets : current.assets));
48
+ return { listing: { slug, homepage, headline: seed.headline ?? null, brand, shots, look: { ...look, shotsAt: selected ? selected.listing.look.shotsAt : shots.length ? look.app : null } },
49
+ assets, shotAssets: selected ? selected.shotAssets : current.assets, shotIdentity: selected ? selected.shotIdentity : current.identity };
50
+ }
51
+
52
+ export function writeListing({ repo, ref = 'HEAD', output }, previousReader) {
53
+ const hydrateFrom = git(repo, 'remote').toString().split('\n').includes('origin') ? git(repo, 'remote', 'get-url', 'origin').toString().trim() : undefined;
54
+ const source = committedSource(repo, ref, { hydrateFrom }), repository = listingRepository(source);
55
+ const content = listingContent(source, repository, previousReader(source, repository));
56
+ output = newOutput(source.repo, output);
57
+ try {
58
+ for (const asset of content.assets) {
59
+ const target = resolve(output, asset.path); mkdirSync(dirname(target), { recursive: true }); writeFileSync(target, asset.bytes, { flag: 'wx' });
60
+ }
61
+ writeFileSync(resolve(output, 'listing.json'), JSON.stringify(content.listing, null, 2) + '\n', { flag: 'wx' });
62
+ } catch (error) { rmSync(output, { recursive: true, force: true }); throw error; }
63
+ return { path: resolve(output, 'listing.json'), revision: source.commit, listing: content.listing };
64
+ }