@runeditrun/rer 0.2.67 → 0.2.68

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -61,10 +61,10 @@ parsing and unexpected errors never expose response text, URLs or key values.
61
61
 
62
62
  Every adapter strips credential values from a parsed response before it emits
63
63
  bytes for upload, keeping the record, by the same rules as the seed's import
64
- door. A field whose name's letters end in a credential word (`apiKey`,
64
+ door. A field whose name's parts are a credential word or phrase (`apiKey`,
65
65
  `httpPassword`, `http_password`, `client_secret`, `X-Api-Key`, `integration_key`,
66
66
  `webhook_url`, `secret_key`, `x-functions-key`, `Ocp-Apim-Subscription-Key`,
67
- `pwd` and the like) or are `X-Auth`, the `value` of a `{name|key, value}` pair
67
+ `pwd`, `X-Auth` and the like), the `value` of a `{name|key, value}` pair
68
68
  naming one (`{name: 'Authorization', value}`), or the value of such a
69
69
  `[name, value]` pair in a list of them (Sentry's header tuples), becomes
70
70
  `"[removed]"`. A plural name (`apiKeys`, `tokens`) counts only when it holds
@@ -76,8 +76,20 @@ and the value of each credential query or fragment parameter (`api_key`,
76
76
  own validated pagination keys (`page_token`). The door drops each marker and
77
77
  reports it per record as `CREDENTIAL_REMOVED`, so the owner sees which record
78
78
  lost a credential. Null, booleans, empty strings and masks such as `****` are
79
- not credentials and stay as they are. Names that merely contain such letters
80
- (`keyword`, `monkey`, `signal`, `design`) are not credential names.
79
+ not credentials and stay as they are. Names are matched by whole parts
80
+ (camelCase, `_`, `-`, digits), so names that merely contain such letters
81
+ (`keyword`, `monkey`, `author`, `signal`, `design`, `primaryKey`) are not
82
+ credential names and a `?keyword=up` URL stays intact. `Signature` and
83
+ `Session-Id` count only as header names (a key under `headers`, a tuple, a
84
+ `{name|key, value}` pair in a header list), so an agent's `signature` field and
85
+ Help Scout's `customFields: [{name: 'Session ID', value}]` stay;
86
+ `auth` and `credentials` count only when they hold text naming no scheme or mode,
87
+ and an object under them loses only its own credential fields
88
+ (`auth: {type: 'oauth2', token: "[removed]"}`) and a `value`, `pass`, `pw` or
89
+ `secret…` member (Keycloak's `credentials: [{type: 'password', value: "[removed]"}]`).
90
+ `passphrase`, `jwt`, `JSESSIONID` and `PHPSESSID` count everywhere. Query and fragment keys also
91
+ count documented API key names such as `token_auth`, `application_key`,
92
+ `user_key` and `hapikey`.
81
93
  Pagination is read from the response as received, so a credential-bearing
82
94
  pagination link is still refused.
83
95
 
@@ -1,5 +1,6 @@
1
1
  import { createHash } from 'node:crypto';
2
2
  import { setTimeout as delay } from 'node:timers/promises';
3
+ import { credentialMember, credentialName, credentialScheme, headerContainer } from './credential-names.mjs';
3
4
 
4
5
  export class VendorExportError extends Error {
5
6
  constructor(code, status) {
@@ -47,14 +48,22 @@ export function reference(value, origin, pathname, allowedQuery = []) {
47
48
  return url;
48
49
  }
49
50
  // The seed's import door applies the same rules (modules/import-kit, removeCredentials).
50
- // A field holds a credential when its name's letters end in a credential word
51
- // (apiKey, httpPassword, http_password, client_secret, X-Api-Key, integration_key,
52
- // webhook_url, secret_key, x-functions-key, Ocp-Apim-Subscription-Key, db_pwd) or
53
- // are X-Auth, unless it is empty: null, a boolean (Calendly's `secret` flag),
54
- // whitespace or a mask such as "****" or "[redacted]". A plural (apiKeys, tokens)
55
- // holds credentials only as text, so a count such as `tokens: 1500` stays.
56
- const credentialName = /(?:authorization|credentials?|webhookurl|calurl)$|^xauth$/u;
57
- const credentialWord = /(?:password|passwd|pwd|secret|token|cookie|(?:api|access|private|auth|client|integration|routing|service|signing|session|refresh|secret|subscription|functions)key)(s?)$/u;
51
+ // A field holds a credential when the parts of its name are a credential word or
52
+ // phrase (apiKey, httpPassword, http_password, client_secret, X-Api-Key,
53
+ // integration_key, webhook_url, secret_key, x-functions-key,
54
+ // Ocp-Apim-Subscription-Key, db_pwd, X-Auth), unless its value is empty: null, a
55
+ // boolean (Calendly's `secret` flag), whitespace or a mask such as "****" or
56
+ // "[redacted]". Names that only contain such letters (keyword, monkey, author,
57
+ // signal, design) are not credentials. A plural (apiKeys, tokens) holds
58
+ // credentials only as text, so a count such as `tokens: 1500` stays. `auth` and
59
+ // `credentials` hold one only as text naming no scheme or mode (`include`,
60
+ // `oauth2`); an object under them keeps its fields and loses its own credential
61
+ // fields (`auth: {token}`) and, directly under them, a `value`, `pass`, `pw` or
62
+ // `secret…` member (Keycloak's `credentials: [{type, value}]`, nodemailer's
63
+ // `{user, pass}`). `passphrase`, `jwt`, `JSESSIONID` and `PHPSESSID` count
64
+ // everywhere. `sig`, `signature` and `sessionId` are credentials as header names
65
+ // (a tuple, a key under `headers`, a `{name|key, value}` pair in a header list),
66
+ // never as fields; a pair elsewhere (Help Scout's `customFields`) labels a field.
58
67
  const placeholder = /^(?:\s*|[*\u2022\u25CF\u00B7x#_.-]{1,64}|[<[({]?\s*(?:redacted|hidden|masked|removed|omitted|none|null|n\/a|placeholder)\s*[>\])}]?)$/iu;
59
68
  /** What a stripped credential value becomes; the seed's import door reports it per record. */
60
69
  export const REMOVED_CREDENTIAL = '[removed]';
@@ -70,15 +79,20 @@ function decoded(text) {
70
79
  for (let depth = 0; depth < 32 && /%[\da-f]{2}/iu.test(text); depth++) text = text.replace(/%([\da-f]{2})/giu, (_, hex) => String.fromCharCode(parseInt(hex, 16)));
71
80
  return text;
72
81
  }
73
- /** Whether a value under this name is a credential, by the door's rule. */
74
- function credential(name, value) {
75
- const letters = decoded(name).toLowerCase().replace(/[^a-z]/gu, ''), word = credentialWord.exec(letters);
76
- if (!word && !credentialName.test(letters)) return false;
77
- return !empty(value) && (!word?.[1] || text(value));
78
- }
79
- // The door's credentialQueryKey rule for URL query and fragment parameter names;
80
- // `sig` (an Azure SAS signature) only as the whole name.
81
- const credentialParameter = key => /token|secret|key|auth|signature|credential|password|cookie/iu.test(decoded(key)) || /^\s*sig\s*$/iu.test(decoded(key));
82
+ /** Text under `auth` or `credentials` naming no scheme or mode, or a list with such a member. */
83
+ const schemeless = value => typeof value === 'string' ? !placeholder.test(value) && !credentialScheme(value) : Array.isArray(value) && value.some(member => typeof member === 'string' && schemeless(member));
84
+ /** Whether a value under this name, a record field or an HTTP header name, is a credential, by the door's rule. */
85
+ function credential(name, value, place) {
86
+ const found = credentialName(decoded(name), 'remove', place);
87
+ return found !== undefined && !empty(value) && (found.container ? schemeless(value) : !found.plural || text(value));
88
+ }
89
+ /** Whether this key is `auth`, `X-Auth`, `credential` or `credentials`, whose members are searched for a credential under a generic name. */
90
+ const container = key => credentialName(decoded(key), 'remove')?.container === true;
91
+ /** A member of such a container holding a credential under a generic name (`value`, `pass`, `pw`, `secretData`). */
92
+ const member = (key, value) => credentialMember(decoded(key)) !== undefined && !empty(value);
93
+ // The door's credentialQueryKey rule for URL query and fragment parameter names:
94
+ // api_key, access_token, a bare `key`, sig and X-Amz-Signature, never keyword or author.
95
+ const credentialParameter = key => credentialName(decoded(key), 'query') !== undefined;
82
96
  const authority = /^(\s*(?:[a-z][a-z\d+.-]*:)?[\\/]{2})[^\\/?#]*@/iu;
83
97
  /** A URL of any scheme with its user name and password replaced by `removed@`,
84
98
  * and the value of each credential query or fragment parameter by `removed`,
@@ -107,17 +121,24 @@ const tuples = list => list.length > 0 && list.every(item => Array.isArray(item)
107
121
  * URLs lose credentials as withoutUrlCredentials describes; `pagination` lists
108
122
  * the adapter's validated query keys, which stay. */
109
123
  export function withoutCredentials(value, pagination = []) {
110
- if (Array.isArray(value)) {
111
- const pairs = tuples(value);
112
- return value.map(child => pairs && credential(child[0], child[1]) ? [child[0], REMOVED_CREDENTIAL] : withoutCredentials(child, pagination));
113
- }
114
- if (value && typeof value === 'object') {
115
- const label = Object.entries(value).find(([key, child]) => /^(?:name|key)$/iu.test(key) && typeof child === 'string');
116
- const pair = !!label && Object.hasOwn(value, 'value');
117
- return Object.fromEntries(Object.entries(value).map(([key, child]) =>
118
- [key, credential(pair && key === 'value' ? label[1] : key, child) ? REMOVED_CREDENTIAL : withoutCredentials(child, pagination)]));
119
- }
120
- return typeof value === 'string' ? withoutUrlCredentials(value, pagination) : value;
124
+ // `headers`: the keys of this object, or of objects in this list, are HTTP header names.
125
+ // `inside`: this object, or the objects in this list, sit under `auth` or `credential(s)`.
126
+ const strip = (value, headers, inside = false) => {
127
+ if (Array.isArray(value)) {
128
+ const pairs = tuples(value);
129
+ return value.map(child => pairs && credential(child[0], child[1], 'header') ? [child[0], REMOVED_CREDENTIAL] : strip(child, headers, inside));
130
+ }
131
+ if (value && typeof value === 'object') {
132
+ // A `{name|key, value}` pair's name is a header name only under a header container; elsewhere it labels a field.
133
+ const place = headers ? 'header' : 'field';
134
+ const label = Object.entries(value).find(([key, child]) => /^(?:name|key)$/iu.test(key) && typeof child === 'string');
135
+ const pair = !!label && Object.hasOwn(value, 'value') && credentialName(decoded(label[1]), 'remove', place) !== undefined;
136
+ return Object.fromEntries(Object.entries(value).map(([key, child]) => [key, (pair && key === 'value' ? credential(label[1], child, place) : credential(key, child, place) || (inside && member(key, child)))
137
+ ? REMOVED_CREDENTIAL : strip(child, headerContainer(key), container(key))]));
138
+ }
139
+ return typeof value === 'string' ? withoutUrlCredentials(value, pagination) : value;
140
+ };
141
+ return strip(value, false);
121
142
  }
122
143
  export function assertCount(value, max = 1000) {
123
144
  if (!Number.isSafeInteger(value) || value < 0 || value > max) fail('INVALID_CHECKPOINT');
@@ -0,0 +1,124 @@
1
+ // A JavaScript copy of modules/import-kit/src/provenance/credential-names.ts, which the
2
+ // seed's import door applies; keep the two the same (rer's vendor exports use `remove` and `query`).
3
+ /** Whether a name (an object key, a header name, the name in a `[name, value]`
4
+ * pair, a URL query or fragment key) names a credential, by its whole parts.
5
+ *
6
+ * A name is split into lower-case parts at camelCase and PascalCase boundaries
7
+ * (`apiKey`, `XAPIKey`), at `_`, `-`, `.`, spaces and any other non-letter, and
8
+ * at digits (`token2` is `token`). A name is a credential when its parts are a
9
+ * credential word or phrase, never because a part only contains one: `keyword`,
10
+ * `monkey`, `author`, `signal`, `design` and `tokenizer` hold the letters of a
11
+ * credential word and are not credentials. */
12
+ // credentialName returns the name's letters, lower-cased (`apiKey` and `api_key` are both
13
+ // `apikey`), and whether it is the word's plural (`tokens`, `apiKeys`: it holds credentials
14
+ // only as text, so a count stays), or undefined. Modes: `refuse`: the whole name is a
15
+ // credential phrase, so a pagination cursor such as `next_page_token` is not one. `remove`:
16
+ // the name ends in a credential word (`pushToken`, `webhook_url`), the wider net of the
17
+ // import door, which drops the value and keeps the record. `query`: a URL query or fragment
18
+ // key, as `remove` plus a bare `key` and `X-Amz-Signature`. It also says whether the name is
19
+ // a container (`auth`, `X-Auth`, `credential(s)`): its value is a credential when it is text
20
+ // that names no scheme or mode (`include`, `oauth2`), or a list with such a member; an object
21
+ // under it is searched field by field. Places, in `remove` mode: a record `field`, or an HTTP
22
+ // `header` name (a `[name, value]` tuple, a key of a `headers` object, a `{name|key, value}`
23
+ // pair in a header list). `sig`, `signature` and `sessionId` are credentials only as header names and URL
24
+ // query or fragment keys, never as record fields, so an agent's `signature` or a chat's
25
+ // `session_id` imports.
26
+ const WORDS = new Set(['token', 'secret', 'password', 'passwd', 'pwd', 'cookie', 'authorization', 'bearer', 'credential']);
27
+ /** Words that make a trailing `key` a credential: `api key`, `secret key`, `Ocp-Apim-Subscription-Key`. A bare `key` or `primaryKey` is not. */
28
+ const KEY_QUALIFIERS = new Set(['api', 'access', 'private', 'secret', 'auth', 'client', 'integration', 'routing', 'service', 'signing', 'session',
29
+ 'refresh', 'subscription', 'functions', 'consumer', 'app', 'license', 'master', 'encryption', 'admin']);
30
+ /** URL query and fragment keys only: `?application_key=` (Datadog), `?user_key=` (3scale), `?account_key=`, `?deploy_key=`, `?shared_key=`, and `?token_auth=` (Matomo), `?basic_auth=`. */
31
+ const QUERY_KEY_QUALIFIERS = new Set(['application', 'user', 'account', 'deploy', 'shared']);
32
+ const QUERY_AUTH_QUALIFIERS = new Set(['token', 'basic', 'http', 'proxy', 'api']);
33
+ const TOKEN_QUALIFIERS = ['access', 'refresh', 'auth', 'api', 'bearer', 'id', 'session', 'csrf', 'xsrf', 'bot', 'push', 'oauth', 'jwt'];
34
+ const SECRET_QUALIFIERS = ['client', 'app', 'api', 'oauth', 'webhook', 'signing', 'shared'];
35
+ const PASSWORD_QUALIFIERS = ['http', 'db', 'ftp', 'smtp', 'admin', 'user', 'basic', 'proxy', 'root', 'mail', 'login'];
36
+ /** One-word credentials however a name splits them: a `passphrase` (`passPhrase`), a `jwt`, and Java's and PHP's session
37
+ * cookies (`JSESSIONID`, `jsessionId`, `PHPSESSID`). They name nothing else, so they count as fields, header names and URL keys. */
38
+ const SPELLED = /^(?:passphrase|jwt|jsessionid|phpsessid)(s?)$/;
39
+ /** What may stand before one in `refuse` mode (`ssh_passphrase`, `id_jwt`). */
40
+ const SPELLED_QUALIFIERS = new Set([...PASSWORD_QUALIFIERS, ...TOKEN_QUALIFIERS, 'ssh', 'key', 'gpg', 'pgp', 'private']);
41
+ /** What may stand before a credential word in `refuse` mode, which names the whole phrase (`access token`, `http password`). */
42
+ const QUALIFIERS = {
43
+ token: new Set(TOKEN_QUALIFIERS), secret: new Set(SECRET_QUALIFIERS),
44
+ password: new Set(PASSWORD_QUALIFIERS), passwd: new Set(PASSWORD_QUALIFIERS), pwd: new Set(PASSWORD_QUALIFIERS),
45
+ key: KEY_QUALIFIERS,
46
+ };
47
+ /** The same phrases written as one lower-case part (`apikey`, `accesstoken`, `clientsecret`, `httppassword`, `webhookurl`). */
48
+ const COMPOUND = new RegExp('^x?(?:'
49
+ + `(?:${[...KEY_QUALIFIERS].join('|')})(?:key)|(?:${TOKEN_QUALIFIERS.join('|')})(?:token)|(?:${SECRET_QUALIFIERS.join('|')})(?:secret)`
50
+ + `|(?:${PASSWORD_QUALIFIERS.join('|')})(?:password|passwd|pwd)|webhook(?:url)|cal(?:url))(s?)$`);
51
+ /** A scheme or mode an `auth` or `credentials` field names, never a credential:
52
+ * fetch's `include`, `same-origin` and `omit`, HTTP and API authentication schemes, and boolean
53
+ * words (Keycloak's `smtpServer.auth: "true"`). */
54
+ const SCHEME = /^(?:omit|sameorigin|include|none|noauth|inherit|anonymous|basic|bearer|digest|negotiate|ntlm|kerberos|oauth(?:1|2|10|20|1a)?|apikey|jwt|hawk|hmac|aws|awsv4|sigv4|awssigv4|edgegrid|mutual|vapid|token|password|cookie|session|header|query|true|false|yes|no|on|off|enabled|disabled)$/;
55
+
56
+ /** Whether text under an `auth` or `credentials` name names a scheme or mode (`include`, `oauth2`, `Bearer`, `same-origin`). */
57
+ export function credentialScheme(text) {
58
+ return SCHEME.test(text.trim().toLowerCase().replace(/[\s_.-]/g, ''));
59
+ }
60
+ /** Whether an object under this key holds HTTP headers by name (`headers`, `customHttpHeaders`, `header`). */
61
+ export function headerContainer(key) {
62
+ const last = nameParts(key).at(-1);
63
+ return last === 'headers' || last === 'header';
64
+ }
65
+ // Under `auth`, `X-Auth`, `credential` or `credentials`, a member holding the credential under a generic name: `value`
66
+ // (Keycloak's `{type: 'password', value}`, `credential: {type: 'apiKey', value}`), `pass` and `pw` (nodemailer's
67
+ // `{user, pass}`), or a name whose first part is `secret` (Keycloak's `secretData`). Its name's letters, or undefined.
68
+ export function credentialMember(name) {
69
+ const parts = nameParts(name);
70
+ return (parts.length === 1 && /^(?:value|pass|pw)$/.test(parts[0])) || parts[0] === 'secret' ? parts.join('') : undefined;
71
+ }
72
+
73
+ /** A name's lower-case parts, without digits. Format characters and accents are folded away first. */
74
+ export function nameParts(name) {
75
+ const text = name.normalize('NFKC').replace(/[\p{Cf}\p{M}]/gu, '');
76
+ return (text.match(/\p{Lu}+(?!\p{Ll})|\p{Lu}?\p{Ll}+|\p{Lo}+/gu) ?? []).map(part => part.toLowerCase());
77
+ }
78
+
79
+ export function credentialName(name, mode, place = 'field') {
80
+ const parts = nameParts(name), last = parts.at(-1), before = parts.at(-2);
81
+ if (last === undefined) return undefined;
82
+ const letters = name.normalize('NFKC').toLowerCase().replace(/[^a-z]/g, '').slice(-64);
83
+ const compound = COMPOUND.exec(last);
84
+ if (compound) return { name: letters, plural: compound[1] === 's', container: false };
85
+ // A header's leading `x-` (`x-api-key`, `X-Auth-Token`) is not part of the phrase; `refuse` mode takes the rest whole.
86
+ const phrase = parts.length > 1 && parts[0] === 'x' ? parts.slice(1) : parts;
87
+ const found = (plural, container = false) => ({ name: letters, plural, container });
88
+ const qualifier = word => word !== undefined && (KEY_QUALIFIERS.has(word) || (word.startsWith('x') && KEY_QUALIFIERS.has(word.slice(1))));
89
+ const strict = mode === 'refuse';
90
+ // A URL key or a header name: where `sig`, `signature` and `sessionId` are credentials.
91
+ const header = mode === 'query' || place === 'header';
92
+ for (let size = 1; size <= Math.min(3, phrase.length); size++) {
93
+ const spelled = SPELLED.exec(phrase.slice(-size).join(''));
94
+ if (spelled && (!strict || phrase.length === size || (phrase.length === size + 1 && SPELLED_QUALIFIERS.has(phrase[0])))) return found(spelled[1] === 's');
95
+ }
96
+ // `credentials` holds a credential object as one value; the other plurals hold a list or map of text.
97
+ const word = last === 'credentials' ? 'credential' : WORDS.has(last) ? last : last.endsWith('s') && WORDS.has(last.slice(0, -1)) ? last.slice(0, -1) : undefined;
98
+ if (word !== undefined) {
99
+ const plural = word !== last && last !== 'credentials';
100
+ // `refuse` mode: the word alone or after one qualifier (`token`, `access_token`), and `pwd` after anything.
101
+ const named = phrase.length === 1 || (phrase.length === 2 && QUALIFIERS[word]?.has(phrase[0]) === true) || word === 'pwd';
102
+ if (!strict || named) return found(plural, word === 'credential' && mode !== 'query');
103
+ }
104
+ if (last === 'key' || last === 'keys') {
105
+ // `refuse` mode: `api_key`, `x-functions-key`, and `secret`/`subscription`/`functions` before the key under any name.
106
+ const named = phrase.length === 2 || (before !== undefined && ['secret', 'subscription', 'functions'].includes(before));
107
+ if (qualifier(before) && (!strict || named)) return found(last === 'keys');
108
+ // A bare `key` is a credential only as a URL query or fragment key (the `?key=` convention).
109
+ if (mode === 'query' && parts.length === 1 && last === 'key') return found(false);
110
+ if (mode === 'query' && before !== undefined && QUERY_KEY_QUALIFIERS.has(before)) return found(last === 'keys');
111
+ }
112
+ // A session id is a credential as a header or URL key (`Session-Id`, `?sessionid=`), never as a record's field (a chat's `session_id`).
113
+ const session = /^x?sessionid(s?)$/.exec(last);
114
+ if (header && session) return found(session[1] === 's');
115
+ if (header && last === 'id' && before === 'session' && (!strict || phrase.length === 2)) return found(false);
116
+ if (!strict && last === 'url' && (before === 'webhook' || before === 'cal')) return found(false);
117
+ // A signature is a credential as a whole header name (`Signature`) and, as a query key, as its last part (an Azure SAS `sig`, `X-Amz-Signature`); an agent's `signature` field is not.
118
+ if ((last === 'sig' || last === 'signature') && (mode === 'query' || (place === 'header' && parts.length === 1))) return found(false);
119
+ if (last === 'auth' && (parts.length === 1 || (parts.length === 2 && before === 'x'))) return found(false, mode !== 'query');
120
+ if (mode === 'query' && last === 'auth' && before !== undefined && QUERY_AUTH_QUALIFIERS.has(before)) return found(false);
121
+ // A query key written as one part that ends in an unambiguous phrase: HubSpot's `?hapikey=`.
122
+ if (mode === 'query' && parts.length === 1 && /(?:apikey|accesstoken|authtoken)$/.test(last)) return found(false);
123
+ return undefined;
124
+ }
@@ -0,0 +1,7 @@
1
+ // Only declared safe replacement values and aggregate counts reach the terminal.
2
+ export function importSafetySummary({safetyChanges}={}) {
3
+ if(!safetyChanges?.length)return undefined;
4
+ return safetyChanges.map(({table,column,value,rows})=>
5
+ `${rows.toLocaleString('en-US')} ${table} imported with ${column}=${JSON.stringify(value)}`
6
+ ).join('\n');
7
+ }
package/cli/output.mjs CHANGED
@@ -1,5 +1,6 @@
1
1
  import { handleBrokenPipes } from './stdio.mjs';
2
2
  import { exportSummary } from './export-summary.mjs';
3
+ import { importSafetySummary } from './import-summary.mjs';
3
4
  export function output(options) {
4
5
  handleBrokenPipes();
5
6
  let finished = false;
@@ -19,6 +20,7 @@ export function output(options) {
19
20
  if (structuredResult) this.event('result', { ok, exitCode, ...details });
20
21
  // A finished export ends with one line naming its archive; the receipt is --json data.
21
22
  else if (ok && options.verb === 'export' && exportSummary(details)) this.human(exportSummary(details));
23
+ else if (ok && options.verb === 'import' && importSafetySummary(details)) this.human('import: complete (exit 0)\n' + importSafetySummary(details));
22
24
  else if (!['describe', 'check', 'dev', 'preview', 'setup'].includes(options.verb) || !ok) this.human(`${options.verb}: ${ok ? 'complete' : 'failed'} (exit ${exitCode})`);
23
25
  },
24
26
  error(error) {
@@ -6,6 +6,7 @@ import { legacyTargetState, requireLegacyEmpty } from '../modules/portability/le
6
6
  import { migrationReplay } from '../modules/portability/migration-history.mjs';
7
7
  import { appliedMigrations as appliedMigrationNames } from '../modules/portability/bridge.mjs';
8
8
  import { join } from 'node:path';
9
+ import { createHash } from 'node:crypto';
9
10
  import { mkdir, readFile, rm, stat, writeFile } from 'node:fs/promises';
10
11
  import { createRequire } from 'node:module';
11
12
  import { createPortability, PortabilityError, readArchive } from '../modules/portability/index.mjs';
@@ -14,13 +15,16 @@ import { nativeFailure, stateBytes } from './portability-progress.mjs';
14
15
  import { parseJsonc } from './jsonc.mjs';
15
16
  import { loadPolicy } from './portability-policy.mjs';
16
17
 
17
- // The migrations this installation applied, in its ledger's order, so a
18
- // restore can place an older archive within them.
18
+ // The migrations this installation applied, in its ledger's order, with each
19
+ // file's hash, so a restore can place an older archive within them.
19
20
  async function appliedMigrations(root, db, binding, split) {
20
21
  const config = parseJsonc(await readFile(join(root, 'wrangler.jsonc'), 'utf8'));
21
22
  const directory = join(root, config.d1_databases.find(entry => entry.binding === binding).migrations_dir ?? 'migrations');
22
23
  const names = await appliedMigrationNames(db) ?? [];
23
- return Promise.all(names.map(async name => ({ name, statements: split(await readFile(join(directory, name), 'utf8')) })));
24
+ return Promise.all(names.map(async name => {
25
+ const bytes = await readFile(join(directory, name));
26
+ return { name, statements: split(bytes.toString('utf8')), sha256: createHash('sha256').update(bytes).digest('hex') };
27
+ }));
24
28
  }
25
29
 
26
30
  // Transfers read only the D1 database and R2 buckets. A configuration with just
@@ -81,8 +85,10 @@ catch (error) {
81
85
  const migration = error instanceof PortabilityError && error.code.startsWith('schema_mismatch:migration_data:') ? error.code.slice('schema_mismatch:migration_data:'.length) : undefined;
82
86
  // A populated destination names its table or bucket: schema names, never row data.
83
87
  const occupied = error instanceof PortabilityError && /^(target|bucket)_not_empty:/.test(error.code) ? error.code.slice(error.code.indexOf(':') + 1) : undefined;
88
+ // Safety diagnostics contain only policy schema names, never source values.
89
+ const safety = error instanceof PortabilityError && error.code.startsWith('import_row_') ? error.code.slice(error.code.indexOf(':')+1) : undefined;
84
90
  // Anything that is not a portability refusal names its class and code, never its message: that can hold row data.
85
91
  const native = error instanceof PortabilityError || error.message === 'archive_json' ? undefined : nativeFailure(error);
86
92
  process.exitCode = exitCode;
87
- await reply({ ok: false, code: exitCode === 2 ? 'invalid_archive' : exitCode === 4 ? 'restore_refused' : 'transfer_failed', reason, ...(migration ? { migration } : {}), ...(occupied ? { occupied } : {}), ...(native ? { native } : {}), exitCode });
93
+ await reply({ ok: false, code: exitCode === 2 ? 'invalid_archive' : exitCode === 4 ? 'restore_refused' : 'transfer_failed', reason, ...(migration ? { migration } : {}), ...(occupied ? { occupied } : {}), ...(safety ? { safety } : {}), ...(native ? { native } : {}), exitCode });
88
94
  }
@@ -82,6 +82,7 @@ async function nativeWorker(store,direction,input,signal) {
82
82
  if (!receipt || typeof receipt !== 'object') throw new RerError('transfer_failed', `Local portability ended before it finished (${status.signal ? `signal ${status.signal}` : `exit code ${status.code}`}). Run the same command again; it starts over${direction === 'import' ? ' and first cleans up anything this attempt staged' : ''}.`, 1);
83
83
  if (status.code !== 0 || !receipt.ok) throw new RerError(receipt.code ?? 'transfer_failed', receipt.reason==='legacy_restored_export_failed'
84
84
  ? 'Legacy data was restored and audited, but its v2 export failed. Preserve this installation, resolve the output problem, then run rer export --output FILE. Do not reset it to retry recovery.'
85
+ : receipt.safety ? `Restore refused (${receipt.reason}): safety normalization for ${receipt.safety} violates the declared import policy.`
85
86
  : receipt.migration ? `Restore refused (schema_mismatch:migration_data): migration ${receipt.migration} changed data after this archive was made, so its rows would skip that change. Restore an archive made after it.`
86
87
  : receipt.occupied ? `Restore refused (${receipt.reason}): ${receipt.reason === 'bucket_not_empty' ? `storage bucket ${receipt.occupied} in this installation already holds files` : `table ${receipt.occupied} in this installation already holds rows its migrations did not insert`}, and a restore only goes into an empty installation. To replace this installation's data with the archive, keep a copy with rer export --output exports/before-restore.rer, then run rer reset --yes and rer import --input ${relative(input.root, input.path)} --yes again.`
87
88
  : `The portability module refused or failed the archive operation (${receipt.reason ?? 'incomplete'}${receipt.native ? `: ${receipt.native.class}${receipt.native.code && receipt.native.code !== receipt.reason ? `, code ${receipt.native.code}` : ''}` : ''}).${receipt.native && receipt.exitCode === 1 ? ' Run the same command again; it starts over.' : ''}`, receipt.exitCode ?? 1);
@@ -1,5 +1,13 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.2.68
4
+
5
+ - **Retained-archive restores (#316, #318).** Placing an older archive treats a CREATE TRIGGER or DROP TRIGGER statement bounded as exactly one trigger as schema-only, and restore drops and recreates the destination's triggers around insertion; a later write that reaches an archived table through a trigger is still refused. Base reads RER Platform's rer_migrations ledger, places an archive without a migration record by the ledger it carries (every archived name and hash must match an applied migration), and admits a later UPDATE or DELETE of a table the archive holds no rows of.
6
+ - **Release scans committed bytes only (#317).** rer release scans the raw committed source in an isolated mirror, so ignored local files such as .dev.vars and dist no longer block a release; a committed credential still does.
7
+ - **Import credential names (#311).** The import door matches credential names by whole name parts (keyword, author and primaryKey are not credentials), catches query-key credentials such as application_key, hapikey and token_auth, and treats signatures and session ids as credentials only as URL keys and header names.
8
+ - **Lobby recovery help on this computer only (#319).** The claimed Lobby shows the project-folder `rer signin` line only on loopback hosts, so hosted owners no longer see it.
9
+ - **Import safety normalization (#322).** A seed's importSafety/importRow hook bounds staged rows to declared safe values (for example connections forced inactive with their tokens cleared), checks determinism and idempotence, and journals per-table, column and value counts into status, receipts and owner CLI output.
10
+
3
11
  ## 0.2.67
4
12
 
5
13
  - **Import door options (#298 with #308).** Seeds choose credential removal modes and a preserve or strict reversal policy when an import is created. Failed applies can be retried, CLI-stripped query and fragment credentials receive per-record notes, more credential shapes are recognized, and a blocked re-baseline is refused at plan time with a named 409.
@@ -41,6 +41,8 @@ schema at construction. Apply the bookkeeping migration first. Policy fields:
41
41
  | `derivedTables` | Virtual/FTS tables to regenerate; their shadow tables are excluded. |
42
42
  | `secretColumns` | Per-table nullable base columns omitted at export and written as NULL. Primary keys and generated columns cannot be secret. |
43
43
  | `exportRow(table,row)` | Pure, synchronous, idempotent transformation of encoded base cells. Restore writes its result verbatim; SQLite recomputes generated cells. |
44
+ | `importSafety` | Per-table, per-column nonempty finite sets of safe encoded cells, paired with `importRow`. Only writable non-key columns may be declared. |
45
+ | `importRow(table,row)` | Pure, synchronous, deterministic, idempotent normalization of staged rows in declared tables, before `validateStaged` and apply. Changes are bounded by `importSafety`. |
44
46
  | `replaceableTables` | Bootstrap rows replaced in the atomic restore batch. Rows the destination's own migrations insert need no entry where the runner passes `migrations` (see "Migration-seeded rows"). Populated excluded dependents reachable through modifying FK actions refuse replacement. |
45
47
  | `requiredConfiguration` | Setting names returned after restore for the owner to re-enter. |
46
48
  | `validateStaged({db,tables,read,manifest})` | Read-only product validation before any object write; false or a thrown error refuses restore. `tables` names the staging tables for bounded SQL queries; `read` is a convenience for small tables. |
@@ -55,7 +57,7 @@ belong to the current runtime or environment and never travel:
55
57
  | Tables (exact names or prefixes ending in `%`) | Reason |
56
58
  | --- | --- |
57
59
  | `sqlite_%`, `_cf_%` | SQLite and Cloudflare internals. |
58
- | `d1_migrations`, `_rer_deploy_migrations`, `_rer_portability_migrations`, `rer_migrations` | Destination migration history. Current Base uses the D1 and `_rer_` ledgers; `rer_migrations` is excluded for historical compatibility. |
60
+ | `d1_migrations`, `_rer_deploy_migrations`, `_rer_portability_migrations`, `rer_migrations` | Destination migration history. Base uses the D1 and `_rer_` ledgers; RER Platform's runner writes `rer_migrations`. An old archive that still carries its source's ledger is placed by it (see below); its rows are never restored. |
59
61
  | `rer_portability_%` | Transfer jobs, snapshots, staging, journals and recovery bookkeeping. |
60
62
  | `rer_mail_development_outbox`, `rer_ai_development_outbox` | Local development effect captures. |
61
63
  | `rer_dependency_keys`, `rer_dependency_pending`, `rer_dependency_codes` | Provider secrets and transient linking credentials. |
@@ -151,6 +153,49 @@ the bytes. Large objects span parts and restore through multipart upload with at
151
153
  most one 5 MiB upload buffer. Extra payload consumers own their own transaction,
152
154
  idempotence and recovery; D1 cannot roll back Durable Object or external state.
153
155
 
156
+ ### Bounded import normalization
157
+
158
+ Use `importSafety` with `importRow` when restored records must start in a safe
159
+ state, even if an older archive contains active connections or credentials:
160
+
161
+ ```js
162
+ export default {
163
+ product: 'my-product',
164
+ importSafety: {connections: {status: ['inactive'], token: [null]}},
165
+ importRow(table, row) {
166
+ return {...row, status: 'inactive', token: null};
167
+ },
168
+ };
169
+ ```
170
+
171
+ The hook is called only for tables declared in `importSafety`, after staging
172
+ and before `validateStaged` or apply. Mandatory development/provider-data refusals
173
+ run first; normalization cannot bypass them. It must return exactly the writable columns
174
+ of its input row, preserve primary keys and every undeclared column, and change
175
+ declared columns only to their listed safe values. Unchanged values need not be
176
+ in the safe set. The hook must be pure, synchronous, deterministic and idempotent;
177
+ keep external effects out of it. Generated columns recompute from the normalized
178
+ base cells. Without this pair, import behavior is unchanged.
179
+
180
+ Construction refuses an unpaired hook or declaration, unknown table or column
181
+ names, primary keys, generated columns, empty safe sets and malformed encoded
182
+ cells. Safe values use the ordinary cell encoding: null, strings, finite numbers
183
+ or `{blob:base64}`. The destination policy supplies the hook; an archive cannot
184
+ supply executable normalization.
185
+
186
+ The import job persistently journals the summary in `metadata.safetyChanges`.
187
+ `imports.status(id)` and the `importArchive` receipt expose it as `safetyChanges`,
188
+ an array of `{table,column,value,rows}`. Each entry counts actual changed rows for
189
+ that column and safe replacement value; a row changed in two columns contributes
190
+ to two entries. The summary contains no original values or row records. Durable
191
+ checkpoints keep retries from counting changes twice. Human `rer import` output
192
+ prints these counts too, for example `3 automations imported with status="inactive"`.
193
+
194
+ Archive integrity checks still verify the original archived contents. This
195
+ summary is separate from the erasure journal: normalization does not bypass its
196
+ fence or change journal adoption. Existing atomic D1 apply, owned-object rollback,
197
+ extra-payload recovery responsibilities and transfer erasure behavior still apply.
198
+
154
199
  ## Archive and durable jobs
155
200
 
156
201
  The UTF-8 JSON-lines archive has a manifest, indexed checksummed parts and a
@@ -210,16 +255,30 @@ table). This applies to normal v2 import and to backup-only v1 recovery.
210
255
 
211
256
  The Base CLI runners (local Workers and container) pass `createPortability` a
212
257
  `migrations` replayer (`migration-history.mjs`, Node only) built from the
213
- destination's applied migration ledger and migration files. Import replays them
214
- into a scratch database to find the archive's place in that history:
258
+ destination's applied migration ledger and migration files, with each file's
259
+ SHA-256. The ledger is `_rer_deploy_migrations`, `d1_migrations` (Wrangler or
260
+ the deploy kit) or, on an installation RER Platform installed, its exact
261
+ `rer_migrations(name, sha256, applied_at)` ledger, read in application order
262
+ (rowid) and refused while it holds a partially applied file. Import replays
263
+ them into a scratch database to find the archive's place in that history:
215
264
 
216
265
  - Archives record the applied migration names in `manifest.migrations` (since
217
266
  additive restore). Every recorded migration must be applied here, and
218
267
  replaying exactly those must give the archive's schema identity.
219
- - Older v2 and all v1 archives do not record them; the latest prefix of the
220
- applied migrations that gives the archive's schema identity is used. Migrations
221
- that change no archived table (indexes, excluded tables) cannot be told apart
222
- there and are reported as later.
268
+ - Older archives that still carry their source's migration ledger as a table
269
+ (`rer_migrations` and the other ledgers above, from before they became
270
+ bookkeeping) are placed by it. Every name it records must be applied here,
271
+ with the same file hash (the destination ledger's, else the replayed file's),
272
+ or the import is refused with `schema_mismatch:migration_history:<migration>`
273
+ naming the first that differs; replaying exactly those must give the archive's
274
+ schema identity. Every other applied migration is later, wherever the
275
+ destination's history orders it. `importArchive` reads the ledger rows from
276
+ the verified file and passes them to `imports.create(manifest, actor,
277
+ {ledgers})`; staging refuses any part whose ledger rows differ from them.
278
+ - Other older v2 and all v1 archives do not record them; the latest prefix of
279
+ the applied migrations that gives the archive's schema identity is used.
280
+ Migrations that change no archived table (indexes, excluded tables) cannot be
281
+ told apart there and are reported as later.
223
282
 
224
283
  Restored rows never pass through the later migrations, so each of them may
225
284
  contain only `CREATE TABLE` (not `AS SELECT`), `CREATE INDEX` and
@@ -227,7 +286,11 @@ contain only `CREATE TABLE` (not `AS SELECT`), `CREATE INDEX` and
227
286
  restorable tables. A later migration with any other statement that names a
228
287
  restorable table, such as an `INSERT`, `UPDATE` or `DELETE` that purges or
229
288
  backfills rows or a trigger, is refused with `schema_mismatch`; statements that
230
- touch only excluded tables are allowed. For archives without a migration record,
289
+ touch only excluded tables are allowed. An archived table the archive holds no
290
+ rows of restores empty, so an `UPDATE` or `DELETE` of it changes nothing and is
291
+ allowed unless it also names a table the archive holds rows of (which it could
292
+ read into its writes); an `INSERT` into it, or any other statement naming it,
293
+ is still refused. For archives without a migration record,
231
294
  every migration after the earliest prefix that gives the archive's schema is
232
295
  checked, since the archive may predate any of them.
233
296
 
@@ -496,11 +559,13 @@ modules/portability/provider-data.mjs
496
559
  modules/portability/generated.mjs
497
560
  modules/portability/restore-safety.mjs
498
561
  modules/portability/bridge.mjs
562
+ modules/portability/ledgers.mjs
499
563
  modules/portability/migration-statements.mjs
500
564
  modules/portability/legacy-safety.mjs
501
565
  modules/portability/export.mjs
502
566
  modules/portability/import.mjs
503
567
  modules/portability/import-staging.mjs
568
+ modules/portability/import-safety.mjs
504
569
  modules/portability/payloads.mjs
505
570
  modules/portability/journal.mjs
506
571
  modules/portability/http.mjs