@forzalabs/remora 1.7.0 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -1
- package/index.js +92 -10
- package/json_schemas/producer-schema.json +8 -0
- package/package.json +1 -1
- package/workers/ExecutorWorker.js +92 -10
package/CHANGELOG.md
CHANGED
|
@@ -6,12 +6,19 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
|
|
|
6
6
|
|
|
7
7
|
## Unreleased
|
|
8
8
|
|
|
9
|
+
## V 1.8.0 - 2026-08-05
|
|
10
|
+
|
|
9
11
|
### Added
|
|
12
|
+
- Added `synth.min` / `synth.max` on a producer dimension to bound the values `synth` generates. Both ends are inclusive and either can be given alone, in which case the other spans the default range's width from it. On a `number` dimension they replace the name-based generator, so a column whose real-world range the field name can't imply (`"synth": { "min": 18, "max": 90, "decimals": 0 }`) stays inside it instead of getting a generic 0–1000 draw; values stay whole unless `decimals` asks for more, except where the range spans less than one whole number (e.g. `0.4`–`0.6`), which is generated fractionally so it doesn't collapse to a single value. On a `datetime` dimension they are date strings that replace the default 2020–2024 window — written the way the column's own values are, so with the dimension's `format` when it declares one (`"min": "20240101"` for `"YYYYMMDD"`) and as a plain/ISO date otherwise. A date bound that names a day covers that whole day, so `20240101`–`20240131` includes the 31st rather than stopping at its first instant; give a time to end the window at a precise one. Bounds apply only to `number`/`datetime` dimensions and only where the value is actually generated: setting them alongside `allowedValues`, `codeSet` or `references`, or on a key field served from a shared key pool, is rejected with an error naming the dimension rather than being silently ignored, as are an inverted range and a bound of the wrong kind for the dimension's type. `synth` only; no effect on a real producer read
|
|
13
|
+
- Synthetic `datetime` values are now rendered in UTC, the timezone their bounds are parsed in, instead of the host's local timezone. A formatted date (e.g. `"format": "YYYYMMDD"`) generated west of UTC could otherwise render as the previous day — which would also put it outside a declared `synth` window — and the same seed produced different output on machines in different timezones
|
|
10
14
|
- Added a new producer input type for **fixed-width (positional) flat files**: `"fileType": "FIXED"`. Such a file has no delimiter and no header, so each dimension declares where its value sits in the line with `position` — `{ "start": 21, "length": 10 }` or `{ "start": 21, "end": 30 }`, 1-based and inclusive, the way a layout spec is written ("columns 21-30"). Values are trimmed, so space- and zero-padded fields read as their real value (`"0000123.45"` → `123.45`) and an all-padding field reads as `null`; gaps (unmapped filler columns) and overlaps (a composite field plus its subfields) are both allowed; `format` still applies, so a date written as bare digits parses with `"format": "YYYYMMDD"`; and a `sourceFilename` dimension needs no position. Works everywhere a flat file already does — local and S3 sources, wildcard `fileKey` patterns, compression, chunked parallel reads, `sample`, `mock` and `synth`
|
|
11
15
|
- Added `settings.recordSelector` for flat files that interleave record types in one file (a header, many detail records, a trailer). It reads only the lines whose marker at a given position matches — `{ "position": { "start": 1, "length": 1 }, "equals": "D" }`, or a list of values — so one producer describes exactly one record layout and every other line is dropped before parsing. Applied per line, so it is unaffected by how a file is chunked across workers
|
|
12
16
|
|
|
13
17
|
### Fixed
|
|
14
|
-
- Fixed a deployed worker producing no logs. The whole run pipeline logged on the debug channel, which is dropped at `prod` level, so a deployment that did not set `REMORA_DEBUG_MODE=true` emitted nothing at all beyond startup lines and errors. The operational lifecycle events of a run are now logged at `INFO`, so they appear without turning debug on: worker-pool startup, consumer launch/completion, producers skipped as optional or empty, export start/finish per destination, downstream (`run-consumer`) triggers and cycle skips, and CRON/queue scheduler setup, per-trigger execution and stats. Per-line, per-chunk and per-worker detail stays on the debug channel, so `REMORA_DEBUG_MODE=true` is still what you set to see it.
|
|
18
|
+
- Fixed a deployed worker producing no logs. The whole run pipeline logged on the debug channel, which is dropped at `prod` level, so a deployment that did not set `REMORA_DEBUG_MODE=true` emitted nothing at all beyond startup lines and errors. The operational lifecycle events of a run are now logged at `INFO`, so they appear without turning debug on: worker-pool startup, consumer launch/completion, producers skipped as optional or empty, export start/finish per destination, downstream (`run-consumer`) triggers and cycle skips, and CRON/queue scheduler setup, per-trigger execution and stats. Per-line, per-chunk and per-worker detail stays on the debug channel, so `REMORA_DEBUG_MODE=true` is still what you set to see it. These lines follow the log file's existing exclusivity, so a deployment that enables file logging still gets a quiet stdout: `INFO` now goes to the file *instead of* the console (previously it went to both), while `warn`/`error` always stay on the console so a failure can never be silent
|
|
19
|
+
- Fixed the CLI progress bar erasing log lines. The bar redraws in place by moving the cursor up one line, which overwrote whatever had been printed since its last frame — now that the run pipeline logs at `INFO`, that was the log messages themselves. The logger counts what it writes to the console and the renderer only reclaims the previous line when it is still its own bar, so a log line printed mid-run survives and the bar continues below it
|
|
20
|
+
- Fixed the periodic configuration refresh ignoring IAM-role authentication for S3. `Environment.refreshIfNeeded` still fetched the config archive over plain HTTP even for an `s3` URI, so a deployment loading its project from S3 with the ambient IAM role (added in 1.7.0) succeeded at startup and then failed on every refresh. Refresh now takes the same S3 path as the initial load; non-S3 remote URIs keep using the HTTP download with ETag caching
|
|
21
|
+
- API queue execution failures now include the underlying error text in the log message, not just the queue URL, matching the poll-failure logging fixed in 1.7.0
|
|
15
22
|
- Fixed JSON/JSONL producer dimensions ignoring their `format` when casting. The flat-file read path passed the dimension's `format` to the type cast but the JSON path did not, so a date written as a formatted string (e.g. `"20211213"` with `"format": "YYYYMMDD"`) failed to parse with `Invalid time value`. Both paths now cast identically
|
|
16
23
|
- Fixed the `sample` command emitting the header row of a CSV/Excel producer as if it were a data row when `hasHeaderRow` was not set explicitly. Deciding where the data starts is now one shared rule for the run path and the sample path, so the two can no longer disagree
|
|
17
24
|
- Fixed a multi-file (wildcard `fileKey`) read of a headerless flat file failing with a "different header line than the other files" error: the first line of each file was compared as a header even for file types that don't have one, where it is data and legitimately differs
|
package/index.js
CHANGED
|
@@ -11256,7 +11256,7 @@ var CONSTANTS, Constants_default;
|
|
|
11256
11256
|
var init_Constants = __esm({
|
|
11257
11257
|
"../../packages/constants/src/Constants.ts"() {
|
|
11258
11258
|
CONSTANTS = {
|
|
11259
|
-
cliVersion: "1.
|
|
11259
|
+
cliVersion: "1.8.0",
|
|
11260
11260
|
backendVersion: 1,
|
|
11261
11261
|
backendPort: 5088,
|
|
11262
11262
|
workerVersion: 2,
|
|
@@ -45240,17 +45240,24 @@ var init_BuiltinCodeSets = __esm({
|
|
|
45240
45240
|
});
|
|
45241
45241
|
|
|
45242
45242
|
// ../../packages/engines/src/mock/MockDataFactory.ts
|
|
45243
|
-
var import_dayjs3, DATE_FROM, DATE_TO, MAX_SEED, MockKeyRegistry, PAYERS, PLAN_SUFFIXES, ORG_SUFFIXES, MBI_LETTERS, pickCh, mbi, npi, icd10, oid, HEALTHCARE_STRING_TOPICS, HEALTHCARE_NUMBER_TOPICS, STRING_TOPICS, NUMBER_TOPICS, MockDataFactoryClass, MockDataFactory, MockDataFactory_default;
|
|
45243
|
+
var import_dayjs3, import_utc2, DATE_FROM, DATE_TO, DATE_FROM_MS, DATE_TO_MS, DATE_SPAN_MS, MAX_SEED, DEFAULT_SPAN, DEFAULT_FRACTION_DIGITS, MockKeyRegistry, PAYERS, PLAN_SUFFIXES, ORG_SUFFIXES, MBI_LETTERS, pickCh, mbi, npi, icd10, oid, HEALTHCARE_STRING_TOPICS, HEALTHCARE_NUMBER_TOPICS, STRING_TOPICS, NUMBER_TOPICS, MockDataFactoryClass, MockDataFactory, MockDataFactory_default;
|
|
45244
45244
|
var init_MockDataFactory = __esm({
|
|
45245
45245
|
"../../packages/engines/src/mock/MockDataFactory.ts"() {
|
|
45246
45246
|
import_dayjs3 = __toESM(require("dayjs"), 1);
|
|
45247
|
+
import_utc2 = __toESM(require("dayjs/plugin/utc"), 1);
|
|
45247
45248
|
init_dist();
|
|
45248
45249
|
init_src();
|
|
45249
45250
|
init_RandomEngine();
|
|
45250
45251
|
init_BuiltinCodeSets();
|
|
45252
|
+
import_dayjs3.default.extend(import_utc2.default);
|
|
45251
45253
|
DATE_FROM = "2020-01-01T00:00:00.000Z";
|
|
45252
45254
|
DATE_TO = "2024-12-31T23:59:59.999Z";
|
|
45255
|
+
DATE_FROM_MS = new Date(DATE_FROM).valueOf();
|
|
45256
|
+
DATE_TO_MS = new Date(DATE_TO).valueOf();
|
|
45257
|
+
DATE_SPAN_MS = DATE_TO_MS - DATE_FROM_MS;
|
|
45253
45258
|
MAX_SEED = 2147483647;
|
|
45259
|
+
DEFAULT_SPAN = 1e3;
|
|
45260
|
+
DEFAULT_FRACTION_DIGITS = 4;
|
|
45254
45261
|
MockKeyRegistry = class {
|
|
45255
45262
|
constructor() {
|
|
45256
45263
|
this.pools = /* @__PURE__ */ new Map();
|
|
@@ -45416,18 +45423,44 @@ var init_MockDataFactory = __esm({
|
|
|
45416
45423
|
return spec.decimals === void 0 ? value : this._round(value, spec.decimals);
|
|
45417
45424
|
};
|
|
45418
45425
|
this._rawNumber = (spec) => {
|
|
45419
|
-
if (spec.min !== void 0 || spec.max !== void 0)
|
|
45420
|
-
return r6.number.int({ min: Math.round(spec.min ?? 0), max: Math.round(spec.max ?? 1e3) });
|
|
45426
|
+
if (spec.min !== void 0 || spec.max !== void 0) return this._bounded(spec);
|
|
45421
45427
|
const topic = this._match(NUMBER_TOPICS, spec.name);
|
|
45422
|
-
return topic ? topic.gen() : r6.number.int({ min: 0, max:
|
|
45428
|
+
return topic ? topic.gen() : r6.number.int({ min: 0, max: DEFAULT_SPAN });
|
|
45429
|
+
};
|
|
45430
|
+
/**
|
|
45431
|
+
* A value inside the declared bounds, both ends inclusive. An open end spans the default width from
|
|
45432
|
+
* the bound that was given — never the given bound itself, which would collapse the range to one
|
|
45433
|
+
* value. An open lower end still starts at 0 where that leaves a usable range, since a column
|
|
45434
|
+
* capped at (say) 12 is far more likely to mean 0-12 than to want negatives. Whole numbers unless
|
|
45435
|
+
* the dimension asks for decimals — or the range spans less than one whole number (e.g. 0.4-0.6),
|
|
45436
|
+
* where an integer draw would collapse to a single value or an empty range.
|
|
45437
|
+
*/
|
|
45438
|
+
this._bounded = (spec) => {
|
|
45439
|
+
const min = spec.min ?? (spec.max >= 0 ? 0 : spec.max - DEFAULT_SPAN);
|
|
45440
|
+
const max = spec.max ?? min + DEFAULT_SPAN;
|
|
45441
|
+
const low = Math.ceil(min);
|
|
45442
|
+
const high = Math.floor(max);
|
|
45443
|
+
if (spec.decimals || low >= high)
|
|
45444
|
+
return r6.number.float({ min, max, fractionDigits: spec.decimals ?? DEFAULT_FRACTION_DIGITS });
|
|
45445
|
+
return r6.number.int({ min: low, max: high });
|
|
45423
45446
|
};
|
|
45424
45447
|
this._round = (value, decimals) => {
|
|
45425
45448
|
const factor = Math.pow(10, decimals);
|
|
45426
45449
|
return Math.round(value * factor) / factor;
|
|
45427
45450
|
};
|
|
45428
45451
|
this._datetime = (spec) => {
|
|
45429
|
-
const date2 = r6.date.between(
|
|
45430
|
-
return spec.format ?
|
|
45452
|
+
const date2 = r6.date.between(this._dateRange(spec));
|
|
45453
|
+
return spec.format ? import_dayjs3.default.utc(date2).format(spec.format) : date2.toISOString();
|
|
45454
|
+
};
|
|
45455
|
+
/**
|
|
45456
|
+
* The window a datetime is drawn from: the default realistic window, moved/narrowed by whichever
|
|
45457
|
+
* bounds the dimension declares. A one-sided bound keeps the default window's width, so a `min`
|
|
45458
|
+
* past the default window still yields a spread of dates rather than an inverted or single-day range.
|
|
45459
|
+
*/
|
|
45460
|
+
this._dateRange = (spec) => {
|
|
45461
|
+
if (spec.min === void 0 && spec.max === void 0) return { from: DATE_FROM_MS, to: DATE_TO_MS };
|
|
45462
|
+
const from = spec.min ?? spec.max - DATE_SPAN_MS;
|
|
45463
|
+
return { from, to: spec.max ?? from + DATE_SPAN_MS };
|
|
45431
45464
|
};
|
|
45432
45465
|
// First topic whose key is a substring of the lowercased field name.
|
|
45433
45466
|
this._match = (topics, name) => {
|
|
@@ -45443,18 +45476,23 @@ var init_MockDataFactory = __esm({
|
|
|
45443
45476
|
});
|
|
45444
45477
|
|
|
45445
45478
|
// ../../packages/engines/src/mock/SyntheticInputEngine.ts
|
|
45446
|
-
var import_promises12, import_os, import_path18, POOL_CAP, SyntheticInputEngineClass, SyntheticInputEngine, SyntheticInputEngine_default;
|
|
45479
|
+
var import_promises12, import_os, import_path18, import_dayjs4, import_customParseFormat3, import_utc3, POOL_CAP, SyntheticInputEngineClass, SyntheticInputEngine, SyntheticInputEngine_default;
|
|
45447
45480
|
var init_SyntheticInputEngine = __esm({
|
|
45448
45481
|
"../../packages/engines/src/mock/SyntheticInputEngine.ts"() {
|
|
45449
45482
|
import_promises12 = __toESM(require("fs/promises"), 1);
|
|
45450
45483
|
import_os = __toESM(require("os"), 1);
|
|
45451
45484
|
import_path18 = __toESM(require("path"), 1);
|
|
45485
|
+
import_dayjs4 = __toESM(require("dayjs"), 1);
|
|
45486
|
+
import_customParseFormat3 = __toESM(require("dayjs/plugin/customParseFormat"), 1);
|
|
45487
|
+
import_utc3 = __toESM(require("dayjs/plugin/utc"), 1);
|
|
45452
45488
|
init_src();
|
|
45453
45489
|
init_src2();
|
|
45454
45490
|
init_src4();
|
|
45455
45491
|
init_MockDataFactory();
|
|
45456
45492
|
init_BuiltinCodeSets();
|
|
45457
45493
|
init_RandomEngine();
|
|
45494
|
+
import_dayjs4.default.extend(import_customParseFormat3.default);
|
|
45495
|
+
import_dayjs4.default.extend(import_utc3.default);
|
|
45458
45496
|
POOL_CAP = 1e5;
|
|
45459
45497
|
SyntheticInputEngineClass = class {
|
|
45460
45498
|
constructor() {
|
|
@@ -45579,19 +45617,63 @@ var init_SyntheticInputEngine = __esm({
|
|
|
45579
45617
|
const { enumValues, codeSet } = this._resolveCodeSet(dim, name, codeSetMap);
|
|
45580
45618
|
const reference = synth2.references ? { producer: synth2.references.producer, field: this._outputName(this._parentDim(synth2.references, producerMap)) } : void 0;
|
|
45581
45619
|
const isValue = enumValues !== void 0 || codeSet !== void 0 || reference !== void 0;
|
|
45620
|
+
const isKey = !isValue && (dim.pk === true || this._isKeyName(name));
|
|
45621
|
+
const refOwner = refTargets.has(this._refKey(producerName, name));
|
|
45622
|
+
const { min, max } = this._resolveBounds(dim, !isValue && !isKey && !refOwner);
|
|
45582
45623
|
return {
|
|
45583
45624
|
name,
|
|
45584
45625
|
type: dim.type ?? "string",
|
|
45585
45626
|
format: dim.format,
|
|
45586
45627
|
decimals: synth2.decimals,
|
|
45628
|
+
min,
|
|
45629
|
+
max,
|
|
45587
45630
|
isPk: dim.pk === true,
|
|
45588
|
-
isKey
|
|
45631
|
+
isKey,
|
|
45589
45632
|
enumValues,
|
|
45590
45633
|
codeSet,
|
|
45591
45634
|
reference,
|
|
45592
|
-
refOwner
|
|
45635
|
+
refOwner
|
|
45593
45636
|
};
|
|
45594
45637
|
};
|
|
45638
|
+
// Bounds for a generated value — a numeric range for a `number`, a date window for a `datetime` —
|
|
45639
|
+
// which replace the name-based topic generator. Silently dropping a declared range would emit
|
|
45640
|
+
// out-of-range data, so a dimension whose value comes from somewhere else (an enum/code set, a
|
|
45641
|
+
// foreign key, or a coordinated key pool) is rejected instead.
|
|
45642
|
+
this._resolveBounds = (dim, isGenerated) => {
|
|
45643
|
+
const { min, max } = this._synth(dim);
|
|
45644
|
+
if (min === void 0 && max === void 0) return {};
|
|
45645
|
+
const type = dim.type ?? "string";
|
|
45646
|
+
Affirm_default(type === "number" || type === "datetime", `Dimension "${dim.name}" sets synth "min"/"max", which only applies to "number" and "datetime" dimensions (its type is "${type}")`);
|
|
45647
|
+
Affirm_default(isGenerated, `Dimension "${dim.name}" cannot combine synth "min"/"max" with a value drawn from elsewhere ("allowedValues", "codeSet", "references", or a key field served from a shared key pool)`);
|
|
45648
|
+
const bounds = { min: this._bound(dim, min, "min"), max: this._bound(dim, max, "max") };
|
|
45649
|
+
if (typeof max === "string" && this._isDayGranular(max, dim.format))
|
|
45650
|
+
bounds.max = import_dayjs4.default.utc(bounds.max).endOf("day").valueOf();
|
|
45651
|
+
Affirm_default(bounds.min === void 0 || bounds.max === void 0 || bounds.min <= bounds.max, `Dimension "${dim.name}" has synth "min" (${min}) ${type === "datetime" ? "later than" : "greater than"} "max" (${max})`);
|
|
45652
|
+
return bounds;
|
|
45653
|
+
};
|
|
45654
|
+
// True when a date bound names a day rather than an instant — neither the value nor the dimension's
|
|
45655
|
+
// format carries a time. Both have to agree: the value alone can't tell `"20240131"` from a compact
|
|
45656
|
+
// `YYYYMMDDHHmmss` instant, and the format alone can't tell that an ISO value overrode it.
|
|
45657
|
+
this._isDayGranular = (value, format3) => !/[T ]\d{1,2}:/.test(value) && !/[HhmsSAa]/.test(format3 ?? "");
|
|
45658
|
+
/**
|
|
45659
|
+
* One bound in the units the generator works in: the number itself, or epoch milliseconds for a
|
|
45660
|
+
* `datetime`. A date bound is read with the dimension's own `format` first, so it can be written the
|
|
45661
|
+
* same way the column's values are (`"20240101"` for `"YYYYMMDD"`), and as a plain/ISO date
|
|
45662
|
+
* otherwise. Parsed in UTC like the real cast path, so the window never shifts with the host's
|
|
45663
|
+
* timezone and generation stays reproducible.
|
|
45664
|
+
*/
|
|
45665
|
+
this._bound = (dim, value, prop) => {
|
|
45666
|
+
if (value === void 0) return void 0;
|
|
45667
|
+
if (dim.type === "number") {
|
|
45668
|
+
Affirm_default(typeof value === "number", `Dimension "${dim.name}" synth "${prop}" must be a number on a "number" dimension (got ${JSON.stringify(value)})`);
|
|
45669
|
+
return value;
|
|
45670
|
+
}
|
|
45671
|
+
Affirm_default(typeof value === "string", `Dimension "${dim.name}" synth "${prop}" must be a date string on a "datetime" dimension (got ${JSON.stringify(value)})`);
|
|
45672
|
+
const formatted = dim.format ? import_dayjs4.default.utc(value, dim.format, true) : void 0;
|
|
45673
|
+
const date2 = formatted?.isValid() ? formatted : import_dayjs4.default.utc(value);
|
|
45674
|
+
Affirm_default(date2.isValid(), `Dimension "${dim.name}" synth "${prop}" ("${value}") is not a valid date${dim.format ? ` for format "${dim.format}" or as an ISO date` : ""}`);
|
|
45675
|
+
return date2.valueOf();
|
|
45676
|
+
};
|
|
45595
45677
|
// Resolve a dimension into a flat enum or a structured binding. Explicit `synth.allowedValues` /
|
|
45596
45678
|
// `synth.codeSet` always win; otherwise the field name is matched against the built-in code sets.
|
|
45597
45679
|
// Fails fast (affirm) on unknown sets or shape/field mismatches so bad config never silently emits
|
|
@@ -122,6 +122,14 @@
|
|
|
122
122
|
"minimum": 0,
|
|
123
123
|
"description": "How many decimal places generated number values should have: 0 forces whole integers (so values load into an integer column); a positive value rounds to that many decimals. Omit to keep the generator's own value."
|
|
124
124
|
},
|
|
125
|
+
"min": {
|
|
126
|
+
"type": ["number", "string"],
|
|
127
|
+
"description": "Inclusive lower bound for generated values, so the column stays inside its real-world range. A number for a 'number' dimension; a date string for a 'datetime' dimension (read with the dimension's own 'format' when it has one, ISO otherwise). Only for 'number' and 'datetime' dimensions, and mutually exclusive with 'allowedValues', 'codeSet' and 'references'."
|
|
128
|
+
},
|
|
129
|
+
"max": {
|
|
130
|
+
"type": ["number", "string"],
|
|
131
|
+
"description": "Inclusive upper bound for generated values, so the column stays inside its real-world range. A number for a 'number' dimension; a date string for a 'datetime' dimension (read with the dimension's own 'format' when it has one, ISO otherwise). Only for 'number' and 'datetime' dimensions, and mutually exclusive with 'allowedValues', 'codeSet' and 'references'."
|
|
132
|
+
},
|
|
125
133
|
"allowedValues": {
|
|
126
134
|
"type": "array",
|
|
127
135
|
"minItems": 1,
|
package/package.json
CHANGED
|
@@ -11255,7 +11255,7 @@ var CONSTANTS, Constants_default;
|
|
|
11255
11255
|
var init_Constants = __esm({
|
|
11256
11256
|
"../../packages/constants/src/Constants.ts"() {
|
|
11257
11257
|
CONSTANTS = {
|
|
11258
|
-
cliVersion: "1.
|
|
11258
|
+
cliVersion: "1.8.0",
|
|
11259
11259
|
backendVersion: 1,
|
|
11260
11260
|
backendPort: 5088,
|
|
11261
11261
|
workerVersion: 2,
|
|
@@ -44947,17 +44947,24 @@ var init_BuiltinCodeSets = __esm({
|
|
|
44947
44947
|
});
|
|
44948
44948
|
|
|
44949
44949
|
// ../../packages/engines/src/mock/MockDataFactory.ts
|
|
44950
|
-
var import_dayjs3, DATE_FROM, DATE_TO, MAX_SEED, MockKeyRegistry, PAYERS, PLAN_SUFFIXES, ORG_SUFFIXES, MBI_LETTERS, pickCh, mbi, npi, icd10, oid, HEALTHCARE_STRING_TOPICS, HEALTHCARE_NUMBER_TOPICS, STRING_TOPICS, NUMBER_TOPICS, MockDataFactoryClass, MockDataFactory, MockDataFactory_default;
|
|
44950
|
+
var import_dayjs3, import_utc2, DATE_FROM, DATE_TO, DATE_FROM_MS, DATE_TO_MS, DATE_SPAN_MS, MAX_SEED, DEFAULT_SPAN, DEFAULT_FRACTION_DIGITS, MockKeyRegistry, PAYERS, PLAN_SUFFIXES, ORG_SUFFIXES, MBI_LETTERS, pickCh, mbi, npi, icd10, oid, HEALTHCARE_STRING_TOPICS, HEALTHCARE_NUMBER_TOPICS, STRING_TOPICS, NUMBER_TOPICS, MockDataFactoryClass, MockDataFactory, MockDataFactory_default;
|
|
44951
44951
|
var init_MockDataFactory = __esm({
|
|
44952
44952
|
"../../packages/engines/src/mock/MockDataFactory.ts"() {
|
|
44953
44953
|
import_dayjs3 = __toESM(require("dayjs"), 1);
|
|
44954
|
+
import_utc2 = __toESM(require("dayjs/plugin/utc"), 1);
|
|
44954
44955
|
init_dist();
|
|
44955
44956
|
init_src();
|
|
44956
44957
|
init_RandomEngine();
|
|
44957
44958
|
init_BuiltinCodeSets();
|
|
44959
|
+
import_dayjs3.default.extend(import_utc2.default);
|
|
44958
44960
|
DATE_FROM = "2020-01-01T00:00:00.000Z";
|
|
44959
44961
|
DATE_TO = "2024-12-31T23:59:59.999Z";
|
|
44962
|
+
DATE_FROM_MS = new Date(DATE_FROM).valueOf();
|
|
44963
|
+
DATE_TO_MS = new Date(DATE_TO).valueOf();
|
|
44964
|
+
DATE_SPAN_MS = DATE_TO_MS - DATE_FROM_MS;
|
|
44960
44965
|
MAX_SEED = 2147483647;
|
|
44966
|
+
DEFAULT_SPAN = 1e3;
|
|
44967
|
+
DEFAULT_FRACTION_DIGITS = 4;
|
|
44961
44968
|
MockKeyRegistry = class {
|
|
44962
44969
|
constructor() {
|
|
44963
44970
|
this.pools = /* @__PURE__ */ new Map();
|
|
@@ -45123,18 +45130,44 @@ var init_MockDataFactory = __esm({
|
|
|
45123
45130
|
return spec.decimals === void 0 ? value : this._round(value, spec.decimals);
|
|
45124
45131
|
};
|
|
45125
45132
|
this._rawNumber = (spec) => {
|
|
45126
|
-
if (spec.min !== void 0 || spec.max !== void 0)
|
|
45127
|
-
return r6.number.int({ min: Math.round(spec.min ?? 0), max: Math.round(spec.max ?? 1e3) });
|
|
45133
|
+
if (spec.min !== void 0 || spec.max !== void 0) return this._bounded(spec);
|
|
45128
45134
|
const topic = this._match(NUMBER_TOPICS, spec.name);
|
|
45129
|
-
return topic ? topic.gen() : r6.number.int({ min: 0, max:
|
|
45135
|
+
return topic ? topic.gen() : r6.number.int({ min: 0, max: DEFAULT_SPAN });
|
|
45136
|
+
};
|
|
45137
|
+
/**
|
|
45138
|
+
* A value inside the declared bounds, both ends inclusive. An open end spans the default width from
|
|
45139
|
+
* the bound that was given — never the given bound itself, which would collapse the range to one
|
|
45140
|
+
* value. An open lower end still starts at 0 where that leaves a usable range, since a column
|
|
45141
|
+
* capped at (say) 12 is far more likely to mean 0-12 than to want negatives. Whole numbers unless
|
|
45142
|
+
* the dimension asks for decimals — or the range spans less than one whole number (e.g. 0.4-0.6),
|
|
45143
|
+
* where an integer draw would collapse to a single value or an empty range.
|
|
45144
|
+
*/
|
|
45145
|
+
this._bounded = (spec) => {
|
|
45146
|
+
const min = spec.min ?? (spec.max >= 0 ? 0 : spec.max - DEFAULT_SPAN);
|
|
45147
|
+
const max = spec.max ?? min + DEFAULT_SPAN;
|
|
45148
|
+
const low = Math.ceil(min);
|
|
45149
|
+
const high = Math.floor(max);
|
|
45150
|
+
if (spec.decimals || low >= high)
|
|
45151
|
+
return r6.number.float({ min, max, fractionDigits: spec.decimals ?? DEFAULT_FRACTION_DIGITS });
|
|
45152
|
+
return r6.number.int({ min: low, max: high });
|
|
45130
45153
|
};
|
|
45131
45154
|
this._round = (value, decimals) => {
|
|
45132
45155
|
const factor = Math.pow(10, decimals);
|
|
45133
45156
|
return Math.round(value * factor) / factor;
|
|
45134
45157
|
};
|
|
45135
45158
|
this._datetime = (spec) => {
|
|
45136
|
-
const date2 = r6.date.between(
|
|
45137
|
-
return spec.format ?
|
|
45159
|
+
const date2 = r6.date.between(this._dateRange(spec));
|
|
45160
|
+
return spec.format ? import_dayjs3.default.utc(date2).format(spec.format) : date2.toISOString();
|
|
45161
|
+
};
|
|
45162
|
+
/**
|
|
45163
|
+
* The window a datetime is drawn from: the default realistic window, moved/narrowed by whichever
|
|
45164
|
+
* bounds the dimension declares. A one-sided bound keeps the default window's width, so a `min`
|
|
45165
|
+
* past the default window still yields a spread of dates rather than an inverted or single-day range.
|
|
45166
|
+
*/
|
|
45167
|
+
this._dateRange = (spec) => {
|
|
45168
|
+
if (spec.min === void 0 && spec.max === void 0) return { from: DATE_FROM_MS, to: DATE_TO_MS };
|
|
45169
|
+
const from = spec.min ?? spec.max - DATE_SPAN_MS;
|
|
45170
|
+
return { from, to: spec.max ?? from + DATE_SPAN_MS };
|
|
45138
45171
|
};
|
|
45139
45172
|
// First topic whose key is a substring of the lowercased field name.
|
|
45140
45173
|
this._match = (topics, name) => {
|
|
@@ -45150,18 +45183,23 @@ var init_MockDataFactory = __esm({
|
|
|
45150
45183
|
});
|
|
45151
45184
|
|
|
45152
45185
|
// ../../packages/engines/src/mock/SyntheticInputEngine.ts
|
|
45153
|
-
var import_promises12, import_os, import_path15, POOL_CAP, SyntheticInputEngineClass, SyntheticInputEngine;
|
|
45186
|
+
var import_promises12, import_os, import_path15, import_dayjs4, import_customParseFormat3, import_utc3, POOL_CAP, SyntheticInputEngineClass, SyntheticInputEngine;
|
|
45154
45187
|
var init_SyntheticInputEngine = __esm({
|
|
45155
45188
|
"../../packages/engines/src/mock/SyntheticInputEngine.ts"() {
|
|
45156
45189
|
import_promises12 = __toESM(require("fs/promises"), 1);
|
|
45157
45190
|
import_os = __toESM(require("os"), 1);
|
|
45158
45191
|
import_path15 = __toESM(require("path"), 1);
|
|
45192
|
+
import_dayjs4 = __toESM(require("dayjs"), 1);
|
|
45193
|
+
import_customParseFormat3 = __toESM(require("dayjs/plugin/customParseFormat"), 1);
|
|
45194
|
+
import_utc3 = __toESM(require("dayjs/plugin/utc"), 1);
|
|
45159
45195
|
init_src();
|
|
45160
45196
|
init_src2();
|
|
45161
45197
|
init_src4();
|
|
45162
45198
|
init_MockDataFactory();
|
|
45163
45199
|
init_BuiltinCodeSets();
|
|
45164
45200
|
init_RandomEngine();
|
|
45201
|
+
import_dayjs4.default.extend(import_customParseFormat3.default);
|
|
45202
|
+
import_dayjs4.default.extend(import_utc3.default);
|
|
45165
45203
|
POOL_CAP = 1e5;
|
|
45166
45204
|
SyntheticInputEngineClass = class {
|
|
45167
45205
|
constructor() {
|
|
@@ -45286,19 +45324,63 @@ var init_SyntheticInputEngine = __esm({
|
|
|
45286
45324
|
const { enumValues, codeSet } = this._resolveCodeSet(dim, name, codeSetMap);
|
|
45287
45325
|
const reference = synth.references ? { producer: synth.references.producer, field: this._outputName(this._parentDim(synth.references, producerMap)) } : void 0;
|
|
45288
45326
|
const isValue = enumValues !== void 0 || codeSet !== void 0 || reference !== void 0;
|
|
45327
|
+
const isKey = !isValue && (dim.pk === true || this._isKeyName(name));
|
|
45328
|
+
const refOwner = refTargets.has(this._refKey(producerName, name));
|
|
45329
|
+
const { min, max } = this._resolveBounds(dim, !isValue && !isKey && !refOwner);
|
|
45289
45330
|
return {
|
|
45290
45331
|
name,
|
|
45291
45332
|
type: dim.type ?? "string",
|
|
45292
45333
|
format: dim.format,
|
|
45293
45334
|
decimals: synth.decimals,
|
|
45335
|
+
min,
|
|
45336
|
+
max,
|
|
45294
45337
|
isPk: dim.pk === true,
|
|
45295
|
-
isKey
|
|
45338
|
+
isKey,
|
|
45296
45339
|
enumValues,
|
|
45297
45340
|
codeSet,
|
|
45298
45341
|
reference,
|
|
45299
|
-
refOwner
|
|
45342
|
+
refOwner
|
|
45300
45343
|
};
|
|
45301
45344
|
};
|
|
45345
|
+
// Bounds for a generated value — a numeric range for a `number`, a date window for a `datetime` —
|
|
45346
|
+
// which replace the name-based topic generator. Silently dropping a declared range would emit
|
|
45347
|
+
// out-of-range data, so a dimension whose value comes from somewhere else (an enum/code set, a
|
|
45348
|
+
// foreign key, or a coordinated key pool) is rejected instead.
|
|
45349
|
+
this._resolveBounds = (dim, isGenerated) => {
|
|
45350
|
+
const { min, max } = this._synth(dim);
|
|
45351
|
+
if (min === void 0 && max === void 0) return {};
|
|
45352
|
+
const type = dim.type ?? "string";
|
|
45353
|
+
Affirm_default(type === "number" || type === "datetime", `Dimension "${dim.name}" sets synth "min"/"max", which only applies to "number" and "datetime" dimensions (its type is "${type}")`);
|
|
45354
|
+
Affirm_default(isGenerated, `Dimension "${dim.name}" cannot combine synth "min"/"max" with a value drawn from elsewhere ("allowedValues", "codeSet", "references", or a key field served from a shared key pool)`);
|
|
45355
|
+
const bounds = { min: this._bound(dim, min, "min"), max: this._bound(dim, max, "max") };
|
|
45356
|
+
if (typeof max === "string" && this._isDayGranular(max, dim.format))
|
|
45357
|
+
bounds.max = import_dayjs4.default.utc(bounds.max).endOf("day").valueOf();
|
|
45358
|
+
Affirm_default(bounds.min === void 0 || bounds.max === void 0 || bounds.min <= bounds.max, `Dimension "${dim.name}" has synth "min" (${min}) ${type === "datetime" ? "later than" : "greater than"} "max" (${max})`);
|
|
45359
|
+
return bounds;
|
|
45360
|
+
};
|
|
45361
|
+
// True when a date bound names a day rather than an instant — neither the value nor the dimension's
|
|
45362
|
+
// format carries a time. Both have to agree: the value alone can't tell `"20240131"` from a compact
|
|
45363
|
+
// `YYYYMMDDHHmmss` instant, and the format alone can't tell that an ISO value overrode it.
|
|
45364
|
+
this._isDayGranular = (value, format3) => !/[T ]\d{1,2}:/.test(value) && !/[HhmsSAa]/.test(format3 ?? "");
|
|
45365
|
+
/**
|
|
45366
|
+
* One bound in the units the generator works in: the number itself, or epoch milliseconds for a
|
|
45367
|
+
* `datetime`. A date bound is read with the dimension's own `format` first, so it can be written the
|
|
45368
|
+
* same way the column's values are (`"20240101"` for `"YYYYMMDD"`), and as a plain/ISO date
|
|
45369
|
+
* otherwise. Parsed in UTC like the real cast path, so the window never shifts with the host's
|
|
45370
|
+
* timezone and generation stays reproducible.
|
|
45371
|
+
*/
|
|
45372
|
+
this._bound = (dim, value, prop) => {
|
|
45373
|
+
if (value === void 0) return void 0;
|
|
45374
|
+
if (dim.type === "number") {
|
|
45375
|
+
Affirm_default(typeof value === "number", `Dimension "${dim.name}" synth "${prop}" must be a number on a "number" dimension (got ${JSON.stringify(value)})`);
|
|
45376
|
+
return value;
|
|
45377
|
+
}
|
|
45378
|
+
Affirm_default(typeof value === "string", `Dimension "${dim.name}" synth "${prop}" must be a date string on a "datetime" dimension (got ${JSON.stringify(value)})`);
|
|
45379
|
+
const formatted = dim.format ? import_dayjs4.default.utc(value, dim.format, true) : void 0;
|
|
45380
|
+
const date2 = formatted?.isValid() ? formatted : import_dayjs4.default.utc(value);
|
|
45381
|
+
Affirm_default(date2.isValid(), `Dimension "${dim.name}" synth "${prop}" ("${value}") is not a valid date${dim.format ? ` for format "${dim.format}" or as an ISO date` : ""}`);
|
|
45382
|
+
return date2.valueOf();
|
|
45383
|
+
};
|
|
45302
45384
|
// Resolve a dimension into a flat enum or a structured binding. Explicit `synth.allowedValues` /
|
|
45303
45385
|
// `synth.codeSet` always win; otherwise the field name is matched against the built-in code sets.
|
|
45304
45386
|
// Fails fast (affirm) on unknown sets or shape/field mismatches so bad config never silently emits
|