bulltrackers-module 1.0.1102 → 1.0.1104
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/functions/computation-system-v3/framework/adapters/StorageManager.js +10 -7
- package/functions/computation-system-v3/framework/core/ContextBuilder.js +24 -4
- package/functions/computation-system-v3/framework/core/ExecutionPipeline.js +7 -0
- package/functions/computation-system-v3/framework/core/Manifest.js +6 -0
- package/functions/computation-system-v3/framework/tests/unit/SkillsPhase.test.js +53 -0
- package/functions/computation-system-v4/computations/SystemHealth_Smoke_Alerter.js +65 -65
- package/functions/computation-system-v4/config/bulltrackers.config.ts +1 -1
- package/functions/computation-system-v4/core/README.md +252 -0
- package/functions/computation-system-v4/{kernel → core/kernel}/ContextAccumulator.ts +50 -0
- package/functions/computation-system-v4/{kernel → core/kernel}/HookRegistry.ts +40 -6
- package/functions/computation-system-v4/{kernel → core/kernel}/Kernel.ts +35 -4
- package/functions/computation-system-v4/{kernel → core/kernel}/PluginRegistry.ts +284 -278
- package/functions/computation-system-v4/core/kernel/utils/AstCore.ts +259 -0
- package/functions/computation-system-v4/core/kernel/utils/AstValidator.ts +55 -0
- package/functions/computation-system-v4/core/kernel/utils/computationUtils.ts +24 -0
- package/functions/computation-system-v4/{plugins → core/plugins}/context/DefaultContextPlugin.ts +257 -3
- package/functions/computation-system-v4/core/plugins/context/UserLibrarySource.ts +82 -0
- package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/dataHelpers.ts +2 -6
- package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/instruments.ts +3 -7
- package/functions/computation-system-v4/core/plugins/context/libs/metrics.ts +59 -0
- package/functions/computation-system-v4/core/plugins/context/libs/rowUtils.ts +33 -0
- package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/userMetrics.ts +3 -9
- package/functions/computation-system-v4/{plugins → core/plugins}/data/BigQueryDataAdapterPlugin.ts +4 -8
- package/functions/computation-system-v4/{plugins → core/plugins}/data/MaterializedViewManager.ts +111 -111
- package/functions/computation-system-v4/core/plugins/diagnostic/DiagnosticTracerPlugin.ts +139 -0
- package/functions/computation-system-v4/{plugins → core/plugins}/dispatch/DefaultDispatchPlugin.ts +3 -7
- package/functions/computation-system-v4/{plugins → core/plugins}/dispatch/PythonDispatchPlugin.ts +3 -7
- package/functions/computation-system-v4/{plugins → core/plugins}/index.ts +2 -0
- package/functions/computation-system-v4/{plugins → core/plugins}/logging/GcpExecutionLogSinkPlugin.ts +57 -3
- package/functions/computation-system-v4/{plugins → core/plugins}/manifest/AstValidator.ts +58 -107
- package/functions/computation-system-v4/{plugins → core/plugins}/manifest/BundleValidator.ts +23 -64
- package/functions/computation-system-v4/{plugins → core/plugins}/manifest/ConfigResolver.ts +21 -9
- package/functions/computation-system-v4/{plugins → core/plugins}/manifest/DefaultManifestPlugin.ts +190 -25
- package/functions/computation-system-v4/{plugins → core/plugins}/manifest/NpmBundleWorker.ts +1 -1
- package/functions/computation-system-v4/{plugins → core/plugins}/manifest/RecipeExtractor.ts +1 -26
- package/functions/computation-system-v4/{plugins → core/plugins}/planner/DefaultPlannerPlugin.ts +1 -1
- package/functions/computation-system-v4/{plugins → core/plugins}/pubsub/DefaultPubSubPlugin.ts +35 -6
- package/functions/computation-system-v4/{plugins → core/plugins}/runtime/WasmRuntimePlugin.ts +4 -4
- package/functions/computation-system-v4/{plugins → core/plugins}/runtime/wasm/WasmWorker.ts +43 -43
- package/functions/computation-system-v4/{plugins → core/plugins}/runtime/wasm/worker-entry.ts +74 -70
- package/functions/computation-system-v4/{plugins → core/plugins}/scheduler/CloudTasksSchedulerPlugin.ts +4 -6
- package/functions/computation-system-v4/core/plugins/shared/UntrustedMetadataExtractor.ts +62 -0
- package/functions/computation-system-v4/core/plugins/shared/untrusted-metadata-worker.js +199 -0
- package/functions/computation-system-v4/{plugins → core/plugins}/state/FirestoreStatePlugin.ts +148 -8
- package/functions/computation-system-v4/{plugins → core/plugins}/storage/GcsStoragePlugin.ts +91 -1
- package/functions/computation-system-v4/{plugins → core/plugins}/telemetry/StageHarness.ts +113 -113
- package/functions/computation-system-v4/core/plugins/userPlugins/UserPluginRuntimePlugin.ts +283 -0
- package/functions/computation-system-v4/core/plugins/userPlugins/UserPluginSource.ts +61 -0
- package/functions/computation-system-v4/core/plugins/verification/DefaultVerificationPlugin.ts +460 -0
- package/functions/computation-system-v4/{types → core/types}/IHook.ts +4 -0
- package/functions/computation-system-v4/{types → core/types}/IHookContext.ts +16 -0
- package/functions/computation-system-v4/{types → core/types}/IPlugin.ts +3 -1
- package/functions/computation-system-v4/{types → core/types}/IStatePlugin.ts +29 -0
- package/functions/computation-system-v4/{types → core/types}/IStoragePlugin.ts +21 -1
- package/functions/computation-system-v4/core/types/IUserLibrary.ts +59 -0
- package/functions/computation-system-v4/core/types/IUserPlugin.ts +88 -0
- package/functions/computation-system-v4/core/types/IVerificationPlugin.ts +84 -0
- package/functions/computation-system-v4/{types → core/types}/index.ts +35 -0
- package/functions/computation-system-v4/{types → core/types}/shared.ts +28 -1
- package/functions/computation-system-v4/docker/README.md +233 -233
- package/functions/computation-system-v4/docker/docker-compose.yml +82 -82
- package/functions/computation-system-v4/docker/dockerfile +20 -20
- package/functions/computation-system-v4/docker/test-runner.ts +29 -1
- package/functions/computation-system-v4/docs/zero-trust-user-plugins.md +64 -0
- package/functions/computation-system-v4/examples/README.md +21 -0
- package/functions/computation-system-v4/examples/event-driven-pubsub/NOTES.md +5 -0
- package/functions/computation-system-v4/examples/event-driven-pubsub/example.js +22 -0
- package/functions/computation-system-v4/examples/global-computation/NOTES.md +4 -0
- package/functions/computation-system-v4/examples/global-computation/example.js +16 -0
- package/functions/computation-system-v4/examples/per-entity-computation/NOTES.md +5 -0
- package/functions/computation-system-v4/examples/per-entity-computation/example.js +23 -0
- package/functions/computation-system-v4/examples/python-computation/NOTES.md +5 -0
- package/functions/computation-system-v4/examples/python-computation/example.js +21 -0
- package/functions/computation-system-v4/examples/standard-computation/NOTES.md +6 -0
- package/functions/computation-system-v4/examples/standard-computation/example.js +18 -0
- package/functions/computation-system-v4/examples/user-computation-with-library/NOTES.md +28 -0
- package/functions/computation-system-v4/examples/user-computation-with-library/example.js +41 -0
- package/functions/computation-system-v4/examples/user-computation-with-plugin/NOTES.md +6 -0
- package/functions/computation-system-v4/examples/user-computation-with-plugin/example.js +30 -0
- package/functions/computation-system-v4/examples/user-library-wrapper/NOTES.md +22 -0
- package/functions/computation-system-v4/examples/user-library-wrapper/library.js +35 -0
- package/functions/computation-system-v4/examples/user-plugin-pubsub-intent/NOTES.md +7 -0
- package/functions/computation-system-v4/examples/user-plugin-pubsub-intent/binding.d.ts +3 -0
- package/functions/computation-system-v4/examples/user-plugin-pubsub-intent/plugin.js +37 -0
- package/functions/computation-system-v4/jest.config.js +1 -1
- package/functions/computation-system-v4/scripts/generate-lifecycle-graph.ts +126 -126
- package/functions/computation-system-v4/scripts/trace.ts +88 -0
- package/functions/computation-system-v4/tests/ContextAccumulator.tests.ts +2 -2
- package/functions/computation-system-v4/tests/DefaultBillingPlugin.tests.ts +51 -187
- package/functions/computation-system-v4/tests/DefaultContextPlugin.tests.ts +114 -143
- package/functions/computation-system-v4/tests/DefaultManifestPlugin.tests.ts +37 -125
- package/functions/computation-system-v4/tests/DefaultTelemetryPlugin.tests.ts +34 -144
- package/functions/computation-system-v4/tests/HookRegistry.tests.ts +2 -2
- package/functions/computation-system-v4/tests/Kernel.tests.ts +108 -195
- package/functions/computation-system-v4/tests/KernelHooks.tests.ts +19 -55
- package/functions/computation-system-v4/tests/PluginRegistry.tests.ts +42 -138
- package/functions/computation-system-v4/tests/UserLibraryContractResolution.tests.ts +112 -0
- package/functions/computation-system-v4/tests/UserLibraryIsolation.tests.ts +48 -0
- package/functions/computation-system-v4/tests/UserPluginIsolation.tests.ts +41 -0
- package/functions/computation-system-v4/tests/VerificationPlugin.tests.ts +270 -0
- package/functions/computation-system-v4/tests/WasmRuntimePlugin.tests.ts +60 -208
- package/functions/computation-system-v4/tests/helpers.ts +276 -0
- package/functions/computation-system-v4/{plugins/data → tests/plugins}/BigQueryDataAdapterPlugin.tests.ts +1 -1
- package/functions/computation-system-v4/{plugins/scheduler → tests/plugins}/CloudTasksSchedulerPlugin.tests.ts +1 -1
- package/functions/computation-system-v4/{plugins/billing → tests/plugins}/DefaultBillingPlugin.tests.ts +1 -1
- package/functions/computation-system-v4/tests/plugins/DefaultContextPlugin.tests.ts +287 -0
- package/functions/computation-system-v4/{plugins/dispatch → tests/plugins}/DefaultDispatchPlugin.tests.ts +1 -1
- package/functions/computation-system-v4/{plugins/manifest → tests/plugins}/DefaultManifestPlugin.tests.ts +3 -3
- package/functions/computation-system-v4/{plugins/planner → tests/plugins}/DefaultPlannerPlugin.tests.ts +1 -1
- package/functions/computation-system-v4/{plugins/telemetry → tests/plugins}/DefaultTelemetryPlugin.tests.ts +1 -1
- package/functions/computation-system-v4/{plugins/state → tests/plugins}/FirestoreStatePlugin.tests.ts +1 -1
- package/functions/computation-system-v4/{plugins/storage → tests/plugins}/GcsStoragePlugin.tests.ts +1 -1
- package/functions/computation-system-v4/{plugins/runtime → tests/plugins}/WasmRuntimePlugin.tests.ts +1 -1
- package/functions/computation-system-v4/trace_output.log +14 -0
- package/functions/computation-system-v4/trace_proxy.log +189 -0
- package/functions/computation-system-v4/tsconfig.json +6 -5
- package/package.json +1 -1
- package/functions/computation-system-v4/TECHNICALS.MD +0 -524
- package/functions/computation-system-v4/docs/parity-report.md +0 -31
- package/functions/computation-system-v4/docs/v3-v4-executable-parity-proof.md +0 -52
- package/functions/computation-system-v4/examples/computations/NpmDateFnsDemo.js +0 -20
- package/functions/computation-system-v4/examples/computations/NpmMathjsDemo.js +0 -19
- package/functions/computation-system-v4/examples/computations/Python_ExternalTask.js +0 -41
- package/functions/computation-system-v4/examples/computations/Python_HelloWorld.js +0 -30
- package/functions/computation-system-v4/examples/computations/Python_MinimalTask.js +0 -24
- package/functions/computation-system-v4/examples/computations/custom-library-demo.ts +0 -42
- package/functions/computation-system-v4/examples/computations/exploit.js +0 -65
- package/functions/computation-system-v4/examples/computations/exploit_computed_property_call.js +0 -21
- package/functions/computation-system-v4/examples/computations/exploit_constructor_escape.js +0 -21
- package/functions/computation-system-v4/examples/computations/exploit_dynamic_require.js +0 -21
- package/functions/computation-system-v4/examples/computations/exploit_global_access.js +0 -21
- package/functions/computation-system-v4/examples/computations/exploit_npm_constructor.js +0 -20
- package/functions/computation-system-v4/examples/computations/exploit_npm_dynamic_require.js +0 -20
- package/functions/computation-system-v4/examples/computations/exploit_npm_env.js +0 -20
- package/functions/computation-system-v4/examples/computations/exploit_npm_eval.js +0 -20
- package/functions/computation-system-v4/examples/computations/exploit_npm_fetch.js +0 -20
- package/functions/computation-system-v4/examples/computations/exploit_npm_obfuscated_constructor.js +0 -15
- package/functions/computation-system-v4/examples/computations/exploit_npm_postinstall.js +0 -20
- package/functions/computation-system-v4/examples/computations/exploit_npm_unicode_constructor.js +0 -15
- package/functions/computation-system-v4/examples/computations/exploit_timer_access.js +0 -20
- package/functions/computation-system-v4/examples/computations/exploit_webassembly.js +0 -20
- package/functions/computation-system-v4/examples/computations/rate-limited-billing-demo.ts +0 -40
- package/functions/computation-system-v4/examples/computations/telemetry-storage-routing-demo.ts +0 -46
- package/functions/computation-system-v4/examples/event-driven-triggers/event-driven-triggered-computation.js +0 -39
- package/functions/computation-system-v4/examples/local-packages/evil-constructor/index.js +0 -5
- package/functions/computation-system-v4/examples/local-packages/evil-constructor/package.json +0 -7
- package/functions/computation-system-v4/examples/local-packages/evil-dynamic-require/index.js +0 -8
- package/functions/computation-system-v4/examples/local-packages/evil-dynamic-require/package.json +0 -7
- package/functions/computation-system-v4/examples/local-packages/evil-env/index.js +0 -4
- package/functions/computation-system-v4/examples/local-packages/evil-env/package.json +0 -7
- package/functions/computation-system-v4/examples/local-packages/evil-eval/index.js +0 -6
- package/functions/computation-system-v4/examples/local-packages/evil-eval/package.json +0 -7
- package/functions/computation-system-v4/examples/local-packages/evil-fetch/index.js +0 -4
- package/functions/computation-system-v4/examples/local-packages/evil-fetch/package.json +0 -7
- package/functions/computation-system-v4/examples/local-packages/evil-obfuscated-constructor/index.js +0 -7
- package/functions/computation-system-v4/examples/local-packages/evil-obfuscated-constructor/package.json +0 -6
- package/functions/computation-system-v4/examples/local-packages/evil-postinstall/index.js +0 -4
- package/functions/computation-system-v4/examples/local-packages/evil-postinstall/package.json +0 -10
- package/functions/computation-system-v4/examples/local-packages/evil-unicode-constructor/index.js +0 -7
- package/functions/computation-system-v4/examples/local-packages/evil-unicode-constructor/package.json +0 -6
- package/functions/computation-system-v4/examples/plugins/README.md +0 -51
- package/functions/computation-system-v4/examples/plugins/audit-trail-middleware.ts +0 -100
- package/functions/computation-system-v4/examples/plugins/datadog-telemetry.ts +0 -124
- package/functions/computation-system-v4/examples/plugins/postgres-s3-storage.ts +0 -166
- package/functions/computation-system-v4/examples/plugins/rate-limited-billing.ts +0 -124
- package/functions/computation-system-v4/examples/plugins/template/TemplatePlugin.ts +0 -93
- package/functions/computation-system-v4/examples/pubsub-trigger/Pubsub_ExternalMLTrigger.js +0 -51
- package/functions/computation-system-v4/examples/python/hello_world.py +0 -16
- package/functions/computation-system-v4/examples/python/sector_correlations.py +0 -19
- package/functions/computation-system-v4/examples/telemetry-demo/VerboseTelemetryDemo.js +0 -80
- package/functions/computation-system-v4/kernel/utils/AstValidator.ts +0 -128
- package/functions/computation-system-v4/plugins/context/DefaultContextPlugin.tests.ts +0 -93
- package/functions/computation-system-v4/plugins/context/libs/metrics.ts +0 -152
- package/functions/computation-system-v4/scripts/parity-harness.js +0 -539
- package/functions/computation-system-v4/tests/v3-v4-parity-proof.tests.ts +0 -478
- package/functions/computation-system-v4/tests/v3-v4-security-parity.tests.ts +0 -718
- /package/functions/computation-system-v4/{kernel → core/kernel}/index.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/billing/DefaultBillingPlugin.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/_types.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/alertMatcher.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/dashboardParsing.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/finance.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/graph.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/math.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/portfolio.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/rankings.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/social.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/libs/trades.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/wrappers/behavioral.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/wrappers/index.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/wrappers/performance.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/wrappers/profileMetrics.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/wrappers/risk.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/context/wrappers/sectorAnalysis.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/manifest/ContractRegistry.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/manifest/ManifestHasher.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/manifest/NpmPolicy.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/manifest/PythonPolicy.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/manifest/ScheduleValidator.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/runtime/wasm/libBundler.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/shared/AlertSubscriptionMatcher.ts +0 -0
- /package/functions/computation-system-v4/{plugins → core/plugins}/telemetry/DefaultTelemetryPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IBillingPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IContextPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IDataAdapterPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IDispatchPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IManifestPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IMiddleware.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IPlannerPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/IRuntimePlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/ISchedulerPlugin.ts +0 -0
- /package/functions/computation-system-v4/{types → core/types}/ITelemetryPlugin.ts +0 -0
|
@@ -59,6 +59,9 @@ class StorageManager {
|
|
|
59
59
|
async commitResults(dateStr, entry, results, depResultHashes = {}) {
|
|
60
60
|
const storageConfig = entry.storage || {};
|
|
61
61
|
const startTime = Date.now();
|
|
62
|
+
|
|
63
|
+
this.logger.log('INFO', `[StorageMgr] commitResults for ${entry.name}. Entities to commit: ${results ? Object.keys(results).length : 0}. Outbox Config Enabled: ${!!storageConfig.outbox?.enabled}`);
|
|
64
|
+
|
|
62
65
|
const gcsTask = storageConfig.bigquery === false
|
|
63
66
|
? Promise.resolve(null)
|
|
64
67
|
: this._stageToGCS(dateStr, entry, results, depResultHashes);
|
|
@@ -102,7 +105,7 @@ class StorageManager {
|
|
|
102
105
|
const table = this.config.tables.results.table;
|
|
103
106
|
const bucketName = this.config.storage.bucket;
|
|
104
107
|
const prefix = this.config.storage.prefix;
|
|
105
|
-
|
|
108
|
+
|
|
106
109
|
// Normalize name to match Worker output exactly
|
|
107
110
|
const compName = String(entry.name || 'unknown').toLowerCase();
|
|
108
111
|
const ownerId = String(entry.ownerId || 'system');
|
|
@@ -127,7 +130,7 @@ class StorageManager {
|
|
|
127
130
|
const nameHash = crypto.createHash('sha1').update(compName).digest('hex').slice(0, 8);
|
|
128
131
|
const dateKey = dateStr.replace(/-/g, '');
|
|
129
132
|
// Append a short unique ID to prevent concurrent run collisions
|
|
130
|
-
const runId = crypto.randomUUID().replace(/-/g, '').slice(0, 8);
|
|
133
|
+
const runId = crypto.randomUUID().replace(/-/g, '').slice(0, 8);
|
|
131
134
|
const tempTableId = `${table}_temp_${safeName}_${nameHash}_${dateKey}_${runId}`;
|
|
132
135
|
|
|
133
136
|
this.logger.log('INFO', `[StorageMgr] Using deterministic temp table: ${tempTableId}`);
|
|
@@ -213,7 +216,7 @@ class StorageManager {
|
|
|
213
216
|
}
|
|
214
217
|
});
|
|
215
218
|
}
|
|
216
|
-
|
|
219
|
+
|
|
217
220
|
async _computeTableHash(tempTableId) {
|
|
218
221
|
const fullTemp = `\`${this.config.bigquery.projectId}.${this.config.bigquery.dataset}.${tempTableId}\``;
|
|
219
222
|
const query = `SELECT CAST(BIT_XOR(FARM_FINGERPRINT(result_hash)) AS STRING) as table_checksum FROM ${fullTemp}`;
|
|
@@ -233,11 +236,11 @@ class StorageManager {
|
|
|
233
236
|
const bucketName = this.config.storage.bucket;
|
|
234
237
|
const prefix = this.config.storage.prefix;
|
|
235
238
|
const ownerId = String(entry.ownerId || 'system');
|
|
236
|
-
|
|
239
|
+
|
|
237
240
|
// Normalize name
|
|
238
241
|
const compName = String(entry.name || 'unknown').toLowerCase();
|
|
239
242
|
const filename = `${prefix}/${dateStr}/${ownerId}/${compName}/${this.newUuid()}.json`;
|
|
240
|
-
|
|
243
|
+
|
|
241
244
|
const file = this.storage.bucket(bucketName).file(filename);
|
|
242
245
|
const gcsUri = `gs://${bucketName}/${filename}`;
|
|
243
246
|
|
|
@@ -458,13 +461,13 @@ class StorageManager {
|
|
|
458
461
|
async _cleanupGCSFiles(bucketName, filePrefix) {
|
|
459
462
|
// NEW: Log the deletion target
|
|
460
463
|
this.logger.log('INFO', `[StorageMgr] Cleaning up GCS files at gs://${bucketName}/${filePrefix}`);
|
|
461
|
-
|
|
464
|
+
|
|
462
465
|
const [files] = await this.storage.bucket(bucketName).getFiles({ prefix: filePrefix });
|
|
463
466
|
const deletePromises = files.map(file => file.delete().catch(e => {
|
|
464
467
|
this.logger.warn(`[StorageMgr] Failed to delete file ${file.name}: ${e.message}`);
|
|
465
468
|
}));
|
|
466
469
|
await Promise.all(deletePromises);
|
|
467
|
-
|
|
470
|
+
|
|
468
471
|
this.logger.log('INFO', `[StorageMgr] Successfully cleaned up ${files.length} file(s) from gs://${bucketName}/${filePrefix}`);
|
|
469
472
|
}
|
|
470
473
|
}
|
|
@@ -16,6 +16,13 @@ const buildWrappers = require('../wrappers/index');
|
|
|
16
16
|
const MAX_LOG_LINE_BYTES = 4096;
|
|
17
17
|
const ANSI_ESCAPE = /\x1b\[[0-9;]*m/g;
|
|
18
18
|
const CONTROL_CHARS = /[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/g;
|
|
19
|
+
const WRAPPER_DEPENDENCIES = Object.freeze({
|
|
20
|
+
performance: ['portfolio', 'math'],
|
|
21
|
+
risk: ['finance', 'math'],
|
|
22
|
+
behavioral: ['math', 'portfolio', 'rankings', 'trades', 'instruments'],
|
|
23
|
+
sectorAnalysis: ['math', 'portfolio', 'instruments'],
|
|
24
|
+
profileMetrics: ['math', 'portfolio', 'rankings', 'social']
|
|
25
|
+
});
|
|
19
26
|
|
|
20
27
|
/**
|
|
21
28
|
* Sanitize a log message: strip ANSI escapes and non-printable control chars, ensure valid UTF-8, cap length.
|
|
@@ -70,18 +77,31 @@ function buildLib(config) {
|
|
|
70
77
|
const allowed = getAllowedLibModules(effectiveConfig);
|
|
71
78
|
if (!allowed || allowed.length === 0) return Object.freeze({});
|
|
72
79
|
|
|
73
|
-
const
|
|
74
|
-
for (const
|
|
80
|
+
const internalModules = new Set(allowed);
|
|
81
|
+
for (const moduleName of allowed) {
|
|
82
|
+
const dependencies = WRAPPER_DEPENDENCIES[moduleName];
|
|
83
|
+
if (!dependencies) continue;
|
|
84
|
+
dependencies.forEach((dependency) => internalModules.add(dependency));
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
const internalLibs = {};
|
|
88
|
+
for (const m of internalModules) {
|
|
75
89
|
if (Object.prototype.hasOwnProperty.call(libFactories, m)) {
|
|
76
90
|
try {
|
|
77
|
-
|
|
91
|
+
internalLibs[m] = libFactories[m]();
|
|
78
92
|
} catch (err) {
|
|
79
93
|
throw new Error(`Failed to instantiate library '${m}': ${err.message}`);
|
|
80
94
|
}
|
|
81
95
|
}
|
|
82
96
|
}
|
|
83
97
|
|
|
84
|
-
const wrappers = buildWrappers(
|
|
98
|
+
const wrappers = buildWrappers(internalLibs);
|
|
99
|
+
const out = {};
|
|
100
|
+
for (const m of allowed) {
|
|
101
|
+
if (Object.prototype.hasOwnProperty.call(internalLibs, m)) {
|
|
102
|
+
out[m] = internalLibs[m];
|
|
103
|
+
}
|
|
104
|
+
}
|
|
85
105
|
Object.assign(out, wrappers);
|
|
86
106
|
|
|
87
107
|
return deepFreeze(out);
|
|
@@ -248,6 +248,11 @@ class ExecutionPipeline {
|
|
|
248
248
|
}
|
|
249
249
|
return;
|
|
250
250
|
}
|
|
251
|
+
|
|
252
|
+
else if (execResult.status === 'completed' && execResult.result === undefined) {
|
|
253
|
+
this.logger.log(`[Batch] ${entry.name} Entity ${entityId} skipped (computation returned undefined)`);
|
|
254
|
+
}
|
|
255
|
+
|
|
251
256
|
if (execResult.status === 'error') {
|
|
252
257
|
const error = execResult.error?.message || String(execResult.error);
|
|
253
258
|
this.logger.error(`[Batch] Entity ${entityId} failed: ${error}`);
|
|
@@ -261,6 +266,8 @@ class ExecutionPipeline {
|
|
|
261
266
|
}
|
|
262
267
|
})));
|
|
263
268
|
|
|
269
|
+
this.logger.log(`[Batch] ${entry.name} batch ${batchNum ?? 'N/A'} completed. Entities OUT (with data): ${successCount} / ${entityIds.length}`);
|
|
270
|
+
|
|
264
271
|
if (!dryRun && heavyEntities.length > 0 && this.stateRepo.logHeavyEntities) {
|
|
265
272
|
await this.stateRepo.logHeavyEntities(entry.name, heavyEntities, entry.ownerId || ownerId);
|
|
266
273
|
}
|
|
@@ -137,6 +137,9 @@ class Manifest {
|
|
|
137
137
|
path: cfg?.firestore?.path || null,
|
|
138
138
|
merge: cfg?.firestore?.merge || false,
|
|
139
139
|
includeMetadata: cfg?.firestore?.includeMetadata !== false
|
|
140
|
+
},
|
|
141
|
+
outbox: {
|
|
142
|
+
enabled: cfg?.outbox?.enabled === true
|
|
140
143
|
}
|
|
141
144
|
};
|
|
142
145
|
}
|
|
@@ -290,6 +293,9 @@ class Manifest {
|
|
|
290
293
|
path: null,
|
|
291
294
|
merge: false,
|
|
292
295
|
includeMetadata: false
|
|
296
|
+
},
|
|
297
|
+
outbox: {
|
|
298
|
+
enabled: false
|
|
293
299
|
}
|
|
294
300
|
};
|
|
295
301
|
}
|
|
@@ -226,6 +226,18 @@ describe('Skills Phase 1 — Manifest', () => {
|
|
|
226
226
|
const m2 = Manifest.build([base2]);
|
|
227
227
|
assert.notStrictEqual(m1[0].hash, m2[0].hash, 'Hash should differ when lib subset differs');
|
|
228
228
|
});
|
|
229
|
+
|
|
230
|
+
it('preserves outbox storage config in manifest entries', () => {
|
|
231
|
+
const recipe = validRecipe({
|
|
232
|
+
storage: {
|
|
233
|
+
bigquery: true,
|
|
234
|
+
firestore: { enabled: true, path: 'owners/{ownerId}/alerts/{date}/TestComp/{entityId}' },
|
|
235
|
+
outbox: { enabled: true }
|
|
236
|
+
}
|
|
237
|
+
});
|
|
238
|
+
const manifest = Manifest.build([recipe]);
|
|
239
|
+
assert.strictEqual(manifest[0].storage.outbox.enabled, true);
|
|
240
|
+
});
|
|
229
241
|
});
|
|
230
242
|
|
|
231
243
|
describe('Skills Phase 1 — ContextBuilder', () => {
|
|
@@ -319,6 +331,47 @@ describe('Skills Phase 1 — ContextBuilder', () => {
|
|
|
319
331
|
assert.strictEqual(Object.isFrozen(ctx.lib), true);
|
|
320
332
|
assert.strictEqual(Object.isFrozen(ctx.lib.math), true);
|
|
321
333
|
});
|
|
334
|
+
|
|
335
|
+
it('wrapper modules can use internal dependencies without exposing them directly', () => {
|
|
336
|
+
const builder = new ContextBuilder({}, null);
|
|
337
|
+
const ctx = builder.build({
|
|
338
|
+
computationName: 'BehavioralTest',
|
|
339
|
+
date: '2024-01-31',
|
|
340
|
+
data: {},
|
|
341
|
+
config: { skills: ['lib', 'bigquery'], lib: ['behavioral', 'dataHelpers'] }
|
|
342
|
+
});
|
|
343
|
+
|
|
344
|
+
assert.ok(ctx.lib.behavioral);
|
|
345
|
+
assert.strictEqual(ctx.lib.portfolio, undefined);
|
|
346
|
+
assert.strictEqual(ctx.lib.rankings, undefined);
|
|
347
|
+
|
|
348
|
+
const result = ctx.lib.behavioral.DetectAnomaly({
|
|
349
|
+
portfolios: [{
|
|
350
|
+
user_id: 'pi-1',
|
|
351
|
+
date: '2024-01-31',
|
|
352
|
+
portfolio_data: { Positions: [] }
|
|
353
|
+
}],
|
|
354
|
+
rankingsData: [{
|
|
355
|
+
pi_id: 'pi-1',
|
|
356
|
+
date: '2024-01-31',
|
|
357
|
+
rankings_data: {}
|
|
358
|
+
}],
|
|
359
|
+
tradeHistory: [],
|
|
360
|
+
tickerMappings: [],
|
|
361
|
+
sectorMappings: [],
|
|
362
|
+
piMasterList: [],
|
|
363
|
+
entityId: 'pi-1',
|
|
364
|
+
date: '2024-01-31',
|
|
365
|
+
getEntityRows: (rows, entityId) => {
|
|
366
|
+
const arr = Array.isArray(rows) ? rows : Object.values(rows || {}).flat();
|
|
367
|
+
return arr.filter((row) => String(row.user_id ?? row.pi_id ?? row.cid) === String(entityId));
|
|
368
|
+
},
|
|
369
|
+
toDateStr: (value) => String(value).slice(0, 10)
|
|
370
|
+
});
|
|
371
|
+
|
|
372
|
+
assert.ok(result);
|
|
373
|
+
assert.notStrictEqual(result.outcome, 'Cannot read properties of undefined');
|
|
374
|
+
});
|
|
322
375
|
});
|
|
323
376
|
|
|
324
377
|
describe('Phase 1 — Worker tier routing', () => {
|
|
@@ -1,66 +1,66 @@
|
|
|
1
|
-
'use strict';
|
|
2
|
-
/**
|
|
3
|
-
* @fileoverview System health smoke alerter
|
|
4
|
-
*/
|
|
5
|
-
|
|
6
|
-
const DEV_CID = '29312236';
|
|
7
|
-
|
|
8
|
-
exports.config = {
|
|
9
|
-
name: 'SystemHealth_Smoke_Alerter',
|
|
10
|
-
type: 'global',
|
|
11
|
-
category: 'alerter',
|
|
12
|
-
dependencies: [],
|
|
13
|
-
schedule: '0 9 * * *',
|
|
14
|
-
|
|
15
|
-
requires: {},
|
|
16
|
-
|
|
17
|
-
storage: {
|
|
18
|
-
bigquery: false,
|
|
19
|
-
firestore: {
|
|
20
|
-
enabled: true,
|
|
21
|
-
path: 'owners/system/alert_notifications/{date}/SystemHealth_Smoke_Alerter',
|
|
22
|
-
merge: false
|
|
23
|
-
},
|
|
24
|
-
// FIX 1: Explicitly enable outbox storage
|
|
25
|
-
outbox: {
|
|
26
|
-
enabled: true
|
|
27
|
-
}
|
|
28
|
-
},
|
|
29
|
-
|
|
30
|
-
alert: {
|
|
31
|
-
id: 'system_health_smoke',
|
|
32
|
-
frontendName: 'System Health (Smoke)',
|
|
33
|
-
description: 'Proves the alert listener and notification pipeline are working',
|
|
34
|
-
messageTemplate: 'System is online and working fine.',
|
|
35
|
-
severity: 'low'
|
|
36
|
-
}
|
|
37
|
-
};
|
|
38
|
-
|
|
39
|
-
exports.process = (ctx) => {
|
|
40
|
-
const date = ctx.date || new Date().toISOString().slice(0, 10);
|
|
41
|
-
|
|
42
|
-
const notification = {
|
|
43
|
-
userCid: DEV_CID,
|
|
44
|
-
piCid: 0,
|
|
45
|
-
alertTypeId: exports.config.alert.id,
|
|
46
|
-
alertTypeName: exports.config.alert.frontendName,
|
|
47
|
-
message: exports.config.alert.messageTemplate,
|
|
48
|
-
computationName: exports.config.name,
|
|
49
|
-
computationDate: date,
|
|
50
|
-
severity: exports.config.alert.severity,
|
|
51
|
-
watchlistId: 'system_smoke',
|
|
52
|
-
watchlistName: 'System Smoke'
|
|
53
|
-
};
|
|
54
|
-
|
|
55
|
-
return {
|
|
56
|
-
// We keep the notification at the root for standard Firestore paths
|
|
57
|
-
results: [notification],
|
|
58
|
-
|
|
59
|
-
// FIX 2: Provide the exact `_outbox` envelope the StorageManager expects
|
|
60
|
-
_outbox: {
|
|
61
|
-
fcmPayloads: [notification], // Route default to FCM
|
|
62
|
-
emailPayloads: [],
|
|
63
|
-
smsPayloads: []
|
|
64
|
-
}
|
|
65
|
-
};
|
|
1
|
+
'use strict';
|
|
2
|
+
/**
|
|
3
|
+
* @fileoverview System health smoke alerter
|
|
4
|
+
*/
|
|
5
|
+
|
|
6
|
+
const DEV_CID = '29312236';
|
|
7
|
+
|
|
8
|
+
exports.config = {
|
|
9
|
+
name: 'SystemHealth_Smoke_Alerter',
|
|
10
|
+
type: 'global',
|
|
11
|
+
category: 'alerter',
|
|
12
|
+
dependencies: [],
|
|
13
|
+
schedule: '0 9 * * *',
|
|
14
|
+
|
|
15
|
+
requires: {},
|
|
16
|
+
|
|
17
|
+
storage: {
|
|
18
|
+
bigquery: false,
|
|
19
|
+
firestore: {
|
|
20
|
+
enabled: true,
|
|
21
|
+
path: 'owners/system/alert_notifications/{date}/SystemHealth_Smoke_Alerter',
|
|
22
|
+
merge: false
|
|
23
|
+
},
|
|
24
|
+
// FIX 1: Explicitly enable outbox storage
|
|
25
|
+
outbox: {
|
|
26
|
+
enabled: true
|
|
27
|
+
}
|
|
28
|
+
},
|
|
29
|
+
|
|
30
|
+
alert: {
|
|
31
|
+
id: 'system_health_smoke',
|
|
32
|
+
frontendName: 'System Health (Smoke)',
|
|
33
|
+
description: 'Proves the alert listener and notification pipeline are working',
|
|
34
|
+
messageTemplate: 'System is online and working fine.',
|
|
35
|
+
severity: 'low'
|
|
36
|
+
}
|
|
37
|
+
};
|
|
38
|
+
|
|
39
|
+
exports.process = (ctx) => {
|
|
40
|
+
const date = ctx.date || new Date().toISOString().slice(0, 10);
|
|
41
|
+
|
|
42
|
+
const notification = {
|
|
43
|
+
userCid: DEV_CID,
|
|
44
|
+
piCid: 0,
|
|
45
|
+
alertTypeId: exports.config.alert.id,
|
|
46
|
+
alertTypeName: exports.config.alert.frontendName,
|
|
47
|
+
message: exports.config.alert.messageTemplate,
|
|
48
|
+
computationName: exports.config.name,
|
|
49
|
+
computationDate: date,
|
|
50
|
+
severity: exports.config.alert.severity,
|
|
51
|
+
watchlistId: 'system_smoke',
|
|
52
|
+
watchlistName: 'System Smoke'
|
|
53
|
+
};
|
|
54
|
+
|
|
55
|
+
return {
|
|
56
|
+
// We keep the notification at the root for standard Firestore paths
|
|
57
|
+
results: [notification],
|
|
58
|
+
|
|
59
|
+
// FIX 2: Provide the exact `_outbox` envelope the StorageManager expects
|
|
60
|
+
_outbox: {
|
|
61
|
+
fcmPayloads: [notification], // Route default to FCM
|
|
62
|
+
emailPayloads: [],
|
|
63
|
+
smsPayloads: []
|
|
64
|
+
}
|
|
65
|
+
};
|
|
66
66
|
};
|
|
@@ -0,0 +1,252 @@
|
|
|
1
|
+
# Core Overview
|
|
2
|
+
|
|
3
|
+
This directory contains the production runtime for the v4 computation system: the kernel, official system plugins, shared sandbox helpers, and the public type contracts they implement.
|
|
4
|
+
|
|
5
|
+
This README gives a file-by-file purpose map for `core/`, excluding the implementation-heavy files under:
|
|
6
|
+
|
|
7
|
+
- `plugins/context/libs/*`
|
|
8
|
+
- `plugins/context/wrappers/*`
|
|
9
|
+
|
|
10
|
+
Those two directories are intentionally omitted from the detailed breakdown below.
|
|
11
|
+
|
|
12
|
+
## Directory Shape
|
|
13
|
+
|
|
14
|
+
- `kernel/`: the orchestration layer that wires hooks, plugins, and execution flow together.
|
|
15
|
+
- `plugins/`: trusted system plugins and shared runtime helpers.
|
|
16
|
+
- `types/`: the TypeScript contracts for the kernel and plugin capability system.
|
|
17
|
+
|
|
18
|
+
## `kernel/`
|
|
19
|
+
|
|
20
|
+
### [`kernel/index.ts`](./kernel/index.ts)
|
|
21
|
+
Barrel export for the kernel package. Re-exports the main kernel classes so callers can import the orchestration surface from one place.
|
|
22
|
+
|
|
23
|
+
### [`kernel/Kernel.ts`](./kernel/Kernel.ts)
|
|
24
|
+
The main runtime orchestrator. It boots the plugin set, resolves capabilities, coordinates lifecycle hooks, builds execution context, and drives a computation run through planning, execution, and finalization.
|
|
25
|
+
|
|
26
|
+
### [`kernel/HookRegistry.ts`](./kernel/HookRegistry.ts)
|
|
27
|
+
Owns lifecycle hook registration and dispatch. System plugins use it to tap into kernel stages without directly coupling to each other.
|
|
28
|
+
|
|
29
|
+
### [`kernel/PluginRegistry.ts`](./kernel/PluginRegistry.ts)
|
|
30
|
+
Registers plugins, indexes them by capability, and resolves capability lookups at runtime. This is the trust-aware directory the kernel uses to find official system services.
|
|
31
|
+
|
|
32
|
+
### [`kernel/ContextAccumulator.ts`](./kernel/ContextAccumulator.ts)
|
|
33
|
+
Builds and merges the evolving runtime context for a computation run. It is responsible for collecting stage outputs into a consistent context object without leaking plugin-local details into unrelated stages.
|
|
34
|
+
|
|
35
|
+
### [`kernel/utils/AstValidator.ts`](./kernel/utils/AstValidator.ts)
|
|
36
|
+
Shared AST policy checker used by kernel-adjacent validation paths. It provides a lightweight static guardrail around unsafe language constructs before execution is attempted.
|
|
37
|
+
|
|
38
|
+
## `plugins/`
|
|
39
|
+
|
|
40
|
+
### [`plugins/index.ts`](./plugins/index.ts)
|
|
41
|
+
Barrel export for the official system plugin set.
|
|
42
|
+
|
|
43
|
+
### Billing
|
|
44
|
+
|
|
45
|
+
#### [`plugins/billing/DefaultBillingPlugin.ts`](./plugins/billing/DefaultBillingPlugin.ts)
|
|
46
|
+
Trusted billing plugin. Applies billing-related lifecycle behavior and keeps charging logic in system-owned code rather than user-owned code.
|
|
47
|
+
|
|
48
|
+
### Context
|
|
49
|
+
|
|
50
|
+
#### [`plugins/context/DefaultContextPlugin.ts`](./plugins/context/DefaultContextPlugin.ts)
|
|
51
|
+
Builds the `ctx` object exposed to computations. It injects system libraries, resolves verified user libraries, enforces alias isolation rules, applies dependency ordering, and prevents user code from shadowing trusted built-ins.
|
|
52
|
+
|
|
53
|
+
#### [`plugins/context/UserLibrarySource.ts`](./plugins/context/UserLibrarySource.ts)
|
|
54
|
+
Loads and interprets verified user library artifacts. It extracts the library manifest/factory metadata through the isolated metadata worker, validates contracts, and turns uploaded code into a runtime-ready user library definition.
|
|
55
|
+
|
|
56
|
+
### Data
|
|
57
|
+
|
|
58
|
+
#### [`plugins/data/BigQueryDataAdapterPlugin.ts`](./plugins/data/BigQueryDataAdapterPlugin.ts)
|
|
59
|
+
Primary data-access plugin for BigQuery-backed computations. It turns manifest/config requirements into actual query execution against the configured warehouse backend.
|
|
60
|
+
|
|
61
|
+
#### [`plugins/data/MaterializedViewManager.ts`](./plugins/data/MaterializedViewManager.ts)
|
|
62
|
+
Support utility for data-layer materialized view workflows. It encapsulates the view management logic used by the data plugin so the main adapter stays focused on execution flow.
|
|
63
|
+
|
|
64
|
+
### Dispatch
|
|
65
|
+
|
|
66
|
+
#### [`plugins/dispatch/DefaultDispatchPlugin.ts`](./plugins/dispatch/DefaultDispatchPlugin.ts)
|
|
67
|
+
Standard dispatch plugin for non-Python computations. It is responsible for selecting and invoking the appropriate execution route once a computation has been planned and prepared.
|
|
68
|
+
|
|
69
|
+
#### [`plugins/dispatch/PythonDispatchPlugin.ts`](./plugins/dispatch/PythonDispatchPlugin.ts)
|
|
70
|
+
Dispatch plugin for Python-oriented computations. It handles the trusted bridge between the v4 orchestration model and Python execution flows.
|
|
71
|
+
|
|
72
|
+
### Logging
|
|
73
|
+
|
|
74
|
+
#### [`plugins/logging/GcpExecutionLogSinkPlugin.ts`](./plugins/logging/GcpExecutionLogSinkPlugin.ts)
|
|
75
|
+
System log sink for pushing execution telemetry and structured logs into Google Cloud logging infrastructure.
|
|
76
|
+
|
|
77
|
+
### Manifest
|
|
78
|
+
|
|
79
|
+
#### [`plugins/manifest/DefaultManifestPlugin.ts`](./plugins/manifest/DefaultManifestPlugin.ts)
|
|
80
|
+
The central manifest-resolution plugin. It parses computation definitions, validates configuration shape, applies verification-aware policy checks, merges library/plugin requirements, and produces the canonical runtime manifest used by the kernel.
|
|
81
|
+
|
|
82
|
+
#### [`plugins/manifest/ContractRegistry.ts`](./plugins/manifest/ContractRegistry.ts)
|
|
83
|
+
Registry for callable contracts discovered from built-in libraries, wrappers, and verified user libraries. It gives manifest resolution a single place to look up what a call requires from the data layer.
|
|
84
|
+
|
|
85
|
+
#### [`plugins/manifest/ConfigResolver.ts`](./plugins/manifest/ConfigResolver.ts)
|
|
86
|
+
Normalizes and merges config-driven requirements into the manifest. It contains the defensive merge rules for `requires`, including field unioning, `mandatory` promotion, and lookback widening.
|
|
87
|
+
|
|
88
|
+
#### [`plugins/manifest/BundleValidator.ts`](./plugins/manifest/BundleValidator.ts)
|
|
89
|
+
Validates uploaded/bundled source artifacts before they are admitted into the verification pipeline. This is part of the “safe to inspect” gate before deeper semantic checks.
|
|
90
|
+
|
|
91
|
+
#### [`plugins/manifest/AstValidator.ts`](./plugins/manifest/AstValidator.ts)
|
|
92
|
+
Manifest-specific AST safety validator used during computation/plugin/library verification. It focuses on policy enforcement for uploaded code rather than kernel internals.
|
|
93
|
+
|
|
94
|
+
#### [`plugins/manifest/NpmPolicy.ts`](./plugins/manifest/NpmPolicy.ts)
|
|
95
|
+
Enforces package/import policy for JavaScript and TypeScript bundles. It is where allowed and disallowed npm dependency rules are expressed.
|
|
96
|
+
|
|
97
|
+
#### [`plugins/manifest/NpmBundleWorker.ts`](./plugins/manifest/NpmBundleWorker.ts)
|
|
98
|
+
Worker-side helper used for npm bundle inspection/processing during verification. It keeps heavier bundle handling out of the main verification flow.
|
|
99
|
+
|
|
100
|
+
#### [`plugins/manifest/ManifestHasher.ts`](./plugins/manifest/ManifestHasher.ts)
|
|
101
|
+
Computes stable hashes for manifests and related execution inputs. These hashes are used to pin verified artifacts and detect when re-verification is required.
|
|
102
|
+
|
|
103
|
+
#### [`plugins/manifest/PythonPolicy.ts`](./plugins/manifest/PythonPolicy.ts)
|
|
104
|
+
Python-specific verification policy module. It applies the equivalent package/import rules for Python computations.
|
|
105
|
+
|
|
106
|
+
#### [`plugins/manifest/RecipeExtractor.ts`](./plugins/manifest/RecipeExtractor.ts)
|
|
107
|
+
Extracts canonical “recipe” metadata from computation definitions so downstream planning and verification work from a normalized shape.
|
|
108
|
+
|
|
109
|
+
#### [`plugins/manifest/ScheduleValidator.ts`](./plugins/manifest/ScheduleValidator.ts)
|
|
110
|
+
Validates scheduling-related manifest configuration such as trigger shape and supported schedule semantics.
|
|
111
|
+
|
|
112
|
+
### Planner
|
|
113
|
+
|
|
114
|
+
#### [`plugins/planner/DefaultPlannerPlugin.ts`](./plugins/planner/DefaultPlannerPlugin.ts)
|
|
115
|
+
Trusted planning plugin that converts a manifest into an executable plan. It handles dependency-aware execution ordering before dispatch/runtime stages begin.
|
|
116
|
+
|
|
117
|
+
### Pub/Sub
|
|
118
|
+
|
|
119
|
+
#### [`plugins/pubsub/DefaultPubSubPlugin.ts`](./plugins/pubsub/DefaultPubSubPlugin.ts)
|
|
120
|
+
System-owned Pub/Sub side-effect plugin. User plugins may emit publish intents, but this plugin remains the trusted component that validates and performs the actual publish.
|
|
121
|
+
|
|
122
|
+
### Runtime
|
|
123
|
+
|
|
124
|
+
#### [`plugins/runtime/WasmRuntimePlugin.ts`](./plugins/runtime/WasmRuntimePlugin.ts)
|
|
125
|
+
Primary runtime plugin for isolated computation execution. It coordinates execution of bundled code inside the chosen runtime boundary and exposes the runtime capability to the rest of the kernel.
|
|
126
|
+
|
|
127
|
+
#### [`plugins/runtime/wasm/WasmWorker.ts`](./plugins/runtime/wasm/WasmWorker.ts)
|
|
128
|
+
Worker/controller class for the isolated runtime process. It encapsulates the communication boundary used to run computation code away from the main orchestrator thread.
|
|
129
|
+
|
|
130
|
+
#### [`plugins/runtime/wasm/worker-entry.ts`](./plugins/runtime/wasm/worker-entry.ts)
|
|
131
|
+
Worker entrypoint for the runtime sandbox. It bootstraps the isolated runtime environment and listens for execution requests from the host plugin.
|
|
132
|
+
|
|
133
|
+
#### [`plugins/runtime/wasm/libBundler.ts`](./plugins/runtime/wasm/libBundler.ts)
|
|
134
|
+
Helper for preparing bundled library/runtime payloads before they are sent into the sandboxed runtime worker.
|
|
135
|
+
|
|
136
|
+
### Scheduler
|
|
137
|
+
|
|
138
|
+
#### [`plugins/scheduler/CloudTasksSchedulerPlugin.ts`](./plugins/scheduler/CloudTasksSchedulerPlugin.ts)
|
|
139
|
+
Scheduler integration for Google Cloud Tasks. It is the trusted queueing layer that turns validated execution plans into scheduled work items.
|
|
140
|
+
|
|
141
|
+
### Shared
|
|
142
|
+
|
|
143
|
+
#### [`plugins/shared/AlertSubscriptionMatcher.ts`](./plugins/shared/AlertSubscriptionMatcher.ts)
|
|
144
|
+
Shared utility used by alert-oriented flows to match produced results against subscription rules and outbox routing needs.
|
|
145
|
+
|
|
146
|
+
#### [`plugins/shared/UntrustedMetadataExtractor.ts`](./plugins/shared/UntrustedMetadataExtractor.ts)
|
|
147
|
+
Host-side wrapper for extracting metadata from untrusted uploaded libraries/plugins without running that extraction in the main thread. It delegates to a dedicated worker and returns sanitized definition metadata to the caller.
|
|
148
|
+
|
|
149
|
+
#### [`plugins/shared/untrusted-metadata-worker.js`](./plugins/shared/untrusted-metadata-worker.js)
|
|
150
|
+
Worker implementation used by `UntrustedMetadataExtractor.ts`. It performs the isolated metadata extraction step for untrusted artifacts during verification and loading.
|
|
151
|
+
|
|
152
|
+
### State
|
|
153
|
+
|
|
154
|
+
#### [`plugins/state/FirestoreStatePlugin.ts`](./plugins/state/FirestoreStatePlugin.ts)
|
|
155
|
+
State plugin backed by Firestore. It persists computation state, verification records, and owner-scoped metadata for computations, user plugins, and user libraries.
|
|
156
|
+
|
|
157
|
+
### Storage
|
|
158
|
+
|
|
159
|
+
#### [`plugins/storage/GcsStoragePlugin.ts`](./plugins/storage/GcsStoragePlugin.ts)
|
|
160
|
+
Storage plugin backed by Google Cloud Storage. It stores and loads immutable, hash-addressed computation/plugin/library artifacts.
|
|
161
|
+
|
|
162
|
+
### Telemetry
|
|
163
|
+
|
|
164
|
+
#### [`plugins/telemetry/DefaultTelemetryPlugin.ts`](./plugins/telemetry/DefaultTelemetryPlugin.ts)
|
|
165
|
+
Trusted telemetry plugin for recording stage-level execution events, metrics, and verification progress.
|
|
166
|
+
|
|
167
|
+
#### [`plugins/telemetry/StageHarness.ts`](./plugins/telemetry/StageHarness.ts)
|
|
168
|
+
Helper for wrapping runtime stages with consistent telemetry emission, timing, and error capture.
|
|
169
|
+
|
|
170
|
+
### User Plugins
|
|
171
|
+
|
|
172
|
+
#### [`plugins/userPlugins/UserPluginRuntimePlugin.ts`](./plugins/userPlugins/UserPluginRuntimePlugin.ts)
|
|
173
|
+
Runs verified user plugins as isolated middleware, validates stage-specific outputs, and converts user-produced results into trusted intent envelopes that system plugins can safely consume.
|
|
174
|
+
|
|
175
|
+
#### [`plugins/userPlugins/UserPluginSource.ts`](./plugins/userPlugins/UserPluginSource.ts)
|
|
176
|
+
Loads verified user plugin source definitions. Like `UserLibrarySource.ts`, it uses the isolated metadata extraction path and turns uploaded code into a validated runtime definition.
|
|
177
|
+
|
|
178
|
+
### Verification
|
|
179
|
+
|
|
180
|
+
#### [`plugins/verification/DefaultVerificationPlugin.ts`](./plugins/verification/DefaultVerificationPlugin.ts)
|
|
181
|
+
First-class verification plugin for computations, user plugins, and user libraries. It runs policy validation, dry-run checks, sandbox-oriented safety checks, artifact hashing, and redacted error reporting before an artifact can become executable.
|
|
182
|
+
|
|
183
|
+
## `types/`
|
|
184
|
+
|
|
185
|
+
### [`types/index.ts`](./types/index.ts)
|
|
186
|
+
Barrel export for the v4 type surface.
|
|
187
|
+
|
|
188
|
+
### [`types/shared.ts`](./types/shared.ts)
|
|
189
|
+
Shared structural types used across manifests, execution context, verification records, and runtime payloads.
|
|
190
|
+
|
|
191
|
+
### [`types/IPlugin.ts`](./types/IPlugin.ts)
|
|
192
|
+
Base contract for all system plugins.
|
|
193
|
+
|
|
194
|
+
### [`types/IHook.ts`](./types/IHook.ts)
|
|
195
|
+
Core lifecycle hook contract used by the hook registry.
|
|
196
|
+
|
|
197
|
+
### [`types/IHookContext.ts`](./types/IHookContext.ts)
|
|
198
|
+
The canonical context shape passed through lifecycle stages.
|
|
199
|
+
|
|
200
|
+
### [`types/IMiddleware.ts`](./types/IMiddleware.ts)
|
|
201
|
+
Contract for middleware-style behaviors that run inside the broader plugin system.
|
|
202
|
+
|
|
203
|
+
### [`types/IManifestPlugin.ts`](./types/IManifestPlugin.ts)
|
|
204
|
+
Capability contract for plugins that resolve and validate computation manifests.
|
|
205
|
+
|
|
206
|
+
### [`types/IPlannerPlugin.ts`](./types/IPlannerPlugin.ts)
|
|
207
|
+
Capability contract for execution planning plugins.
|
|
208
|
+
|
|
209
|
+
### [`types/IRuntimePlugin.ts`](./types/IRuntimePlugin.ts)
|
|
210
|
+
Capability contract for runtime execution plugins.
|
|
211
|
+
|
|
212
|
+
### [`types/IDispatchPlugin.ts`](./types/IDispatchPlugin.ts)
|
|
213
|
+
Capability contract for dispatch-layer plugins.
|
|
214
|
+
|
|
215
|
+
### [`types/IDataAdapterPlugin.ts`](./types/IDataAdapterPlugin.ts)
|
|
216
|
+
Capability contract for data source adapter plugins.
|
|
217
|
+
|
|
218
|
+
### [`types/IContextPlugin.ts`](./types/IContextPlugin.ts)
|
|
219
|
+
Capability contract for plugins that construct the computation context.
|
|
220
|
+
|
|
221
|
+
### [`types/IBillingPlugin.ts`](./types/IBillingPlugin.ts)
|
|
222
|
+
Capability contract for billing enforcement plugins.
|
|
223
|
+
|
|
224
|
+
### [`types/ISchedulerPlugin.ts`](./types/ISchedulerPlugin.ts)
|
|
225
|
+
Capability contract for scheduler/queue integration plugins.
|
|
226
|
+
|
|
227
|
+
### [`types/IStatePlugin.ts`](./types/IStatePlugin.ts)
|
|
228
|
+
Capability contract for persistence/state plugins.
|
|
229
|
+
|
|
230
|
+
### [`types/IStoragePlugin.ts`](./types/IStoragePlugin.ts)
|
|
231
|
+
Capability contract for blob/object storage plugins.
|
|
232
|
+
|
|
233
|
+
### [`types/ITelemetryPlugin.ts`](./types/ITelemetryPlugin.ts)
|
|
234
|
+
Capability contract for telemetry and stage reporting plugins.
|
|
235
|
+
|
|
236
|
+
### [`types/IVerificationPlugin.ts`](./types/IVerificationPlugin.ts)
|
|
237
|
+
Capability contract for artifact verification services.
|
|
238
|
+
|
|
239
|
+
### [`types/IUserPlugin.ts`](./types/IUserPlugin.ts)
|
|
240
|
+
Type contract for verified user plugin artifacts and their runtime shape.
|
|
241
|
+
|
|
242
|
+
### [`types/IUserLibrary.ts`](./types/IUserLibrary.ts)
|
|
243
|
+
Type contract for verified user library artifacts, bindings, contracts, and runtime resolution.
|
|
244
|
+
|
|
245
|
+
## Omitted Files
|
|
246
|
+
|
|
247
|
+
The following implementation groups are intentionally not covered file-by-file in this README because you asked to exclude the lib/wrapper files:
|
|
248
|
+
|
|
249
|
+
- `plugins/context/libs/*`
|
|
250
|
+
- `plugins/context/wrappers/*`
|
|
251
|
+
|
|
252
|
+
They are still part of the runtime context surface, but their internal implementation details are out of scope for this map.
|