@captiveorigins/onephylogeny-engine 0.0.0-stage → 0.1.0-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +75 -0
- package/README.md +4 -2
- package/dist/index.d.ts +200 -0
- package/dist/index.js +769 -0
- package/package.json +44 -3
package/LICENSE
ADDED
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
Required Notice: Copyright 2026 Captive Origins (https://captiveorigins.com)
|
|
2
|
+
|
|
3
|
+
# PolyForm Noncommercial License 1.0.0
|
|
4
|
+
|
|
5
|
+
<https://polyformproject.org/licenses/noncommercial/1.0.0>
|
|
6
|
+
|
|
7
|
+
## Acceptance
|
|
8
|
+
|
|
9
|
+
In order to get any license under these terms, you must agree to them as both strict obligations and conditions to all your licenses.
|
|
10
|
+
|
|
11
|
+
## Copyright License
|
|
12
|
+
|
|
13
|
+
The licensor grants you a copyright license for the software to do everything you might do with the software that would otherwise infringe the licensor's copyright in it for any permitted purpose. However, you may only distribute the software according to [Distribution License](#distribution-license) and make changes or new works based on the software according to [Changes and New Works License](#changes-and-new-works-license).
|
|
14
|
+
|
|
15
|
+
## Distribution License
|
|
16
|
+
|
|
17
|
+
The licensor grants you an additional copyright license to distribute copies of the software. Your license to distribute covers distributing the software with changes and new works permitted by [Changes and New Works License](#changes-and-new-works-license).
|
|
18
|
+
|
|
19
|
+
## Notices
|
|
20
|
+
|
|
21
|
+
You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms or the URL for them above, as well as copies of any plain-text lines beginning with `Required Notice:` that the licensor provided with the software. For example:
|
|
22
|
+
|
|
23
|
+
> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)
|
|
24
|
+
|
|
25
|
+
## Changes and New Works License
|
|
26
|
+
|
|
27
|
+
The licensor grants you an additional copyright license to make changes and new works based on the software for any permitted purpose.
|
|
28
|
+
|
|
29
|
+
## Patent License
|
|
30
|
+
|
|
31
|
+
The licensor grants you a patent license for the software that covers patent claims the licensor can license, or becomes able to license, that you would infringe by using the software.
|
|
32
|
+
|
|
33
|
+
## Noncommercial Purposes
|
|
34
|
+
|
|
35
|
+
Any noncommercial purpose is a permitted purpose.
|
|
36
|
+
|
|
37
|
+
## Personal Uses
|
|
38
|
+
|
|
39
|
+
Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, is use for a permitted purpose.
|
|
40
|
+
|
|
41
|
+
## Noncommercial Organizations
|
|
42
|
+
|
|
43
|
+
Use by any charitable organization, educational institution, public research organization, public safety or health organization, environmental protection organization, or government institution is use for a permitted purpose regardless of the source of funding or obligations resulting from the funding.
|
|
44
|
+
|
|
45
|
+
## Fair Use
|
|
46
|
+
|
|
47
|
+
You may have "fair use" rights for the software under the law. These terms do not limit them.
|
|
48
|
+
|
|
49
|
+
## No Other Rights
|
|
50
|
+
|
|
51
|
+
These terms do not allow you to sublicense or transfer any of your licenses to anyone else, or prevent the licensor from granting licenses to anyone else. These terms do not imply any other licenses.
|
|
52
|
+
|
|
53
|
+
## Patent Defense
|
|
54
|
+
|
|
55
|
+
If you make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.
|
|
56
|
+
|
|
57
|
+
## Violations
|
|
58
|
+
|
|
59
|
+
The first time you are notified in writing that you have violated any of these terms, or done anything with the software not covered by your licenses, your licenses can nonetheless continue if you come into full compliance with these terms, and take practical steps to correct past violations, within 32 days of receiving notice. Otherwise, all your licenses end immediately.
|
|
60
|
+
|
|
61
|
+
## No Liability
|
|
62
|
+
|
|
63
|
+
***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.***
|
|
64
|
+
|
|
65
|
+
## Definitions
|
|
66
|
+
|
|
67
|
+
The **licensor** is the individual or entity offering these terms, and the **software** is the software the licensor makes available under these terms.
|
|
68
|
+
|
|
69
|
+
**You** refers to the individual or entity agreeing to these terms.
|
|
70
|
+
|
|
71
|
+
**Your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization. **Control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise. Control can be direct or indirect.
|
|
72
|
+
|
|
73
|
+
**Your licenses** are all the licenses granted to you for the software under these terms.
|
|
74
|
+
|
|
75
|
+
**Use** means anything you do with the software requiring one of your licenses.
|
package/README.md
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @captiveorigins/onephylogeny-engine
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
OnePhylogeny engine: runs profiles end to end with immutable run records and structured logs: fetch, anchor, gate, build panels, validate, test samples and build trees through replaceable ports.
|
|
4
|
+
|
|
5
|
+
See the [repository](https://github.com/CaptiveOrigins/onephylogeny) for documentation.
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
import { ValidationInput, FullValidation, RunEvent, Run, SequenceRecord, Diagnostic, PanelBundle, Result, Profile, Curation, RunComparison, LinkedRecord } from '@captiveorigins/onephylogeny-core';
|
|
2
|
+
import { NcbiOptions, FoundPaper } from '@captiveorigins/onephylogeny-acquire';
|
|
3
|
+
|
|
4
|
+
/** Where files live. A folder on disk, in the CLI and the Mac app. */
|
|
5
|
+
interface Store {
|
|
6
|
+
read(path: string): Promise<string>;
|
|
7
|
+
/** Must be atomic: a crash leaves the old file or the new one, never half of it. */
|
|
8
|
+
write(path: string, text: string): Promise<void>;
|
|
9
|
+
exists(path: string): Promise<boolean>;
|
|
10
|
+
/** Adds to a log without rewriting it. Optional: without it, logs are written whole. */
|
|
11
|
+
append?(path: string, text: string): Promise<void>;
|
|
12
|
+
}
|
|
13
|
+
/** How native programs run. Only tree building needs one. */
|
|
14
|
+
interface ToolRunner {
|
|
15
|
+
version(tool: string): Promise<string | null>;
|
|
16
|
+
run(tool: string, args: string[], options: {
|
|
17
|
+
cwd: string;
|
|
18
|
+
}): Promise<{
|
|
19
|
+
code: number;
|
|
20
|
+
stdout: string;
|
|
21
|
+
stderr: string;
|
|
22
|
+
}>;
|
|
23
|
+
}
|
|
24
|
+
type ValidationReports = FullValidation;
|
|
25
|
+
/** Heavy pure calculations, which an app can run off its interface thread. */
|
|
26
|
+
type Compute = {
|
|
27
|
+
validate?: (input: ValidationInput, signal?: AbortSignal) => Promise<ValidationReports>;
|
|
28
|
+
};
|
|
29
|
+
type EngineContext = {
|
|
30
|
+
store: Store;
|
|
31
|
+
ncbi: Omit<NcbiOptions, 'log'>;
|
|
32
|
+
tools?: ToolRunner;
|
|
33
|
+
now?: () => Date;
|
|
34
|
+
onEvent?: (event: RunEvent) => void;
|
|
35
|
+
/** Cancels a run at its next safe point. */
|
|
36
|
+
signal?: AbortSignal;
|
|
37
|
+
compute?: Compute;
|
|
38
|
+
};
|
|
39
|
+
|
|
40
|
+
type ArtifactType = 'accession-list' | 'raw-records' | 'anchored-records' | 'gated-records' | 'panel' | 'validation' | 'sample-result' | 'tree';
|
|
41
|
+
type ToolIdentity = {
|
|
42
|
+
name: string;
|
|
43
|
+
version: string;
|
|
44
|
+
};
|
|
45
|
+
type ArtifactRef = {
|
|
46
|
+
type: ArtifactType;
|
|
47
|
+
key: string;
|
|
48
|
+
};
|
|
49
|
+
/** Written last. Its presence is what says a step finished, so a crash can't leave something that looks complete. */
|
|
50
|
+
type ArtifactManifest = {
|
|
51
|
+
schemaVersion: 1;
|
|
52
|
+
type: ArtifactType;
|
|
53
|
+
key: string;
|
|
54
|
+
step: {
|
|
55
|
+
name: string;
|
|
56
|
+
version: number;
|
|
57
|
+
};
|
|
58
|
+
inputs: ArtifactRef[];
|
|
59
|
+
parameters: unknown;
|
|
60
|
+
tools: ToolIdentity[];
|
|
61
|
+
files: {
|
|
62
|
+
name: string;
|
|
63
|
+
sha256: string;
|
|
64
|
+
}[];
|
|
65
|
+
createdAt: string;
|
|
66
|
+
};
|
|
67
|
+
type Artifact = {
|
|
68
|
+
manifest: ArtifactManifest;
|
|
69
|
+
read: (name: string) => Promise<string>;
|
|
70
|
+
};
|
|
71
|
+
/** Loads an artifact for use, refusing the wrong type or anything that has changed since it was made. */
|
|
72
|
+
declare function loadArtifact(store: Store, ref: ArtifactRef): Promise<Artifact>;
|
|
73
|
+
type TraceNode = {
|
|
74
|
+
type: ArtifactType;
|
|
75
|
+
key: string;
|
|
76
|
+
step: string;
|
|
77
|
+
parameters: unknown;
|
|
78
|
+
tools: ToolIdentity[];
|
|
79
|
+
createdAt: string;
|
|
80
|
+
inputs: TraceNode[];
|
|
81
|
+
};
|
|
82
|
+
/** Follows an artifact back through everything that produced it. */
|
|
83
|
+
declare function traceArtifact(store: Store, ref: ArtifactRef): Promise<TraceNode>;
|
|
84
|
+
|
|
85
|
+
/** Every failure has a stable code, plain words, and what to do next. */
|
|
86
|
+
type ErrorCode = 'invalid-profile' | 'empty-search' | 'empty-panel' | 'source-failed' | 'reference-missing' | 'raw-file-changed' | 'artifact-missing' | 'artifact-wrong-type' | 'artifact-corrupt' | 'step-output-invalid' | 'panel-version-exists' | 'panel-outdated' | 'panel-missing' | 'panel-changed' | 'tool-missing' | 'tool-failed' | 'run-missing' | 'cancelled';
|
|
87
|
+
declare class EngineError extends Error {
|
|
88
|
+
readonly code: ErrorCode;
|
|
89
|
+
readonly action?: string | undefined;
|
|
90
|
+
constructor(code: ErrorCode, message: string, action?: string | undefined);
|
|
91
|
+
}
|
|
92
|
+
declare const isCancelled: (e: unknown) => boolean;
|
|
93
|
+
|
|
94
|
+
/** Bump with each engine release. Recorded in every run, panel and result. */
|
|
95
|
+
declare const ENGINE_VERSION = "0.1.0-alpha.1";
|
|
96
|
+
declare const profilePaths: (profileId: string) => {
|
|
97
|
+
profile: string;
|
|
98
|
+
curation: string;
|
|
99
|
+
latest: string;
|
|
100
|
+
run: (id: string) => string;
|
|
101
|
+
panel: (id: string, version: string) => string;
|
|
102
|
+
results: string;
|
|
103
|
+
};
|
|
104
|
+
|
|
105
|
+
type RecordOutcome = {
|
|
106
|
+
record: string;
|
|
107
|
+
organism?: string;
|
|
108
|
+
status: 'passed' | 'excluded';
|
|
109
|
+
diagnostics: Diagnostic[];
|
|
110
|
+
columns: number;
|
|
111
|
+
identity: number;
|
|
112
|
+
extractedFrom: 'annotation' | 'located' | 'whole-record';
|
|
113
|
+
};
|
|
114
|
+
type FetchOutcome = {
|
|
115
|
+
run: Run;
|
|
116
|
+
records: SequenceRecord[];
|
|
117
|
+
outcomes: RecordOutcome[];
|
|
118
|
+
};
|
|
119
|
+
/**
|
|
120
|
+
* Fetches everything a profile selects, keeps raw files, anchors and checks each record, and writes the sheet and log.
|
|
121
|
+
* Each stage reuses earlier work when its inputs and settings are unchanged.
|
|
122
|
+
*/
|
|
123
|
+
declare function fetchAndAnchor(ctx: EngineContext, profile: Profile, options?: {
|
|
124
|
+
batchSize?: number;
|
|
125
|
+
allowEmpty?: boolean;
|
|
126
|
+
}): Promise<FetchOutcome>;
|
|
127
|
+
/** Studies built from GenBank records of a profile's species and marker, placed on the reference. */
|
|
128
|
+
declare function findRecordStudies(ctx: EngineContext, profile: Profile, onProgress?: (done: number, total: number) => void): Promise<{
|
|
129
|
+
papers: FoundPaper[];
|
|
130
|
+
columns: number;
|
|
131
|
+
query: string;
|
|
132
|
+
}>;
|
|
133
|
+
/** Looks up what PubMed links to one paper, then sorts those records by species and gene and places them. */
|
|
134
|
+
declare function findPaperRecords(ctx: EngineContext, profile: Profile, pmid: string): Promise<{
|
|
135
|
+
linked: LinkedRecord[];
|
|
136
|
+
columns: number;
|
|
137
|
+
}>;
|
|
138
|
+
type PanelBuildOutcome = {
|
|
139
|
+
bundle: PanelBundle;
|
|
140
|
+
diagnostics: Diagnostic[];
|
|
141
|
+
validation: ValidationReports;
|
|
142
|
+
unassigned: number;
|
|
143
|
+
run: Run;
|
|
144
|
+
published: boolean;
|
|
145
|
+
};
|
|
146
|
+
/** Builds and validates a candidate panel from a finished fetch run and the curation file. */
|
|
147
|
+
declare function buildPanelFromRun(ctx: EngineContext, profile: Profile, curation: Curation, id?: string): Promise<PanelBuildOutcome>;
|
|
148
|
+
declare function loadPanel(store: Store, dir: string): Promise<PanelBundle>;
|
|
149
|
+
/** The versions built for a profile's panel, oldest first. Every one is listed, so no attempt can be left out. */
|
|
150
|
+
declare function panelVersions(profile: Profile, folders: string[]): string[];
|
|
151
|
+
/** Compares two built versions. Both must still match their seals. */
|
|
152
|
+
declare function comparePanelVersions(store: Store, profile: Profile, a: string, b: string): Promise<RunComparison>;
|
|
153
|
+
type SampleOutcome = {
|
|
154
|
+
result: Result;
|
|
155
|
+
diagnostics: Diagnostic[];
|
|
156
|
+
aligned: string;
|
|
157
|
+
insertions: number;
|
|
158
|
+
deletions: number;
|
|
159
|
+
identity: number;
|
|
160
|
+
clipped: {
|
|
161
|
+
start: number;
|
|
162
|
+
end: number;
|
|
163
|
+
};
|
|
164
|
+
} | {
|
|
165
|
+
result: null;
|
|
166
|
+
diagnostics: Diagnostic[];
|
|
167
|
+
};
|
|
168
|
+
/** Tests one sequence against one panel. Pure TypeScript: runs in the app, the CLI or a hosted site. */
|
|
169
|
+
declare function testSample(sequence: string, queryId: string, bundle: PanelBundle, options?: {
|
|
170
|
+
runId?: string;
|
|
171
|
+
}): SampleOutcome;
|
|
172
|
+
/** Tests a sequence with a full record: a saved result that traces back to its panel, the records and raw files. */
|
|
173
|
+
declare function runSample(ctx: EngineContext, profile: Profile, sequence: string, queryId: string): Promise<SampleOutcome & {
|
|
174
|
+
run: Run;
|
|
175
|
+
artifact: ArtifactRef;
|
|
176
|
+
}>;
|
|
177
|
+
type TreeOutcome = {
|
|
178
|
+
tool: string;
|
|
179
|
+
version: string;
|
|
180
|
+
newick: string;
|
|
181
|
+
tips: {
|
|
182
|
+
id: string;
|
|
183
|
+
members: string[];
|
|
184
|
+
}[];
|
|
185
|
+
run: Run;
|
|
186
|
+
artifact: ArtifactRef;
|
|
187
|
+
};
|
|
188
|
+
/** Builds a tree with IQ-TREE from one coverage window of a run's records. */
|
|
189
|
+
declare function buildTree(ctx: EngineContext, profile: Profile, options: {
|
|
190
|
+
runId?: string;
|
|
191
|
+
absoluteDir: string;
|
|
192
|
+
tool?: string;
|
|
193
|
+
window?: {
|
|
194
|
+
start: number;
|
|
195
|
+
end: number;
|
|
196
|
+
};
|
|
197
|
+
minCoverage?: number;
|
|
198
|
+
}): Promise<TreeOutcome>;
|
|
199
|
+
|
|
200
|
+
export { type ArtifactManifest, type ArtifactRef, type ArtifactType, type Compute, ENGINE_VERSION, type EngineContext, EngineError, type ErrorCode, type FetchOutcome, type PanelBuildOutcome, type RecordOutcome, type SampleOutcome, type Store, type ToolRunner, type TraceNode, type TreeOutcome, type ValidationReports, buildPanelFromRun, buildTree, comparePanelVersions, fetchAndAnchor, findPaperRecords, findRecordStudies, isCancelled, loadArtifact, loadPanel, panelVersions, profilePaths, runSample, testSample, traceArtifact };
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,769 @@
|
|
|
1
|
+
// src/index.ts
|
|
2
|
+
import {
|
|
3
|
+
ALIGNER,
|
|
4
|
+
alignToReference,
|
|
5
|
+
LOCATE_FLANK,
|
|
6
|
+
locateMarker,
|
|
7
|
+
assessAlignment,
|
|
8
|
+
buildPanel,
|
|
9
|
+
canonicalJson as canonicalJson2,
|
|
10
|
+
checkNumts,
|
|
11
|
+
collapseHaplotypes,
|
|
12
|
+
compareRuns,
|
|
13
|
+
DEFAULT_GATES,
|
|
14
|
+
extractLocation,
|
|
15
|
+
findGeneFeatures,
|
|
16
|
+
fullValidation,
|
|
17
|
+
gatesForRecord,
|
|
18
|
+
parseFasta,
|
|
19
|
+
parseGenBank,
|
|
20
|
+
evaluateQuery,
|
|
21
|
+
toRecordSheet,
|
|
22
|
+
toSequenceRecord,
|
|
23
|
+
validatePanel,
|
|
24
|
+
verifyBundle,
|
|
25
|
+
validatedScopeFromReport,
|
|
26
|
+
validateProfile,
|
|
27
|
+
validateResult,
|
|
28
|
+
writeFasta
|
|
29
|
+
} from "@captiveorigins/onephylogeny-core";
|
|
30
|
+
import { classifyAccessions, genomesHoldingMarkerQuery, linkedAccessions, NcbiClient, selectionQuery, studiesFromRecords } from "@captiveorigins/onephylogeny-acquire";
|
|
31
|
+
|
|
32
|
+
// src/artifacts.ts
|
|
33
|
+
import { canonicalJson, sha256Hex } from "@captiveorigins/onephylogeny-core";
|
|
34
|
+
|
|
35
|
+
// src/errors.ts
|
|
36
|
+
var EngineError = class extends Error {
|
|
37
|
+
constructor(code, message, action) {
|
|
38
|
+
super(message);
|
|
39
|
+
this.code = code;
|
|
40
|
+
this.action = action;
|
|
41
|
+
this.name = "EngineError";
|
|
42
|
+
}
|
|
43
|
+
code;
|
|
44
|
+
action;
|
|
45
|
+
};
|
|
46
|
+
var isCancelled = (e) => e instanceof EngineError && e.code === "cancelled";
|
|
47
|
+
|
|
48
|
+
// src/artifacts.ts
|
|
49
|
+
var hashText = (text) => `sha256:${sha256Hex(text)}`;
|
|
50
|
+
var artifactDir = ({ type, key }) => `artifacts/${type}/${key.replace("sha256:", "")}`;
|
|
51
|
+
function stepKey(spec) {
|
|
52
|
+
return hashText(canonicalJson({ type: spec.type, step: spec.step, inputs: spec.inputs, parameters: spec.parameters, tools: spec.tools }));
|
|
53
|
+
}
|
|
54
|
+
async function readManifest(store, ref2) {
|
|
55
|
+
const path = `${artifactDir(ref2)}/manifest.json`;
|
|
56
|
+
if (!await store.exists(path)) return null;
|
|
57
|
+
let manifest;
|
|
58
|
+
try {
|
|
59
|
+
manifest = JSON.parse(await store.read(path));
|
|
60
|
+
} catch {
|
|
61
|
+
throw new EngineError("artifact-corrupt", `The record of ${ref2.type} ${ref2.key} is damaged`);
|
|
62
|
+
}
|
|
63
|
+
if (manifest.type !== ref2.type) throw new EngineError("artifact-wrong-type", `Expected ${ref2.type}, but this artifact is ${manifest.type}`);
|
|
64
|
+
return manifest;
|
|
65
|
+
}
|
|
66
|
+
async function verifyFiles(store, manifest) {
|
|
67
|
+
for (const file of manifest.files) {
|
|
68
|
+
const path = `${artifactDir(manifest)}/${file.name}`;
|
|
69
|
+
if (!await store.exists(path)) return false;
|
|
70
|
+
if (hashText(await store.read(path)) !== file.sha256) return false;
|
|
71
|
+
}
|
|
72
|
+
return true;
|
|
73
|
+
}
|
|
74
|
+
var open = (store, manifest) => ({
|
|
75
|
+
manifest,
|
|
76
|
+
read: async (name) => {
|
|
77
|
+
if (!manifest.files.some((f) => f.name === name)) throw new EngineError("artifact-missing", `${manifest.type} has no file ${name}`);
|
|
78
|
+
return store.read(`${artifactDir(manifest)}/${name}`);
|
|
79
|
+
}
|
|
80
|
+
});
|
|
81
|
+
async function loadArtifact(store, ref2) {
|
|
82
|
+
const manifest = await readManifest(store, ref2);
|
|
83
|
+
if (!manifest) throw new EngineError("artifact-missing", `${ref2.type} ${ref2.key} doesn't exist`, "Run the earlier step again");
|
|
84
|
+
if (!await verifyFiles(store, manifest)) throw new EngineError("artifact-corrupt", `Files of ${ref2.type} ${ref2.key} have changed or gone missing`, "Run the step again to rebuild them");
|
|
85
|
+
return open(store, manifest);
|
|
86
|
+
}
|
|
87
|
+
var openVerified = open;
|
|
88
|
+
async function traceArtifact(store, ref2) {
|
|
89
|
+
const manifest = await readManifest(store, ref2);
|
|
90
|
+
if (!manifest) throw new EngineError("artifact-missing", `${ref2.type} ${ref2.key} doesn't exist`);
|
|
91
|
+
const inputs = [];
|
|
92
|
+
for (const input of manifest.inputs) inputs.push(await traceArtifact(store, input));
|
|
93
|
+
return { type: manifest.type, key: manifest.key, step: `${manifest.step.name}@${manifest.step.version}`, parameters: manifest.parameters, tools: manifest.tools, createdAt: manifest.createdAt, inputs };
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
// src/run.ts
|
|
97
|
+
import { AcquireError } from "@captiveorigins/onephylogeny-acquire";
|
|
98
|
+
var ENGINE_VERSION = "0.1.0-alpha.1";
|
|
99
|
+
var ENGINE = { name: "onephylogeny-engine", version: ENGINE_VERSION };
|
|
100
|
+
var profilePaths = (profileId) => ({
|
|
101
|
+
profile: `profiles/${profileId}/profile.json`,
|
|
102
|
+
curation: `profiles/${profileId}/curation.json`,
|
|
103
|
+
latest: `profiles/${profileId}/latest-run.txt`,
|
|
104
|
+
run: (id) => `profiles/${profileId}/runs/${id}`,
|
|
105
|
+
panel: (id, version) => `profiles/${profileId}/panels/${id}-${version}`,
|
|
106
|
+
results: `profiles/${profileId}/results`
|
|
107
|
+
});
|
|
108
|
+
var json = (v) => `${JSON.stringify(v, null, 2)}
|
|
109
|
+
`;
|
|
110
|
+
var stamp = (ctx) => (ctx.now?.() ?? /* @__PURE__ */ new Date()).toISOString();
|
|
111
|
+
var RunLog = class {
|
|
112
|
+
constructor(ctx, file) {
|
|
113
|
+
this.ctx = ctx;
|
|
114
|
+
this.file = file;
|
|
115
|
+
}
|
|
116
|
+
ctx;
|
|
117
|
+
file;
|
|
118
|
+
events = [];
|
|
119
|
+
pending = Promise.resolve();
|
|
120
|
+
add(level, step, message, data) {
|
|
121
|
+
const event = { at: stamp(this.ctx), level, step, message, ...data ? { data } : {} };
|
|
122
|
+
this.events.push(event);
|
|
123
|
+
this.ctx.onEvent?.(event);
|
|
124
|
+
const { store } = this.ctx;
|
|
125
|
+
const write = store.append ? () => store.append(this.file, `${JSON.stringify(event)}
|
|
126
|
+
`) : () => store.write(this.file, `${this.events.map((e) => JSON.stringify(e)).join("\n")}
|
|
127
|
+
`);
|
|
128
|
+
this.pending = this.pending.then(write).catch(() => {
|
|
129
|
+
});
|
|
130
|
+
}
|
|
131
|
+
flush() {
|
|
132
|
+
return this.pending;
|
|
133
|
+
}
|
|
134
|
+
};
|
|
135
|
+
async function runStep(scope, spec) {
|
|
136
|
+
const { store } = scope.ctx;
|
|
137
|
+
const tools = spec.tools ?? [];
|
|
138
|
+
const key = stepKey({ type: spec.type, step: spec.step, inputs: spec.inputs, parameters: spec.parameters, tools });
|
|
139
|
+
const ref2 = { type: spec.type, key };
|
|
140
|
+
scope.checkpoint();
|
|
141
|
+
const existing = await readManifest(store, ref2);
|
|
142
|
+
if (existing && await verifyFiles(store, existing)) {
|
|
143
|
+
scope.log("info", spec.step.name, "Reused earlier output: inputs and settings are unchanged", { status: "reused", key });
|
|
144
|
+
scope.run.artifacts.push({ ...ref2, reused: true });
|
|
145
|
+
return openVerified(store, existing);
|
|
146
|
+
}
|
|
147
|
+
const dir = artifactDir(ref2);
|
|
148
|
+
scope.log("info", spec.step.name, "Running", { status: "running", key });
|
|
149
|
+
const files = [];
|
|
150
|
+
const put = async (name, text) => {
|
|
151
|
+
await store.write(`${dir}/${name}`, text);
|
|
152
|
+
files.push({ name, sha256: hashText(text) });
|
|
153
|
+
};
|
|
154
|
+
const register = async (name) => {
|
|
155
|
+
files.push({ name, sha256: hashText(await store.read(`${dir}/${name}`)) });
|
|
156
|
+
};
|
|
157
|
+
await spec.run(put, dir, register);
|
|
158
|
+
scope.checkpoint();
|
|
159
|
+
if (spec.ensure) {
|
|
160
|
+
const problem = await spec.ensure((name) => store.read(`${dir}/${name}`));
|
|
161
|
+
if (problem) throw new EngineError("step-output-invalid", `${spec.step.name} produced invalid output: ${problem}`);
|
|
162
|
+
}
|
|
163
|
+
const manifest = { schemaVersion: 1, type: spec.type, key, step: spec.step, inputs: spec.inputs, parameters: spec.parameters, tools, files, createdAt: stamp(scope.ctx) };
|
|
164
|
+
await store.write(`${dir}/manifest.json`, json(manifest));
|
|
165
|
+
scope.log("info", spec.step.name, "Finished", { status: "finished", key });
|
|
166
|
+
scope.run.artifacts.push({ ...ref2, reused: false });
|
|
167
|
+
return openVerified(store, manifest);
|
|
168
|
+
}
|
|
169
|
+
var newRunId = (ctx, kind) => `${stamp(ctx).replace(/[:.]/g, "-")}-${kind}`;
|
|
170
|
+
var toEngineError = (e) => {
|
|
171
|
+
if (e instanceof AcquireError) return new EngineError("source-failed", e.message, "Check your connection and try again. Anything already downloaded is kept");
|
|
172
|
+
return e;
|
|
173
|
+
};
|
|
174
|
+
async function withRun(ctx, profile, kind, work) {
|
|
175
|
+
const P = profilePaths(profile.id);
|
|
176
|
+
const id = newRunId(ctx, kind);
|
|
177
|
+
const dir = P.run(id);
|
|
178
|
+
const run = { schemaVersion: 1, id, kind, profile, artifacts: [], tools: [ENGINE], status: "running", startedAt: stamp(ctx), events: [], outputs: [], diagnostics: [] };
|
|
179
|
+
await ctx.store.write(`${dir}/run.json`, json(run));
|
|
180
|
+
const log = new RunLog(ctx, `${dir}/events.jsonl`);
|
|
181
|
+
const checkpoint = () => {
|
|
182
|
+
if (ctx.signal?.aborted) throw new EngineError("cancelled", "Cancelled");
|
|
183
|
+
};
|
|
184
|
+
let lastBreath = Date.now();
|
|
185
|
+
const scope = {
|
|
186
|
+
ctx,
|
|
187
|
+
run,
|
|
188
|
+
dir,
|
|
189
|
+
log: (level, step, message, data) => log.add(level, step, message, data),
|
|
190
|
+
checkpoint,
|
|
191
|
+
breathe: async () => {
|
|
192
|
+
checkpoint();
|
|
193
|
+
if (Date.now() - lastBreath > 30) {
|
|
194
|
+
await new Promise((r) => setTimeout(r, 0));
|
|
195
|
+
lastBreath = Date.now();
|
|
196
|
+
checkpoint();
|
|
197
|
+
}
|
|
198
|
+
},
|
|
199
|
+
step: (spec) => runStep(scope, spec)
|
|
200
|
+
};
|
|
201
|
+
const finish = async (status) => {
|
|
202
|
+
await log.flush();
|
|
203
|
+
run.status = status;
|
|
204
|
+
run.finishedAt = stamp(ctx);
|
|
205
|
+
run.events = log.events;
|
|
206
|
+
await ctx.store.write(`${dir}/run.json`, json(run));
|
|
207
|
+
};
|
|
208
|
+
try {
|
|
209
|
+
const value = await work(scope);
|
|
210
|
+
await finish("succeeded");
|
|
211
|
+
return { value, run };
|
|
212
|
+
} catch (raw) {
|
|
213
|
+
const e = toEngineError(raw);
|
|
214
|
+
const cancelled = e instanceof EngineError && e.code === "cancelled";
|
|
215
|
+
scope.log(cancelled ? "warning" : "error", "run", cancelled ? "Cancelled. Finished steps are kept and will be reused" : e instanceof Error ? e.message : String(e));
|
|
216
|
+
await finish(cancelled ? "cancelled" : "failed").catch(() => {
|
|
217
|
+
});
|
|
218
|
+
throw e;
|
|
219
|
+
}
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
// src/index.ts
|
|
223
|
+
var cachePath = (version) => `cache/genbank/${version.replace(/[^A-Za-z0-9._-]/g, "_")}.gb`;
|
|
224
|
+
var ref = (a) => ({ type: a.manifest.type, key: a.manifest.key });
|
|
225
|
+
var artifactOf = (run, type) => run.artifacts.find((a) => a.type === type);
|
|
226
|
+
function geneSequence(rec, profile) {
|
|
227
|
+
const marker = profile.selection.marker;
|
|
228
|
+
const names = [marker.gene, ...marker.synonyms ?? [], ...marker.products ?? []].filter(Boolean);
|
|
229
|
+
if (marker.kind === "gene") {
|
|
230
|
+
const feature = findGeneFeatures(rec, names)[0];
|
|
231
|
+
if (feature) {
|
|
232
|
+
try {
|
|
233
|
+
return { sequence: extractLocation(rec.sequence, feature.location), from: "annotation" };
|
|
234
|
+
} catch {
|
|
235
|
+
}
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
return { sequence: rec.sequence, from: "whole-record" };
|
|
239
|
+
}
|
|
240
|
+
var where = (p) => `bases ${p.start + 1} to ${p.end} on the ${p.strand === "+" ? "given" : "reverse"} strand${p.wrapsOrigin ? ", across the origin" : ""}`;
|
|
241
|
+
function locatedSequence(rec, reference, profile, found) {
|
|
242
|
+
const factor = panelGates(profile).containerRecordFactor ?? Infinity;
|
|
243
|
+
if (found.from !== "whole-record" || profile.selection.marker.kind !== "gene" || found.sequence.length < factor * reference.length) return found;
|
|
244
|
+
const location = locateMarker(found.sequence, reference, { ...rec.circular ? { circular: true } : {} });
|
|
245
|
+
if (location.status === "located") return { sequence: location.sequence, from: "located", located: location.place };
|
|
246
|
+
if (location.status === "multiple") return { ...found, manyPlaces: location.places };
|
|
247
|
+
return found;
|
|
248
|
+
}
|
|
249
|
+
async function searchStep(scope, profile, client, options) {
|
|
250
|
+
const include = profile.selection.include;
|
|
251
|
+
const term = include?.length ? `chosen accessions, ${include.length} records, ${hashText([...include].sort().join("\n"))}` : selectionQuery(profile.selection);
|
|
252
|
+
scope.log("info", "search", include?.length ? "Using the accessions chosen in a Study" : "Searching NCBI", { term });
|
|
253
|
+
const found = include?.length ? include : await client.searchAccessions(term);
|
|
254
|
+
if (!found.length && !options.allowEmpty) throw new EngineError("empty-search", "NCBI found no records for this taxon and marker", "Check the taxon, gene names and organelle in Setup");
|
|
255
|
+
const genomes = include?.length ? null : genomesHoldingMarkerQuery(profile.selection);
|
|
256
|
+
if (genomes) {
|
|
257
|
+
try {
|
|
258
|
+
const missed = (await client.searchAccessions(genomes)).filter((a) => !found.includes(a));
|
|
259
|
+
if (missed.length) scope.log("warning", "search", `${missed.length} mitochondrial genomes of these taxa aren't in this search, probably because they don't annotate the marker. They may hold it, and they aren't used unless chosen in a Study, which cuts the marker out by similarity: ${missed.join(", ")}`, { missed });
|
|
260
|
+
} catch (e) {
|
|
261
|
+
if (isCancelled(e)) throw e;
|
|
262
|
+
scope.log("warning", "search", `Couldn't check for mitochondrial genomes the search missed: ${e instanceof Error ? e.message : String(e)}`);
|
|
263
|
+
}
|
|
264
|
+
}
|
|
265
|
+
const extra = [profile.reference.accession, ...profile.selection.outgroups.map((o) => o.accession).filter((a) => Boolean(a))];
|
|
266
|
+
const accessions = [.../* @__PURE__ */ new Set([...found, ...extra])].sort();
|
|
267
|
+
scope.log("info", "search", `Found ${found.length} records, plus ${extra.length} named in the profile`, { found: found.length });
|
|
268
|
+
const artifact = await scope.step({
|
|
269
|
+
type: "accession-list",
|
|
270
|
+
step: { name: "search", version: 1 },
|
|
271
|
+
inputs: [],
|
|
272
|
+
parameters: { term, accessions: hashText(accessions.join("\n")) },
|
|
273
|
+
run: async (put) => {
|
|
274
|
+
await put("accessions.json", json({ term, accessions }));
|
|
275
|
+
}
|
|
276
|
+
});
|
|
277
|
+
return { artifact, accessions, term };
|
|
278
|
+
}
|
|
279
|
+
function rawStep(scope, list, accessions, client, batchSize) {
|
|
280
|
+
const { store } = scope.ctx;
|
|
281
|
+
return scope.step({
|
|
282
|
+
type: "raw-records",
|
|
283
|
+
step: { name: "fetch", version: 1 },
|
|
284
|
+
inputs: [ref(list)],
|
|
285
|
+
parameters: {},
|
|
286
|
+
run: async (put) => {
|
|
287
|
+
const texts = /* @__PURE__ */ new Map();
|
|
288
|
+
const missing = [];
|
|
289
|
+
for (const version of accessions) {
|
|
290
|
+
const path = cachePath(version);
|
|
291
|
+
if (await store.exists(path)) texts.set(version, await store.read(path));
|
|
292
|
+
else missing.push(version);
|
|
293
|
+
}
|
|
294
|
+
scope.log("info", "fetch", `${texts.size} records already downloaded, ${missing.length} to fetch`, { done: texts.size, total: accessions.length });
|
|
295
|
+
for (let i = 0; i < missing.length; i += batchSize) {
|
|
296
|
+
scope.checkpoint();
|
|
297
|
+
const batch = missing.slice(i, i + batchSize);
|
|
298
|
+
const { text, records } = await client.fetchGenBankText(batch);
|
|
299
|
+
const blocks = text.split(/^\/\/\s*$/m).filter((b) => b.includes("LOCUS"));
|
|
300
|
+
for (const [n, rec] of records.entries()) {
|
|
301
|
+
const raw = `${blocks[n].trimStart()}//
|
|
302
|
+
`;
|
|
303
|
+
const key = rec.version || rec.accession;
|
|
304
|
+
await store.write(cachePath(key), raw);
|
|
305
|
+
texts.set(key, raw);
|
|
306
|
+
}
|
|
307
|
+
scope.log("info", "fetch", `Fetched ${Math.min(i + batchSize, missing.length)} of ${missing.length}`, { done: texts.size, total: accessions.length });
|
|
308
|
+
}
|
|
309
|
+
const index = Object.fromEntries([...texts].map(([k, t]) => [k, hashText(t)]).sort(([a], [b]) => String(a).localeCompare(String(b))));
|
|
310
|
+
await put("index.json", json(index));
|
|
311
|
+
},
|
|
312
|
+
ensure: async (read) => Object.keys(JSON.parse(await read("index.json"))).length ? null : "no records were fetched"
|
|
313
|
+
});
|
|
314
|
+
}
|
|
315
|
+
var markerParameters = (profile) => ({ marker: profile.selection.marker, reference: profile.reference?.accession });
|
|
316
|
+
function anchorStep(scope, profile, raw) {
|
|
317
|
+
const { store } = scope.ctx;
|
|
318
|
+
return scope.step({
|
|
319
|
+
type: "anchored-records",
|
|
320
|
+
step: { name: "anchor", version: 2 },
|
|
321
|
+
inputs: [ref(raw)],
|
|
322
|
+
parameters: markerParameters(profile),
|
|
323
|
+
tools: [ALIGNER],
|
|
324
|
+
run: async (put) => {
|
|
325
|
+
const index = JSON.parse(await raw.read("index.json"));
|
|
326
|
+
const parsed = /* @__PURE__ */ new Map();
|
|
327
|
+
for (const [version, sha] of Object.entries(index)) {
|
|
328
|
+
const text = await store.read(cachePath(version));
|
|
329
|
+
if (hashText(text) !== sha) throw new EngineError("raw-file-changed", `The downloaded file for ${version} changed after it was fetched`, `Delete ${cachePath(version)} and fetch again`);
|
|
330
|
+
const rec = parseGenBank(text)[0];
|
|
331
|
+
if (!rec) throw new EngineError("raw-file-changed", `The downloaded file for ${version} has no record in it`, `Delete ${cachePath(version)} and fetch again`);
|
|
332
|
+
parsed.set(version, rec);
|
|
333
|
+
}
|
|
334
|
+
const accession = profile.reference.accession;
|
|
335
|
+
const refRec = [...parsed.values()].find((r) => r.version === accession || r.accession === accession);
|
|
336
|
+
if (!refRec) throw new EngineError("reference-missing", `The reference ${accession} wasn't retrieved`, "Check the reference accession in Setup");
|
|
337
|
+
const reference = geneSequence(refRec, profile).sequence;
|
|
338
|
+
scope.log("info", "anchor", `Reference ${refRec.version}: ${reference.length} columns`);
|
|
339
|
+
const anchored = [];
|
|
340
|
+
const records = [];
|
|
341
|
+
let done = 0;
|
|
342
|
+
for (const [version, rec] of parsed) {
|
|
343
|
+
await scope.breathe();
|
|
344
|
+
const { sequence, from, located, manyPlaces } = locatedSequence(rec, reference, profile, geneSequence(rec, profile));
|
|
345
|
+
anchored.push({
|
|
346
|
+
...located ? { located } : {},
|
|
347
|
+
...manyPlaces ? { manyPlaces } : {},
|
|
348
|
+
record: rec.version || version,
|
|
349
|
+
...rec.organism ? { organism: rec.organism } : {},
|
|
350
|
+
alignment: alignToReference(sequence, reference),
|
|
351
|
+
extractedFrom: from,
|
|
352
|
+
length: sequence.length,
|
|
353
|
+
ambiguous: (sequence.match(/[^ACGTacgt]/g) ?? []).length,
|
|
354
|
+
unverified: rec.definition.startsWith("UNVERIFIED")
|
|
355
|
+
});
|
|
356
|
+
records.push(toSequenceRecord(rec));
|
|
357
|
+
if (++done % 100 === 0) scope.log("debug", "anchor", `Anchored ${done} of ${parsed.size}`, { done, total: parsed.size });
|
|
358
|
+
}
|
|
359
|
+
await put("reference.fasta", writeFasta([{ id: refRec.version, sequence: reference }]));
|
|
360
|
+
await put("anchored.json", JSON.stringify(anchored));
|
|
361
|
+
await put("records.json", JSON.stringify(records));
|
|
362
|
+
},
|
|
363
|
+
ensure: async (read) => parseFasta(await read("reference.fasta"))[0]?.sequence.length ? null : "the reference has no sequence"
|
|
364
|
+
});
|
|
365
|
+
}
|
|
366
|
+
var panelGates = (profile) => ({ ...DEFAULT_GATES, ...profile.gates, frameErrors: true, ...profile.reference?.codonStart !== void 0 ? { codonStart: profile.reference.codonStart } : {}, ...profile.reference?.geneticCode ? { geneticCode: profile.reference.geneticCode } : {} });
|
|
367
|
+
function gateStep(scope, profile, anchoredArtifact) {
|
|
368
|
+
const { filters, exclude } = profile.selection;
|
|
369
|
+
const gates = panelGates(profile);
|
|
370
|
+
return scope.step({
|
|
371
|
+
type: "gated-records",
|
|
372
|
+
step: { name: "gate", version: 2 },
|
|
373
|
+
inputs: [ref(anchoredArtifact)],
|
|
374
|
+
parameters: { filters, exclude, gates },
|
|
375
|
+
run: async (put) => {
|
|
376
|
+
const excluded = new Map(exclude.map((e) => [e.accession, e.reason]));
|
|
377
|
+
const anchored = JSON.parse(await anchoredArtifact.read("anchored.json"));
|
|
378
|
+
const records = JSON.parse(await anchoredArtifact.read("records.json"));
|
|
379
|
+
const outcomes = [];
|
|
380
|
+
const aligned = [];
|
|
381
|
+
for (const a of anchored) {
|
|
382
|
+
const assessed = assessAlignment(a.alignment, gatesForRecord(gates, { wholeRecord: a.extractedFrom === "whole-record", length: a.length, referenceLength: a.alignment?.aligned.length ?? 0 }));
|
|
383
|
+
const onlyFlank = a.located && a.alignment && a.alignment.clipped.start <= LOCATE_FLANK && a.alignment.clipped.end <= LOCATE_FLANK;
|
|
384
|
+
const diagnostics = onlyFlank ? assessed.filter((d) => d.code !== "clipped-ends") : assessed;
|
|
385
|
+
const fail = (code, message) => diagnostics.push({ severity: "error", code, message });
|
|
386
|
+
const reason = excluded.get(a.record) ?? excluded.get(a.record.split(".")[0]);
|
|
387
|
+
if (reason) fail("excluded-by-profile", reason);
|
|
388
|
+
if (filters.minLength !== void 0 && a.length < filters.minLength) fail("too-short", `${a.length} bp is under the minimum of ${filters.minLength}`);
|
|
389
|
+
if (filters.maxLength !== void 0 && a.length > filters.maxLength) fail("too-long", `${a.length} bp is over the maximum of ${filters.maxLength}`);
|
|
390
|
+
if (filters.maxAmbiguous !== void 0 && a.ambiguous > filters.maxAmbiguous) fail("too-ambiguous", `${a.ambiguous} ambiguous bases is over the maximum of ${filters.maxAmbiguous}`);
|
|
391
|
+
if (a.unverified) fail("unverified", "NCBI marks this record UNVERIFIED");
|
|
392
|
+
if (a.located) diagnostics.push({ severity: "info", code: "located-by-similarity", message: `The record doesn't annotate this marker, so it was cut out by similarity: ${where(a.located)}. ${a.located.seedHits} shared seeds` });
|
|
393
|
+
if (a.manyPlaces) fail("marker-in-several-places", `The marker looks present in ${a.manyPlaces.length} places, so none was chosen: ${a.manyPlaces.map(where).join("; ")}`);
|
|
394
|
+
const passed = !diagnostics.some((d) => d.severity === "error");
|
|
395
|
+
outcomes.push({ record: a.record, ...a.organism ? { organism: a.organism } : {}, status: passed ? "passed" : "excluded", diagnostics, columns: a.alignment ? a.alignment.matches + a.alignment.mismatches : 0, identity: a.alignment?.identity ?? 0, extractedFrom: a.extractedFrom });
|
|
396
|
+
if (a.alignment && passed) aligned.push({ id: a.record, sequence: a.alignment.aligned });
|
|
397
|
+
}
|
|
398
|
+
const passing = outcomes.filter((o) => o.status === "passed").length;
|
|
399
|
+
scope.log("info", "gate", `${passing} records passed, ${outcomes.length - passing} excluded`);
|
|
400
|
+
const counts = /* @__PURE__ */ new Map();
|
|
401
|
+
for (const o of outcomes) for (const d of o.diagnostics) counts.set(d.code, (counts.get(d.code) ?? 0) + 1);
|
|
402
|
+
for (const [code, n] of counts) scope.log("info", "gate", `${code}: ${n}`);
|
|
403
|
+
await put("outcomes.json", json(outcomes));
|
|
404
|
+
await put("aligned.fasta", writeFasta(aligned));
|
|
405
|
+
await put("reference.fasta", await anchoredArtifact.read("reference.fasta"));
|
|
406
|
+
await put("records.csv", toRecordSheet(records, { checks: outcomes.map((o) => ({ record: o.record, status: o.status, reasons: o.diagnostics.map((d) => d.message) })) }));
|
|
407
|
+
}
|
|
408
|
+
});
|
|
409
|
+
}
|
|
410
|
+
async function fetchAndAnchor(ctx, profile, options = {}) {
|
|
411
|
+
const errors = validateProfile(profile).filter((d) => d.severity === "error");
|
|
412
|
+
if (errors.length) throw new EngineError("invalid-profile", `The profile can't run: ${errors.map((e) => e.message).join("; ")}`, "Fix it in Setup");
|
|
413
|
+
if (!profile.reference?.accession) throw new EngineError("invalid-profile", "The profile needs a reference accession to anchor records to", "Add one in Setup");
|
|
414
|
+
const P = profilePaths(profile.id);
|
|
415
|
+
const { value, run } = await withRun(ctx, profile, "fetch", async (scope) => {
|
|
416
|
+
scope.run.tools.push(ALIGNER);
|
|
417
|
+
const client = new NcbiClient({ ...ctx.ncbi, log: (level, message, data) => scope.log(level, "ncbi", message, data) });
|
|
418
|
+
const { artifact: list, accessions, term } = await searchStep(scope, profile, client, options);
|
|
419
|
+
const raw = await rawStep(scope, list, accessions, client, options.batchSize ?? 100);
|
|
420
|
+
const anchored = await anchorStep(scope, profile, raw);
|
|
421
|
+
const gated = await gateStep(scope, profile, anchored);
|
|
422
|
+
const records = JSON.parse(await anchored.read("records.json"));
|
|
423
|
+
const outcomes = JSON.parse(await gated.read("outcomes.json"));
|
|
424
|
+
const index = JSON.parse(await raw.read("index.json"));
|
|
425
|
+
const snapshot = {
|
|
426
|
+
schemaVersion: 1,
|
|
427
|
+
id: scope.run.id,
|
|
428
|
+
createdAt: scope.run.startedAt,
|
|
429
|
+
sources: [{ name: "NCBI nuccore", query: term, retrievedAt: scope.run.startedAt }],
|
|
430
|
+
records,
|
|
431
|
+
rawFiles: records.map((r) => ({ path: cachePath(r.version || r.accession), sha256: index[r.version || r.accession] ?? "" }))
|
|
432
|
+
};
|
|
433
|
+
snapshot.checksum = hashText(canonicalJson2(snapshot));
|
|
434
|
+
scope.run.snapshot = { id: scope.run.id, checksum: snapshot.checksum };
|
|
435
|
+
const publish = async (name, text) => {
|
|
436
|
+
await ctx.store.write(`${scope.dir}/${name}`, text);
|
|
437
|
+
scope.run.outputs.push({ path: name, sha256: hashText(text) });
|
|
438
|
+
};
|
|
439
|
+
await publish("snapshot.json", json(snapshot));
|
|
440
|
+
for (const name of ["records.csv", "outcomes.json", "aligned.fasta", "reference.fasta"]) await publish(name, await gated.read(name));
|
|
441
|
+
return { records, outcomes };
|
|
442
|
+
});
|
|
443
|
+
await ctx.store.write(P.latest, run.id);
|
|
444
|
+
return { run, records: value.records, outcomes: value.outcomes };
|
|
445
|
+
}
|
|
446
|
+
function spanOf(alignment) {
|
|
447
|
+
if (!alignment) return void 0;
|
|
448
|
+
const start = alignment.aligned.search(/[^-]/);
|
|
449
|
+
if (start < 0) return void 0;
|
|
450
|
+
return [start, alignment.aligned.length - (alignment.aligned.match(/-*$/)?.[0].length ?? 0)];
|
|
451
|
+
}
|
|
452
|
+
async function studyRecords(ctx, client, accessions, onProgress) {
|
|
453
|
+
const stop = () => {
|
|
454
|
+
if (ctx.signal?.aborted) throw new EngineError("cancelled", "Cancelled");
|
|
455
|
+
};
|
|
456
|
+
const out = [];
|
|
457
|
+
const missing = [];
|
|
458
|
+
for (const a of accessions) {
|
|
459
|
+
stop();
|
|
460
|
+
if (await ctx.store.exists(cachePath(a))) {
|
|
461
|
+
const r = parseGenBank(await ctx.store.read(cachePath(a)))[0];
|
|
462
|
+
if (r) {
|
|
463
|
+
out.push(r);
|
|
464
|
+
continue;
|
|
465
|
+
}
|
|
466
|
+
}
|
|
467
|
+
missing.push(a);
|
|
468
|
+
}
|
|
469
|
+
for (let i = 0; i < missing.length; i += 100) {
|
|
470
|
+
stop();
|
|
471
|
+
out.push(...(await client.fetchGenBankText(missing.slice(i, i + 100))).records);
|
|
472
|
+
onProgress?.(out.length, accessions.length);
|
|
473
|
+
}
|
|
474
|
+
return out;
|
|
475
|
+
}
|
|
476
|
+
async function studyReference(ctx, client, profile) {
|
|
477
|
+
const accession = profile.reference?.accession;
|
|
478
|
+
if (!accession) throw new EngineError("invalid-profile", "A Study places records on the reference, and this profile has none", "Add a reference accession in Setup");
|
|
479
|
+
const [rec] = await studyRecords(ctx, client, [accession]);
|
|
480
|
+
if (!rec) throw new EngineError("reference-missing", `The reference ${accession} wasn't retrieved`, "Check the reference accession in Setup");
|
|
481
|
+
return geneSequence(rec, profile).sequence;
|
|
482
|
+
}
|
|
483
|
+
var countryOf = (rec) => toSequenceRecord(rec).geoLocName?.split(":")[0]?.trim() || void 0;
|
|
484
|
+
async function placeRecords(ctx, profile, records, reference, classified) {
|
|
485
|
+
const placed = /* @__PURE__ */ new Map();
|
|
486
|
+
for (const rec of records) {
|
|
487
|
+
if (ctx.signal?.aborted) throw new EngineError("cancelled", "Cancelled");
|
|
488
|
+
await new Promise((r) => setTimeout(r, 0));
|
|
489
|
+
const hasMarker = classified.hasMarker.has(rec.version);
|
|
490
|
+
const span = hasMarker ? spanOf(alignToReference(geneSequence(rec, profile).sequence, reference)) : void 0;
|
|
491
|
+
const country = countryOf(rec);
|
|
492
|
+
placed.set(rec.version, { accession: rec.version, organism: rec.organism, length: rec.sequence.length, title: rec.definition, inTarget: classified.inTarget.has(rec.version), hasMarker, ...country ? { country } : {}, ...span ? { span } : {} });
|
|
493
|
+
}
|
|
494
|
+
return placed;
|
|
495
|
+
}
|
|
496
|
+
async function findRecordStudies(ctx, profile, onProgress) {
|
|
497
|
+
const client = new NcbiClient(ctx.ncbi);
|
|
498
|
+
const query = selectionQuery(profile.selection);
|
|
499
|
+
const genomes = genomesHoldingMarkerQuery(profile.selection);
|
|
500
|
+
const accessions = [.../* @__PURE__ */ new Set([...await client.searchAccessions(query), ...genomes ? await client.searchAccessions(genomes) : []])];
|
|
501
|
+
const reference = await studyReference(ctx, client, profile);
|
|
502
|
+
const records = await studyRecords(ctx, client, accessions, onProgress);
|
|
503
|
+
const all = new Set(records.map((r) => r.version));
|
|
504
|
+
const placed = await placeRecords(ctx, profile, records, reference, { inTarget: all, hasMarker: all });
|
|
505
|
+
return { papers: studiesFromRecords(records, (r) => placed.get(r.version)), columns: reference.length, query };
|
|
506
|
+
}
|
|
507
|
+
async function findPaperRecords(ctx, profile, pmid) {
|
|
508
|
+
const client = new NcbiClient(ctx.ncbi);
|
|
509
|
+
const accessions = await linkedAccessions(client, pmid);
|
|
510
|
+
const reference = await studyReference(ctx, client, profile);
|
|
511
|
+
if (!accessions.length) return { linked: [], columns: reference.length };
|
|
512
|
+
const classified = await classifyAccessions(client, accessions, profile.selection);
|
|
513
|
+
const records = await studyRecords(ctx, client, accessions);
|
|
514
|
+
return { linked: [...(await placeRecords(ctx, profile, records, reference, classified)).values()], columns: reference.length };
|
|
515
|
+
}
|
|
516
|
+
var defaultValidate = async (input) => fullValidation(input);
|
|
517
|
+
async function buildPanelFromRun(ctx, profile, curation, id) {
|
|
518
|
+
const settings = profile.panel;
|
|
519
|
+
if (!settings) throw new EngineError("invalid-profile", "The profile has no panel settings", "Add groups in the Panel tab");
|
|
520
|
+
if (!settings.rules) throw new EngineError("invalid-profile", "This profile predates evaluation rules, so a panel built from it would use values nobody chose", "Set the rules in the Panel tab and give the panel a new version");
|
|
521
|
+
const P = profilePaths(profile.id);
|
|
522
|
+
const sourceId = id ?? (await ctx.store.exists(P.latest) ? (await ctx.store.read(P.latest)).trim() : "");
|
|
523
|
+
if (!sourceId) throw new EngineError("run-missing", "No finished fetch run exists yet", "Fetch records first");
|
|
524
|
+
const source = JSON.parse(await ctx.store.read(`${P.run(sourceId)}/run.json`));
|
|
525
|
+
const gatedRef = artifactOf(source, "gated-records");
|
|
526
|
+
if (!gatedRef) throw new EngineError("run-missing", `Run ${sourceId} didn't finish fetching`, "Fetch records again");
|
|
527
|
+
const { value, run } = await withRun(ctx, profile, "panel", async (scope) => {
|
|
528
|
+
scope.run.tools.push(ALIGNER);
|
|
529
|
+
const gated = await loadArtifact(ctx.store, { type: "gated-records", key: gatedRef.key });
|
|
530
|
+
const aligned = parseFasta(await gated.read("aligned.fasta"));
|
|
531
|
+
const reference = parseFasta(await gated.read("reference.fasta"))[0];
|
|
532
|
+
const groupIds = new Set(settings.groups.map((g) => g.id));
|
|
533
|
+
const records = JSON.parse(await (await loadArtifact(ctx.store, gated.manifest.inputs[0])).read("records.json"));
|
|
534
|
+
const studyOf = new Map(records.map((r) => [r.version, r.publications.find((p) => p.title && p.title !== "Direct Submission")?.title]));
|
|
535
|
+
const members = [];
|
|
536
|
+
const diagnostics = [];
|
|
537
|
+
let unassigned = 0;
|
|
538
|
+
for (const e of aligned) {
|
|
539
|
+
const decision = curation.records[e.id];
|
|
540
|
+
if (decision?.exclude) continue;
|
|
541
|
+
if (!decision?.group) {
|
|
542
|
+
unassigned++;
|
|
543
|
+
continue;
|
|
544
|
+
}
|
|
545
|
+
if (!groupIds.has(decision.group)) {
|
|
546
|
+
diagnostics.push({ severity: "error", code: "unknown-group", message: `${e.id} names group ${decision.group}, which isn't in the panel settings`, ids: [e.id] });
|
|
547
|
+
continue;
|
|
548
|
+
}
|
|
549
|
+
const study = decision.study ?? studyOf.get(e.id);
|
|
550
|
+
members.push({ record: e.id, group: decision.group, aligned: e.sequence, provenanceClass: decision.provenanceClass ?? "unknown", ...decision.use ? { use: decision.use } : {}, ...decision.site ? { block: decision.site } : {}, ...decision.originSource ? { originSource: decision.originSource } : {}, ...study ? { study } : {} });
|
|
551
|
+
}
|
|
552
|
+
if (!members.length && !diagnostics.some((d) => d.severity === "error")) throw new EngineError("empty-panel", "No records are assigned to a group, so there is nothing to build", "Assign records to groups in the Records tab");
|
|
553
|
+
const memberIds = new Set(members.map((m) => m.record));
|
|
554
|
+
const citations = /* @__PURE__ */ new Map();
|
|
555
|
+
for (const r of records) if (memberIds.has(r.version)) {
|
|
556
|
+
for (const p of r.publications) if (p.title) citations.set(p.title, { citation: [p.authors, p.title, p.journal].filter(Boolean).join(". ") });
|
|
557
|
+
}
|
|
558
|
+
const numtPath = `profiles/${profile.id}/numt-screen/numt-library.fasta`;
|
|
559
|
+
const numtCopies = await ctx.store.exists(numtPath) ? parseFasta(await ctx.store.read(numtPath)).map((e) => ({ id: e.id, aligned: e.sequence })) : [];
|
|
560
|
+
const input = {
|
|
561
|
+
settings,
|
|
562
|
+
taxa: profile.selection.taxa,
|
|
563
|
+
marker: profile.selection.marker,
|
|
564
|
+
reference: { accession: reference.id, length: reference.sequence.length, ...profile.reference?.codonStart !== void 0 ? { codonStart: profile.reference.codonStart } : {}, ...profile.reference?.geneticCode ? { geneticCode: profile.reference.geneticCode } : {} },
|
|
565
|
+
sources: [...citations.values()],
|
|
566
|
+
builtBy: gated.manifest.key,
|
|
567
|
+
referenceSequence: reference.sequence,
|
|
568
|
+
builtWith: [ENGINE, ALIGNER],
|
|
569
|
+
// The checks that actually ran on these records, even if the profile has changed since.
|
|
570
|
+
gates: gated.manifest.parameters.gates,
|
|
571
|
+
...numtCopies.length ? { numtCopies } : {},
|
|
572
|
+
// From the fetch these records came from, not today's profile, which may have moved to another Study.
|
|
573
|
+
...source.profile.selection.fromStudy ? { study: source.profile.selection.fromStudy } : {}
|
|
574
|
+
};
|
|
575
|
+
if (settings.validatedScope.length) diagnostics.push({ severity: "warning", code: "hand-entered-scope-ignored", message: "The scope written in the panel settings is ignored. A panel's scope now comes only from claims that pass validation", path: "panel.validatedScope" });
|
|
576
|
+
const draft = buildPanel({ ...input, members });
|
|
577
|
+
diagnostics.push(...validatePanel(draft.panel).filter((d) => d.code !== "not-validated"));
|
|
578
|
+
const empty = { scheme: "leave-one-block-out", tested: 0, skipped: 0, groups: [], untestedGroups: [], diagnostics: [] };
|
|
579
|
+
const hasErrors = diagnostics.some((d) => d.severity === "error");
|
|
580
|
+
if (hasErrors) return { bundle: draft, diagnostics, validation: { blocks: empty, groups: { ...empty, scheme: "leave-one-group-out" }, ladder: [], claims: [], controls: { shuffles: 0, simulations: 0, seed: 0 } }, unassigned, published: false };
|
|
581
|
+
const fingerprint = hashText(canonicalJson2({ settings: { ...settings, version: "", id: "" }, members: members.map(({ aligned: aligned2, ...m }) => ({ ...m, aligned: hashText(aligned2) })) }));
|
|
582
|
+
const validationArtifact = await scope.step({
|
|
583
|
+
type: "validation",
|
|
584
|
+
step: { name: "validate", version: 3 },
|
|
585
|
+
inputs: [ref(gated)],
|
|
586
|
+
parameters: { fingerprint },
|
|
587
|
+
run: async (put) => {
|
|
588
|
+
const reports = await (ctx.compute?.validate ?? defaultValidate)({ ...input, members }, ctx.signal);
|
|
589
|
+
await put("validation.json", json(reports));
|
|
590
|
+
}
|
|
591
|
+
});
|
|
592
|
+
const validation = JSON.parse(await validationArtifact.read("validation.json"));
|
|
593
|
+
const bundle = buildPanel({ ...input, members, claims: validatedScopeFromReport(validation.claims) });
|
|
594
|
+
if (!bundle.panel.claims.length) diagnostics.push({ severity: "warning", code: "not-validated", message: "No claim passed validation, so this panel has no validated scope", path: "validatedScope" });
|
|
595
|
+
const out = P.panel(bundle.panel.id, bundle.panel.version);
|
|
596
|
+
if (await ctx.store.exists(`${out}/panel.json`)) {
|
|
597
|
+
const existing = JSON.parse(await ctx.store.read(`${out}/panel.json`));
|
|
598
|
+
if (existing.checksum !== bundle.panel.checksum) throw new EngineError("panel-version-exists", `Panel ${bundle.panel.id} version ${bundle.panel.version} already exists with different content`, "Give this build a new version, so earlier results stay true");
|
|
599
|
+
}
|
|
600
|
+
const panelArtifact = await scope.step({
|
|
601
|
+
type: "panel",
|
|
602
|
+
step: { name: "panel", version: 2 },
|
|
603
|
+
inputs: [ref(gated), ref(validationArtifact)],
|
|
604
|
+
parameters: { checksum: bundle.panel.checksum },
|
|
605
|
+
tools: [ENGINE, ALIGNER],
|
|
606
|
+
run: async (put) => {
|
|
607
|
+
await put("panel.json", json(bundle.panel));
|
|
608
|
+
await put("panel.fasta", writeFasta([{ id: "__reference__", sequence: reference.sequence }, ...Object.entries(bundle.sequences).map(([k, sequence]) => ({ id: k, sequence }))]));
|
|
609
|
+
}
|
|
610
|
+
});
|
|
611
|
+
await ctx.store.write(`${out}/panel.json`, await panelArtifact.read("panel.json"));
|
|
612
|
+
await ctx.store.write(`${out}/panel.fasta`, await panelArtifact.read("panel.fasta"));
|
|
613
|
+
await ctx.store.write(`${out}/validation.json`, json(validation));
|
|
614
|
+
await ctx.store.write(`${out}/diagnostics.json`, json({ diagnostics, unassigned }));
|
|
615
|
+
await ctx.store.write(`${out}/artifact.json`, json(ref(panelArtifact)));
|
|
616
|
+
return { bundle, diagnostics, validation, unassigned, published: true };
|
|
617
|
+
});
|
|
618
|
+
return { ...value, run };
|
|
619
|
+
}
|
|
620
|
+
async function loadPanel(store, dir) {
|
|
621
|
+
if (!await store.exists(`${dir}/panel.json`)) throw new EngineError("panel-missing", "This panel hasn't been built yet", "Build it in the Panel tab");
|
|
622
|
+
const panel = JSON.parse(await store.read(`${dir}/panel.json`));
|
|
623
|
+
if (panel.schemaVersion !== 2) throw new EngineError("panel-outdated", `Panel ${panel.id} version ${panel.version} was built with an earlier format, so it can't be read by the current evaluator`, "Build it again under a new version");
|
|
624
|
+
const fasta = parseFasta(await store.read(`${dir}/panel.fasta`));
|
|
625
|
+
const referenceSequence = fasta.find((f) => f.id === "__reference__")?.sequence;
|
|
626
|
+
const bundle = { panel, sequences: Object.fromEntries(fasta.filter((f) => f.id !== "__reference__").map((f) => [f.id, f.sequence])), ...referenceSequence ? { referenceSequence } : {} };
|
|
627
|
+
const problems = verifyBundle(bundle);
|
|
628
|
+
if (problems.length) throw new EngineError("panel-changed", `Panel ${panel.id} version ${panel.version} doesn't match its seal: ${problems.map((p) => p.message).join("; ")}`, "Restore it from its source, or build it again under a new version");
|
|
629
|
+
return bundle;
|
|
630
|
+
}
|
|
631
|
+
function panelVersions(profile, folders) {
|
|
632
|
+
const prefix = `${profile.panel?.id ?? ""}-`;
|
|
633
|
+
const parts = (v) => v.split(".").map(Number);
|
|
634
|
+
const order = (a, b) => {
|
|
635
|
+
const x = parts(a), y = parts(b);
|
|
636
|
+
for (let i = 0; i < Math.max(x.length, y.length); i++) {
|
|
637
|
+
const d = (x[i] ?? 0) - (y[i] ?? 0);
|
|
638
|
+
if (d) return d;
|
|
639
|
+
}
|
|
640
|
+
return a.localeCompare(b);
|
|
641
|
+
};
|
|
642
|
+
return folders.filter((f) => f.startsWith(prefix)).map((f) => f.slice(prefix.length)).sort(order);
|
|
643
|
+
}
|
|
644
|
+
async function comparePanelVersions(store, profile, a, b) {
|
|
645
|
+
if (!profile.panel) throw new EngineError("invalid-profile", "The profile has no panel settings", "Add groups in the Panel tab");
|
|
646
|
+
const P = profilePaths(profile.id);
|
|
647
|
+
const load = async (version) => {
|
|
648
|
+
const dir = P.panel(profile.panel.id, version);
|
|
649
|
+
const { panel } = await loadPanel(store, dir);
|
|
650
|
+
const report = await store.exists(`${dir}/validation.json`) ? JSON.parse(await store.read(`${dir}/validation.json`)).claims : void 0;
|
|
651
|
+
return { panel, report };
|
|
652
|
+
};
|
|
653
|
+
const [x, y] = [await load(a), await load(b)];
|
|
654
|
+
return compareRuns(x.panel, y.panel, x.report, y.report);
|
|
655
|
+
}
|
|
656
|
+
function testSample(sequence, queryId, bundle, options = {}) {
|
|
657
|
+
if (!bundle.referenceSequence) throw new EngineError("panel-outdated", "This panel has no reference sequence", "Build it again");
|
|
658
|
+
const a = alignToReference(sequence, bundle.referenceSequence);
|
|
659
|
+
const diagnostics = assessAlignment(a, { ...bundle.panel.gates, frameErrors: false });
|
|
660
|
+
if (!a || diagnostics.some((d) => d.severity === "error")) return { result: null, diagnostics };
|
|
661
|
+
const numtCaveats = [];
|
|
662
|
+
if (bundle.panel.numtCopies?.length) {
|
|
663
|
+
const numt = checkNumts(a.aligned, bundle.panel.numtCopies, bundle.referenceSequence);
|
|
664
|
+
for (const f of numt.flagged) diagnostics.push({ severity: "warning", code: "numt-flagged", message: `This sequence is closer to nuclear copy ${f.copy} than to the reference, over ${f.shared} columns (${(f.toCopy * 100).toFixed(1)}% against ${(f.toReference * 100).toFixed(1)}%). Review it. This doesn't prove it's a nuclear copy` });
|
|
665
|
+
numtCaveats.push(numt.flagged.length ? "Flagged for review as possibly a nuclear copy of the marker" : `Compared with ${numt.tested} of ${bundle.panel.numtCopies.length} known nuclear copies with no flag. That doesn't clear it, since copies absent from the screened genome can't be seen`);
|
|
666
|
+
}
|
|
667
|
+
const result = evaluateQuery(a.aligned, bundle, {
|
|
668
|
+
queryId,
|
|
669
|
+
runId: options.runId ?? "sample",
|
|
670
|
+
querySha256: hashText(sequence.toUpperCase()),
|
|
671
|
+
producedBy: [ENGINE, ALIGNER],
|
|
672
|
+
caveats: ["Rules are configured values and these statuses are not probabilities", ...diagnostics.filter((d) => d.severity === "warning").map((d) => d.message), ...numtCaveats]
|
|
673
|
+
});
|
|
674
|
+
diagnostics.push(...validateResult(result, bundle.panel));
|
|
675
|
+
return { result, diagnostics, aligned: a.aligned, insertions: a.insertions.reduce((s, x) => s + x.length, 0), deletions: a.deletions.reduce((s, x) => s + x.length, 0), identity: a.identity, clipped: a.clipped };
|
|
676
|
+
}
|
|
677
|
+
async function runSample(ctx, profile, sequence, queryId) {
|
|
678
|
+
if (!profile.panel) throw new EngineError("invalid-profile", "The profile has no panel settings", "Add groups in the Panel tab");
|
|
679
|
+
const P = profilePaths(profile.id);
|
|
680
|
+
const dir = P.panel(profile.panel.id, profile.panel.version);
|
|
681
|
+
const bundle = await loadPanel(ctx.store, dir);
|
|
682
|
+
const panelRef = JSON.parse(await ctx.store.read(`${dir}/artifact.json`));
|
|
683
|
+
const { value, run } = await withRun(ctx, profile, "sample", async (scope) => {
|
|
684
|
+
scope.run.tools.push(ALIGNER);
|
|
685
|
+
const panelArtifact = await loadArtifact(ctx.store, panelRef);
|
|
686
|
+
if (JSON.parse(await panelArtifact.read("panel.json")).checksum !== bundle.panel.checksum) throw new EngineError("panel-changed", `The saved copy of panel ${bundle.panel.id} isn't the one its record traces back to`, "Build it again under a new version");
|
|
687
|
+
const artifact = await scope.step({
|
|
688
|
+
type: "sample-result",
|
|
689
|
+
step: { name: "sample", version: 1 },
|
|
690
|
+
inputs: [ref(panelArtifact)],
|
|
691
|
+
parameters: { queryId, sequence: hashText(sequence.toUpperCase()) },
|
|
692
|
+
tools: [ENGINE, ALIGNER],
|
|
693
|
+
run: async (put) => {
|
|
694
|
+
await put("outcome.json", json(testSample(sequence, queryId, bundle, { runId: scope.run.id })));
|
|
695
|
+
}
|
|
696
|
+
});
|
|
697
|
+
const text = await artifact.read("outcome.json");
|
|
698
|
+
await ctx.store.write(`${P.results}/${bundle.panel.id}-${bundle.panel.version}/${queryId.replace(/[^A-Za-z0-9._-]/g, "_")}.json`, text);
|
|
699
|
+
return { outcome: JSON.parse(text), artifact: ref(artifact) };
|
|
700
|
+
});
|
|
701
|
+
return { ...value.outcome, run, artifact: value.artifact };
|
|
702
|
+
}
|
|
703
|
+
async function buildTree(ctx, profile, options) {
|
|
704
|
+
if (!ctx.tools) throw new EngineError("tool-missing", "Building a tree needs a tool runner");
|
|
705
|
+
const settings = profile.tree;
|
|
706
|
+
if (!settings) throw new EngineError("invalid-profile", "The profile has no tree settings");
|
|
707
|
+
const tool = options.tool ?? "iqtree3";
|
|
708
|
+
const version = await ctx.tools.version(tool);
|
|
709
|
+
if (!version) throw new EngineError("tool-missing", `${tool} isn't installed or can't be found`, `Install it with: brew install ${tool}`);
|
|
710
|
+
const tools = ctx.tools;
|
|
711
|
+
const P = profilePaths(profile.id);
|
|
712
|
+
const sourceId = options.runId ?? (await ctx.store.exists(P.latest) ? (await ctx.store.read(P.latest)).trim() : "");
|
|
713
|
+
if (!sourceId) throw new EngineError("run-missing", "No finished fetch run exists yet", "Fetch records first");
|
|
714
|
+
const source = JSON.parse(await ctx.store.read(`${P.run(sourceId)}/run.json`));
|
|
715
|
+
const gatedRef = artifactOf(source, "gated-records");
|
|
716
|
+
if (!gatedRef) throw new EngineError("run-missing", `Run ${sourceId} didn't finish fetching`, "Fetch records again");
|
|
717
|
+
const { value, run } = await withRun(ctx, profile, "tree", async (scope) => {
|
|
718
|
+
scope.run.tools.push({ name: tool, version });
|
|
719
|
+
const gated = await loadArtifact(ctx.store, { type: "gated-records", key: gatedRef.key });
|
|
720
|
+
const aligned = parseFasta(await gated.read("aligned.fasta"));
|
|
721
|
+
const w = options.window ?? { start: 0, end: aligned[0]?.sequence.length ?? 0 };
|
|
722
|
+
const min = options.minCoverage ?? 0.85;
|
|
723
|
+
const usable = aligned.map((e) => ({ id: e.id, aligned: e.sequence.slice(w.start, w.end) })).filter((e) => (e.aligned.match(/[ACGT]/gi) ?? []).length >= min * (w.end - w.start));
|
|
724
|
+
const haps = collapseHaplotypes(usable);
|
|
725
|
+
if (haps.length < 4) throw new EngineError("step-output-invalid", `Only ${haps.length} haplotypes cover this window, and a tree needs at least 4`, "Choose another window, or lower the coverage needed");
|
|
726
|
+
const outgroups = profile.selection.outgroups.map((o) => o.accession).filter((a) => Boolean(a));
|
|
727
|
+
const names = new Map(haps.map((h, i) => [h, `H${String(i + 1).padStart(4, "0")}`]));
|
|
728
|
+
const og = haps.filter((h) => h.members.some((m) => outgroups.includes(m))).map((h) => names.get(h));
|
|
729
|
+
const artifact = await scope.step({
|
|
730
|
+
type: "tree",
|
|
731
|
+
step: { name: "tree", version: 1 },
|
|
732
|
+
inputs: [ref(gated)],
|
|
733
|
+
parameters: { window: w, minCoverage: min, tree: settings, outgroups: og },
|
|
734
|
+
tools: [{ name: tool, version }],
|
|
735
|
+
run: async (put, dir, register) => {
|
|
736
|
+
await put("input.fasta", writeFasta(haps.map((h) => ({ id: names.get(h), sequence: h.representative }))));
|
|
737
|
+
await put("tips.json", json(haps.map((h) => ({ id: names.get(h), members: h.members }))));
|
|
738
|
+
const args = ["-s", "input.fasta", "-m", settings.model, "-B", String(settings.bootstrap), "--alrt", String(settings.shAlrt), "--seed", String(settings.seed), "-T", String(settings.threads), "-pre", "tree", "-redo", ...og.length ? ["-o", og.join(",")] : []];
|
|
739
|
+
const res = await tools.run(tool, args, { cwd: `${options.absoluteDir}/${dir}` });
|
|
740
|
+
await ctx.store.write(`${dir}/tool.log`, `${res.stdout}
|
|
741
|
+
${res.stderr}`);
|
|
742
|
+
if (res.code !== 0) throw new EngineError("tool-failed", `${tool} failed: ${(res.stderr || res.stdout).slice(0, 600)}`, `See ${dir}/tool.log`);
|
|
743
|
+
await register("tree.treefile");
|
|
744
|
+
await register("tree.iqtree");
|
|
745
|
+
},
|
|
746
|
+
ensure: async (read) => (await read("tree.treefile")).includes("(") ? null : "the tree file has no tree in it"
|
|
747
|
+
});
|
|
748
|
+
return { newick: await artifact.read("tree.treefile"), tips: JSON.parse(await artifact.read("tips.json")), artifact: ref(artifact) };
|
|
749
|
+
});
|
|
750
|
+
return { tool, version, ...value, run };
|
|
751
|
+
}
|
|
752
|
+
export {
|
|
753
|
+
ENGINE_VERSION,
|
|
754
|
+
EngineError,
|
|
755
|
+
buildPanelFromRun,
|
|
756
|
+
buildTree,
|
|
757
|
+
comparePanelVersions,
|
|
758
|
+
fetchAndAnchor,
|
|
759
|
+
findPaperRecords,
|
|
760
|
+
findRecordStudies,
|
|
761
|
+
isCancelled,
|
|
762
|
+
loadArtifact,
|
|
763
|
+
loadPanel,
|
|
764
|
+
panelVersions,
|
|
765
|
+
profilePaths,
|
|
766
|
+
runSample,
|
|
767
|
+
testSample,
|
|
768
|
+
traceArtifact
|
|
769
|
+
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,47 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@captiveorigins/onephylogeny-engine",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
3
|
+
"version": "0.1.0-alpha.1",
|
|
4
|
+
"description": "OnePhylogeny engine: runs profiles end to end with immutable run records and structured logs: fetch, anchor, gate, build panels, validate, test samples and build trees through replaceable ports.",
|
|
5
|
+
"keywords": [
|
|
6
|
+
"phylogenetics",
|
|
7
|
+
"reference-panel",
|
|
8
|
+
"genbank"
|
|
9
|
+
],
|
|
10
|
+
"homepage": "https://github.com/CaptiveOrigins/onephylogeny#readme",
|
|
11
|
+
"bugs": {
|
|
12
|
+
"url": "https://github.com/CaptiveOrigins/onephylogeny/issues"
|
|
13
|
+
},
|
|
14
|
+
"license": "PolyForm-Noncommercial-1.0.0",
|
|
15
|
+
"repository": {
|
|
16
|
+
"type": "git",
|
|
17
|
+
"url": "git+https://github.com/CaptiveOrigins/onephylogeny.git",
|
|
18
|
+
"directory": "packages/engine"
|
|
19
|
+
},
|
|
20
|
+
"type": "module",
|
|
21
|
+
"sideEffects": false,
|
|
22
|
+
"exports": {
|
|
23
|
+
".": {
|
|
24
|
+
"types": "./dist/index.d.ts",
|
|
25
|
+
"import": "./dist/index.js"
|
|
26
|
+
}
|
|
27
|
+
},
|
|
28
|
+
"main": "./dist/index.js",
|
|
29
|
+
"types": "./dist/index.d.ts",
|
|
30
|
+
"files": [
|
|
31
|
+
"dist",
|
|
32
|
+
"README.md",
|
|
33
|
+
"LICENSE"
|
|
34
|
+
],
|
|
35
|
+
"engines": {
|
|
36
|
+
"node": ">=22"
|
|
37
|
+
},
|
|
38
|
+
"dependencies": {
|
|
39
|
+
"@captiveorigins/onephylogeny-acquire": "^0.1.0-alpha.1",
|
|
40
|
+
"@captiveorigins/onephylogeny-core": "^0.1.0-alpha.1"
|
|
41
|
+
},
|
|
42
|
+
"scripts": {
|
|
43
|
+
"build": "tsup src/index.ts --format esm --dts --clean --target es2022",
|
|
44
|
+
"typecheck": "tsc --noEmit",
|
|
45
|
+
"test": "vitest run --passWithNoTests"
|
|
46
|
+
}
|
|
6
47
|
}
|