@platforma-open/milaboratories.mixcr-clonotyping-2 2.23.4 → 2.23.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/block-pack/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,77 @@
|
|
|
1
1
|
# @platforma-open/milaboratories.mixcr-clonotyping
|
|
2
2
|
|
|
3
|
+
## 2.23.6
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- 231f62f: fix: the aggregation shard grant ignores `perProcessMemGB`
|
|
8
|
+
|
|
9
|
+
A shard's grant is already the larger of 64 GiB and its need computed from the bytes it
|
|
10
|
+
keeps, so the override cannot help it and only reduces how many shards run at once. It
|
|
11
|
+
still raises the single unsharded run taken when the backend cannot report blob sizes,
|
|
12
|
+
and it still applies to the MiXCR steps.
|
|
13
|
+
|
|
14
|
+
## 2.23.5
|
|
15
|
+
|
|
16
|
+
### Patch Changes
|
|
17
|
+
|
|
18
|
+
- fd14efa: fix: aggregate the cohort clonotype table in shards
|
|
19
|
+
|
|
20
|
+
`aggregate-by-clonotype-key` groups every sample's clonotype rows by `clonotypeKey` and keeps,
|
|
21
|
+
per column, the value from the most abundant sample. ptabler lowers that `maxBy` to
|
|
22
|
+
`top_k_by(k=1).first()`, which polars runs as an in-memory group-by, so one run held every
|
|
23
|
+
group of the cohort at once. The peak followed the number of input rows and no grant could
|
|
24
|
+
change it: a large cohort can need hundreds of GiB in a single run, above any cluster ceiling.
|
|
25
|
+
The step also asked for `max(samples, 32)` cores, and the
|
|
26
|
+
memory need of this plan shape rises with the polars thread count.
|
|
27
|
+
|
|
28
|
+
The aggregation now runs in shards. Rows are bucketed by the first letter of `clonotypeKey`
|
|
29
|
+
after digits are removed, upper-cased — the same first letter the clonotype label's five-letter
|
|
30
|
+
prefix uses — so every key lands in exactly one shard, every group is complete inside its shard
|
|
31
|
+
and the label is computed per shard. The shard outputs are disjoint and a final streaming run
|
|
32
|
+
concatenates them. A cohort small enough for one shard runs unfiltered, as before.
|
|
33
|
+
|
|
34
|
+
The shard count is chosen from the input volume. The template reads the blob size of every
|
|
35
|
+
input TSV through the backend's `getBlobSize` (the same call `f.size()` resolves through) and
|
|
36
|
+
takes the smallest shard count whose largest shard stays under 64 GiB at 6 GiB of RAM per GiB
|
|
37
|
+
of TSV plus a 2 GiB intercept -- the concat + `maxBy` law measured under MILAB-6874 (4.94 at
|
|
38
|
+
eight threads), at the slope the SDK's default ptabler sizing uses. The memory override raises
|
|
39
|
+
every shard's grant but never the shard count: a request the backend cannot satisfy is clamped
|
|
40
|
+
without notice, so a larger target would only recreate the single oversized run. Each shard
|
|
41
|
+
runs on 8 cores. A backend without `getBlobSize` gets a single shard.
|
|
42
|
+
|
|
43
|
+
The `byCloneKey` Parquet import that follows was a flat 24 GiB, which the measured `write_frame`
|
|
44
|
+
law (`4.13 x^0.68` GiB for x GiB of TSV) says holds about 13 GiB of aggregated TSV. Its memory
|
|
45
|
+
is now left to the SDK, which sizes the ptabler run from the blob size of the aggregated TSV
|
|
46
|
+
(`2 GiB + 6 x size`, capped at 256 GiB in workflow-tengo 6.11.0, above the measured need at
|
|
47
|
+
every size it can express). The memory override does not apply to this import: the Xsv output
|
|
48
|
+
passes no floor through, and a fixed request would replace the formula.
|
|
49
|
+
|
|
50
|
+
The six other Parquet imports of the block had flat grants of 12, 16 or 24 GiB: the per-sample
|
|
51
|
+
`byCloneKeyBySample` table and the single-cell abundance, aggregates, properties, cell-linker
|
|
52
|
+
and SHM tables. A 16 GiB grant holds about 7 GiB of TSV under the same law, and one deep sample
|
|
53
|
+
can export twice that. Their memory is now left to the same SDK sizing. All seven imports run in
|
|
54
|
+
the medium queue; the Xsv import default is the light queue.
|
|
55
|
+
|
|
56
|
+
The QC report run in `export-report` framed every sample's full clonotype TSV and every filter
|
|
57
|
+
TSV in one 8 GiB ptabler run to count clonotypes, reads, out-of-frame and stop-codon clones per
|
|
58
|
+
sample. Those counts are now computed by one ptabler run per sample, sized by the SDK from that
|
|
59
|
+
sample's files with an 8 GiB floor, each reading only that sample's files and writing a one-row
|
|
60
|
+
table; the cohort run frames those rows and the qc
|
|
61
|
+
report, so its input no longer grows with the clonotype or cell count. In single-cell mode the
|
|
62
|
+
per-sample run also computes that sample's cell-pairing statistics from its single-cell chain
|
|
63
|
+
TSVs. The `exportClones` filter runs are unchanged.
|
|
64
|
+
|
|
65
|
+
The single-cell per-cell preprocessing run asked for one core and one GiB per sample, with
|
|
66
|
+
floors of 16 and 32. Its memory is now left to the SDK, which sizes the run from the blob size
|
|
67
|
+
of its input TSVs with a 32 GiB floor, and its cpu is pinned at 12, which the formula's slope
|
|
68
|
+
still covers.
|
|
69
|
+
|
|
70
|
+
The block moves to workflow-tengo 6.11.0, which ships that sizing formula and `memFloor`.
|
|
71
|
+
|
|
72
|
+
The hash override of `aggregate-by-clonotype-key` is new, so a failed aggregation is not
|
|
73
|
+
recovered from its old identity.
|
|
74
|
+
|
|
3
75
|
## 2.23.4
|
|
4
76
|
|
|
5
77
|
### Patch Changes
|
package/block-pack/main.plj.gz
CHANGED
|
Binary file
|
package/block-pack/manifest.json
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"schema":"v2","description":{"id":{"organization":"milaboratories","name":"mixcr-clonotyping-2","version":"2.23.
|
|
1
|
+
{"schema":"v2","description":{"id":{"organization":"milaboratories","name":"mixcr-clonotyping-2","version":"2.23.6"},"components":{"workflow":{"type":"workflow-v1","main":{"type":"relative","path":"main.plj.gz"}},"model":{"type":"relative","path":"model.json"},"ui":{"type":"relative","path":"ui.tgz"}},"meta":{"title":"MiXCR Clonotyping","description":"Extract TCR / BCR clonotypes from next-generation sequencing data","longDescription":{"type":"relative","path":"description.md"},"changelog":{"type":"relative","path":"CHANGELOG.md"},"logo":{"type":"relative","path":"block-logo.png"},"url":"https://github.com/platforma-open/mixcr-clonotyping-2","support":"mailto:support@milaboratories.com","tags":["upstream","airr","vdj","single-cell"],"organization":{"name":"MiLaboratories Inc","url":"https://milaboratories.com/","logo":{"type":"relative","path":"organization-logo.png"}},"marketplaceRanking":16900},"featureFlags":{"supportsLazyState":true,"supportsPframeQueryRanking":true,"requiresUIAPIVersion":3,"requiresModelAPIVersion":2,"requiresCreatePTable":2,"requiresPFramesVersion":1001031,"requiresPFrameSpec":true,"requiresPFrame":true,"requiresDialog":true,"requiresColumnsCollection":true},"kind":"@platforma-open/milaboratories.mixcr-clonotyping-2.kind@1.1.0"},"timestamp":1789812483535,"files":[{"name":"main.plj.gz","size":1547103,"sha256":"00B3B18033DD04E6CF105B2040937486E64612483CD947732D36F281C0BDFB42"},{"name":"model.json","size":575939,"sha256":"ABE8B2BA1F3818F65E7E58D81B8C72FB3274CFEDA89BC2F0F5D9AF8FE0D6C416"},{"name":"ui.tgz","size":4040762,"sha256":"6ED0E753D1BD1E2ADF34160EBD302742CED05457BF50B47A248DD09B3622B546"},{"name":"organization-logo.png","size":24439,"sha256":"FA71390C77C91E4B7FAAE5640D00F92F1E3F2869296F68B6040DD7CC549A50B5"},{"name":"description.md","size":1148,"sha256":"B319CBECC5055A89194C4D7B5768E1ABDE225053885800179439839408ECBA54"},{"name":"CHANGELOG.md","size":53383,"sha256":"FD12F5D4E5ADAFDEABD5F030EE5A45B3B0A47C4C7DB9E4AEB24B3E4225BB6DA2"},{"name":"block-logo.png","size":21527,"sha256":"6BB33BAF0CD039549661B51AE490373BE60D1811EC71F5023400928293BC2427"}]}
|
package/block-pack/ui.tgz
CHANGED
|
Binary file
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@platforma-open/milaboratories.mixcr-clonotyping-2",
|
|
3
|
-
"version": "2.23.
|
|
3
|
+
"version": "2.23.6",
|
|
4
4
|
"files": [
|
|
5
5
|
"dist",
|
|
6
6
|
"block-pack"
|
|
@@ -26,8 +26,8 @@
|
|
|
26
26
|
"typescript": "~5.6.3",
|
|
27
27
|
"@platforma-open/milaboratories.mixcr-clonotyping-2.kind": "1.1.0",
|
|
28
28
|
"@platforma-open/milaboratories.mixcr-clonotyping-2.model": "1.28.0",
|
|
29
|
-
"@platforma-open/milaboratories.mixcr-clonotyping-2.
|
|
30
|
-
"@platforma-open/milaboratories.mixcr-clonotyping-2.
|
|
29
|
+
"@platforma-open/milaboratories.mixcr-clonotyping-2.workflow": "3.29.5",
|
|
30
|
+
"@platforma-open/milaboratories.mixcr-clonotyping-2.ui": "1.27.3"
|
|
31
31
|
},
|
|
32
32
|
"block": {
|
|
33
33
|
"components": {
|