foca-plugin-tes-data-crypt4gh 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- foca_plugin_tes_data_crypt4gh-0.1.0/LICENSE +201 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/PKG-INFO +406 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/README.md +377 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/pyproject.toml +65 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/setup.cfg +4 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugin_tes_data_crypt4gh.egg-info/PKG-INFO +406 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugin_tes_data_crypt4gh.egg-info/SOURCES.txt +17 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugin_tes_data_crypt4gh.egg-info/dependency_links.txt +1 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugin_tes_data_crypt4gh.egg-info/entry_points.txt +2 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugin_tes_data_crypt4gh.egg-info/requires.txt +2 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugin_tes_data_crypt4gh.egg-info/top_level.txt +1 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugins/tes/data/crypt4gh/__init__.py +226 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugins/tes/data/crypt4gh/decrypt.py +312 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugins/tes/data/crypt4gh/inputs.py +188 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugins/tes/data/crypt4gh/py.typed +0 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/src/foca_plugins/tes/data/crypt4gh/task.py +344 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/tests/test_crypt4gh.py +784 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/tests/test_decrypt.py +801 -0
- foca_plugin_tes_data_crypt4gh-0.1.0/tests/test_inputs.py +130 -0
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
6
|
+
|
|
7
|
+
1. Definitions.
|
|
8
|
+
|
|
9
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
10
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
11
|
+
|
|
12
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
13
|
+
the copyright owner that is granting the License.
|
|
14
|
+
|
|
15
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
16
|
+
other entities that control, are controlled by, or are under common
|
|
17
|
+
control with that entity. For the purposes of this definition,
|
|
18
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
19
|
+
direction or management of such entity, whether by contract or
|
|
20
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
21
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
22
|
+
|
|
23
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
24
|
+
exercising permissions granted by this License.
|
|
25
|
+
|
|
26
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
27
|
+
including but not limited to software source code, documentation
|
|
28
|
+
source, and configuration files.
|
|
29
|
+
|
|
30
|
+
"Object" form shall mean any form resulting from mechanical
|
|
31
|
+
transformation or translation of a Source form, including but
|
|
32
|
+
not limited to compiled object code, generated documentation,
|
|
33
|
+
and conversions to other media types.
|
|
34
|
+
|
|
35
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
36
|
+
Object form, made available under the License, as indicated by a
|
|
37
|
+
copyright notice that is included in or attached to the work
|
|
38
|
+
(an example is provided in the Appendix below).
|
|
39
|
+
|
|
40
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
41
|
+
form, that is based on (or derived from) the Work and for which the
|
|
42
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
43
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
44
|
+
of this License, Derivative Works shall not include works that remain
|
|
45
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
46
|
+
the Work and Derivative Works thereof.
|
|
47
|
+
|
|
48
|
+
"Contribution" shall mean any work of authorship, including
|
|
49
|
+
the original version of the Work and any modifications or additions
|
|
50
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
51
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
52
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
53
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
54
|
+
means any form of electronic, verbal, or written communication sent
|
|
55
|
+
to the Licensor or its representatives, including but not limited to
|
|
56
|
+
communication on electronic mailing lists, source code control systems,
|
|
57
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
58
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
59
|
+
excluding communication that is conspicuously marked or otherwise
|
|
60
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
61
|
+
|
|
62
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
63
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
64
|
+
subsequently incorporated within the Work.
|
|
65
|
+
|
|
66
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
67
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
68
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
69
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
70
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
71
|
+
Work and such Derivative Works in Source or Object form.
|
|
72
|
+
|
|
73
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
74
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
75
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
76
|
+
(except as stated in this section) patent license to make, have made,
|
|
77
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
78
|
+
where such license applies only to those patent claims licensable
|
|
79
|
+
by such Contributor that are necessarily infringed by their
|
|
80
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
81
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
82
|
+
institute patent litigation against any entity (including a
|
|
83
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
84
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
85
|
+
or contributory patent infringement, then any patent licenses
|
|
86
|
+
granted to You under this License for that Work shall terminate
|
|
87
|
+
as of the date such litigation is filed.
|
|
88
|
+
|
|
89
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
90
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
91
|
+
modifications, and in Source or Object form, provided that You
|
|
92
|
+
meet the following conditions:
|
|
93
|
+
|
|
94
|
+
(a) You must give any other recipients of the Work or
|
|
95
|
+
Derivative Works a copy of this License; and
|
|
96
|
+
|
|
97
|
+
(b) You must cause any modified files to carry prominent notices
|
|
98
|
+
stating that You changed the files; and
|
|
99
|
+
|
|
100
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
101
|
+
that You distribute, all copyright, patent, trademark, and
|
|
102
|
+
attribution notices from the Source form of the Work,
|
|
103
|
+
excluding those notices that do not pertain to any part of
|
|
104
|
+
the Derivative Works; and
|
|
105
|
+
|
|
106
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
107
|
+
distribution, then any Derivative Works that You distribute must
|
|
108
|
+
include a readable copy of the attribution notices contained
|
|
109
|
+
within such NOTICE file, excluding those notices that do not
|
|
110
|
+
pertain to any part of the Derivative Works, in at least one
|
|
111
|
+
of the following places: within a NOTICE text file distributed
|
|
112
|
+
as part of the Derivative Works; within the Source form or
|
|
113
|
+
documentation, if provided along with the Derivative Works; or,
|
|
114
|
+
within a display generated by the Derivative Works, if and
|
|
115
|
+
wherever such third-party notices normally appear. The contents
|
|
116
|
+
of the NOTICE file are for informational purposes only and
|
|
117
|
+
do not modify the License. You may add Your own attribution
|
|
118
|
+
notices within Derivative Works that You distribute, alongside
|
|
119
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
120
|
+
that such additional attribution notices cannot be construed
|
|
121
|
+
as modifying the License.
|
|
122
|
+
|
|
123
|
+
You may add Your own copyright statement to Your modifications and
|
|
124
|
+
may provide additional or different license terms and conditions
|
|
125
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
126
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
127
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
128
|
+
the conditions stated in this License.
|
|
129
|
+
|
|
130
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
131
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
132
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
133
|
+
this License, without any additional terms or conditions.
|
|
134
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
135
|
+
the terms of any separate license agreement you may have executed
|
|
136
|
+
with Licensor regarding such Contributions.
|
|
137
|
+
|
|
138
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
139
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
140
|
+
except as required for reasonable and customary use in describing the
|
|
141
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
142
|
+
|
|
143
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
144
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
145
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
146
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
147
|
+
implied, including, without limitation, any warranties or conditions
|
|
148
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
149
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
150
|
+
appropriateness of using or redistributing the Work and assume any
|
|
151
|
+
risks associated with Your exercise of permissions under this License.
|
|
152
|
+
|
|
153
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
154
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
155
|
+
unless required by applicable law (such as deliberate and grossly
|
|
156
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
157
|
+
liable to You for damages, including any direct, indirect, special,
|
|
158
|
+
incidental, or consequential damages of any character arising as a
|
|
159
|
+
result of this License or out of the use or inability to use the
|
|
160
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
161
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
162
|
+
other commercial damages or losses), even if such Contributor
|
|
163
|
+
has been advised of the possibility of such damages.
|
|
164
|
+
|
|
165
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
166
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
167
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
168
|
+
or other liability obligations and/or rights consistent with this
|
|
169
|
+
License. However, in accepting such obligations, You may act only
|
|
170
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
171
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
172
|
+
defend, and hold each Contributor harmless for any liability
|
|
173
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
174
|
+
of your accepting any such warranty or additional liability.
|
|
175
|
+
|
|
176
|
+
END OF TERMS AND CONDITIONS
|
|
177
|
+
|
|
178
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
179
|
+
|
|
180
|
+
To apply the Apache License to your work, attach the following
|
|
181
|
+
boilerplate notice, with the fields enclosed by brackets "{}"
|
|
182
|
+
replaced with your own identifying information. (Don't include
|
|
183
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
184
|
+
comment syntax for the file format. We also recommend that a
|
|
185
|
+
file or class name and description of purpose be included on the
|
|
186
|
+
same "printed page" as the copyright notice for easier
|
|
187
|
+
identification within third-party archives.
|
|
188
|
+
|
|
189
|
+
Copyright 2026 ELIXIR On-Cloud
|
|
190
|
+
|
|
191
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
192
|
+
you may not use this file except in compliance with the License.
|
|
193
|
+
You may obtain a copy of the License at
|
|
194
|
+
|
|
195
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
196
|
+
|
|
197
|
+
Unless required by applicable law or agreed to in writing, software
|
|
198
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
199
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
200
|
+
See the License for the specific language governing permissions and
|
|
201
|
+
limitations under the License.
|
|
@@ -0,0 +1,406 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: foca-plugin-tes-data-crypt4gh
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: FOCA gateway plugin that decrypts the Crypt4GH inputs of TES tasks in an extra executor
|
|
5
|
+
Author-email: ELIXIR On-Cloud <cloud-service@elixir-europe.org>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/elixir-cloud-aai/foca-plugins/tree/dev/plugins/tes/data/crypt4gh
|
|
8
|
+
Project-URL: Documentation, https://foca-plugins.readthedocs.io/
|
|
9
|
+
Project-URL: Changelog, https://github.com/elixir-cloud-aai/foca-plugins/blob/dev/plugins/tes/data/crypt4gh/CHANGELOG.md
|
|
10
|
+
Project-URL: Issues, https://github.com/elixir-cloud-aai/foca-plugins/issues
|
|
11
|
+
Keywords: foca,foca-plugin,gateway,ga4gh,tes,crypt4gh,encryption
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Intended Audience :: System Administrators
|
|
15
|
+
Classifier: Natural Language :: English
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.15
|
|
22
|
+
Classifier: Typing :: Typed
|
|
23
|
+
Requires-Python: >=3.11
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE
|
|
26
|
+
Requires-Dist: foca<0.23,>=0.19.0
|
|
27
|
+
Requires-Dist: pydantic<3,>=2.7
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
|
|
30
|
+
# tes.data.crypt4gh
|
|
31
|
+
|
|
32
|
+
<!-- --8<-- [start:summary] -->
|
|
33
|
+
Decrypts the [Crypt4GH](https://www.ga4gh.org/news_item/crypt4gh-a-secure-method-for-sharing-human-genetic-data/)
|
|
34
|
+
inputs of TES tasks, which TES implementations can't read themselves: adds
|
|
35
|
+
an executor before the task's own ones that decrypts the Crypt4GH inputs
|
|
36
|
+
with the secret keys among the task's inputs to a volume of its own,
|
|
37
|
+
points the task's executors to the decrypted files, and adds an executor
|
|
38
|
+
after them that empties that volume again. The successor of the Crypt4GH
|
|
39
|
+
middleware of proTES.
|
|
40
|
+
<!-- --8<-- [end:summary] -->
|
|
41
|
+
|
|
42
|
+
| | |
|
|
43
|
+
| --- | --- |
|
|
44
|
+
| Plugin ID | `tes.data.crypt4gh` |
|
|
45
|
+
| Distribution | `foca-plugin-tes-data-crypt4gh` |
|
|
46
|
+
| Import path | `foca_plugins.tes.data.crypt4gh` |
|
|
47
|
+
| APIs | `tes` |
|
|
48
|
+
| Phases | `request` |
|
|
49
|
+
| Capabilities | `crypt4gh-decryption` |
|
|
50
|
+
| Maturity | experimental |
|
|
51
|
+
| Data transfers | none |
|
|
52
|
+
| Data stored | none (no plugin store) |
|
|
53
|
+
| FOCA | 0.19, 0.20, 0.21, 0.22 (`>=0.19.0,<0.23`) |
|
|
54
|
+
|
|
55
|
+
## What it does
|
|
56
|
+
|
|
57
|
+
`request`:
|
|
58
|
+
|
|
59
|
+
1. Leaves tasks without executors and tasks whose `executors` or
|
|
60
|
+
`volumes` have the wrong type as they are, without checking them: the
|
|
61
|
+
gateway app rejects them.
|
|
62
|
+
2. Finds the Crypt4GH files and the secret keys among the task's `inputs`
|
|
63
|
+
(files only, not `type: DIRECTORY`; a null `type` is a file), by the
|
|
64
|
+
suffixes of their `path`s:
|
|
65
|
+
`encrypted_suffixes` (`.c4gh`) and `key_suffixes` (`.sec`). The gateway
|
|
66
|
+
can't read the inputs, so their names decide; the decryption executor
|
|
67
|
+
checks the files when the task runs.
|
|
68
|
+
3. Leaves tasks without Crypt4GH inputs as they are, even if they have
|
|
69
|
+
keys. Tasks that it changed already (they have `volume` once and, as
|
|
70
|
+
their first executor, exactly the decryption executor that it would add)
|
|
71
|
+
are checked as in step 4, except for that volume, and only their other
|
|
72
|
+
executors are changed (steps 6 and 7), so that a task sent through the
|
|
73
|
+
gateway twice gets one decryption executor and one teardown executor.
|
|
74
|
+
This holds for gateways with the same settings only: chain gateways with
|
|
75
|
+
this plugin only if their settings are the same.
|
|
76
|
+
4. Otherwise, it rejects the task with `422 Unprocessable Content` if:
|
|
77
|
+
- it has no secret key among its inputs;
|
|
78
|
+
- a Crypt4GH input or key has no absolute, normalized path (e.g.,
|
|
79
|
+
`/in/./reads.fq.c4gh` or `//in/reads.fq.c4gh`), or another input, an
|
|
80
|
+
output or a volume has a relative path;
|
|
81
|
+
- a Crypt4GH input has no name besides its suffix (e.g., `/in/.c4gh`),
|
|
82
|
+
or two decrypt to the same name (e.g., `/a/x.c4gh` and `/b/x.c4gh`);
|
|
83
|
+
- an input, output or volume of the task is the `volume` of the
|
|
84
|
+
decrypted files, inside it or one of its parent directories (e.g.,
|
|
85
|
+
`/vol` or `/`), also in another spelling (e.g., `/vol/x/../crypt4gh`);
|
|
86
|
+
for an output with wildcards, the directory before its first wildcard
|
|
87
|
+
counts (e.g., `/` for `/v*/crypt4gh/*`);
|
|
88
|
+
- an output has the path of a Crypt4GH input, also in another spelling,
|
|
89
|
+
or has wildcards that match one (e.g., `/in/*.c4gh`).
|
|
90
|
+
5. Adds `volume` (`/vol/crypt4gh`) to the task's `volumes`, and, as the
|
|
91
|
+
task's first executor, the decryption executor (`image`, `command`), which
|
|
92
|
+
decrypts each Crypt4GH input to `<volume>/<name without suffix>`, e.g.,
|
|
93
|
+
`/in/reads.fq.c4gh` to `/vol/crypt4gh/reads.fq`, with any of the keys.
|
|
94
|
+
The paths are arguments of the executor, so a task's Crypt4GH inputs
|
|
95
|
+
and keys are bounded by the argument limit of the backend's nodes
|
|
96
|
+
(usually 2 MiB in all, i.e., thousands of paths).
|
|
97
|
+
6. Changes the arguments of the task's executors (`command`, and `stdin`)
|
|
98
|
+
that are the path of a Crypt4GH input, in any spelling of the absolute
|
|
99
|
+
path (e.g., `/in/./reads.fq.c4gh`), to the path of its decrypted file.
|
|
100
|
+
Only whole, absolute arguments are changed: a relative path, or an
|
|
101
|
+
argument such as
|
|
102
|
+
`--in=/in/reads.fq.c4gh` keeps the encrypted file, as do `env` and
|
|
103
|
+
`workdir`.
|
|
104
|
+
7. With `teardown` (the default), adds the teardown executor after the
|
|
105
|
+
task's own executors, which empties the volume (see [The teardown
|
|
106
|
+
executor](#the-teardown-executor)). Exact copies of it among the task's
|
|
107
|
+
own executors are moved to the end, not kept where they are, so that it
|
|
108
|
+
runs once, after all of them. With `teardown_on_error`, the task's own
|
|
109
|
+
executors get `ignore_error: true`.
|
|
110
|
+
|
|
111
|
+
For example, a task with the inputs `/in/reads.fq.c4gh` and
|
|
112
|
+
`/keys/alice.sec` and the executor `align /in/reads.fq.c4gh` gets these
|
|
113
|
+
executors:
|
|
114
|
+
|
|
115
|
+
```json
|
|
116
|
+
[
|
|
117
|
+
{
|
|
118
|
+
"image": "<image>",
|
|
119
|
+
"command": ["crypt4gh-decrypt", "--key", "/keys/alice.sec",
|
|
120
|
+
"--file", "/in/reads.fq.c4gh", "/vol/crypt4gh/reads.fq"]
|
|
121
|
+
},
|
|
122
|
+
{"image": "aligner", "command": ["align", "/vol/crypt4gh/reads.fq"]},
|
|
123
|
+
{
|
|
124
|
+
"image": "<image>",
|
|
125
|
+
"command": ["crypt4gh-decrypt", "--clear", "/vol/crypt4gh"]
|
|
126
|
+
}
|
|
127
|
+
]
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Annotations, counts only: `encrypted_inputs` and `key_inputs`, for all
|
|
131
|
+
tasks but those left to the app in step 1. Other
|
|
132
|
+
inputs (e.g., a reference genome) are left where they are.
|
|
133
|
+
|
|
134
|
+
### The decryption executor
|
|
135
|
+
|
|
136
|
+
The executor runs `crypt4gh-decrypt` (`decrypt.py` of the package) in the
|
|
137
|
+
decryption image. It reads the keys (Crypt4GH or OpenSSH ed25519 secret
|
|
138
|
+
keys, **without passphrase**), then decrypts each file with any of them,
|
|
139
|
+
writing it to a temporary file next to its target first. Decrypted files
|
|
140
|
+
are readable by all (0644, their directory 0755 where the executor may
|
|
141
|
+
change it), deliberately: the task's next executors may run as other users
|
|
142
|
+
than the decryption executor (e.g., images with a non-root user), and the
|
|
143
|
+
volume belongs to the task alone. Inputs
|
|
144
|
+
are only read, so they may be mounted read-only. If a key can't be read or
|
|
145
|
+
a file can't be decrypted (e.g., no key fits, or it is no Crypt4GH file),
|
|
146
|
+
the executor removes the files it decrypted and exits with status 1, so
|
|
147
|
+
that the task fails before its own executors run. Existing files at the
|
|
148
|
+
targets are replaced, not refused, so that a restarted executor (e.g., a
|
|
149
|
+
Kubernetes pod) runs again: the volume belongs to one task, whose
|
|
150
|
+
executors run once each, in order.
|
|
151
|
+
|
|
152
|
+
### The teardown executor
|
|
153
|
+
|
|
154
|
+
The teardown executor runs the same script, in the same image, with
|
|
155
|
+
`--clear` and the `volume`. It removes everything in the volume, but not
|
|
156
|
+
the volume itself (a mount point): the decrypted files, temporary files
|
|
157
|
+
that a stopped decryption may have left, and files that the task's
|
|
158
|
+
executors derived from the decrypted files there (e.g., an index written
|
|
159
|
+
next to its input). The volume is reserved for the plugin (no input,
|
|
160
|
+
output or volume of the task may be in it), so nothing there is the
|
|
161
|
+
task's result. Symbolic links in it are removed, not followed, and a
|
|
162
|
+
volume that is a link is refused. An empty or
|
|
163
|
+
missing volume is fine, so that a restarted executor runs again. If an
|
|
164
|
+
entry can't be removed, the executor removes the others and exits with
|
|
165
|
+
status 1, so that the task fails (`EXECUTOR_ERROR`) and the entries that
|
|
166
|
+
remain show in its logs; backends may then skip uploading the task's
|
|
167
|
+
outputs. Copies of the decrypted data that the task's executors write
|
|
168
|
+
outside the volume are not removed.
|
|
169
|
+
|
|
170
|
+
TES runs a task's executors in order and stops at the first that fails
|
|
171
|
+
(TES 1.1, `ignore_error`). So, by default, the teardown executor runs only
|
|
172
|
+
if all of the task's own executors succeed: after a failed executor, a
|
|
173
|
+
canceled task or a system error, the decrypted files stay in the volume
|
|
174
|
+
until the backend removes the task's working directory (see below). A
|
|
175
|
+
failed decryption removes its own files.
|
|
176
|
+
|
|
177
|
+
`teardown_on_error` sets `ignore_error: true` on the task's own executors
|
|
178
|
+
(not on the decryption and teardown executors), so that the teardown
|
|
179
|
+
executor runs also after one of them fails. It is off by default, as it
|
|
180
|
+
changes how the task runs and how it is reported, and not all backends
|
|
181
|
+
support it (as of the sources of Funnel and TESK in November 2026):
|
|
182
|
+
|
|
183
|
+
- The task's next executors run after one of them fails, also those that
|
|
184
|
+
need the failed executor's results.
|
|
185
|
+
- The task may no longer end with `EXECUTOR_ERROR` when one of its own
|
|
186
|
+
executors fails: TES says that the backend records the exit code and
|
|
187
|
+
goes on. The final state differs between backends: the current sources
|
|
188
|
+
of Funnel report the task as `COMPLETE` if the teardown succeeds, while
|
|
189
|
+
Funnel 0.11.14 (the image that the [end-to-end test](#end-to-end-test)
|
|
190
|
+
ran in November 2026) still ends it with `EXECUTOR_ERROR`, after running
|
|
191
|
+
the teardown. Clients must check the exit codes in the executor logs.
|
|
192
|
+
- TESK accepts only `ignore_error: false` and refuses other tasks with
|
|
193
|
+
`400 Bad Request`: don't set `teardown_on_error` with TESK backends.
|
|
194
|
+
- Backends that implement TES 1.0, which has no `ignore_error`, may ignore
|
|
195
|
+
the field and stop at the first failed executor as before.
|
|
196
|
+
- Executors that set `ignore_error: false` themselves get `true`.
|
|
197
|
+
|
|
198
|
+
Without the teardown, or where it doesn't run, the decrypted files are
|
|
199
|
+
removed with the task's working directory, as far as the backend removes
|
|
200
|
+
it: Funnel removes it after the task unless its worker setting
|
|
201
|
+
`LeaveWorkDir` is set; TESK deletes the task's volume claim after the
|
|
202
|
+
task, unless `KEEP_FAILED_PVC` keeps that of tasks that don't complete.
|
|
203
|
+
|
|
204
|
+
### The decryption image
|
|
205
|
+
|
|
206
|
+
Each release of the plugin publishes its image on Docker Hub, as
|
|
207
|
+
[`elixircloud/foca-plugin-tes-data-crypt4gh-decrypt`](https://hub.docker.com/r/elixircloud/foca-plugin-tes-data-crypt4gh-decrypt),
|
|
208
|
+
tagged with the plugin's version (e.g., `0.1.0`; there is no `latest`).
|
|
209
|
+
Use the image of the plugin version that you install, and set `image` to
|
|
210
|
+
it by digest, which Docker Hub shows for the tag, as does:
|
|
211
|
+
|
|
212
|
+
```bash
|
|
213
|
+
docker buildx imagetools inspect docker.io/elixircloud/foca-plugin-tes-data-crypt4gh-decrypt:<version>
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
Before the release workflow signs the image (Sigstore keyless signing, with
|
|
217
|
+
the workflow's identity), attests its build provenance and tags it, it
|
|
218
|
+
scans the image for known vulnerabilities and runs its smoke test on it.
|
|
219
|
+
Verify the signature and the attestation before you use it:
|
|
220
|
+
|
|
221
|
+
```bash
|
|
222
|
+
image=docker.io/elixircloud/foca-plugin-tes-data-crypt4gh-decrypt@sha256:<digest>
|
|
223
|
+
cosign verify "${image}" \
|
|
224
|
+
--certificate-identity-regexp '^https://github\.com/elixir-cloud-aai/foca-plugins/\.github/workflows/release\.yml@refs/tags/foca-plugin-tes-data-crypt4gh-v' \
|
|
225
|
+
--certificate-oidc-issuer https://token.actions.githubusercontent.com
|
|
226
|
+
gh attestation verify "oci://${image}" --repo elixir-cloud-aai/foca-plugins
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
The image is built from [`image/Dockerfile`](image/Dockerfile) (Python 3.13,
|
|
230
|
+
`crypt4gh` from [`image/requirements.txt`](image/requirements.txt), pinned
|
|
231
|
+
with hashes, without pip, `linux/amd64`), with the plugin's directory as
|
|
232
|
+
the build context. To build it yourself (e.g., for a registry that your TES backends
|
|
233
|
+
can pull from, if they can't pull from Docker Hub), from the repository
|
|
234
|
+
root:
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
docker build -f plugins/tes/data/crypt4gh/image/Dockerfile \
|
|
238
|
+
-t <registry>/<name>:<tag> plugins/tes/data/crypt4gh
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
CI builds the image for every pull request, runs
|
|
242
|
+
[`image/smoke.sh`](image/smoke.sh) on it and scans it. The image runs as
|
|
243
|
+
root, as some TES implementations (e.g., Funnel with host directories)
|
|
244
|
+
make the task's volumes writable for root only.
|
|
245
|
+
|
|
246
|
+
## Configuration
|
|
247
|
+
|
|
248
|
+
Admin settings, in the gateway app's configuration (FOCA's `plugins`
|
|
249
|
+
section), e.g., for proTES (overlays replace lists, so repeat the route
|
|
250
|
+
step of your pipeline; the decryption step comes first, as it changes the
|
|
251
|
+
task):
|
|
252
|
+
|
|
253
|
+
```yaml
|
|
254
|
+
plugins:
|
|
255
|
+
available:
|
|
256
|
+
tes.data.crypt4gh:
|
|
257
|
+
distribution: foca-plugin-tes-data-crypt4gh
|
|
258
|
+
config:
|
|
259
|
+
# The image of the installed plugin version, by digest
|
|
260
|
+
image: docker.io/elixircloud/foca-plugin-tes-data-crypt4gh-decrypt@sha256:<digest>
|
|
261
|
+
pipeline:
|
|
262
|
+
- id: decrypt
|
|
263
|
+
alternatives:
|
|
264
|
+
- plugin: tes.data.crypt4gh
|
|
265
|
+
- id: route
|
|
266
|
+
alternatives:
|
|
267
|
+
- plugin: generic.routing.random
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
proTES's image doesn't contain the plugin: install it in an image derived
|
|
271
|
+
from that of proTES (see "Installing" in the docs, and proTES's "Plugin
|
|
272
|
+
images"). In proTES's Helm chart, set the same under `plugins.available`
|
|
273
|
+
and `plugins.pipeline`.
|
|
274
|
+
|
|
275
|
+
| Setting | Type | Default | Description |
|
|
276
|
+
| --- | --- | --- | --- |
|
|
277
|
+
| `config.image` | string, 1 to 512 characters, no spaces | none; required | image of the decryption executor (see [The decryption image](#the-decryption-image)) |
|
|
278
|
+
| `config.command` | 1 to 16 strings | `["crypt4gh-decrypt"]` | command of the decryption script in the image; the plugin appends its arguments |
|
|
279
|
+
| `config.volume` | absolute, normalized path below `/` | `/vol/crypt4gh` | volume of the decrypted files; tasks may not use it or paths in it |
|
|
280
|
+
| `config.encrypted_suffixes` | 1 to 16 suffixes (a dot and 1 to 32 characters, no `/` or spaces) | `[".c4gh"]` | suffixes of the names of Crypt4GH inputs; the longest that matches is removed from the decrypted file's name |
|
|
281
|
+
| `config.key_suffixes` | as `encrypted_suffixes`; none may end with one of them or the other way round (e.g., `.key.c4gh` and `.c4gh`) | `[".sec"]` | suffixes of the names of secret keys among the inputs |
|
|
282
|
+
| `config.teardown` | boolean | `true` | add the teardown executor, which empties the volume after the task's own executors (see [The teardown executor](#the-teardown-executor)) |
|
|
283
|
+
| `config.teardown_on_error` | boolean; needs `teardown` | `false` | set `ignore_error: true` on the task's own executors, so that the teardown executor runs after they fail; not for all backends (see [The teardown executor](#the-teardown-executor)) |
|
|
284
|
+
|
|
285
|
+
The plugin has no parameters.
|
|
286
|
+
|
|
287
|
+
## Keys and decrypted data
|
|
288
|
+
|
|
289
|
+
The plugin moves no data itself, but the tasks that it changes do:
|
|
290
|
+
|
|
291
|
+
- **Secret keys are inputs of the task**, as with the middleware: they are
|
|
292
|
+
downloaded by the TES backend, available to all of the task's executors,
|
|
293
|
+
and, if a client sends them as inline `content`, stored with the task by
|
|
294
|
+
the gateway and the backend. Prefer keys made for a single task or
|
|
295
|
+
dataset (re-encrypt the data for them), and pass them by URL from storage
|
|
296
|
+
that only the task can read.
|
|
297
|
+
- **Decrypted files are written to the task's volume**, on the backend's
|
|
298
|
+
disks. The teardown executor removes them, with everything else in the
|
|
299
|
+
volume, after the task's own executors, if they all succeed (or with
|
|
300
|
+
`teardown_on_error`, where the backend supports it); otherwise they are
|
|
301
|
+
removed with the task's working directory, as far as the backend
|
|
302
|
+
removes it (see [The teardown executor](#the-teardown-executor)). Removing a file doesn't overwrite
|
|
303
|
+
its blocks on the disk, and copies that the task's executors make
|
|
304
|
+
outside the volume stay.
|
|
305
|
+
|
|
306
|
+
Use the plugin only with TES backends that you trust with the decrypted
|
|
307
|
+
data and the keys, e.g., in a trusted research environment.
|
|
308
|
+
|
|
309
|
+
## Data transfers
|
|
310
|
+
|
|
311
|
+
None: the plugin calls no services, and the decryption executor, which
|
|
312
|
+
runs on the TES backend, needs no network.
|
|
313
|
+
|
|
314
|
+
## Changes from the middleware
|
|
315
|
+
|
|
316
|
+
The plugin replaces the Crypt4GH middleware of proTES
|
|
317
|
+
(`protes-middleware-crypt4gh`), which proTES can't load since it adopted
|
|
318
|
+
FOCA's plugin engine:
|
|
319
|
+
|
|
320
|
+
- Only inputs named as Crypt4GH files (`.c4gh`) are decrypted, and only
|
|
321
|
+
keys named as keys (`.sec`) are passed to the executor; other inputs stay
|
|
322
|
+
where they are, instead of being moved to the volume.
|
|
323
|
+
- Decrypted files lose their suffix (`reads.fq.c4gh` decrypts to
|
|
324
|
+
`reads.fq`), are readable by all (0644), and are written to the volume;
|
|
325
|
+
inputs are only read, so they may be mounted read-only.
|
|
326
|
+
- A failed decryption fails the task (exit status 1), instead of passing
|
|
327
|
+
the encrypted file on.
|
|
328
|
+
- The volume is a setting (`/vol/crypt4gh`), not a random path per process,
|
|
329
|
+
and tasks that use it are rejected (`422`), as are tasks with Crypt4GH
|
|
330
|
+
inputs but no key.
|
|
331
|
+
- The image is a setting, built from this repository, instead of a fixed
|
|
332
|
+
image.
|
|
333
|
+
- A teardown executor empties the volume after the task's own
|
|
334
|
+
executors (`teardown`).
|
|
335
|
+
|
|
336
|
+
## Installation
|
|
337
|
+
|
|
338
|
+
```bash
|
|
339
|
+
pip install foca-plugin-tes-data-crypt4gh
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
See "Installing" in the docs for release trains and derived images.
|
|
343
|
+
|
|
344
|
+
## Development
|
|
345
|
+
|
|
346
|
+
From the repository root, with the development environment set up (see
|
|
347
|
+
"Contributing" in the docs):
|
|
348
|
+
|
|
349
|
+
```bash
|
|
350
|
+
uv run ruff check plugins/tes/data/crypt4gh
|
|
351
|
+
uv run python -m scripts.packages mypy plugins/tes/data/crypt4gh
|
|
352
|
+
uv run coverage run --source plugins/tes/data/crypt4gh/src \
|
|
353
|
+
-m pytest plugins/tes/data/crypt4gh
|
|
354
|
+
uv run coverage report
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
The tests make their keys and Crypt4GH files with `crypt4gh` (a
|
|
358
|
+
development dependency), in temporary directories: the repository holds no
|
|
359
|
+
keys. With Docker, build the image and run its smoke test as CI does:
|
|
360
|
+
|
|
361
|
+
```bash
|
|
362
|
+
docker build -f plugins/tes/data/crypt4gh/image/Dockerfile \
|
|
363
|
+
-t crypt4gh-decrypt:dev plugins/tes/data/crypt4gh
|
|
364
|
+
plugins/tes/data/crypt4gh/image/smoke.sh crypt4gh-decrypt:dev
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
### End-to-end test
|
|
368
|
+
|
|
369
|
+
[`e2e/check.sh`](e2e/check.sh) runs the plugin end to end, as CI does (job
|
|
370
|
+
`e2e`): proTES (a release image, with the plugin's wheel built from the
|
|
371
|
+
checkout) sends tasks to Funnel, a real TES backend, in Docker Compose
|
|
372
|
+
([`e2e/compose.yaml`](e2e/compose.yaml); the versions are pinned in
|
|
373
|
+
`e2e/Dockerfile.protes` and `e2e/Dockerfile.funnel`). Their inputs, a Crypt4GH
|
|
374
|
+
file and a secret key made in the test with the decryption image (also built
|
|
375
|
+
from the checkout), are served over HTTP, and Funnel downloads them as from
|
|
376
|
+
any server. It checks that:
|
|
377
|
+
|
|
378
|
+
1. a task completes, its executor reads the decrypted content, and the
|
|
379
|
+
teardown executor leaves the volume of the decrypted files empty;
|
|
380
|
+
2. a failing decryption (no key for one of two files) fails the task before
|
|
381
|
+
its own executors run, and leaves no decrypted files;
|
|
382
|
+
3. without `teardown_on_error`, a failing executor of the task stops it
|
|
383
|
+
before the teardown, and the decrypted file stays in the volume (until
|
|
384
|
+
the backend removes the task's working directory, see [The teardown
|
|
385
|
+
executor](#the-teardown-executor));
|
|
386
|
+
4. with `teardown_on_error`, the teardown runs after the failing executor
|
|
387
|
+
and empties the volume (the task's final state is the backend's, so
|
|
388
|
+
either `EXECUTOR_ERROR` or `COMPLETE` passes).
|
|
389
|
+
|
|
390
|
+
Funnel keeps the tasks' working directories in the test, so that it can
|
|
391
|
+
check the volume on the host. TESK refuses tasks with `ignore_error: true`,
|
|
392
|
+
so `teardown_on_error` is tested with Funnel only, and the test has no TESK
|
|
393
|
+
variant. It needs Docker with Compose, uv, curl and jq (without them, or
|
|
394
|
+
with a bad work directory, it exits with status 2 before starting
|
|
395
|
+
anything), but no credentials:
|
|
396
|
+
|
|
397
|
+
```bash
|
|
398
|
+
plugins/tes/data/crypt4gh/e2e/check.sh
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
It removes the stack, the task containers and its work directory when it
|
|
402
|
+
ends; see "Contributing" in the docs for its settings.
|
|
403
|
+
|
|
404
|
+
## Changelog
|
|
405
|
+
|
|
406
|
+
See [`CHANGELOG.md`](CHANGELOG.md).
|