pybiolib 0.2.951__py3-none-any.whl → 1.2.1890__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (262) hide show
  1. biolib/__init__.py +357 -11
  2. biolib/_data_record/data_record.py +380 -0
  3. biolib/_index/__init__.py +0 -0
  4. biolib/_index/index.py +55 -0
  5. biolib/_index/query_result.py +103 -0
  6. biolib/_internal/__init__.py +0 -0
  7. biolib/_internal/add_copilot_prompts.py +58 -0
  8. biolib/_internal/add_gui_files.py +81 -0
  9. biolib/_internal/data_record/__init__.py +1 -0
  10. biolib/_internal/data_record/data_record.py +85 -0
  11. biolib/_internal/data_record/push_data.py +116 -0
  12. biolib/_internal/data_record/remote_storage_endpoint.py +43 -0
  13. biolib/_internal/errors.py +5 -0
  14. biolib/_internal/file_utils.py +125 -0
  15. biolib/_internal/fuse_mount/__init__.py +1 -0
  16. biolib/_internal/fuse_mount/experiment_fuse_mount.py +209 -0
  17. biolib/_internal/http_client.py +159 -0
  18. biolib/_internal/lfs/__init__.py +1 -0
  19. biolib/_internal/lfs/cache.py +51 -0
  20. biolib/_internal/libs/__init__.py +1 -0
  21. biolib/_internal/libs/fusepy/__init__.py +1257 -0
  22. biolib/_internal/push_application.py +488 -0
  23. biolib/_internal/runtime.py +22 -0
  24. biolib/_internal/string_utils.py +13 -0
  25. biolib/_internal/templates/__init__.py +1 -0
  26. biolib/_internal/templates/copilot_template/.github/instructions/general-app-knowledge.instructions.md +10 -0
  27. biolib/_internal/templates/copilot_template/.github/instructions/style-general.instructions.md +20 -0
  28. biolib/_internal/templates/copilot_template/.github/instructions/style-python.instructions.md +16 -0
  29. biolib/_internal/templates/copilot_template/.github/instructions/style-react-ts.instructions.md +47 -0
  30. biolib/_internal/templates/copilot_template/.github/prompts/biolib_app_inputs.prompt.md +11 -0
  31. biolib/_internal/templates/copilot_template/.github/prompts/biolib_onboard_repo.prompt.md +19 -0
  32. biolib/_internal/templates/copilot_template/.github/prompts/biolib_run_apps.prompt.md +12 -0
  33. biolib/_internal/templates/dashboard_template/.biolib/config.yml +5 -0
  34. biolib/_internal/templates/github_workflow_template/.github/workflows/biolib.yml +21 -0
  35. biolib/_internal/templates/gitignore_template/.gitignore +10 -0
  36. biolib/_internal/templates/gui_template/.yarnrc.yml +1 -0
  37. biolib/_internal/templates/gui_template/App.tsx +53 -0
  38. biolib/_internal/templates/gui_template/Dockerfile +27 -0
  39. biolib/_internal/templates/gui_template/biolib-sdk.ts +82 -0
  40. biolib/_internal/templates/gui_template/dev-data/output.json +7 -0
  41. biolib/_internal/templates/gui_template/index.css +5 -0
  42. biolib/_internal/templates/gui_template/index.html +13 -0
  43. biolib/_internal/templates/gui_template/index.tsx +10 -0
  44. biolib/_internal/templates/gui_template/package.json +27 -0
  45. biolib/_internal/templates/gui_template/tsconfig.json +24 -0
  46. biolib/_internal/templates/gui_template/vite-plugin-dev-data.ts +50 -0
  47. biolib/_internal/templates/gui_template/vite.config.mts +10 -0
  48. biolib/_internal/templates/init_template/.biolib/config.yml +19 -0
  49. biolib/_internal/templates/init_template/Dockerfile +14 -0
  50. biolib/_internal/templates/init_template/requirements.txt +1 -0
  51. biolib/_internal/templates/init_template/run.py +12 -0
  52. biolib/_internal/templates/init_template/run.sh +4 -0
  53. biolib/_internal/templates/templates.py +25 -0
  54. biolib/_internal/tree_utils.py +106 -0
  55. biolib/_internal/utils/__init__.py +65 -0
  56. biolib/_internal/utils/auth.py +46 -0
  57. biolib/_internal/utils/job_url.py +33 -0
  58. biolib/_internal/utils/multinode.py +263 -0
  59. biolib/_runtime/runtime.py +157 -0
  60. biolib/_session/session.py +44 -0
  61. biolib/_shared/__init__.py +0 -0
  62. biolib/_shared/types/__init__.py +74 -0
  63. biolib/_shared/types/account.py +12 -0
  64. biolib/_shared/types/account_member.py +8 -0
  65. biolib/_shared/types/app.py +9 -0
  66. biolib/_shared/types/data_record.py +40 -0
  67. biolib/_shared/types/experiment.py +32 -0
  68. biolib/_shared/types/file_node.py +17 -0
  69. biolib/_shared/types/push.py +6 -0
  70. biolib/_shared/types/resource.py +37 -0
  71. biolib/_shared/types/resource_deploy_key.py +11 -0
  72. biolib/_shared/types/resource_permission.py +14 -0
  73. biolib/_shared/types/resource_version.py +19 -0
  74. biolib/_shared/types/result.py +14 -0
  75. biolib/_shared/types/typing.py +10 -0
  76. biolib/_shared/types/user.py +19 -0
  77. biolib/_shared/utils/__init__.py +7 -0
  78. biolib/_shared/utils/resource_uri.py +75 -0
  79. biolib/api/__init__.py +6 -0
  80. biolib/api/client.py +168 -0
  81. biolib/app/app.py +252 -49
  82. biolib/app/search_apps.py +45 -0
  83. biolib/biolib_api_client/api_client.py +126 -31
  84. biolib/biolib_api_client/app_types.py +24 -4
  85. biolib/biolib_api_client/auth.py +31 -8
  86. biolib/biolib_api_client/biolib_app_api.py +147 -52
  87. biolib/biolib_api_client/biolib_job_api.py +161 -141
  88. biolib/biolib_api_client/job_types.py +21 -5
  89. biolib/biolib_api_client/lfs_types.py +7 -23
  90. biolib/biolib_api_client/user_state.py +56 -0
  91. biolib/biolib_binary_format/__init__.py +1 -4
  92. biolib/biolib_binary_format/file_in_container.py +105 -0
  93. biolib/biolib_binary_format/module_input.py +24 -7
  94. biolib/biolib_binary_format/module_output_v2.py +149 -0
  95. biolib/biolib_binary_format/remote_endpoints.py +34 -0
  96. biolib/biolib_binary_format/remote_stream_seeker.py +59 -0
  97. biolib/biolib_binary_format/saved_job.py +3 -2
  98. biolib/biolib_binary_format/{attestation_document.py → stdout_and_stderr.py} +8 -8
  99. biolib/biolib_binary_format/system_status_update.py +3 -2
  100. biolib/biolib_binary_format/utils.py +175 -0
  101. biolib/biolib_docker_client/__init__.py +11 -2
  102. biolib/biolib_errors.py +36 -0
  103. biolib/biolib_logging.py +27 -10
  104. biolib/cli/__init__.py +38 -0
  105. biolib/cli/auth.py +46 -0
  106. biolib/cli/data_record.py +164 -0
  107. biolib/cli/index.py +32 -0
  108. biolib/cli/init.py +421 -0
  109. biolib/cli/lfs.py +101 -0
  110. biolib/cli/push.py +50 -0
  111. biolib/cli/run.py +63 -0
  112. biolib/cli/runtime.py +14 -0
  113. biolib/cli/sdk.py +16 -0
  114. biolib/cli/start.py +56 -0
  115. biolib/compute_node/cloud_utils/cloud_utils.py +110 -161
  116. biolib/compute_node/job_worker/cache_state.py +66 -88
  117. biolib/compute_node/job_worker/cache_types.py +1 -6
  118. biolib/compute_node/job_worker/docker_image_cache.py +112 -37
  119. biolib/compute_node/job_worker/executors/__init__.py +0 -3
  120. biolib/compute_node/job_worker/executors/docker_executor.py +532 -199
  121. biolib/compute_node/job_worker/executors/docker_types.py +9 -1
  122. biolib/compute_node/job_worker/executors/types.py +19 -9
  123. biolib/compute_node/job_worker/job_legacy_input_wait_timeout_thread.py +30 -0
  124. biolib/compute_node/job_worker/job_max_runtime_timer_thread.py +3 -5
  125. biolib/compute_node/job_worker/job_storage.py +108 -0
  126. biolib/compute_node/job_worker/job_worker.py +397 -212
  127. biolib/compute_node/job_worker/large_file_system.py +87 -38
  128. biolib/compute_node/job_worker/network_alloc.py +99 -0
  129. biolib/compute_node/job_worker/network_buffer.py +240 -0
  130. biolib/compute_node/job_worker/utilization_reporter_thread.py +197 -0
  131. biolib/compute_node/job_worker/utils.py +9 -24
  132. biolib/compute_node/remote_host_proxy.py +400 -98
  133. biolib/compute_node/utils.py +31 -9
  134. biolib/compute_node/webserver/compute_node_results_proxy.py +189 -0
  135. biolib/compute_node/webserver/proxy_utils.py +28 -0
  136. biolib/compute_node/webserver/webserver.py +130 -44
  137. biolib/compute_node/webserver/webserver_types.py +2 -6
  138. biolib/compute_node/webserver/webserver_utils.py +77 -12
  139. biolib/compute_node/webserver/worker_thread.py +183 -42
  140. biolib/experiments/__init__.py +0 -0
  141. biolib/experiments/experiment.py +356 -0
  142. biolib/jobs/__init__.py +1 -0
  143. biolib/jobs/job.py +741 -0
  144. biolib/jobs/job_result.py +185 -0
  145. biolib/jobs/types.py +50 -0
  146. biolib/py.typed +0 -0
  147. biolib/runtime/__init__.py +14 -0
  148. biolib/sdk/__init__.py +91 -0
  149. biolib/tables.py +34 -0
  150. biolib/typing_utils.py +2 -7
  151. biolib/user/__init__.py +1 -0
  152. biolib/user/sign_in.py +54 -0
  153. biolib/utils/__init__.py +162 -0
  154. biolib/utils/cache_state.py +94 -0
  155. biolib/utils/multipart_uploader.py +194 -0
  156. biolib/utils/seq_util.py +150 -0
  157. biolib/utils/zip/remote_zip.py +640 -0
  158. pybiolib-1.2.1890.dist-info/METADATA +41 -0
  159. pybiolib-1.2.1890.dist-info/RECORD +177 -0
  160. {pybiolib-0.2.951.dist-info → pybiolib-1.2.1890.dist-info}/WHEEL +1 -1
  161. pybiolib-1.2.1890.dist-info/entry_points.txt +2 -0
  162. README.md +0 -17
  163. biolib/app/app_result.py +0 -68
  164. biolib/app/utils.py +0 -62
  165. biolib/biolib-js/0-biolib.worker.js +0 -1
  166. biolib/biolib-js/1-biolib.worker.js +0 -1
  167. biolib/biolib-js/2-biolib.worker.js +0 -1
  168. biolib/biolib-js/3-biolib.worker.js +0 -1
  169. biolib/biolib-js/4-biolib.worker.js +0 -1
  170. biolib/biolib-js/5-biolib.worker.js +0 -1
  171. biolib/biolib-js/6-biolib.worker.js +0 -1
  172. biolib/biolib-js/index.html +0 -10
  173. biolib/biolib-js/main-biolib.js +0 -1
  174. biolib/biolib_api_client/biolib_account_api.py +0 -21
  175. biolib/biolib_api_client/biolib_large_file_system_api.py +0 -108
  176. biolib/biolib_binary_format/aes_encrypted_package.py +0 -42
  177. biolib/biolib_binary_format/module_output.py +0 -58
  178. biolib/biolib_binary_format/rsa_encrypted_aes_package.py +0 -57
  179. biolib/biolib_push.py +0 -114
  180. biolib/cli.py +0 -203
  181. biolib/cli_utils.py +0 -273
  182. biolib/compute_node/cloud_utils/enclave_parent_types.py +0 -7
  183. biolib/compute_node/enclave/__init__.py +0 -2
  184. biolib/compute_node/enclave/enclave_remote_hosts.py +0 -53
  185. biolib/compute_node/enclave/nitro_secure_module_utils.py +0 -64
  186. biolib/compute_node/job_worker/executors/base_executor.py +0 -18
  187. biolib/compute_node/job_worker/executors/pyppeteer_executor.py +0 -173
  188. biolib/compute_node/job_worker/executors/remote/__init__.py +0 -1
  189. biolib/compute_node/job_worker/executors/remote/nitro_enclave_utils.py +0 -81
  190. biolib/compute_node/job_worker/executors/remote/remote_executor.py +0 -51
  191. biolib/lfs.py +0 -196
  192. biolib/pyppeteer/.circleci/config.yml +0 -100
  193. biolib/pyppeteer/.coveragerc +0 -3
  194. biolib/pyppeteer/.gitignore +0 -89
  195. biolib/pyppeteer/.pre-commit-config.yaml +0 -28
  196. biolib/pyppeteer/CHANGES.md +0 -253
  197. biolib/pyppeteer/CONTRIBUTING.md +0 -26
  198. biolib/pyppeteer/LICENSE +0 -12
  199. biolib/pyppeteer/README.md +0 -137
  200. biolib/pyppeteer/docs/Makefile +0 -177
  201. biolib/pyppeteer/docs/_static/custom.css +0 -28
  202. biolib/pyppeteer/docs/_templates/layout.html +0 -10
  203. biolib/pyppeteer/docs/changes.md +0 -1
  204. biolib/pyppeteer/docs/conf.py +0 -299
  205. biolib/pyppeteer/docs/index.md +0 -21
  206. biolib/pyppeteer/docs/make.bat +0 -242
  207. biolib/pyppeteer/docs/reference.md +0 -211
  208. biolib/pyppeteer/docs/server.py +0 -60
  209. biolib/pyppeteer/poetry.lock +0 -1699
  210. biolib/pyppeteer/pyppeteer/__init__.py +0 -135
  211. biolib/pyppeteer/pyppeteer/accessibility.py +0 -286
  212. biolib/pyppeteer/pyppeteer/browser.py +0 -401
  213. biolib/pyppeteer/pyppeteer/browser_fetcher.py +0 -194
  214. biolib/pyppeteer/pyppeteer/command.py +0 -22
  215. biolib/pyppeteer/pyppeteer/connection/__init__.py +0 -242
  216. biolib/pyppeteer/pyppeteer/connection/cdpsession.py +0 -101
  217. biolib/pyppeteer/pyppeteer/coverage.py +0 -346
  218. biolib/pyppeteer/pyppeteer/device_descriptors.py +0 -787
  219. biolib/pyppeteer/pyppeteer/dialog.py +0 -79
  220. biolib/pyppeteer/pyppeteer/domworld.py +0 -597
  221. biolib/pyppeteer/pyppeteer/emulation_manager.py +0 -53
  222. biolib/pyppeteer/pyppeteer/errors.py +0 -48
  223. biolib/pyppeteer/pyppeteer/events.py +0 -63
  224. biolib/pyppeteer/pyppeteer/execution_context.py +0 -156
  225. biolib/pyppeteer/pyppeteer/frame/__init__.py +0 -299
  226. biolib/pyppeteer/pyppeteer/frame/frame_manager.py +0 -306
  227. biolib/pyppeteer/pyppeteer/helpers.py +0 -245
  228. biolib/pyppeteer/pyppeteer/input.py +0 -371
  229. biolib/pyppeteer/pyppeteer/jshandle.py +0 -598
  230. biolib/pyppeteer/pyppeteer/launcher.py +0 -683
  231. biolib/pyppeteer/pyppeteer/lifecycle_watcher.py +0 -169
  232. biolib/pyppeteer/pyppeteer/models/__init__.py +0 -103
  233. biolib/pyppeteer/pyppeteer/models/_protocol.py +0 -12460
  234. biolib/pyppeteer/pyppeteer/multimap.py +0 -82
  235. biolib/pyppeteer/pyppeteer/network_manager.py +0 -678
  236. biolib/pyppeteer/pyppeteer/options.py +0 -8
  237. biolib/pyppeteer/pyppeteer/page.py +0 -1728
  238. biolib/pyppeteer/pyppeteer/pipe_transport.py +0 -59
  239. biolib/pyppeteer/pyppeteer/target.py +0 -147
  240. biolib/pyppeteer/pyppeteer/task_queue.py +0 -24
  241. biolib/pyppeteer/pyppeteer/timeout_settings.py +0 -36
  242. biolib/pyppeteer/pyppeteer/tracing.py +0 -93
  243. biolib/pyppeteer/pyppeteer/us_keyboard_layout.py +0 -305
  244. biolib/pyppeteer/pyppeteer/util.py +0 -18
  245. biolib/pyppeteer/pyppeteer/websocket_transport.py +0 -47
  246. biolib/pyppeteer/pyppeteer/worker.py +0 -101
  247. biolib/pyppeteer/pyproject.toml +0 -97
  248. biolib/pyppeteer/spell.txt +0 -137
  249. biolib/pyppeteer/tox.ini +0 -72
  250. biolib/pyppeteer/utils/generate_protocol_types.py +0 -603
  251. biolib/start_cli.py +0 -7
  252. biolib/utils.py +0 -47
  253. biolib/validators/validate_app_version.py +0 -183
  254. biolib/validators/validate_argument.py +0 -134
  255. biolib/validators/validate_module.py +0 -323
  256. biolib/validators/validate_zip_file.py +0 -40
  257. biolib/validators/validator_utils.py +0 -103
  258. pybiolib-0.2.951.dist-info/LICENSE +0 -21
  259. pybiolib-0.2.951.dist-info/METADATA +0 -61
  260. pybiolib-0.2.951.dist-info/RECORD +0 -153
  261. pybiolib-0.2.951.dist-info/entry_points.txt +0 -3
  262. /LICENSE → /pybiolib-1.2.1890.dist-info/licenses/LICENSE +0 -0
@@ -0,0 +1,488 @@
1
+ import json
2
+ import os
3
+ import re
4
+ import sys
5
+ import time
6
+ from pathlib import Path
7
+
8
+ import rich.progress
9
+ import yaml
10
+
11
+ from biolib import api, utils
12
+ from biolib._internal.data_record.push_data import (
13
+ push_data_path,
14
+ validate_data_path_and_get_files_and_size_of_directory,
15
+ )
16
+ from biolib._internal.errors import AuthenticationError
17
+ from biolib._internal.file_utils import get_files_and_size_of_directory, get_iterable_zip_stream
18
+ from biolib._shared.types import PushResponseDict
19
+ from biolib._shared.utils import parse_resource_uri
20
+ from biolib.biolib_api_client import BiolibApiClient
21
+ from biolib.biolib_api_client.biolib_app_api import BiolibAppApi
22
+ from biolib.biolib_docker_client import BiolibDockerClient
23
+ from biolib.biolib_errors import BioLibError
24
+ from biolib.biolib_logging import logger
25
+ from biolib.typing_utils import Dict, Iterable, Optional, Set, TypedDict, Union
26
+
27
+ REGEX_MARKDOWN_INLINE_IMAGE = re.compile(r'!\[(?P<alt>.*)\]\((?P<src>.*)\)')
28
+
29
+
30
+ class DockerProgressDetail(TypedDict):
31
+ current: int
32
+ total: int
33
+
34
+
35
+ class DockerStatusUpdate(TypedDict, total=False):
36
+ status: str
37
+ progressDetail: DockerProgressDetail
38
+ progress: str
39
+ id: str
40
+
41
+
42
+ def process_docker_status_updates(status_updates: Iterable[DockerStatusUpdate], action: str) -> None:
43
+ if sys.stdout.isatty():
44
+ _process_docker_status_updates_with_progress_bar(status_updates, action)
45
+ else:
46
+ _process_docker_status_updates_with_logging(status_updates, action)
47
+
48
+
49
+ def _process_docker_status_updates_with_progress_bar(status_updates: Iterable[DockerStatusUpdate], action: str) -> None:
50
+ with rich.progress.Progress() as progress:
51
+ layer_id_to_task_id = {}
52
+ overall_task_id = progress.add_task(description=f'[bold blue]{action} Docker image', total=None)
53
+
54
+ for update in status_updates:
55
+ if 'progressDetail' in update and 'id' in update:
56
+ layer_id = update['id']
57
+ progress_detail = update['progressDetail']
58
+
59
+ if layer_id not in layer_id_to_task_id:
60
+ layer_id_to_task_id[layer_id] = progress.add_task(description=f'[cyan]{action} layer {layer_id}')
61
+
62
+ if progress_detail and 'current' in progress_detail and 'total' in progress_detail:
63
+ progress.update(
64
+ task_id=layer_id_to_task_id[layer_id],
65
+ completed=progress_detail['current'],
66
+ total=progress_detail['total'],
67
+ )
68
+ elif update['status'] == 'Layer already exists':
69
+ progress.update(
70
+ completed=100,
71
+ task_id=layer_id_to_task_id[layer_id],
72
+ total=100,
73
+ )
74
+ elif 'status' in update and 'id' in update:
75
+ layer_id = update['id']
76
+ status = update['status']
77
+
78
+ if layer_id not in layer_id_to_task_id:
79
+ layer_id_to_task_id[layer_id] = progress.add_task(description=f'[cyan]{action} layer {layer_id}')
80
+
81
+ if status in ['Preparing', 'Waiting']:
82
+ progress.update(
83
+ task_id=layer_id_to_task_id[layer_id], description=f'[yellow]{status} layer {layer_id}'
84
+ )
85
+ elif status in ['Pushing', 'Uploading']:
86
+ progress.update(
87
+ task_id=layer_id_to_task_id[layer_id], description=f'[cyan]{status} layer {layer_id}'
88
+ )
89
+ elif status in ['Pushed', 'Uploaded']:
90
+ progress.update(
91
+ task_id=layer_id_to_task_id[layer_id],
92
+ description=f'[green]{status} layer {layer_id}',
93
+ completed=100,
94
+ total=100,
95
+ )
96
+ elif status == 'Layer already exists':
97
+ progress.update(
98
+ task_id=layer_id_to_task_id[layer_id],
99
+ description=f'[green]{status} - layer {layer_id}',
100
+ completed=100,
101
+ total=100,
102
+ )
103
+ elif 'status' in update and update['status']:
104
+ status = update['status']
105
+ if status not in ['Preparing', 'Pushing', 'Pushed', 'Waiting', 'Layer already exists']:
106
+ progress.update(task_id=overall_task_id, description=f'[bold blue]{action} Docker image - {status}')
107
+ elif 'status' not in update and 'progressDetail' not in update:
108
+ print(update)
109
+
110
+
111
+ def _process_docker_status_updates_with_logging(status_updates: Iterable[DockerStatusUpdate], action: str) -> None:
112
+ layer_progress: Dict[str, float] = {}
113
+ layer_status: Dict[str, str] = {}
114
+ layer_details: Dict[str, Dict[str, int]] = {}
115
+ layer_bytes_at_last_log: Dict[str, int] = {}
116
+ last_log_time = time.time()
117
+
118
+ logger.info(f'{action} Docker image...')
119
+
120
+ for update in status_updates:
121
+ current_time = time.time()
122
+
123
+ if 'progressDetail' in update and 'id' in update:
124
+ layer_id = update['id']
125
+ progress_detail = update['progressDetail']
126
+
127
+ if progress_detail and 'current' in progress_detail and 'total' in progress_detail:
128
+ current = progress_detail['current']
129
+ total = progress_detail['total']
130
+ percentage = (current / total * 100) if total > 0 else 0
131
+ layer_progress[layer_id] = percentage
132
+ layer_status[layer_id] = f'{action.lower()}'
133
+ layer_details[layer_id] = {'current': current, 'total': total}
134
+ elif update.get('status') == 'Layer already exists':
135
+ layer_progress[layer_id] = 100
136
+ layer_status[layer_id] = 'already exists'
137
+
138
+ elif 'status' in update and 'id' in update:
139
+ layer_id = update['id']
140
+ status = update['status']
141
+ layer_status[layer_id] = status.lower()
142
+
143
+ if status in ['Pushed', 'Uploaded'] or status == 'Layer already exists':
144
+ layer_progress[layer_id] = 100
145
+
146
+ elif 'status' in update and update['status']:
147
+ status = update['status']
148
+ if status not in ['Preparing', 'Pushing', 'Pushed', 'Waiting', 'Layer already exists']:
149
+ logger.info(f'{action} Docker image - {status}')
150
+
151
+ if current_time - last_log_time >= 10.0:
152
+ _log_progress_summary(
153
+ action,
154
+ layer_progress,
155
+ layer_status,
156
+ layer_details,
157
+ layer_bytes_at_last_log,
158
+ current_time - last_log_time,
159
+ )
160
+ layer_bytes_at_last_log = {lid: details['current'] for lid, details in layer_details.items()}
161
+ last_log_time = current_time
162
+
163
+ _log_progress_summary(
164
+ action, layer_progress, layer_status, layer_details, layer_bytes_at_last_log, time.time() - last_log_time
165
+ )
166
+ if action == 'Pushing':
167
+ logger.info('Pushing final image manifest...')
168
+ logger.info(f'{action} Docker image completed')
169
+
170
+
171
+ def _log_progress_summary(
172
+ action: str,
173
+ layer_progress: Dict[str, float],
174
+ layer_status: Dict[str, str],
175
+ layer_details: Dict[str, Dict[str, int]],
176
+ layer_bytes_at_last_log: Dict[str, int],
177
+ time_delta: float,
178
+ ) -> None:
179
+ if not layer_progress and not layer_status:
180
+ return
181
+
182
+ completed_layers = sum(1 for progress in layer_progress.values() if progress >= 100)
183
+ total_layers = len(layer_progress) if layer_progress else len(layer_status)
184
+
185
+ if total_layers > 0:
186
+ overall_percentage = completed_layers / total_layers * 100
187
+ logger.info(
188
+ f'{action} progress: {completed_layers}/{total_layers} layers completed ({overall_percentage:.1f}%)'
189
+ )
190
+
191
+ active_layers = [
192
+ layer_id
193
+ for layer_id, status in layer_status.items()
194
+ if status in ['preparing', 'waiting', 'pushing', 'uploading'] and layer_progress.get(layer_id, 0) < 100
195
+ ]
196
+
197
+ if active_layers and layer_details:
198
+ total_bytes_transferred = 0
199
+ layer_info_parts = []
200
+
201
+ for layer_id in active_layers[:5]:
202
+ if layer_id in layer_details:
203
+ details = layer_details[layer_id]
204
+ current = details['current']
205
+ total = details['total']
206
+ percentage = layer_progress.get(layer_id, 0)
207
+
208
+ bytes_since_last = current - layer_bytes_at_last_log.get(layer_id, 0)
209
+ total_bytes_transferred += bytes_since_last
210
+
211
+ current_mb = current / (1024 * 1024)
212
+ total_mb = total / (1024 * 1024)
213
+ layer_info_parts.append(f'{layer_id}: {current_mb:.1f}/{total_mb:.1f} MB ({percentage:.1f}%)')
214
+
215
+ speed_info = ''
216
+ if time_delta > 0 and total_bytes_transferred > 0:
217
+ speed_mbps = (total_bytes_transferred / (1024 * 1024)) / time_delta
218
+ speed_info = f' @ {speed_mbps:.2f} MB/s'
219
+
220
+ more_layers_info = ''
221
+ if len(active_layers) > 5:
222
+ more_layers_info = f' (+ {len(active_layers) - 5} more)'
223
+
224
+ if layer_info_parts:
225
+ logger.info(f'Active layers: {", ".join(layer_info_parts)}{speed_info}{more_layers_info}')
226
+ elif active_layers:
227
+ logger.info(f'Active layers: {", ".join(active_layers[:5])}{"..." if len(active_layers) > 5 else ""}')
228
+
229
+
230
+ def set_app_version_as_active(
231
+ app_version_uuid: str,
232
+ ):
233
+ logger.debug(f'Setting app version {app_version_uuid} as active.')
234
+ api.client.patch(
235
+ path=f'/app_versions/{app_version_uuid}/',
236
+ data={'set_as_active': True},
237
+ )
238
+
239
+
240
+ def push_application(
241
+ app_uri: str,
242
+ app_path: str,
243
+ app_version_to_copy_images_from: Optional[str],
244
+ set_as_active: bool,
245
+ set_as_published: bool,
246
+ dry_run: bool = False,
247
+ ) -> Optional[PushResponseDict]:
248
+ app_uri = app_uri.rstrip('/')
249
+ parsed_uri = parse_resource_uri(app_uri)
250
+ resource_name = parsed_uri['resource_name']
251
+
252
+ app_uri_prefix = f"@{parsed_uri['resource_prefix']}/" if parsed_uri['resource_prefix'] is not None else ''
253
+ app_uri_to_fetch = f"{app_uri_prefix}{parsed_uri['account_handle_normalized']}/{resource_name}"
254
+
255
+ version = parsed_uri['version']
256
+ semantic_version = f"{version['major']}.{version['minor']}.{version['patch']}" if version else None
257
+
258
+ app_path_absolute = Path(app_path).resolve()
259
+
260
+ api_client = BiolibApiClient.get()
261
+ if not api_client.is_signed_in:
262
+ github_repository = os.getenv('GITHUB_REPOSITORY')
263
+ if github_repository and not api_client.resource_deploy_key:
264
+ github_secrets_url = f'https://github.com/{github_repository}/settings/secrets/actions/new'
265
+ raise AuthenticationError(
266
+ 'You must be authenticated to push an application.\n'
267
+ 'Please set the environment variable "BIOLIB_TOKEN=[your_deploy_token]"\n'
268
+ f'You can get a deploy key at: {api_client.base_url}/{app_uri_to_fetch}/settings/keys/\n'
269
+ f'Then add it to your GitHub repository at: {github_secrets_url}'
270
+ )
271
+ else:
272
+ raise AuthenticationError(
273
+ 'You must be authenticated to push an application.\n'
274
+ 'Please set the environment variable "BIOLIB_TOKEN=[your_deploy_token]"\n'
275
+ f'You can get a deploy key at: {api_client.base_url}/{app_uri_to_fetch}/settings/keys/'
276
+ )
277
+
278
+ # prepare zip file
279
+ config_yml_path = app_path_absolute.joinpath('.biolib/config.yml')
280
+ if not config_yml_path.is_file():
281
+ raise BioLibError('The file .biolib/config.yml was not found in the application directory')
282
+
283
+ zip_filters: Set[str] = set()
284
+ zip_filters.add('.biolib/config.yml')
285
+
286
+ input_files_maps_to_root = False
287
+ app_data_path: Optional[Path] = None
288
+ try:
289
+ with open(config_yml_path) as config_yml_file:
290
+ try:
291
+ config = json.loads(json.dumps(yaml.safe_load(config_yml_file.read())))
292
+ except (TypeError, ValueError) as e:
293
+ raise BioLibError(
294
+ f'The .biolib/config.yml file contains data types that are not supported '
295
+ f'(must be JSON-serializable). Please ensure only standard JSON types '
296
+ f'(str, int, float, bool, list, dict, null) are used. Original error: {e}'
297
+ ) from e
298
+
299
+ if 'assets' in config and 'app_data' not in config:
300
+ config['app_data'] = config.pop('assets')
301
+ elif 'assets' in config and 'app_data' in config:
302
+ raise BioLibError(
303
+ 'In .biolib/config.yml you cannot specify both "app_data" and "assets" fields. Please use only one.'
304
+ )
305
+
306
+ app_data = config.get('app_data')
307
+ if app_data:
308
+ field_name = 'app_data' if 'app_data' in config else 'assets'
309
+ if not isinstance(app_data, str):
310
+ raise BioLibError(
311
+ f'In .biolib/config.yml the value of "{field_name}" must be a string but got {type(app_data)}'
312
+ )
313
+
314
+ app_data_path = app_path_absolute.joinpath(app_data).resolve()
315
+ if not app_data_path.is_dir():
316
+ raise BioLibError(
317
+ f'In .biolib/config.yml the value of "{field_name}" must be a path to a directory '
318
+ 'in the application directory'
319
+ )
320
+
321
+ license_file_relative_path = config.get('license_file', 'LICENSE')
322
+ if app_path_absolute.joinpath(license_file_relative_path).is_file():
323
+ zip_filters.add(license_file_relative_path)
324
+
325
+ description_file_relative_path = config.get('description_file', 'README.md')
326
+ description_file_absolute_path = app_path_absolute.joinpath(description_file_relative_path)
327
+ if not description_file_absolute_path.is_file():
328
+ raise BioLibError(f'Could not find {description_file_relative_path}')
329
+
330
+ zip_filters.add(description_file_relative_path)
331
+ with open(description_file_absolute_path) as description_file:
332
+ description_file_content = description_file.read()
333
+
334
+ for _, img_src_path in re.findall(REGEX_MARKDOWN_INLINE_IMAGE, description_file_content):
335
+ zip_filters.add(img_src_path)
336
+
337
+ for _, module in config['modules'].items():
338
+ if module.get('source_files'):
339
+ zip_filters.add('*')
340
+
341
+ for mapping in module.get('input_files', []):
342
+ mapping_parts = mapping.split(' ')
343
+ if len(mapping_parts) == 3 and mapping_parts[2] == '/':
344
+ input_files_maps_to_root = True
345
+
346
+ except BioLibError as error:
347
+ raise error from None
348
+
349
+ except Exception as error:
350
+ raise BioLibError('Failed to parse the .biolib/config.yml file') from error
351
+
352
+ if input_files_maps_to_root:
353
+ logger.error(
354
+ 'Error: In your config.yml some module maps input files to "/" (root). '
355
+ 'This is potentially an unsafe operation as it allows the user to '
356
+ 'overwrite system executables in the module.'
357
+ )
358
+ exit(1)
359
+
360
+ files_in_app_dir, _ = get_files_and_size_of_directory(directory=str(app_path_absolute))
361
+ files_to_zip: Set[str] = set()
362
+
363
+ for file_path in files_in_app_dir:
364
+ for pattern in zip_filters:
365
+ if pattern == '*' or pattern == file_path:
366
+ files_to_zip.add(file_path)
367
+ break
368
+
369
+ original_directory = os.getcwd()
370
+ os.chdir(app_path_absolute.parent)
371
+ files_with_app_dir_prefixed = [f'{app_path_absolute.stem}/{path}' for path in files_to_zip]
372
+
373
+ # Workaround as backend currently expects directory objects for root level and .biolib directory
374
+ files_with_app_dir_prefixed.append(f'{app_path_absolute.stem}/')
375
+ files_with_app_dir_prefixed.append(f'{app_path_absolute.stem}/.biolib/')
376
+
377
+ byte_iterator = get_iterable_zip_stream(files_with_app_dir_prefixed, chunk_size=50_000_000)
378
+ source_files_zip_bytes = b''.join(byte_iterator)
379
+ os.chdir(original_directory)
380
+
381
+ if app_version_to_copy_images_from and app_version_to_copy_images_from != 'active':
382
+ # Get app with `app_version_to_copy_images_from` in app_uri_to_fetch to get the app version public id.
383
+ app_uri_to_fetch += f':{app_version_to_copy_images_from}'
384
+
385
+ app_response = BiolibAppApi.get_by_uri(app_uri_to_fetch)
386
+ app = app_response['app']
387
+
388
+ if dry_run:
389
+ logger.info('Successfully completed dry-run. No new version was pushed.')
390
+ return None
391
+
392
+ new_app_version_json = BiolibAppApi.push_app_version(
393
+ semantic_version=semantic_version,
394
+ app_id=app['public_id'],
395
+ app_name=app['name'],
396
+ author=app['account_handle'],
397
+ set_as_active=False,
398
+ zip_binary=source_files_zip_bytes,
399
+ app_version_id_to_copy_images_from=app_response['app_version']['public_id']
400
+ if app_version_to_copy_images_from
401
+ else None,
402
+ )
403
+
404
+ if app_data_path:
405
+ app_data_files_to_zip, app_data_size_in_bytes = validate_data_path_and_get_files_and_size_of_directory(
406
+ data_path=str(app_data_path),
407
+ )
408
+ push_data_path(
409
+ resource_version_uuid=new_app_version_json['public_id'],
410
+ data_path=str(app_data_path),
411
+ data_size_in_bytes=app_data_size_in_bytes,
412
+ files_to_zip=app_data_files_to_zip,
413
+ )
414
+
415
+ # Don't push docker images if copying from another app version
416
+ docker_tags = new_app_version_json.get('docker_tags', {})
417
+ if not app_version_to_copy_images_from and docker_tags:
418
+ logger.info('Found docker images to push.')
419
+ docker_client = BiolibDockerClient.get_docker_client()
420
+
421
+ for module_name, repo_and_tag in docker_tags.items():
422
+ docker_image_definition = config['modules'][module_name]['image']
423
+ repo, tag = repo_and_tag.split(':')
424
+
425
+ if docker_image_definition.startswith('dockerhub://'):
426
+ docker_image_name = docker_image_definition.replace('dockerhub://', 'docker.io/', 1)
427
+ logger.info(f'Pulling image {docker_image_name} defined on module {module_name} from Dockerhub.')
428
+ dockerhub_repo, dockerhub_tag = docker_image_name.split(':')
429
+ pull_status_updates: Iterable[DockerStatusUpdate] = docker_client.api.pull(
430
+ decode=True,
431
+ platform='linux/amd64',
432
+ repository=dockerhub_repo,
433
+ stream=True,
434
+ tag=dockerhub_tag,
435
+ )
436
+
437
+ process_docker_status_updates(pull_status_updates, action='Pulling')
438
+
439
+ elif docker_image_definition.startswith('local-docker://'):
440
+ docker_image_name = docker_image_definition.replace('local-docker://', '', 1)
441
+
442
+ try:
443
+ logger.info(f'Trying to push image {docker_image_name} defined on module {module_name}.')
444
+ image = docker_client.images.get(docker_image_name)
445
+ architecture = image.attrs.get('Architecture')
446
+ if architecture != 'amd64':
447
+ print(f"Error: '{docker_image_name}' is compiled for {architecture}, expected x86 (amd64).")
448
+ print('If you are on an ARM processor, try passing --platform linux/amd64 to docker build.')
449
+ exit(1)
450
+ absolute_repo_uri = f'{utils.BIOLIB_SITE_HOSTNAME}/{repo}'
451
+ image.tag(absolute_repo_uri, tag)
452
+
453
+ push_status_updates: Iterable[DockerStatusUpdate] = docker_client.images.push(
454
+ absolute_repo_uri,
455
+ tag=tag,
456
+ stream=True,
457
+ decode=True,
458
+ auth_config={
459
+ 'username': 'biolib',
460
+ # For legacy reasons access token is sent with trailing comma ','
461
+ 'password': api_client.resource_deploy_key or f'{api_client.access_token},',
462
+ },
463
+ )
464
+
465
+ process_docker_status_updates(push_status_updates, action='Pushing')
466
+
467
+ except Exception as exception:
468
+ raise BioLibError(f'Failed to tag and push image {docker_image_name}.') from exception
469
+
470
+ logger.info(f'Successfully pushed {docker_image_name}')
471
+
472
+ app_version_uuid = new_app_version_json['public_id']
473
+ complete_push_data: Dict[str, Union[bool, str]] = {
474
+ 'set_as_active': set_as_active,
475
+ 'set_as_published': set_as_published,
476
+ }
477
+ if parsed_uri['tag']:
478
+ complete_push_data['tag'] = parsed_uri['tag']
479
+ api.client.post(
480
+ path=f'/app-versions/{app_version_uuid}/complete-push/',
481
+ data=complete_push_data,
482
+ )
483
+
484
+ sematic_version = f"{new_app_version_json['major']}.{new_app_version_json['minor']}.{new_app_version_json['patch']}"
485
+ version_name = 'development ' if not set_as_published else ''
486
+ logger.info(f'Successfully pushed new {version_name}version {sematic_version} of {app_uri}.')
487
+
488
+ return {'app_uri': app_uri, 'sematic_version': sematic_version}
@@ -0,0 +1,22 @@
1
+ from biolib.typing_utils import TypedDict
2
+
3
+
4
+ class RuntimeJobDataDict(TypedDict):
5
+ version: str
6
+ job_requested_machine: str
7
+ job_requested_machine_spot: bool
8
+ job_uuid: str
9
+ job_auth_token: str
10
+ app_uri: str
11
+ is_environment_biolib_cloud: bool
12
+ job_reserved_machines: int
13
+
14
+
15
+ class BioLibRuntimeError(Exception):
16
+ pass
17
+
18
+
19
+ class BioLibRuntimeNotRecognizedError(BioLibRuntimeError):
20
+ def __init__(self, message='The runtime is not recognized as a BioLib app'):
21
+ self.message = message
22
+ super().__init__(self.message)
@@ -0,0 +1,13 @@
1
+ import re
2
+
3
+
4
+ def normalize_for_docker_tag(name: str) -> str:
5
+ if not name:
6
+ return ''
7
+
8
+ normalized = re.sub(r'[^a-z0-9-]', '-', name.lower())
9
+
10
+ normalized = re.sub(r'-+', '-', normalized)
11
+ normalized = normalized.strip('-')
12
+
13
+ return normalized
@@ -0,0 +1 @@
1
+ from . import templates
@@ -0,0 +1,10 @@
1
+ ---
2
+ applyTo: "**"
3
+ ---
4
+
5
+ You are writing code that runs inside a BioLib app. In general, most BioLib apps are structured such that there is a `run.py` or `main.py` file that contains the main function.
6
+ Other files are usually helper files that contain functions that are called from the main function.
7
+
8
+ BioLib apps often contain a Vite React Typescript project that compiles to a single HTML file used to render interactive and visual output.
9
+
10
+ BioLib apps run inside a Docker container, which is built from the `Dockerfile` in the root of the app.
@@ -0,0 +1,20 @@
1
+ ---
2
+ applyTo: '**'
3
+ ---
4
+
5
+ # General Coding Guidelines
6
+ - Variable names are allowed to be verbose, and should be descriptive.
7
+ - Unit tests are not necessary. Tests should instead be written as simple examples demonstrating the functionality of relevant functions.
8
+ - Always use 4 spaces for indentation.
9
+ - Function definitions should be typed as specifically as possible.
10
+ - Always explicitly type function parameters and return values.
11
+ - Avoid using the `Any` type unless absolutely necessary; use specific types instead.
12
+ - Write code with the expectation that a strict type checker will be used.
13
+ - Prefer failing fast with clear errors over silently providing fallback values.
14
+ - Always lock down the exact version of external dependencies (e.g., use `package==1.2.3` instead of `package>=1.2.3`).
15
+
16
+ # Code Comment Guidelines
17
+ - Code comments should only be added for complex logic or unintuitive code that is not adequately explained by the function names themselves.
18
+ - Avoid comments that simply restate what the code does.
19
+ - When writing comments explaining some non-trivial logic, explain blocks of code rather than individual lines.
20
+ - Explain changes to the code in chat, not in comments.
@@ -0,0 +1,16 @@
1
+ ---
2
+ applyTo: '**/*.py'
3
+ ---
4
+
5
+ Apply the [general coding guidelines](./style-general.instructions.md) to all code.
6
+
7
+ # Python-specific code style guidelines
8
+ - Use snake_case for function and variable names, and PascalCase for class names.
9
+ - Place all imports at the top of the file, grouped in the following order: standard library imports, third-party imports, local application imports. Separate each group with a blank line.
10
+ - Limit all lines to a maximum of 120 characters.
11
+ - Prefer single quotes for strings unless the string contains a single quote, in which case use double quotes.
12
+ - Use blank lines to separate functions, class definitions, and logical sections within functions.
13
+ - Use type hints for function arguments and return values where possible.
14
+ - Use docstrings to document functions that are used in the main python script, but not elsewhere.
15
+ - Avoid using bare except clauses; always specify the exception being caught.
16
+ - Use list comprehensions and generator expressions where appropriate for clarity and conciseness.
@@ -0,0 +1,47 @@
1
+ ---
2
+ applyTo: "**/*.ts,**/*.tsx,**/package.json,**/vite.config.*,**/.yarnrc.yml,**/yarn.lock,**/gui/**"
3
+ ---
4
+
5
+ Apply the [general coding guidelines](./style-general.instructions.md) to all code.
6
+
7
+ # General Project Guidelines
8
+ - Prefer using `export default function` over exporting at the end of the file.
9
+
10
+ # Package Management
11
+ - **Always use yarn instead of npm** for all package management operations
12
+ - Use `yarn install` instead of `npm install`
13
+ - Use `yarn add <package>` instead of `npm install <package>`
14
+ - Use `yarn remove <package>` instead of `npm uninstall <package>`
15
+ - Use `yarn dev` instead of `npm run dev`
16
+ - Use `yarn build` instead of `npm run build`
17
+
18
+ # Build Process
19
+ - BioLib GUI projects use Vite for building and development
20
+ - The build process compiles TypeScript and React into a single HTML file
21
+ - Always run `yarn build` to create the production build before deployment
22
+ - Use `yarn dev` for local development with hot reloading
23
+
24
+ # Configuration Files
25
+ - Respect the `.yarnrc.yml` configuration for yarn settings
26
+ - The `package.json` should specify `"packageManager": "yarn@4.6.0"` or similar
27
+ - Never modify yarn.lock manually - let yarn manage it automatically
28
+
29
+ # Dependencies
30
+ - Add new dependencies using `yarn add <package>` for runtime dependencies
31
+ - Add development dependencies using `yarn add -D <package>`
32
+ - Keep dependencies up to date but test thoroughly after updates
33
+
34
+ # TypeScript Guidelines
35
+ - Use TypeScript for all new code
36
+ - Follow functional programming principles where possible
37
+ - Use interfaces for data structures prefixed with I like `interface IRecord`
38
+ - Prefer immutable data (const, readonly)
39
+ - Use optional chaining (?.) and nullish coalescing (??) operators
40
+
41
+ # React Guidelines
42
+ - Use functional components with hooks
43
+ - Follow the React hooks rules (no conditional hooks)
44
+ - Prefer one component per file
45
+ - Use Tailwindcss for styling
46
+ - Extract props in components with object destructuring like `const { prop1, prop2 } = props;`
47
+ - Instantiate functional components with props like `export default function MyComponent(props: IProps) { ... }`.
@@ -0,0 +1,11 @@
1
+ ---
2
+ mode: 'agent'
3
+ tools: ['codebase', 'fetch']
4
+ description: 'Handle changing inputs for a biolib app, by adding, removing or modifying inputs in the config.yml and the python script.'
5
+ ---
6
+
7
+ # Main task
8
+ Your task is to make sure that all inputs are handled correctly in the give Python script and/or biolib config file.
9
+ Read the documentation [here](https://biolib.com/docs/building-applications/syntax-of-config-yml/) to understand how the inputs are handled in the config.yml file.
10
+ Inputs in the Python script should be parsed with argparse. If a default is given in the config.yml file, it it is not necessary to set a default in the Python script.
11
+ Otherwise, the two files should be consistent.
@@ -0,0 +1,19 @@
1
+ ---
2
+ mode: 'agent'
3
+ tools: ['githubRepo', 'codebase']
4
+ description: 'Handle onboarding and implementing code from a GitHub repository into a biolib application, with focus on creating easily editable and maintainable code structure.'
5
+ ---
6
+
7
+ # Main task
8
+ Your task is to help onboard and implement code from a GitHub repository into a biolib application. This involves understanding the repository structure, implementing the core functionality, and ensuring the code is easily editable for future iterations.
9
+ Generally, you can do this by adding the repository into an src folder as a submodule, and reading the README.md file to understand how to run the code.
10
+ You will then call the relevant functions or classes from the cloned repository in your biolib application
11
+
12
+ ## Key requirements:
13
+ - Always ask the user for the GitHub repository if not already provided. Inform them that it needs to be in the format `author/repo_name`.
14
+ - Use the #githubRepo tool to examine the repository structure, README, and key files to understand the project.
15
+ - Focus on creating code that is easily editable and maintainable, as it's likely the implementation won't be perfect on the first attempt.
16
+ - Structure the code in a modular way that allows for easy modifications and improvements.
17
+ - Include clear comments for complex logic, but avoid over-commenting obvious code.
18
+ - Follow the existing biolib application patterns and conventions.
19
+ - Ensure all dependencies are properly specified in requirements.txt with versions locked down.
@@ -0,0 +1,12 @@
1
+ ---
2
+ mode: 'agent'
3
+ tools: ['githubRepo', 'codebase', 'fetch']
4
+ description: 'Handle running biolib apps, including login, running apps, and managing jobs and results.'
5
+ ---
6
+
7
+ # Main task
8
+ Your task is to run one or more biolib apps, using the biolib Python API. You can find general instructions [here](https://biolib.com/docs/using-applications/python/)
9
+ A few relevant notes:
10
+ - You will be running this from inside a biolib app, so login is not necessary.
11
+ - Always look at the relevant app's #githubRepo. Ask the user for the repo, and inform them that it needs to be in the format `author/app_name`.
12
+ - If you do look at the repo, look at the config.yml to see how it expects inputs to be formatted.