idmtools-platform-container 0.0.0.dev0__tar.gz → 0.0.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. idmtools_platform_container-0.0.2/.bumpversion.cfg +28 -0
  2. idmtools_platform_container-0.0.2/LICENSE.TXT +3 -0
  3. idmtools_platform_container-0.0.2/MANIFEST.in +2 -0
  4. idmtools_platform_container-0.0.2/Makefile +12 -0
  5. idmtools_platform_container-0.0.2/PKG-INFO +212 -0
  6. idmtools_platform_container-0.0.2/README.md +182 -0
  7. idmtools_platform_container-0.0.2/docker_image/BASE_VERSION +1 -0
  8. idmtools_platform_container-0.0.2/docker_image/Dockerfile +48 -0
  9. idmtools_platform_container-0.0.2/docker_image/Dockerfile_buildenv +46 -0
  10. idmtools_platform_container-0.0.2/docker_image/ImageName +1 -0
  11. idmtools_platform_container-0.0.2/docker_image/README.md +78 -0
  12. idmtools_platform_container-0.0.2/docker_image/__init__.py +6 -0
  13. idmtools_platform_container-0.0.2/docker_image/build_docker_image.py +145 -0
  14. idmtools_platform_container-0.0.2/docker_image/debian/BASE_VERSION +1 -0
  15. idmtools_platform_container-0.0.2/docker_image/debian/Dockerfile +40 -0
  16. idmtools_platform_container-0.0.2/docker_image/debian/ImageName +1 -0
  17. idmtools_platform_container-0.0.2/docker_image/debian/README.md +48 -0
  18. idmtools_platform_container-0.0.2/docker_image/debian/pip.conf +3 -0
  19. idmtools_platform_container-0.0.2/docker_image/debian/requirements.txt +1 -0
  20. idmtools_platform_container-0.0.2/docker_image/docker_image_history.py +101 -0
  21. idmtools_platform_container-0.0.2/docker_image/pip.conf +3 -0
  22. idmtools_platform_container-0.0.2/docker_image/push_docker_image.py +62 -0
  23. idmtools_platform_container-0.0.2/docker_image/requirements.txt +1 -0
  24. idmtools_platform_container-0.0.2/docker_image/rocky_meta_runtime.txt +37 -0
  25. idmtools_platform_container-0.0.2/idmtools_platform_container/__init__.py +18 -0
  26. idmtools_platform_container-0.0.2/idmtools_platform_container/cli/__init__.py +5 -0
  27. idmtools_platform_container-0.0.2/idmtools_platform_container/cli/container.py +682 -0
  28. idmtools_platform_container-0.0.2/idmtools_platform_container/container_operations/__init__.py +5 -0
  29. idmtools_platform_container-0.0.2/idmtools_platform_container/container_operations/docker_operations.py +593 -0
  30. idmtools_platform_container-0.0.2/idmtools_platform_container/container_platform.py +375 -0
  31. idmtools_platform_container-0.0.2/idmtools_platform_container/platform_operations/__init__.py +5 -0
  32. idmtools_platform_container-0.0.2/idmtools_platform_container/platform_operations/experiment_operations.py +112 -0
  33. idmtools_platform_container-0.0.2/idmtools_platform_container/platform_operations/simulation_operations.py +58 -0
  34. idmtools_platform_container-0.0.2/idmtools_platform_container/plugin_info.py +79 -0
  35. idmtools_platform_container-0.0.2/idmtools_platform_container/utils/__init__.py +5 -0
  36. idmtools_platform_container-0.0.2/idmtools_platform_container/utils/general.py +136 -0
  37. idmtools_platform_container-0.0.2/idmtools_platform_container/utils/status.py +130 -0
  38. idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/PKG-INFO +212 -0
  39. idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/SOURCES.txt +95 -0
  40. idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/entry_points.txt +5 -0
  41. idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/requires.txt +15 -0
  42. {idmtools_platform_container-0.0.0.dev0 → idmtools_platform_container-0.0.2}/idmtools_platform_container.egg-info/top_level.txt +2 -0
  43. idmtools_platform_container-0.0.2/pyproject.toml +83 -0
  44. {idmtools_platform_container-0.0.0.dev0 → idmtools_platform_container-0.0.2}/setup.cfg +4 -4
  45. idmtools_platform_container-0.0.2/tests/Makefile +8 -0
  46. idmtools_platform_container-0.0.2/tests/__init__.py +4 -0
  47. idmtools_platform_container-0.0.2/tests/cleanup_containers.py +13 -0
  48. idmtools_platform_container-0.0.2/tests/helps.py +36 -0
  49. idmtools_platform_container-0.0.2/tests/idmtools_container_exp_dir.ini +3 -0
  50. idmtools_platform_container-0.0.2/tests/idmtools_container_sim_dir.ini +2 -0
  51. idmtools_platform_container-0.0.2/tests/inputs/Assets/MyLib/functions.py +2 -0
  52. idmtools_platform_container-0.0.2/tests/inputs/__init__.py +0 -0
  53. idmtools_platform_container-0.0.2/tests/inputs/model.py +28 -0
  54. idmtools_platform_container-0.0.2/tests/inputs/model1.py +31 -0
  55. idmtools_platform_container-0.0.2/tests/inputs/model3.py +21 -0
  56. idmtools_platform_container-0.0.2/tests/inputs/model_file.py +18 -0
  57. idmtools_platform_container-0.0.2/tests/inputs/run.sh +1 -0
  58. idmtools_platform_container-0.0.2/tests/inputs/sleep.py +9 -0
  59. idmtools_platform_container-0.0.2/tests/pytest.ini +17 -0
  60. idmtools_platform_container-0.0.2/tests/test_analyzers.py +144 -0
  61. idmtools_platform_container-0.0.2/tests/test_bash_sh.py +133 -0
  62. idmtools_platform_container-0.0.2/tests/test_container_cli/__init__.py +0 -0
  63. idmtools_platform_container-0.0.2/tests/test_container_cli/helper.py +57 -0
  64. idmtools_platform_container-0.0.2/tests/test_container_cli/test_base.py +14 -0
  65. idmtools_platform_container-0.0.2/tests/test_container_cli/test_cancel.py +96 -0
  66. idmtools_platform_container-0.0.2/tests/test_container_cli/test_clear_results.py +54 -0
  67. idmtools_platform_container-0.0.2/tests/test_container_cli/test_container.py +72 -0
  68. idmtools_platform_container-0.0.2/tests/test_container_cli/test_file_container_cli.py +121 -0
  69. idmtools_platform_container-0.0.2/tests/test_container_cli/test_get_detail.py +60 -0
  70. idmtools_platform_container-0.0.2/tests/test_container_cli/test_history.py +136 -0
  71. idmtools_platform_container-0.0.2/tests/test_container_cli/test_history_count.py +53 -0
  72. idmtools_platform_container-0.0.2/tests/test_container_cli/test_inspect.py +53 -0
  73. idmtools_platform_container-0.0.2/tests/test_container_cli/test_install.py +48 -0
  74. idmtools_platform_container-0.0.2/tests/test_container_cli/test_is_running.py +69 -0
  75. idmtools_platform_container-0.0.2/tests/test_container_cli/test_jobs.py +138 -0
  76. idmtools_platform_container-0.0.2/tests/test_container_cli/test_list_containers.py +99 -0
  77. idmtools_platform_container-0.0.2/tests/test_container_cli/test_packages.py +41 -0
  78. idmtools_platform_container-0.0.2/tests/test_container_cli/test_path.py +96 -0
  79. idmtools_platform_container-0.0.2/tests/test_container_cli/test_ps.py +47 -0
  80. idmtools_platform_container-0.0.2/tests/test_container_cli/test_remove_container.py +78 -0
  81. idmtools_platform_container-0.0.2/tests/test_container_cli/test_status.py +149 -0
  82. idmtools_platform_container-0.0.2/tests/test_container_cli/test_stop_container.py +71 -0
  83. idmtools_platform_container-0.0.2/tests/test_container_cli/test_sync_history.py +98 -0
  84. idmtools_platform_container-0.0.2/tests/test_container_cli/test_verify_docker.py +28 -0
  85. idmtools_platform_container-0.0.2/tests/test_container_cli/test_volume.py +28 -0
  86. idmtools_platform_container-0.0.2/tests/test_container_platform.py +441 -0
  87. idmtools_platform_container-0.0.2/tests/test_create_sim_directory_map.py +64 -0
  88. idmtools_platform_container-0.0.2/tests/test_docker_operations.py +812 -0
  89. idmtools_platform_container-0.0.2/tests/test_filter_by_tags.py +121 -0
  90. idmtools_platform_container-0.0.2/tests/test_filter_item.py +141 -0
  91. idmtools_platform_container-0.0.2/tests/test_flatten_item.py +122 -0
  92. idmtools_platform_container-0.0.2/tests/test_get_files.py +135 -0
  93. idmtools_platform_container-0.0.2/tests/test_get_methods.py +184 -0
  94. idmtools_platform_container-0.0.2/tests/test_max_file_path.py +37 -0
  95. idmtools_platform_container-0.0.2/tests/test_platform_experiment.py +419 -0
  96. idmtools_platform_container-0.0.2/tests/utils.py +90 -0
  97. idmtools_platform_container-0.0.0.dev0/PKG-INFO +0 -41
  98. idmtools_platform_container-0.0.0.dev0/README.md +0 -21
  99. idmtools_platform_container-0.0.0.dev0/idmtools_platform_container/__init__.py +0 -8
  100. idmtools_platform_container-0.0.0.dev0/idmtools_platform_container.egg-info/PKG-INFO +0 -41
  101. idmtools_platform_container-0.0.0.dev0/idmtools_platform_container.egg-info/SOURCES.txt +0 -7
  102. idmtools_platform_container-0.0.0.dev0/pyproject.toml +0 -32
  103. {idmtools_platform_container-0.0.0.dev0 → idmtools_platform_container-0.0.2}/idmtools_platform_container.egg-info/dependency_links.txt +0 -0
@@ -0,0 +1,28 @@
1
+ [bumpversion]
2
+ current_version = 3.0.0+nightly
3
+ commit = False
4
+ tag = False
5
+ parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(?P<release>\+\w+)?(\.(?P<build>\d+))?
6
+ serialize =
7
+ {major}.{minor}.{patch}{release}.{build}
8
+ {major}.{minor}.{patch}{release}
9
+ {major}.{minor}.{patch}
10
+
11
+ [bumpversion:part:release]
12
+ optional_value = prod
13
+ first_value = prod
14
+ values =
15
+ prod
16
+ +nightly
17
+
18
+ [bumpversion:file:./idmtools_platform_container/__init__.py]
19
+ search = __version__ = "{current_version}"
20
+ replace = __version__ = "{new_version}"
21
+ serialize = {major}.{minor}.{patch}{release}
22
+ {major}.{minor}.{patch}
23
+
24
+ [bumpversion:file:./pyproject.toml]
25
+ search = version = "{current_version}"
26
+ replace = version = "{new_version}"
27
+ serialize = {major}.{minor}.{patch}{release}
28
+ {major}.{minor}.{patch}
@@ -0,0 +1,3 @@
1
+ idmtools is licensed under the Creative Commons Attribution-Noncommercial-ShareAlike 4.0 License.
2
+
3
+ To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode
@@ -0,0 +1,2 @@
1
+ include README.md
2
+ include LICENSE.TXT
@@ -0,0 +1,12 @@
1
+ PACKAGE_NAME=idmtools_platform_container
2
+ include $(abspath ../dev_scripts/package_general.mk)
3
+
4
+ help:
5
+ help-from-makefile -f ../dev_scripts/package_general.mk -f ./Makefile
6
+
7
+ # Release related rules
8
+ container-image: ## Build container Docker image
9
+ $(MAKE) -C docker_image docker
10
+
11
+ container-image-release: ## Push image to staging
12
+ $(MAKE) -C docker_image docker-release-staging
@@ -0,0 +1,212 @@
1
+ Metadata-Version: 2.4
2
+ Name: idmtools_platform_container
3
+ Version: 0.0.2
4
+ Summary: Container platform for IDM-Tools
5
+ Author-email: Zhaowei Du <zdu@idmod.org>, Sharon Chen <shchen@idmod.org>, Clinton Collins <ccollins@idmod.org>, Benoit Raybaud <braybaud@idmod.org>, Clark Kirkman IV <ckirkman@idmod.org>, Ye Chen <yechen@idmod.org>, Mary Fisher <mafisher@idmod.org>, Mandy Izzo <mizzo@idmod.org>, Jen Schripsema <jschripsema@idmod.org>, Ross Carter <rcarter@idmod.org>
6
+ Project-URL: Homepage, https://github.com/InstituteforDiseaseModeling/idmtools
7
+ Keywords: modeling,IDM
8
+ Classifier: Programming Language :: Python :: 3.8
9
+ Classifier: Programming Language :: Python :: 3.9
10
+ Classifier: Programming Language :: Python :: 3.10
11
+ Classifier: Programming Language :: Python :: 3.11
12
+ Classifier: Programming Language :: Python :: 3.12
13
+ Requires-Python: >=3.8
14
+ Description-Content-Type: text/markdown
15
+ License-File: LICENSE.TXT
16
+ Requires-Dist: idmtools_platform_general<1.0.0,>=0.0.0
17
+ Requires-Dist: docker>5.0
18
+ Requires-Dist: rich
19
+ Provides-Extra: test
20
+ Requires-Dist: idmtools[test]; extra == "test"
21
+ Requires-Dist: idmtools_test; extra == "test"
22
+ Requires-Dist: idmtools_models; extra == "test"
23
+ Provides-Extra: packaging
24
+ Requires-Dist: flake8; extra == "packaging"
25
+ Requires-Dist: coverage; extra == "packaging"
26
+ Requires-Dist: bump2version; extra == "packaging"
27
+ Requires-Dist: twine; extra == "packaging"
28
+ Requires-Dist: natsort; extra == "packaging"
29
+ Dynamic: license-file
30
+
31
+ ![Staging: idmtools-platform-container](https://github.com/InstituteforDiseaseModeling/idmtools/workflows/Staging:%20idmtools-platform-container/badge.svg?branch=dev)
32
+
33
+ # Idmtools platform container
34
+
35
+ <!-- START doctoc generated TOC please keep comment here to allow auto update -->
36
+ <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
37
+ **Table of Contents**
38
+
39
+ - [Introduction](#introduction)
40
+ - [Pre-requisites](#pre-requisites)
41
+ - [Installation](#installation)
42
+ - [Examples for container platform](#examples-for-container-platform)
43
+ - [Initialize platform](#initialize-platform)
44
+ - [Container Examples](#container-examples)
45
+ - [Check result with CLI commands](#check-result-with-cli-commands)
46
+ - [Check result files](#check-result-files)
47
+ - [Folder structure](#folder-structure)
48
+ - [Basic CLI commands](#basic-cli-commands)
49
+ - [List running jobs](#list-running-jobs)
50
+ - [Check status](#check-status)
51
+ - [Cancel job](#cancel-job)
52
+ - [View Experiments history](#view-experiments-history)
53
+ - [Note](#note)
54
+
55
+ <!-- END doctoc generated TOC please keep comment here to allow auto update -->
56
+
57
+ # Introduction
58
+
59
+ **ContainerPlatform** is a platform designed to facilitate the execution of experiments and simulations within Docker containers. It provides a robust environment with all necessary tools and dependencies installed, allowing for seamless integration and execution of computational tasks.
60
+
61
+ ## Pre-requisites
62
+ - Python 3.8/3.9/3.10/3.11/3.12 x64-bit
63
+ - OS:
64
+ - Windows 10 Pro or Enterprise
65
+ - Linux
66
+ - macOS (10.15 Catalina or later)
67
+ - Docker or Docker Desktop(required for the container platform)
68
+ On Windows, please use Docker Desktop 4.0.0 or later
69
+ - **Mac user**: Only support Intel based **x86_64** architecture if you want to run emodpy related disease models on **Docker** container platform. Apple based ARM architecture currently is not supported.
70
+
71
+ ## Installation
72
+
73
+ - **Install python**
74
+
75
+ Ensure you have Python 3.8+ installed on your system.
76
+
77
+ - Create and activate a virtual environment:
78
+ ```
79
+ python -m venv venv
80
+ source venv/bin/activate # On macOS/Linux
81
+ venv\Scripts\activate # On Windows
82
+ ```
83
+
84
+ - Install all **container** platform related packages
85
+ ```bash
86
+ pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
87
+ ```
88
+ - Optional: Install all **idmtools** packages
89
+ ```bash
90
+ pip install idmtools[full] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
91
+ ```
92
+ - To **override** existing idmtools container related packages after installing emodpy, run this command
93
+ ```bash
94
+ pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple --force-reinstall --no-cache-dir --upgrade
95
+ ```
96
+ **Mac user**: You map need to escape the square brackets with a backslash like `\[container\]` or `\[full\]` in above command.
97
+
98
+ - Extra steps for Windows user:
99
+ - Enable **Developer Mode** on Windows
100
+
101
+ If you are running the script on Windows, you need to enable Developer Mode. To enable Developer Mode, go to Settings -> Update & Security -> For developers and select Developer Mode on, or refer to this [guide](https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development).
102
+
103
+ - Enable **long file path** for Windows
104
+
105
+ Due to the file/folder structure design outlined in the section below, if running the script on Windows, be aware of the file path length limitation (less than 255 characters).
106
+
107
+ To allow longer file paths, you can enable Long Path Support in the Windows Group Policy Editor.
108
+ Refer to this [guide](https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/The-Windows-10-default-path-length-limitation-MAX-PATH-is-256-characters.html) for detailed instructions.
109
+
110
+ ## Examples for container platform
111
+
112
+ ### Initialize platform
113
+ - This is the example using Container Platform
114
+ ```python
115
+ from idmtools.core.platform_factory import Platform
116
+ platform = Platform('CONTAINER', job_directory='<user job directory>')
117
+ ```
118
+ - To trigger MPI, use ntasks >=2:
119
+ ```python
120
+ from idmtools.core.platform_factory import Platform
121
+ platform = Platform('CONTAINER', job_directory='<user job directory>', ntasks=2)
122
+ ```
123
+ - More options for container platform initialization:
124
+ refer to [ContainerPlatform attributes](hhttps://docs.idmod.org/projects/idmtools/en/latest/platforms/container/index.html#containerplatform-attributes)
125
+
126
+ ### Container Examples
127
+ - idmtools examples: https://github.com/InstituteforDiseaseModeling/idmtools/tree/main/examples/platform_container
128
+ - emodpy-malaria examples: https://github.com/EMOD-Hub/emodpy-malaria/tree/main/examples-container
129
+
130
+
131
+ ### Check result with CLI commands
132
+ ```bash
133
+ idmtools container status <experiment id>
134
+ ```
135
+ ### Check result files
136
+ - on host: `<job_directory>/<suite_path>/<experiment_path>/<simulation_path>/`
137
+ - in container: `/home/container-data/<suite_path>/<experiment_path>/<simulation_path>/`
138
+
139
+
140
+ ### Folder structure
141
+ By default, `idmtools` now generates simulations with the following structure:
142
+ `job_directory/s_<suite_name>_<suite_uuid>/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
143
+ - `job_directory` — The base directory that contains all suite, experiment, and simulation folders.
144
+ - `s_<suite_name>_<suite_uuid>` — The suite directory, where the suite name (truncated to a maximum of 30 characters) is prefixed with s_, followed by its unique suite UUID.
145
+ - `e_<experiment_name>_<experiment_uuid>` — The experiment directory, where the experiment name (also truncated to 30 characters) is prefixed with e_, followed by its unique experiment UUID.
146
+ - `simulation_uuid` — The simulation folder identified only by its UUID.
147
+
148
+ Suite is optional. If the user does not specify a suite, the folder will be:
149
+ `job_directory/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
150
+
151
+ Examples:
152
+
153
+ If you create a suite named: `my_very_long_suite_name_for_malaria_experiment`
154
+
155
+ and an experiment named: `test_experiment_with_calibration_phase`
156
+ `idmtools` will automatically truncate both names to a maximum of 30 characters and apply the prefixes s_ for suites and e_ for experiments, resulting in a path like:
157
+ ```
158
+ job_directory/
159
+ └── s_my_very_long_suite_name_for_m_12345678-9abc-def0-1234-56789abcdef0/
160
+ └── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
161
+ └── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
162
+ ```
163
+
164
+ Or for no suite case:
165
+ ```
166
+ job_directory/
167
+ └── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
168
+ └── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
169
+ ```
170
+
171
+ Users can customize this structure through the idmtools.ini configuration file:
172
+ - `name_directory = False` — Excludes the suite and experiment names (and their prefixes) from the simulation path.
173
+ - `sim_name_directory = True` — Includes the simulation name in the simulation folder path when name_directory = True.
174
+
175
+ ## Basic CLI commands
176
+
177
+ **ContainerPlatform** provides several CLI commands to manage and monitor experiments and simulations. Below are some basic commands:
178
+
179
+ ### List running jobs
180
+
181
+ To list running experiment or simulation jobs:
182
+ ```bash
183
+ idmtools container jobs [<container-id>] [-l <limit>] [-n <next>]
184
+ ```
185
+
186
+ ### Check status
187
+
188
+ To check the status of an experiment or simulation:
189
+ ```bash
190
+ idmtools container status <item-id> [-c <container_id>] [-l <limit>] [--verbose/--no-verbose]
191
+ ```
192
+
193
+ ### Cancel job
194
+
195
+ To cancel an experiment or simulation job:
196
+ ```bash
197
+ idmtools container cancel <item-id> [-c <container_id>]
198
+ ```
199
+
200
+ ### View Experiments history
201
+
202
+ To view experiments history:
203
+ ```bash
204
+ idmtools container history [<container-id>] [-l <limit>] [-n <next>]
205
+ ```
206
+
207
+
208
+ ## Note
209
+
210
+ - **WorkItem** is not supported on the Container Platform as it is not needed in most cases since the code already runs on user's local computer.
211
+ - **AssetCollection** creation or referencing to an existing AssetCollection are not supported on the Container Platform with current release. If you've used the COMPS Platform, you may have scripts using these objects. You would need to update these scripts without using these objects in order to run them on the Container Platform.
212
+ - Run with **Singularity** is not needed with Container Platform. If you take existing COMPS example and try to run it with Container Platform, you may need to remove the code that setups the singularity image.
@@ -0,0 +1,182 @@
1
+ ![Staging: idmtools-platform-container](https://github.com/InstituteforDiseaseModeling/idmtools/workflows/Staging:%20idmtools-platform-container/badge.svg?branch=dev)
2
+
3
+ # Idmtools platform container
4
+
5
+ <!-- START doctoc generated TOC please keep comment here to allow auto update -->
6
+ <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
7
+ **Table of Contents**
8
+
9
+ - [Introduction](#introduction)
10
+ - [Pre-requisites](#pre-requisites)
11
+ - [Installation](#installation)
12
+ - [Examples for container platform](#examples-for-container-platform)
13
+ - [Initialize platform](#initialize-platform)
14
+ - [Container Examples](#container-examples)
15
+ - [Check result with CLI commands](#check-result-with-cli-commands)
16
+ - [Check result files](#check-result-files)
17
+ - [Folder structure](#folder-structure)
18
+ - [Basic CLI commands](#basic-cli-commands)
19
+ - [List running jobs](#list-running-jobs)
20
+ - [Check status](#check-status)
21
+ - [Cancel job](#cancel-job)
22
+ - [View Experiments history](#view-experiments-history)
23
+ - [Note](#note)
24
+
25
+ <!-- END doctoc generated TOC please keep comment here to allow auto update -->
26
+
27
+ # Introduction
28
+
29
+ **ContainerPlatform** is a platform designed to facilitate the execution of experiments and simulations within Docker containers. It provides a robust environment with all necessary tools and dependencies installed, allowing for seamless integration and execution of computational tasks.
30
+
31
+ ## Pre-requisites
32
+ - Python 3.8/3.9/3.10/3.11/3.12 x64-bit
33
+ - OS:
34
+ - Windows 10 Pro or Enterprise
35
+ - Linux
36
+ - macOS (10.15 Catalina or later)
37
+ - Docker or Docker Desktop(required for the container platform)
38
+ On Windows, please use Docker Desktop 4.0.0 or later
39
+ - **Mac user**: Only support Intel based **x86_64** architecture if you want to run emodpy related disease models on **Docker** container platform. Apple based ARM architecture currently is not supported.
40
+
41
+ ## Installation
42
+
43
+ - **Install python**
44
+
45
+ Ensure you have Python 3.8+ installed on your system.
46
+
47
+ - Create and activate a virtual environment:
48
+ ```
49
+ python -m venv venv
50
+ source venv/bin/activate # On macOS/Linux
51
+ venv\Scripts\activate # On Windows
52
+ ```
53
+
54
+ - Install all **container** platform related packages
55
+ ```bash
56
+ pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
57
+ ```
58
+ - Optional: Install all **idmtools** packages
59
+ ```bash
60
+ pip install idmtools[full] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
61
+ ```
62
+ - To **override** existing idmtools container related packages after installing emodpy, run this command
63
+ ```bash
64
+ pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple --force-reinstall --no-cache-dir --upgrade
65
+ ```
66
+ **Mac user**: You map need to escape the square brackets with a backslash like `\[container\]` or `\[full\]` in above command.
67
+
68
+ - Extra steps for Windows user:
69
+ - Enable **Developer Mode** on Windows
70
+
71
+ If you are running the script on Windows, you need to enable Developer Mode. To enable Developer Mode, go to Settings -> Update & Security -> For developers and select Developer Mode on, or refer to this [guide](https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development).
72
+
73
+ - Enable **long file path** for Windows
74
+
75
+ Due to the file/folder structure design outlined in the section below, if running the script on Windows, be aware of the file path length limitation (less than 255 characters).
76
+
77
+ To allow longer file paths, you can enable Long Path Support in the Windows Group Policy Editor.
78
+ Refer to this [guide](https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/The-Windows-10-default-path-length-limitation-MAX-PATH-is-256-characters.html) for detailed instructions.
79
+
80
+ ## Examples for container platform
81
+
82
+ ### Initialize platform
83
+ - This is the example using Container Platform
84
+ ```python
85
+ from idmtools.core.platform_factory import Platform
86
+ platform = Platform('CONTAINER', job_directory='<user job directory>')
87
+ ```
88
+ - To trigger MPI, use ntasks >=2:
89
+ ```python
90
+ from idmtools.core.platform_factory import Platform
91
+ platform = Platform('CONTAINER', job_directory='<user job directory>', ntasks=2)
92
+ ```
93
+ - More options for container platform initialization:
94
+ refer to [ContainerPlatform attributes](hhttps://docs.idmod.org/projects/idmtools/en/latest/platforms/container/index.html#containerplatform-attributes)
95
+
96
+ ### Container Examples
97
+ - idmtools examples: https://github.com/InstituteforDiseaseModeling/idmtools/tree/main/examples/platform_container
98
+ - emodpy-malaria examples: https://github.com/EMOD-Hub/emodpy-malaria/tree/main/examples-container
99
+
100
+
101
+ ### Check result with CLI commands
102
+ ```bash
103
+ idmtools container status <experiment id>
104
+ ```
105
+ ### Check result files
106
+ - on host: `<job_directory>/<suite_path>/<experiment_path>/<simulation_path>/`
107
+ - in container: `/home/container-data/<suite_path>/<experiment_path>/<simulation_path>/`
108
+
109
+
110
+ ### Folder structure
111
+ By default, `idmtools` now generates simulations with the following structure:
112
+ `job_directory/s_<suite_name>_<suite_uuid>/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
113
+ - `job_directory` — The base directory that contains all suite, experiment, and simulation folders.
114
+ - `s_<suite_name>_<suite_uuid>` — The suite directory, where the suite name (truncated to a maximum of 30 characters) is prefixed with s_, followed by its unique suite UUID.
115
+ - `e_<experiment_name>_<experiment_uuid>` — The experiment directory, where the experiment name (also truncated to 30 characters) is prefixed with e_, followed by its unique experiment UUID.
116
+ - `simulation_uuid` — The simulation folder identified only by its UUID.
117
+
118
+ Suite is optional. If the user does not specify a suite, the folder will be:
119
+ `job_directory/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
120
+
121
+ Examples:
122
+
123
+ If you create a suite named: `my_very_long_suite_name_for_malaria_experiment`
124
+
125
+ and an experiment named: `test_experiment_with_calibration_phase`
126
+ `idmtools` will automatically truncate both names to a maximum of 30 characters and apply the prefixes s_ for suites and e_ for experiments, resulting in a path like:
127
+ ```
128
+ job_directory/
129
+ └── s_my_very_long_suite_name_for_m_12345678-9abc-def0-1234-56789abcdef0/
130
+ └── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
131
+ └── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
132
+ ```
133
+
134
+ Or for no suite case:
135
+ ```
136
+ job_directory/
137
+ └── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
138
+ └── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
139
+ ```
140
+
141
+ Users can customize this structure through the idmtools.ini configuration file:
142
+ - `name_directory = False` — Excludes the suite and experiment names (and their prefixes) from the simulation path.
143
+ - `sim_name_directory = True` — Includes the simulation name in the simulation folder path when name_directory = True.
144
+
145
+ ## Basic CLI commands
146
+
147
+ **ContainerPlatform** provides several CLI commands to manage and monitor experiments and simulations. Below are some basic commands:
148
+
149
+ ### List running jobs
150
+
151
+ To list running experiment or simulation jobs:
152
+ ```bash
153
+ idmtools container jobs [<container-id>] [-l <limit>] [-n <next>]
154
+ ```
155
+
156
+ ### Check status
157
+
158
+ To check the status of an experiment or simulation:
159
+ ```bash
160
+ idmtools container status <item-id> [-c <container_id>] [-l <limit>] [--verbose/--no-verbose]
161
+ ```
162
+
163
+ ### Cancel job
164
+
165
+ To cancel an experiment or simulation job:
166
+ ```bash
167
+ idmtools container cancel <item-id> [-c <container_id>]
168
+ ```
169
+
170
+ ### View Experiments history
171
+
172
+ To view experiments history:
173
+ ```bash
174
+ idmtools container history [<container-id>] [-l <limit>] [-n <next>]
175
+ ```
176
+
177
+
178
+ ## Note
179
+
180
+ - **WorkItem** is not supported on the Container Platform as it is not needed in most cases since the code already runs on user's local computer.
181
+ - **AssetCollection** creation or referencing to an existing AssetCollection are not supported on the Container Platform with current release. If you've used the COMPS Platform, you may have scripts using these objects. You would need to update these scripts without using these objects in order to run them on the Container Platform.
182
+ - Run with **Singularity** is not needed with Container Platform. If you take existing COMPS example and try to run it with Container Platform, you may need to remove the code that setups the singularity image.
@@ -0,0 +1,48 @@
1
+ # FROM docker-production.packages.idmod.org/dtk-rocky-buildenv:0.2-dev
2
+ # Replace the following lines with above line after the offical image is available
3
+ #############################################################
4
+ FROM rockylinux:9.2
5
+ # Set the timezone and the frontend
6
+ ENV TZ=UTC DEBIAN_FRONTEND=noninteractive
7
+ # To suppress warning without altering the installation when pip install with root user
8
+ ENV PIP_ROOT_USER_ACTION=ignore
9
+ # Install the necessary packages
10
+ RUN yum -y install rpm dnf-plugins-core \
11
+ && yum -y update \
12
+ && dnf -y install \
13
+ python3 \
14
+ python3-devel \
15
+ mpich \
16
+ snappy \
17
+ glibc-devel \
18
+ epel-release \
19
+ wget \
20
+ nano
21
+
22
+ # Update the system
23
+ RUN yum clean all
24
+
25
+ RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone # This is for setting up the timezone
26
+
27
+ # Install pip
28
+ RUN python3 -m pip install pip --upgrade
29
+ # Create a symbolic link to python3
30
+ RUN ln -s /usr/bin/python3 /usr/bin/python
31
+ # Create a directory for pip configurations
32
+ RUN mkdir /root/.pip
33
+ # Copy the pip configuration file
34
+ COPY pip.conf /root/.pip
35
+
36
+ # Set the path and library path
37
+ ENV PATH ${PATH}:/usr/lib64/mpich/bin
38
+ ENV LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/usr/lib64/mpich/lib
39
+
40
+ # Add the requirements file
41
+ ADD requirements.txt /tmp/
42
+
43
+ # make the PIP index configurable so we can build against staging, production, or a local PyPI server
44
+ ARG CONTAINER_VERSION
45
+
46
+ # Install the packages
47
+ RUN bash -c "pip3 install -r /tmp/requirements.txt"
48
+
@@ -0,0 +1,46 @@
1
+ #FROM docker-production.packages.idmod.org/dtk-rocky-buildenv:0.2-dev
2
+ # Replace the following lines with above line after the offical image is available
3
+ #############################################################
4
+ FROM rockylinux:9.2
5
+ ENV TZ=UTC DEBIAN_FRONTEND=noninteractive
6
+ ENV GNUMAKEFLAGS=-j4
7
+
8
+ RUN yum -y install rpm dnf-plugins-core \
9
+ && yum -y update \
10
+ && dnf -y install \
11
+ python3 \
12
+ python3-devel \
13
+ snappy \
14
+ mpich \
15
+ sudo \
16
+ epel-release \
17
+ git \
18
+ wget \
19
+ nano
20
+
21
+ RUN yum -y update
22
+ RUN yum -y group install "Development Tools" \
23
+ && yum clean all
24
+
25
+ RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone # This is for setting up the timezone
26
+
27
+ RUN python3 -m pip install pip --upgrade
28
+ RUN ln -s /usr/bin/python3 /usr/bin/python
29
+ RUN mkdir /root/.pip
30
+ COPY pip.conf /root/.pip
31
+ # RUN dnf install -y R
32
+ ENV PATH ${PATH}:/usr/lib64/mpich/bin
33
+ ENV LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/usr/lib64/mpich/lib
34
+
35
+ #############################################################
36
+ RUN dnf -y install dos2unix
37
+
38
+ # Add the requirements file
39
+ ADD requirements.txt /tmp/
40
+
41
+ # make the PIP index configurable so we can build against staging, production, or a local PyPI server
42
+ ARG CONTAINER_VERSION
43
+
44
+ # Install the packages
45
+ RUN bash -c "pip3 install -r /tmp/requirements.txt"
46
+
@@ -0,0 +1 @@
1
+ container-rocky-runtime
@@ -0,0 +1,78 @@
1
+ <!-- START doctoc generated TOC please keep comment here to allow auto update -->
2
+ <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
3
+ **Table of Contents**
4
+
5
+ - [idmtools_platform_container Docker image](#idmtools_platform_container-docker-image)
6
+ - [Introduction](#introduction)
7
+ - [Prerequisites](#prerequisites)
8
+ - [Build Docker image](#build-docker-image)
9
+ - [Docker image versioning](#docker-image-versioning)
10
+ - [Docker image usage](#docker-image-usage)
11
+ - [Publish Docker image](#publish-docker-image)
12
+ - [Extend the Docker image](#extend-the-docker-image)
13
+
14
+ <!-- END doctoc generated TOC please keep comment here to allow auto update -->
15
+
16
+ # idmtools_platform_container Docker image
17
+
18
+ **Note**: This document is primarily for developers. It is not necessary for users to build and push the Docker image. However, if you wish to use your own Docker image, please follow the instructions below at [Extend the Docker Image](#extend-the-docker-image).
19
+
20
+ ## Introduction
21
+ This Docker image is designed to facilitate idmtools_platform_container. It serves as local running platform with all necessary tools and dependencies installed. The image is based on rockylinux 9.2 and includes the following tools:
22
+ - python 3.9
23
+ - mipch 4.1.1
24
+ - emod-api 1.33.3
25
+ - emod-api's dependencies like numpy, pandas, scipy, matplotlib etc.
26
+
27
+ ## Prerequisites
28
+ - Docker
29
+ On Windows or Mac, please use Docker Desktop 2.1.0.5 or 2.2.0.1
30
+
31
+ ## Build Docker image
32
+ Note, you do not need to build the image locally. The image is auto built in github action and pushed to idmod artifactory.
33
+
34
+ To build Docker image locally, run:
35
+ ```bash
36
+ python build_docker_image.py --username <username> --password <password>
37
+ ```
38
+ where `<username>` and `<password>` are the username and password for the idmod artifactory account.
39
+
40
+ You can also build the image with different docker file and image name by specifying `--dockerfile` and `--image_name` arguments.
41
+ ```bash
42
+ python build_docker_image.py --username <username> --password <password> --dockerfile Dockerfile_buildenv --image_name container-rocky-buildenv
43
+ ```
44
+ which will build image name as 'idm-docker-staging.packages.idmod.org/idmtools/container-rocky-buildenv:x.x.x'
45
+
46
+ ## Docker image versioning
47
+ The Docker image version is determined by the version in idm docker-staging artifactory. It will advance the version number by 0.0.1 for each new build.
48
+
49
+ ## Docker image usage
50
+ By default, you DO NOT need to worry about the image build and how to use image. The image is auto built in github action and pushed to idmod artifactory. The image is used in idmtools_platform_container Platform object. For example,
51
+ ```python
52
+ from idmtools.core.platform_factory import Platform
53
+ platform = Platform('CONTAINER', job_directory='any_dir', docker_image='idm-docker-public.packages.idmod.org/idmtools/container-rocky-runtime:x.x.x')
54
+ ```
55
+ where docker_image can be your local built image or the image in idmod artifactory.
56
+ If you do not provide docker_image, the default image will be used.
57
+
58
+ ## Publish Docker image
59
+ Note, you do not need to push docker image to artifactory. The image is auto built in github action and pushed to idmod artifactory.
60
+ If you want to push the image to artifactory, run:
61
+ ```bash
62
+ python push_docker_image.py --username <username> --password <password>
63
+ ```
64
+ where `<username>` and `<password>` are the username and password for the idmod artifactory account.
65
+
66
+ ## Extend the Docker image
67
+ If you want to build you own Docker image, please use our ``container-rocky-runtime`` as your baseline image and add the following line to the top of your Dockerfile:
68
+
69
+ ```bash
70
+ FROM docker-production-public/idmtools/container-rocky-runtime/0.0.3
71
+ ```
72
+ You can use general Docker build [command](https://docs.docker.com/reference/cli/docker/buildx/build/) to build your own Docker image.
73
+ Then use that image in the Platform object. For example:
74
+
75
+ ```bash
76
+ from idmtools_platform_container import Platform
77
+ platform = Platform('CONTAINER', job_directory='any_dir', docker_image='your_own_image_name:x.x.x')
78
+ ```
@@ -0,0 +1,6 @@
1
+ """idmtools comps platform.
2
+
3
+ Build docker image.
4
+
5
+ Copyright 2021, Bill & Melinda Gates Foundation. All rights reserved.
6
+ """