idmtools-platform-container 0.0.0.dev0__tar.gz → 0.0.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- idmtools_platform_container-0.0.2/.bumpversion.cfg +28 -0
- idmtools_platform_container-0.0.2/LICENSE.TXT +3 -0
- idmtools_platform_container-0.0.2/MANIFEST.in +2 -0
- idmtools_platform_container-0.0.2/Makefile +12 -0
- idmtools_platform_container-0.0.2/PKG-INFO +212 -0
- idmtools_platform_container-0.0.2/README.md +182 -0
- idmtools_platform_container-0.0.2/docker_image/BASE_VERSION +1 -0
- idmtools_platform_container-0.0.2/docker_image/Dockerfile +48 -0
- idmtools_platform_container-0.0.2/docker_image/Dockerfile_buildenv +46 -0
- idmtools_platform_container-0.0.2/docker_image/ImageName +1 -0
- idmtools_platform_container-0.0.2/docker_image/README.md +78 -0
- idmtools_platform_container-0.0.2/docker_image/__init__.py +6 -0
- idmtools_platform_container-0.0.2/docker_image/build_docker_image.py +145 -0
- idmtools_platform_container-0.0.2/docker_image/debian/BASE_VERSION +1 -0
- idmtools_platform_container-0.0.2/docker_image/debian/Dockerfile +40 -0
- idmtools_platform_container-0.0.2/docker_image/debian/ImageName +1 -0
- idmtools_platform_container-0.0.2/docker_image/debian/README.md +48 -0
- idmtools_platform_container-0.0.2/docker_image/debian/pip.conf +3 -0
- idmtools_platform_container-0.0.2/docker_image/debian/requirements.txt +1 -0
- idmtools_platform_container-0.0.2/docker_image/docker_image_history.py +101 -0
- idmtools_platform_container-0.0.2/docker_image/pip.conf +3 -0
- idmtools_platform_container-0.0.2/docker_image/push_docker_image.py +62 -0
- idmtools_platform_container-0.0.2/docker_image/requirements.txt +1 -0
- idmtools_platform_container-0.0.2/docker_image/rocky_meta_runtime.txt +37 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/__init__.py +18 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/cli/__init__.py +5 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/cli/container.py +682 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/container_operations/__init__.py +5 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/container_operations/docker_operations.py +593 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/container_platform.py +375 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/platform_operations/__init__.py +5 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/platform_operations/experiment_operations.py +112 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/platform_operations/simulation_operations.py +58 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/plugin_info.py +79 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/utils/__init__.py +5 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/utils/general.py +136 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container/utils/status.py +130 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/PKG-INFO +212 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/SOURCES.txt +95 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/entry_points.txt +5 -0
- idmtools_platform_container-0.0.2/idmtools_platform_container.egg-info/requires.txt +15 -0
- {idmtools_platform_container-0.0.0.dev0 → idmtools_platform_container-0.0.2}/idmtools_platform_container.egg-info/top_level.txt +2 -0
- idmtools_platform_container-0.0.2/pyproject.toml +83 -0
- {idmtools_platform_container-0.0.0.dev0 → idmtools_platform_container-0.0.2}/setup.cfg +4 -4
- idmtools_platform_container-0.0.2/tests/Makefile +8 -0
- idmtools_platform_container-0.0.2/tests/__init__.py +4 -0
- idmtools_platform_container-0.0.2/tests/cleanup_containers.py +13 -0
- idmtools_platform_container-0.0.2/tests/helps.py +36 -0
- idmtools_platform_container-0.0.2/tests/idmtools_container_exp_dir.ini +3 -0
- idmtools_platform_container-0.0.2/tests/idmtools_container_sim_dir.ini +2 -0
- idmtools_platform_container-0.0.2/tests/inputs/Assets/MyLib/functions.py +2 -0
- idmtools_platform_container-0.0.2/tests/inputs/__init__.py +0 -0
- idmtools_platform_container-0.0.2/tests/inputs/model.py +28 -0
- idmtools_platform_container-0.0.2/tests/inputs/model1.py +31 -0
- idmtools_platform_container-0.0.2/tests/inputs/model3.py +21 -0
- idmtools_platform_container-0.0.2/tests/inputs/model_file.py +18 -0
- idmtools_platform_container-0.0.2/tests/inputs/run.sh +1 -0
- idmtools_platform_container-0.0.2/tests/inputs/sleep.py +9 -0
- idmtools_platform_container-0.0.2/tests/pytest.ini +17 -0
- idmtools_platform_container-0.0.2/tests/test_analyzers.py +144 -0
- idmtools_platform_container-0.0.2/tests/test_bash_sh.py +133 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/__init__.py +0 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/helper.py +57 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_base.py +14 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_cancel.py +96 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_clear_results.py +54 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_container.py +72 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_file_container_cli.py +121 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_get_detail.py +60 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_history.py +136 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_history_count.py +53 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_inspect.py +53 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_install.py +48 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_is_running.py +69 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_jobs.py +138 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_list_containers.py +99 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_packages.py +41 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_path.py +96 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_ps.py +47 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_remove_container.py +78 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_status.py +149 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_stop_container.py +71 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_sync_history.py +98 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_verify_docker.py +28 -0
- idmtools_platform_container-0.0.2/tests/test_container_cli/test_volume.py +28 -0
- idmtools_platform_container-0.0.2/tests/test_container_platform.py +441 -0
- idmtools_platform_container-0.0.2/tests/test_create_sim_directory_map.py +64 -0
- idmtools_platform_container-0.0.2/tests/test_docker_operations.py +812 -0
- idmtools_platform_container-0.0.2/tests/test_filter_by_tags.py +121 -0
- idmtools_platform_container-0.0.2/tests/test_filter_item.py +141 -0
- idmtools_platform_container-0.0.2/tests/test_flatten_item.py +122 -0
- idmtools_platform_container-0.0.2/tests/test_get_files.py +135 -0
- idmtools_platform_container-0.0.2/tests/test_get_methods.py +184 -0
- idmtools_platform_container-0.0.2/tests/test_max_file_path.py +37 -0
- idmtools_platform_container-0.0.2/tests/test_platform_experiment.py +419 -0
- idmtools_platform_container-0.0.2/tests/utils.py +90 -0
- idmtools_platform_container-0.0.0.dev0/PKG-INFO +0 -41
- idmtools_platform_container-0.0.0.dev0/README.md +0 -21
- idmtools_platform_container-0.0.0.dev0/idmtools_platform_container/__init__.py +0 -8
- idmtools_platform_container-0.0.0.dev0/idmtools_platform_container.egg-info/PKG-INFO +0 -41
- idmtools_platform_container-0.0.0.dev0/idmtools_platform_container.egg-info/SOURCES.txt +0 -7
- idmtools_platform_container-0.0.0.dev0/pyproject.toml +0 -32
- {idmtools_platform_container-0.0.0.dev0 → idmtools_platform_container-0.0.2}/idmtools_platform_container.egg-info/dependency_links.txt +0 -0
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
[bumpversion]
|
|
2
|
+
current_version = 3.0.0+nightly
|
|
3
|
+
commit = False
|
|
4
|
+
tag = False
|
|
5
|
+
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(?P<release>\+\w+)?(\.(?P<build>\d+))?
|
|
6
|
+
serialize =
|
|
7
|
+
{major}.{minor}.{patch}{release}.{build}
|
|
8
|
+
{major}.{minor}.{patch}{release}
|
|
9
|
+
{major}.{minor}.{patch}
|
|
10
|
+
|
|
11
|
+
[bumpversion:part:release]
|
|
12
|
+
optional_value = prod
|
|
13
|
+
first_value = prod
|
|
14
|
+
values =
|
|
15
|
+
prod
|
|
16
|
+
+nightly
|
|
17
|
+
|
|
18
|
+
[bumpversion:file:./idmtools_platform_container/__init__.py]
|
|
19
|
+
search = __version__ = "{current_version}"
|
|
20
|
+
replace = __version__ = "{new_version}"
|
|
21
|
+
serialize = {major}.{minor}.{patch}{release}
|
|
22
|
+
{major}.{minor}.{patch}
|
|
23
|
+
|
|
24
|
+
[bumpversion:file:./pyproject.toml]
|
|
25
|
+
search = version = "{current_version}"
|
|
26
|
+
replace = version = "{new_version}"
|
|
27
|
+
serialize = {major}.{minor}.{patch}{release}
|
|
28
|
+
{major}.{minor}.{patch}
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
PACKAGE_NAME=idmtools_platform_container
|
|
2
|
+
include $(abspath ../dev_scripts/package_general.mk)
|
|
3
|
+
|
|
4
|
+
help:
|
|
5
|
+
help-from-makefile -f ../dev_scripts/package_general.mk -f ./Makefile
|
|
6
|
+
|
|
7
|
+
# Release related rules
|
|
8
|
+
container-image: ## Build container Docker image
|
|
9
|
+
$(MAKE) -C docker_image docker
|
|
10
|
+
|
|
11
|
+
container-image-release: ## Push image to staging
|
|
12
|
+
$(MAKE) -C docker_image docker-release-staging
|
|
@@ -0,0 +1,212 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: idmtools_platform_container
|
|
3
|
+
Version: 0.0.2
|
|
4
|
+
Summary: Container platform for IDM-Tools
|
|
5
|
+
Author-email: Zhaowei Du <zdu@idmod.org>, Sharon Chen <shchen@idmod.org>, Clinton Collins <ccollins@idmod.org>, Benoit Raybaud <braybaud@idmod.org>, Clark Kirkman IV <ckirkman@idmod.org>, Ye Chen <yechen@idmod.org>, Mary Fisher <mafisher@idmod.org>, Mandy Izzo <mizzo@idmod.org>, Jen Schripsema <jschripsema@idmod.org>, Ross Carter <rcarter@idmod.org>
|
|
6
|
+
Project-URL: Homepage, https://github.com/InstituteforDiseaseModeling/idmtools
|
|
7
|
+
Keywords: modeling,IDM
|
|
8
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
9
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
10
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
13
|
+
Requires-Python: >=3.8
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
License-File: LICENSE.TXT
|
|
16
|
+
Requires-Dist: idmtools_platform_general<1.0.0,>=0.0.0
|
|
17
|
+
Requires-Dist: docker>5.0
|
|
18
|
+
Requires-Dist: rich
|
|
19
|
+
Provides-Extra: test
|
|
20
|
+
Requires-Dist: idmtools[test]; extra == "test"
|
|
21
|
+
Requires-Dist: idmtools_test; extra == "test"
|
|
22
|
+
Requires-Dist: idmtools_models; extra == "test"
|
|
23
|
+
Provides-Extra: packaging
|
|
24
|
+
Requires-Dist: flake8; extra == "packaging"
|
|
25
|
+
Requires-Dist: coverage; extra == "packaging"
|
|
26
|
+
Requires-Dist: bump2version; extra == "packaging"
|
|
27
|
+
Requires-Dist: twine; extra == "packaging"
|
|
28
|
+
Requires-Dist: natsort; extra == "packaging"
|
|
29
|
+
Dynamic: license-file
|
|
30
|
+
|
|
31
|
+

|
|
32
|
+
|
|
33
|
+
# Idmtools platform container
|
|
34
|
+
|
|
35
|
+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
|
36
|
+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
|
37
|
+
**Table of Contents**
|
|
38
|
+
|
|
39
|
+
- [Introduction](#introduction)
|
|
40
|
+
- [Pre-requisites](#pre-requisites)
|
|
41
|
+
- [Installation](#installation)
|
|
42
|
+
- [Examples for container platform](#examples-for-container-platform)
|
|
43
|
+
- [Initialize platform](#initialize-platform)
|
|
44
|
+
- [Container Examples](#container-examples)
|
|
45
|
+
- [Check result with CLI commands](#check-result-with-cli-commands)
|
|
46
|
+
- [Check result files](#check-result-files)
|
|
47
|
+
- [Folder structure](#folder-structure)
|
|
48
|
+
- [Basic CLI commands](#basic-cli-commands)
|
|
49
|
+
- [List running jobs](#list-running-jobs)
|
|
50
|
+
- [Check status](#check-status)
|
|
51
|
+
- [Cancel job](#cancel-job)
|
|
52
|
+
- [View Experiments history](#view-experiments-history)
|
|
53
|
+
- [Note](#note)
|
|
54
|
+
|
|
55
|
+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
|
56
|
+
|
|
57
|
+
# Introduction
|
|
58
|
+
|
|
59
|
+
**ContainerPlatform** is a platform designed to facilitate the execution of experiments and simulations within Docker containers. It provides a robust environment with all necessary tools and dependencies installed, allowing for seamless integration and execution of computational tasks.
|
|
60
|
+
|
|
61
|
+
## Pre-requisites
|
|
62
|
+
- Python 3.8/3.9/3.10/3.11/3.12 x64-bit
|
|
63
|
+
- OS:
|
|
64
|
+
- Windows 10 Pro or Enterprise
|
|
65
|
+
- Linux
|
|
66
|
+
- macOS (10.15 Catalina or later)
|
|
67
|
+
- Docker or Docker Desktop(required for the container platform)
|
|
68
|
+
On Windows, please use Docker Desktop 4.0.0 or later
|
|
69
|
+
- **Mac user**: Only support Intel based **x86_64** architecture if you want to run emodpy related disease models on **Docker** container platform. Apple based ARM architecture currently is not supported.
|
|
70
|
+
|
|
71
|
+
## Installation
|
|
72
|
+
|
|
73
|
+
- **Install python**
|
|
74
|
+
|
|
75
|
+
Ensure you have Python 3.8+ installed on your system.
|
|
76
|
+
|
|
77
|
+
- Create and activate a virtual environment:
|
|
78
|
+
```
|
|
79
|
+
python -m venv venv
|
|
80
|
+
source venv/bin/activate # On macOS/Linux
|
|
81
|
+
venv\Scripts\activate # On Windows
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
- Install all **container** platform related packages
|
|
85
|
+
```bash
|
|
86
|
+
pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
|
|
87
|
+
```
|
|
88
|
+
- Optional: Install all **idmtools** packages
|
|
89
|
+
```bash
|
|
90
|
+
pip install idmtools[full] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
|
|
91
|
+
```
|
|
92
|
+
- To **override** existing idmtools container related packages after installing emodpy, run this command
|
|
93
|
+
```bash
|
|
94
|
+
pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple --force-reinstall --no-cache-dir --upgrade
|
|
95
|
+
```
|
|
96
|
+
**Mac user**: You map need to escape the square brackets with a backslash like `\[container\]` or `\[full\]` in above command.
|
|
97
|
+
|
|
98
|
+
- Extra steps for Windows user:
|
|
99
|
+
- Enable **Developer Mode** on Windows
|
|
100
|
+
|
|
101
|
+
If you are running the script on Windows, you need to enable Developer Mode. To enable Developer Mode, go to Settings -> Update & Security -> For developers and select Developer Mode on, or refer to this [guide](https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development).
|
|
102
|
+
|
|
103
|
+
- Enable **long file path** for Windows
|
|
104
|
+
|
|
105
|
+
Due to the file/folder structure design outlined in the section below, if running the script on Windows, be aware of the file path length limitation (less than 255 characters).
|
|
106
|
+
|
|
107
|
+
To allow longer file paths, you can enable Long Path Support in the Windows Group Policy Editor.
|
|
108
|
+
Refer to this [guide](https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/The-Windows-10-default-path-length-limitation-MAX-PATH-is-256-characters.html) for detailed instructions.
|
|
109
|
+
|
|
110
|
+
## Examples for container platform
|
|
111
|
+
|
|
112
|
+
### Initialize platform
|
|
113
|
+
- This is the example using Container Platform
|
|
114
|
+
```python
|
|
115
|
+
from idmtools.core.platform_factory import Platform
|
|
116
|
+
platform = Platform('CONTAINER', job_directory='<user job directory>')
|
|
117
|
+
```
|
|
118
|
+
- To trigger MPI, use ntasks >=2:
|
|
119
|
+
```python
|
|
120
|
+
from idmtools.core.platform_factory import Platform
|
|
121
|
+
platform = Platform('CONTAINER', job_directory='<user job directory>', ntasks=2)
|
|
122
|
+
```
|
|
123
|
+
- More options for container platform initialization:
|
|
124
|
+
refer to [ContainerPlatform attributes](hhttps://docs.idmod.org/projects/idmtools/en/latest/platforms/container/index.html#containerplatform-attributes)
|
|
125
|
+
|
|
126
|
+
### Container Examples
|
|
127
|
+
- idmtools examples: https://github.com/InstituteforDiseaseModeling/idmtools/tree/main/examples/platform_container
|
|
128
|
+
- emodpy-malaria examples: https://github.com/EMOD-Hub/emodpy-malaria/tree/main/examples-container
|
|
129
|
+
|
|
130
|
+
|
|
131
|
+
### Check result with CLI commands
|
|
132
|
+
```bash
|
|
133
|
+
idmtools container status <experiment id>
|
|
134
|
+
```
|
|
135
|
+
### Check result files
|
|
136
|
+
- on host: `<job_directory>/<suite_path>/<experiment_path>/<simulation_path>/`
|
|
137
|
+
- in container: `/home/container-data/<suite_path>/<experiment_path>/<simulation_path>/`
|
|
138
|
+
|
|
139
|
+
|
|
140
|
+
### Folder structure
|
|
141
|
+
By default, `idmtools` now generates simulations with the following structure:
|
|
142
|
+
`job_directory/s_<suite_name>_<suite_uuid>/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
|
|
143
|
+
- `job_directory` — The base directory that contains all suite, experiment, and simulation folders.
|
|
144
|
+
- `s_<suite_name>_<suite_uuid>` — The suite directory, where the suite name (truncated to a maximum of 30 characters) is prefixed with s_, followed by its unique suite UUID.
|
|
145
|
+
- `e_<experiment_name>_<experiment_uuid>` — The experiment directory, where the experiment name (also truncated to 30 characters) is prefixed with e_, followed by its unique experiment UUID.
|
|
146
|
+
- `simulation_uuid` — The simulation folder identified only by its UUID.
|
|
147
|
+
|
|
148
|
+
Suite is optional. If the user does not specify a suite, the folder will be:
|
|
149
|
+
`job_directory/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
|
|
150
|
+
|
|
151
|
+
Examples:
|
|
152
|
+
|
|
153
|
+
If you create a suite named: `my_very_long_suite_name_for_malaria_experiment`
|
|
154
|
+
|
|
155
|
+
and an experiment named: `test_experiment_with_calibration_phase`
|
|
156
|
+
`idmtools` will automatically truncate both names to a maximum of 30 characters and apply the prefixes s_ for suites and e_ for experiments, resulting in a path like:
|
|
157
|
+
```
|
|
158
|
+
job_directory/
|
|
159
|
+
└── s_my_very_long_suite_name_for_m_12345678-9abc-def0-1234-56789abcdef0/
|
|
160
|
+
└── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
|
|
161
|
+
└── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Or for no suite case:
|
|
165
|
+
```
|
|
166
|
+
job_directory/
|
|
167
|
+
└── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
|
|
168
|
+
└── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Users can customize this structure through the idmtools.ini configuration file:
|
|
172
|
+
- `name_directory = False` — Excludes the suite and experiment names (and their prefixes) from the simulation path.
|
|
173
|
+
- `sim_name_directory = True` — Includes the simulation name in the simulation folder path when name_directory = True.
|
|
174
|
+
|
|
175
|
+
## Basic CLI commands
|
|
176
|
+
|
|
177
|
+
**ContainerPlatform** provides several CLI commands to manage and monitor experiments and simulations. Below are some basic commands:
|
|
178
|
+
|
|
179
|
+
### List running jobs
|
|
180
|
+
|
|
181
|
+
To list running experiment or simulation jobs:
|
|
182
|
+
```bash
|
|
183
|
+
idmtools container jobs [<container-id>] [-l <limit>] [-n <next>]
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
### Check status
|
|
187
|
+
|
|
188
|
+
To check the status of an experiment or simulation:
|
|
189
|
+
```bash
|
|
190
|
+
idmtools container status <item-id> [-c <container_id>] [-l <limit>] [--verbose/--no-verbose]
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
### Cancel job
|
|
194
|
+
|
|
195
|
+
To cancel an experiment or simulation job:
|
|
196
|
+
```bash
|
|
197
|
+
idmtools container cancel <item-id> [-c <container_id>]
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
### View Experiments history
|
|
201
|
+
|
|
202
|
+
To view experiments history:
|
|
203
|
+
```bash
|
|
204
|
+
idmtools container history [<container-id>] [-l <limit>] [-n <next>]
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
|
|
208
|
+
## Note
|
|
209
|
+
|
|
210
|
+
- **WorkItem** is not supported on the Container Platform as it is not needed in most cases since the code already runs on user's local computer.
|
|
211
|
+
- **AssetCollection** creation or referencing to an existing AssetCollection are not supported on the Container Platform with current release. If you've used the COMPS Platform, you may have scripts using these objects. You would need to update these scripts without using these objects in order to run them on the Container Platform.
|
|
212
|
+
- Run with **Singularity** is not needed with Container Platform. If you take existing COMPS example and try to run it with Container Platform, you may need to remove the code that setups the singularity image.
|
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+

|
|
2
|
+
|
|
3
|
+
# Idmtools platform container
|
|
4
|
+
|
|
5
|
+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
|
6
|
+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
|
7
|
+
**Table of Contents**
|
|
8
|
+
|
|
9
|
+
- [Introduction](#introduction)
|
|
10
|
+
- [Pre-requisites](#pre-requisites)
|
|
11
|
+
- [Installation](#installation)
|
|
12
|
+
- [Examples for container platform](#examples-for-container-platform)
|
|
13
|
+
- [Initialize platform](#initialize-platform)
|
|
14
|
+
- [Container Examples](#container-examples)
|
|
15
|
+
- [Check result with CLI commands](#check-result-with-cli-commands)
|
|
16
|
+
- [Check result files](#check-result-files)
|
|
17
|
+
- [Folder structure](#folder-structure)
|
|
18
|
+
- [Basic CLI commands](#basic-cli-commands)
|
|
19
|
+
- [List running jobs](#list-running-jobs)
|
|
20
|
+
- [Check status](#check-status)
|
|
21
|
+
- [Cancel job](#cancel-job)
|
|
22
|
+
- [View Experiments history](#view-experiments-history)
|
|
23
|
+
- [Note](#note)
|
|
24
|
+
|
|
25
|
+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
|
26
|
+
|
|
27
|
+
# Introduction
|
|
28
|
+
|
|
29
|
+
**ContainerPlatform** is a platform designed to facilitate the execution of experiments and simulations within Docker containers. It provides a robust environment with all necessary tools and dependencies installed, allowing for seamless integration and execution of computational tasks.
|
|
30
|
+
|
|
31
|
+
## Pre-requisites
|
|
32
|
+
- Python 3.8/3.9/3.10/3.11/3.12 x64-bit
|
|
33
|
+
- OS:
|
|
34
|
+
- Windows 10 Pro or Enterprise
|
|
35
|
+
- Linux
|
|
36
|
+
- macOS (10.15 Catalina or later)
|
|
37
|
+
- Docker or Docker Desktop(required for the container platform)
|
|
38
|
+
On Windows, please use Docker Desktop 4.0.0 or later
|
|
39
|
+
- **Mac user**: Only support Intel based **x86_64** architecture if you want to run emodpy related disease models on **Docker** container platform. Apple based ARM architecture currently is not supported.
|
|
40
|
+
|
|
41
|
+
## Installation
|
|
42
|
+
|
|
43
|
+
- **Install python**
|
|
44
|
+
|
|
45
|
+
Ensure you have Python 3.8+ installed on your system.
|
|
46
|
+
|
|
47
|
+
- Create and activate a virtual environment:
|
|
48
|
+
```
|
|
49
|
+
python -m venv venv
|
|
50
|
+
source venv/bin/activate # On macOS/Linux
|
|
51
|
+
venv\Scripts\activate # On Windows
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
- Install all **container** platform related packages
|
|
55
|
+
```bash
|
|
56
|
+
pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
|
|
57
|
+
```
|
|
58
|
+
- Optional: Install all **idmtools** packages
|
|
59
|
+
```bash
|
|
60
|
+
pip install idmtools[full] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
|
|
61
|
+
```
|
|
62
|
+
- To **override** existing idmtools container related packages after installing emodpy, run this command
|
|
63
|
+
```bash
|
|
64
|
+
pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple --force-reinstall --no-cache-dir --upgrade
|
|
65
|
+
```
|
|
66
|
+
**Mac user**: You map need to escape the square brackets with a backslash like `\[container\]` or `\[full\]` in above command.
|
|
67
|
+
|
|
68
|
+
- Extra steps for Windows user:
|
|
69
|
+
- Enable **Developer Mode** on Windows
|
|
70
|
+
|
|
71
|
+
If you are running the script on Windows, you need to enable Developer Mode. To enable Developer Mode, go to Settings -> Update & Security -> For developers and select Developer Mode on, or refer to this [guide](https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development).
|
|
72
|
+
|
|
73
|
+
- Enable **long file path** for Windows
|
|
74
|
+
|
|
75
|
+
Due to the file/folder structure design outlined in the section below, if running the script on Windows, be aware of the file path length limitation (less than 255 characters).
|
|
76
|
+
|
|
77
|
+
To allow longer file paths, you can enable Long Path Support in the Windows Group Policy Editor.
|
|
78
|
+
Refer to this [guide](https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/The-Windows-10-default-path-length-limitation-MAX-PATH-is-256-characters.html) for detailed instructions.
|
|
79
|
+
|
|
80
|
+
## Examples for container platform
|
|
81
|
+
|
|
82
|
+
### Initialize platform
|
|
83
|
+
- This is the example using Container Platform
|
|
84
|
+
```python
|
|
85
|
+
from idmtools.core.platform_factory import Platform
|
|
86
|
+
platform = Platform('CONTAINER', job_directory='<user job directory>')
|
|
87
|
+
```
|
|
88
|
+
- To trigger MPI, use ntasks >=2:
|
|
89
|
+
```python
|
|
90
|
+
from idmtools.core.platform_factory import Platform
|
|
91
|
+
platform = Platform('CONTAINER', job_directory='<user job directory>', ntasks=2)
|
|
92
|
+
```
|
|
93
|
+
- More options for container platform initialization:
|
|
94
|
+
refer to [ContainerPlatform attributes](hhttps://docs.idmod.org/projects/idmtools/en/latest/platforms/container/index.html#containerplatform-attributes)
|
|
95
|
+
|
|
96
|
+
### Container Examples
|
|
97
|
+
- idmtools examples: https://github.com/InstituteforDiseaseModeling/idmtools/tree/main/examples/platform_container
|
|
98
|
+
- emodpy-malaria examples: https://github.com/EMOD-Hub/emodpy-malaria/tree/main/examples-container
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
### Check result with CLI commands
|
|
102
|
+
```bash
|
|
103
|
+
idmtools container status <experiment id>
|
|
104
|
+
```
|
|
105
|
+
### Check result files
|
|
106
|
+
- on host: `<job_directory>/<suite_path>/<experiment_path>/<simulation_path>/`
|
|
107
|
+
- in container: `/home/container-data/<suite_path>/<experiment_path>/<simulation_path>/`
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
### Folder structure
|
|
111
|
+
By default, `idmtools` now generates simulations with the following structure:
|
|
112
|
+
`job_directory/s_<suite_name>_<suite_uuid>/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
|
|
113
|
+
- `job_directory` — The base directory that contains all suite, experiment, and simulation folders.
|
|
114
|
+
- `s_<suite_name>_<suite_uuid>` — The suite directory, where the suite name (truncated to a maximum of 30 characters) is prefixed with s_, followed by its unique suite UUID.
|
|
115
|
+
- `e_<experiment_name>_<experiment_uuid>` — The experiment directory, where the experiment name (also truncated to 30 characters) is prefixed with e_, followed by its unique experiment UUID.
|
|
116
|
+
- `simulation_uuid` — The simulation folder identified only by its UUID.
|
|
117
|
+
|
|
118
|
+
Suite is optional. If the user does not specify a suite, the folder will be:
|
|
119
|
+
`job_directory/e_<experiment_name>_<experiment_uuid>/simulation_uuid`
|
|
120
|
+
|
|
121
|
+
Examples:
|
|
122
|
+
|
|
123
|
+
If you create a suite named: `my_very_long_suite_name_for_malaria_experiment`
|
|
124
|
+
|
|
125
|
+
and an experiment named: `test_experiment_with_calibration_phase`
|
|
126
|
+
`idmtools` will automatically truncate both names to a maximum of 30 characters and apply the prefixes s_ for suites and e_ for experiments, resulting in a path like:
|
|
127
|
+
```
|
|
128
|
+
job_directory/
|
|
129
|
+
└── s_my_very_long_suite_name_for_m_12345678-9abc-def0-1234-56789abcdef0/
|
|
130
|
+
└── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
|
|
131
|
+
└── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Or for no suite case:
|
|
135
|
+
```
|
|
136
|
+
job_directory/
|
|
137
|
+
└── e_test_experiment_with_calibrati_abcd1234-5678-90ef-abcd-1234567890ef/
|
|
138
|
+
└── 7c9e6679-7425-40de-944b-e07fc1f90ae7/
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Users can customize this structure through the idmtools.ini configuration file:
|
|
142
|
+
- `name_directory = False` — Excludes the suite and experiment names (and their prefixes) from the simulation path.
|
|
143
|
+
- `sim_name_directory = True` — Includes the simulation name in the simulation folder path when name_directory = True.
|
|
144
|
+
|
|
145
|
+
## Basic CLI commands
|
|
146
|
+
|
|
147
|
+
**ContainerPlatform** provides several CLI commands to manage and monitor experiments and simulations. Below are some basic commands:
|
|
148
|
+
|
|
149
|
+
### List running jobs
|
|
150
|
+
|
|
151
|
+
To list running experiment or simulation jobs:
|
|
152
|
+
```bash
|
|
153
|
+
idmtools container jobs [<container-id>] [-l <limit>] [-n <next>]
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
### Check status
|
|
157
|
+
|
|
158
|
+
To check the status of an experiment or simulation:
|
|
159
|
+
```bash
|
|
160
|
+
idmtools container status <item-id> [-c <container_id>] [-l <limit>] [--verbose/--no-verbose]
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
### Cancel job
|
|
164
|
+
|
|
165
|
+
To cancel an experiment or simulation job:
|
|
166
|
+
```bash
|
|
167
|
+
idmtools container cancel <item-id> [-c <container_id>]
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
### View Experiments history
|
|
171
|
+
|
|
172
|
+
To view experiments history:
|
|
173
|
+
```bash
|
|
174
|
+
idmtools container history [<container-id>] [-l <limit>] [-n <next>]
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
## Note
|
|
179
|
+
|
|
180
|
+
- **WorkItem** is not supported on the Container Platform as it is not needed in most cases since the code already runs on user's local computer.
|
|
181
|
+
- **AssetCollection** creation or referencing to an existing AssetCollection are not supported on the Container Platform with current release. If you've used the COMPS Platform, you may have scripts using these objects. You would need to update these scripts without using these objects in order to run them on the Container Platform.
|
|
182
|
+
- Run with **Singularity** is not needed with Container Platform. If you take existing COMPS example and try to run it with Container Platform, you may need to remove the code that setups the singularity image.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
0.0
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# FROM docker-production.packages.idmod.org/dtk-rocky-buildenv:0.2-dev
|
|
2
|
+
# Replace the following lines with above line after the offical image is available
|
|
3
|
+
#############################################################
|
|
4
|
+
FROM rockylinux:9.2
|
|
5
|
+
# Set the timezone and the frontend
|
|
6
|
+
ENV TZ=UTC DEBIAN_FRONTEND=noninteractive
|
|
7
|
+
# To suppress warning without altering the installation when pip install with root user
|
|
8
|
+
ENV PIP_ROOT_USER_ACTION=ignore
|
|
9
|
+
# Install the necessary packages
|
|
10
|
+
RUN yum -y install rpm dnf-plugins-core \
|
|
11
|
+
&& yum -y update \
|
|
12
|
+
&& dnf -y install \
|
|
13
|
+
python3 \
|
|
14
|
+
python3-devel \
|
|
15
|
+
mpich \
|
|
16
|
+
snappy \
|
|
17
|
+
glibc-devel \
|
|
18
|
+
epel-release \
|
|
19
|
+
wget \
|
|
20
|
+
nano
|
|
21
|
+
|
|
22
|
+
# Update the system
|
|
23
|
+
RUN yum clean all
|
|
24
|
+
|
|
25
|
+
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone # This is for setting up the timezone
|
|
26
|
+
|
|
27
|
+
# Install pip
|
|
28
|
+
RUN python3 -m pip install pip --upgrade
|
|
29
|
+
# Create a symbolic link to python3
|
|
30
|
+
RUN ln -s /usr/bin/python3 /usr/bin/python
|
|
31
|
+
# Create a directory for pip configurations
|
|
32
|
+
RUN mkdir /root/.pip
|
|
33
|
+
# Copy the pip configuration file
|
|
34
|
+
COPY pip.conf /root/.pip
|
|
35
|
+
|
|
36
|
+
# Set the path and library path
|
|
37
|
+
ENV PATH ${PATH}:/usr/lib64/mpich/bin
|
|
38
|
+
ENV LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/usr/lib64/mpich/lib
|
|
39
|
+
|
|
40
|
+
# Add the requirements file
|
|
41
|
+
ADD requirements.txt /tmp/
|
|
42
|
+
|
|
43
|
+
# make the PIP index configurable so we can build against staging, production, or a local PyPI server
|
|
44
|
+
ARG CONTAINER_VERSION
|
|
45
|
+
|
|
46
|
+
# Install the packages
|
|
47
|
+
RUN bash -c "pip3 install -r /tmp/requirements.txt"
|
|
48
|
+
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
#FROM docker-production.packages.idmod.org/dtk-rocky-buildenv:0.2-dev
|
|
2
|
+
# Replace the following lines with above line after the offical image is available
|
|
3
|
+
#############################################################
|
|
4
|
+
FROM rockylinux:9.2
|
|
5
|
+
ENV TZ=UTC DEBIAN_FRONTEND=noninteractive
|
|
6
|
+
ENV GNUMAKEFLAGS=-j4
|
|
7
|
+
|
|
8
|
+
RUN yum -y install rpm dnf-plugins-core \
|
|
9
|
+
&& yum -y update \
|
|
10
|
+
&& dnf -y install \
|
|
11
|
+
python3 \
|
|
12
|
+
python3-devel \
|
|
13
|
+
snappy \
|
|
14
|
+
mpich \
|
|
15
|
+
sudo \
|
|
16
|
+
epel-release \
|
|
17
|
+
git \
|
|
18
|
+
wget \
|
|
19
|
+
nano
|
|
20
|
+
|
|
21
|
+
RUN yum -y update
|
|
22
|
+
RUN yum -y group install "Development Tools" \
|
|
23
|
+
&& yum clean all
|
|
24
|
+
|
|
25
|
+
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone # This is for setting up the timezone
|
|
26
|
+
|
|
27
|
+
RUN python3 -m pip install pip --upgrade
|
|
28
|
+
RUN ln -s /usr/bin/python3 /usr/bin/python
|
|
29
|
+
RUN mkdir /root/.pip
|
|
30
|
+
COPY pip.conf /root/.pip
|
|
31
|
+
# RUN dnf install -y R
|
|
32
|
+
ENV PATH ${PATH}:/usr/lib64/mpich/bin
|
|
33
|
+
ENV LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/usr/lib64/mpich/lib
|
|
34
|
+
|
|
35
|
+
#############################################################
|
|
36
|
+
RUN dnf -y install dos2unix
|
|
37
|
+
|
|
38
|
+
# Add the requirements file
|
|
39
|
+
ADD requirements.txt /tmp/
|
|
40
|
+
|
|
41
|
+
# make the PIP index configurable so we can build against staging, production, or a local PyPI server
|
|
42
|
+
ARG CONTAINER_VERSION
|
|
43
|
+
|
|
44
|
+
# Install the packages
|
|
45
|
+
RUN bash -c "pip3 install -r /tmp/requirements.txt"
|
|
46
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
container-rocky-runtime
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
|
2
|
+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
|
3
|
+
**Table of Contents**
|
|
4
|
+
|
|
5
|
+
- [idmtools_platform_container Docker image](#idmtools_platform_container-docker-image)
|
|
6
|
+
- [Introduction](#introduction)
|
|
7
|
+
- [Prerequisites](#prerequisites)
|
|
8
|
+
- [Build Docker image](#build-docker-image)
|
|
9
|
+
- [Docker image versioning](#docker-image-versioning)
|
|
10
|
+
- [Docker image usage](#docker-image-usage)
|
|
11
|
+
- [Publish Docker image](#publish-docker-image)
|
|
12
|
+
- [Extend the Docker image](#extend-the-docker-image)
|
|
13
|
+
|
|
14
|
+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
|
15
|
+
|
|
16
|
+
# idmtools_platform_container Docker image
|
|
17
|
+
|
|
18
|
+
**Note**: This document is primarily for developers. It is not necessary for users to build and push the Docker image. However, if you wish to use your own Docker image, please follow the instructions below at [Extend the Docker Image](#extend-the-docker-image).
|
|
19
|
+
|
|
20
|
+
## Introduction
|
|
21
|
+
This Docker image is designed to facilitate idmtools_platform_container. It serves as local running platform with all necessary tools and dependencies installed. The image is based on rockylinux 9.2 and includes the following tools:
|
|
22
|
+
- python 3.9
|
|
23
|
+
- mipch 4.1.1
|
|
24
|
+
- emod-api 1.33.3
|
|
25
|
+
- emod-api's dependencies like numpy, pandas, scipy, matplotlib etc.
|
|
26
|
+
|
|
27
|
+
## Prerequisites
|
|
28
|
+
- Docker
|
|
29
|
+
On Windows or Mac, please use Docker Desktop 2.1.0.5 or 2.2.0.1
|
|
30
|
+
|
|
31
|
+
## Build Docker image
|
|
32
|
+
Note, you do not need to build the image locally. The image is auto built in github action and pushed to idmod artifactory.
|
|
33
|
+
|
|
34
|
+
To build Docker image locally, run:
|
|
35
|
+
```bash
|
|
36
|
+
python build_docker_image.py --username <username> --password <password>
|
|
37
|
+
```
|
|
38
|
+
where `<username>` and `<password>` are the username and password for the idmod artifactory account.
|
|
39
|
+
|
|
40
|
+
You can also build the image with different docker file and image name by specifying `--dockerfile` and `--image_name` arguments.
|
|
41
|
+
```bash
|
|
42
|
+
python build_docker_image.py --username <username> --password <password> --dockerfile Dockerfile_buildenv --image_name container-rocky-buildenv
|
|
43
|
+
```
|
|
44
|
+
which will build image name as 'idm-docker-staging.packages.idmod.org/idmtools/container-rocky-buildenv:x.x.x'
|
|
45
|
+
|
|
46
|
+
## Docker image versioning
|
|
47
|
+
The Docker image version is determined by the version in idm docker-staging artifactory. It will advance the version number by 0.0.1 for each new build.
|
|
48
|
+
|
|
49
|
+
## Docker image usage
|
|
50
|
+
By default, you DO NOT need to worry about the image build and how to use image. The image is auto built in github action and pushed to idmod artifactory. The image is used in idmtools_platform_container Platform object. For example,
|
|
51
|
+
```python
|
|
52
|
+
from idmtools.core.platform_factory import Platform
|
|
53
|
+
platform = Platform('CONTAINER', job_directory='any_dir', docker_image='idm-docker-public.packages.idmod.org/idmtools/container-rocky-runtime:x.x.x')
|
|
54
|
+
```
|
|
55
|
+
where docker_image can be your local built image or the image in idmod artifactory.
|
|
56
|
+
If you do not provide docker_image, the default image will be used.
|
|
57
|
+
|
|
58
|
+
## Publish Docker image
|
|
59
|
+
Note, you do not need to push docker image to artifactory. The image is auto built in github action and pushed to idmod artifactory.
|
|
60
|
+
If you want to push the image to artifactory, run:
|
|
61
|
+
```bash
|
|
62
|
+
python push_docker_image.py --username <username> --password <password>
|
|
63
|
+
```
|
|
64
|
+
where `<username>` and `<password>` are the username and password for the idmod artifactory account.
|
|
65
|
+
|
|
66
|
+
## Extend the Docker image
|
|
67
|
+
If you want to build you own Docker image, please use our ``container-rocky-runtime`` as your baseline image and add the following line to the top of your Dockerfile:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
FROM docker-production-public/idmtools/container-rocky-runtime/0.0.3
|
|
71
|
+
```
|
|
72
|
+
You can use general Docker build [command](https://docs.docker.com/reference/cli/docker/buildx/build/) to build your own Docker image.
|
|
73
|
+
Then use that image in the Platform object. For example:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
from idmtools_platform_container import Platform
|
|
77
|
+
platform = Platform('CONTAINER', job_directory='any_dir', docker_image='your_own_image_name:x.x.x')
|
|
78
|
+
```
|