devstack-cli 9.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- devstack-cli-9.0.0/LICENSE +19 -0
- devstack-cli-9.0.0/PKG-INFO +80 -0
- devstack-cli-9.0.0/README.md +63 -0
- devstack-cli-9.0.0/pyproject.toml +89 -0
- devstack-cli-9.0.0/setup.cfg +4 -0
- devstack-cli-9.0.0/src/__init__.py +0 -0
- devstack-cli-9.0.0/src/cli.py +846 -0
- devstack-cli-9.0.0/src/devstack_cli.egg-info/PKG-INFO +80 -0
- devstack-cli-9.0.0/src/devstack_cli.egg-info/SOURCES.txt +12 -0
- devstack-cli-9.0.0/src/devstack_cli.egg-info/dependency_links.txt +1 -0
- devstack-cli-9.0.0/src/devstack_cli.egg-info/entry_points.txt +2 -0
- devstack-cli-9.0.0/src/devstack_cli.egg-info/requires.txt +3 -0
- devstack-cli-9.0.0/src/devstack_cli.egg-info/top_level.txt +3 -0
- devstack-cli-9.0.0/src/version.py +9 -0
@@ -0,0 +1,19 @@
|
|
1
|
+
Copyright (c) 2024 Starflows OG
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
4
|
+
of this software and associated documentation files (the "Software"), to deal
|
5
|
+
in the Software without restriction, including without limitation the rights
|
6
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
7
|
+
copies of the Software, and to permit persons to whom the Software is
|
8
|
+
furnished to do so, subject to the following conditions:
|
9
|
+
|
10
|
+
The above copyright notice and this permission notice shall be included in all
|
11
|
+
copies or substantial portions of the Software.
|
12
|
+
|
13
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
14
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
15
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
16
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
17
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
18
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
19
|
+
SOFTWARE.
|
@@ -0,0 +1,80 @@
|
|
1
|
+
Metadata-Version: 2.1
|
2
|
+
Name: devstack-cli
|
3
|
+
Version: 9.0.0
|
4
|
+
Summary: Command-line access to Remote Development Environments (RDEs) created by Cloudomation DevStack
|
5
|
+
Author-email: Stefan Mückstein <stefan@cloudomation.com>
|
6
|
+
Project-URL: Homepage, https://cloudomation.com/
|
7
|
+
Project-URL: Documentation, https://docs.cloudomation.com/devstack/
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
10
|
+
Classifier: Operating System :: OS Independent
|
11
|
+
Requires-Python: >=3.8
|
12
|
+
Description-Content-Type: text/markdown
|
13
|
+
License-File: LICENSE
|
14
|
+
Requires-Dist: paramiko
|
15
|
+
Requires-Dist: watchdog
|
16
|
+
Requires-Dist: rich
|
17
|
+
|
18
|
+
# DevStack CLI
|
19
|
+
|
20
|
+
The `devstack-cli` provides command-line access to Remote Development Environments (RDEs) created by Cloudomation DevStack. Learn more about [Cloudomation DevStack](https://docs.cloudomation.com/devstack/docs/overview-and-concept).
|
21
|
+
|
22
|
+
## Installation
|
23
|
+
|
24
|
+
The following binaries must be installed to be able to use `devstack-cli`:
|
25
|
+
|
26
|
+
* `ssh`
|
27
|
+
* `ssh-keyscan`
|
28
|
+
* `rsync`
|
29
|
+
* `git`
|
30
|
+
|
31
|
+
On Debian/Ubuntu the packages can be installed with
|
32
|
+
|
33
|
+
```
|
34
|
+
apt install openssh-client rsync git
|
35
|
+
```
|
36
|
+
|
37
|
+
Then you can install `devstack-cli` by running
|
38
|
+
|
39
|
+
```
|
40
|
+
python -m pip install devstack-cli
|
41
|
+
```
|
42
|
+
|
43
|
+
## Usage
|
44
|
+
|
45
|
+
```
|
46
|
+
usage: devstack-cli [-h] -H HOSTNAME -s SOURCE_DIRECTORY [-o OUTPUT_DIRECTORY] [-v]
|
47
|
+
```
|
48
|
+
|
49
|
+
You need to specify the hostname/IP of the RDE created by Cloudomation DevStack as well as the path to a directory where the sources will be cached locally. Optionally you can specify an output directory where artifacts created on the RDE will be stored locally.
|
50
|
+
The `-v` switch enables debug logging.
|
51
|
+
|
52
|
+
## Support
|
53
|
+
|
54
|
+
`devstack-cli` is part of Cloudomation DevStack and support is provided to you with an active subscription.
|
55
|
+
|
56
|
+
## Authors and acknowledgment
|
57
|
+
|
58
|
+
Cloudomation actively maintains the `devstack-cli` command line tool as part of Cloudomation DevStack
|
59
|
+
|
60
|
+
## License
|
61
|
+
|
62
|
+
Copyright (c) 2024 Starflows OG
|
63
|
+
|
64
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
65
|
+
of this software and associated documentation files (the "Software"), to deal
|
66
|
+
in the Software without restriction, including without limitation the rights
|
67
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
68
|
+
copies of the Software, and to permit persons to whom the Software is
|
69
|
+
furnished to do so, subject to the following conditions:
|
70
|
+
|
71
|
+
The above copyright notice and this permission notice shall be included in all
|
72
|
+
copies or substantial portions of the Software.
|
73
|
+
|
74
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
75
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
76
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
77
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
78
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
79
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
80
|
+
SOFTWARE.
|
@@ -0,0 +1,63 @@
|
|
1
|
+
# DevStack CLI
|
2
|
+
|
3
|
+
The `devstack-cli` provides command-line access to Remote Development Environments (RDEs) created by Cloudomation DevStack. Learn more about [Cloudomation DevStack](https://docs.cloudomation.com/devstack/docs/overview-and-concept).
|
4
|
+
|
5
|
+
## Installation
|
6
|
+
|
7
|
+
The following binaries must be installed to be able to use `devstack-cli`:
|
8
|
+
|
9
|
+
* `ssh`
|
10
|
+
* `ssh-keyscan`
|
11
|
+
* `rsync`
|
12
|
+
* `git`
|
13
|
+
|
14
|
+
On Debian/Ubuntu the packages can be installed with
|
15
|
+
|
16
|
+
```
|
17
|
+
apt install openssh-client rsync git
|
18
|
+
```
|
19
|
+
|
20
|
+
Then you can install `devstack-cli` by running
|
21
|
+
|
22
|
+
```
|
23
|
+
python -m pip install devstack-cli
|
24
|
+
```
|
25
|
+
|
26
|
+
## Usage
|
27
|
+
|
28
|
+
```
|
29
|
+
usage: devstack-cli [-h] -H HOSTNAME -s SOURCE_DIRECTORY [-o OUTPUT_DIRECTORY] [-v]
|
30
|
+
```
|
31
|
+
|
32
|
+
You need to specify the hostname/IP of the RDE created by Cloudomation DevStack as well as the path to a directory where the sources will be cached locally. Optionally you can specify an output directory where artifacts created on the RDE will be stored locally.
|
33
|
+
The `-v` switch enables debug logging.
|
34
|
+
|
35
|
+
## Support
|
36
|
+
|
37
|
+
`devstack-cli` is part of Cloudomation DevStack and support is provided to you with an active subscription.
|
38
|
+
|
39
|
+
## Authors and acknowledgment
|
40
|
+
|
41
|
+
Cloudomation actively maintains the `devstack-cli` command line tool as part of Cloudomation DevStack
|
42
|
+
|
43
|
+
## License
|
44
|
+
|
45
|
+
Copyright (c) 2024 Starflows OG
|
46
|
+
|
47
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
48
|
+
of this software and associated documentation files (the "Software"), to deal
|
49
|
+
in the Software without restriction, including without limitation the rights
|
50
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
51
|
+
copies of the Software, and to permit persons to whom the Software is
|
52
|
+
furnished to do so, subject to the following conditions:
|
53
|
+
|
54
|
+
The above copyright notice and this permission notice shall be included in all
|
55
|
+
copies or substantial portions of the Software.
|
56
|
+
|
57
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
58
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
59
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
60
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
61
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
62
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
63
|
+
SOFTWARE.
|
@@ -0,0 +1,89 @@
|
|
1
|
+
[project]
|
2
|
+
name = "devstack-cli"
|
3
|
+
version = "9.0.0"
|
4
|
+
authors = [
|
5
|
+
{ name="Stefan Mückstein", email="stefan@cloudomation.com" },
|
6
|
+
]
|
7
|
+
description = "Command-line access to Remote Development Environments (RDEs) created by Cloudomation DevStack"
|
8
|
+
readme = "README.md"
|
9
|
+
requires-python = ">=3.8"
|
10
|
+
classifiers = [
|
11
|
+
"Programming Language :: Python :: 3",
|
12
|
+
"License :: OSI Approved :: MIT License",
|
13
|
+
"Operating System :: OS Independent",
|
14
|
+
]
|
15
|
+
dependencies = [
|
16
|
+
"paramiko",
|
17
|
+
"watchdog",
|
18
|
+
"rich",
|
19
|
+
]
|
20
|
+
|
21
|
+
[project.urls]
|
22
|
+
Homepage = "https://cloudomation.com/"
|
23
|
+
Documentation = "https://docs.cloudomation.com/devstack/"
|
24
|
+
|
25
|
+
[build-system]
|
26
|
+
requires = ["setuptools"]
|
27
|
+
build-backend = "setuptools.build_meta"
|
28
|
+
|
29
|
+
[tool.setuptools]
|
30
|
+
package-dir = {"" = "src"}
|
31
|
+
|
32
|
+
[project.scripts]
|
33
|
+
devstack-cli = "cli:main"
|
34
|
+
|
35
|
+
[tool.ruff]
|
36
|
+
select = ["ALL"]
|
37
|
+
ignore = ["E501", "ERA001", "TRY003", "EM102", "EM101", "D100", "ARG002", "D300", "D205", "D212", "D", "TRY301", "PLC1901", "BLE001", "PERF203", "FA100", "ANN401", "PLR0913", "ANN002", "ANN003", "RET504", "PLR2004"]
|
38
|
+
|
39
|
+
# Allow autofix for all enabled rules (when `--fix`) is provided.
|
40
|
+
fixable = ["A", "B", "C", "D", "E", "F", "G", "I", "N", "Q", "S", "T", "W", "ANN", "ARG", "BLE", "COM", "DJ", "DTZ", "EM", "ERA", "EXE", "FBT", "ICN", "INP", "ISC", "NPY", "PD", "PGH", "PIE", "PL", "PT", "PTH", "PYI", "RET", "RSE", "RUF", "SIM", "SLF", "TCH", "TID", "TRY", "UP", "YTT"]
|
41
|
+
unfixable = ["D", "Q001", "Q002", "F841"]
|
42
|
+
|
43
|
+
# Exclude a variety of commonly ignored directories.
|
44
|
+
exclude = [
|
45
|
+
".bzr",
|
46
|
+
".direnv",
|
47
|
+
".eggs",
|
48
|
+
".git",
|
49
|
+
".hg",
|
50
|
+
".mypy_cache",
|
51
|
+
".nox",
|
52
|
+
".pants.d",
|
53
|
+
".pytype",
|
54
|
+
".ruff_cache",
|
55
|
+
".svn",
|
56
|
+
".tox",
|
57
|
+
".venv",
|
58
|
+
"__pypackages__",
|
59
|
+
"_build",
|
60
|
+
"buck-out",
|
61
|
+
"build",
|
62
|
+
"dist",
|
63
|
+
"node_modules",
|
64
|
+
"venv",
|
65
|
+
"tests/",
|
66
|
+
"tests_sandbox/",
|
67
|
+
"alembic/",
|
68
|
+
"map_async_iterator.py",
|
69
|
+
"flow_api/",
|
70
|
+
]
|
71
|
+
per-file-ignores = {}
|
72
|
+
|
73
|
+
# Same as Black.
|
74
|
+
line-length = 88
|
75
|
+
|
76
|
+
# Allow unused variables when underscore-prefixed.
|
77
|
+
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
|
78
|
+
|
79
|
+
# Assume Python 3.8.
|
80
|
+
target-version = "py38"
|
81
|
+
|
82
|
+
[tool.ruff.mccabe]
|
83
|
+
# Unlike Flake8, default to a complexity level of 10.
|
84
|
+
max-complexity = 10
|
85
|
+
|
86
|
+
[tool.ruff.flake8-quotes]
|
87
|
+
inline-quotes = "single"
|
88
|
+
multiline-quotes = "single"
|
89
|
+
docstring-quotes = "double"
|
File without changes
|
@@ -0,0 +1,846 @@
|
|
1
|
+
import argparse
|
2
|
+
import asyncio
|
3
|
+
import contextlib
|
4
|
+
import functools
|
5
|
+
import io
|
6
|
+
import logging
|
7
|
+
import pathlib
|
8
|
+
import shlex
|
9
|
+
import stat
|
10
|
+
import string
|
11
|
+
import sys
|
12
|
+
import tempfile
|
13
|
+
import termios
|
14
|
+
import time
|
15
|
+
import tty
|
16
|
+
import typing
|
17
|
+
|
18
|
+
import paramiko
|
19
|
+
import paramiko.sftp_client
|
20
|
+
import rich.console
|
21
|
+
import rich.logging
|
22
|
+
import rich.markup
|
23
|
+
import rich.progress
|
24
|
+
import version
|
25
|
+
import watchdog.events
|
26
|
+
import watchdog.observers
|
27
|
+
|
28
|
+
REMOTE_USERNAME = 'devstack-user'
|
29
|
+
REMOTE_SOURCE_DIRECTORY = '/home/devstack-user/starflows'
|
30
|
+
REMOTE_OUTPUT_DIRECTORY = '/home/devstack-user/starflows-output'
|
31
|
+
EVENT_DEBOUNCE_SECONDS = .5
|
32
|
+
|
33
|
+
logging.basicConfig(level=logging.INFO, handlers=[rich.logging.RichHandler()], format='%(message)s')
|
34
|
+
logger = logging.getLogger('cli')
|
35
|
+
|
36
|
+
class SubprocessError(Exception):
|
37
|
+
"""A subprocess call returned with non-zero."""
|
38
|
+
|
39
|
+
|
40
|
+
class FileSystemEventHandlerToQueue(watchdog.events.FileSystemEventHandler):
|
41
|
+
def __init__(
|
42
|
+
self: 'FileSystemEventHandlerToQueue',
|
43
|
+
queue: asyncio.Queue,
|
44
|
+
loop: asyncio.BaseEventLoop,
|
45
|
+
*args,
|
46
|
+
**kwargs,
|
47
|
+
) -> None:
|
48
|
+
self._loop = loop
|
49
|
+
self._queue = queue
|
50
|
+
super(*args, **kwargs)
|
51
|
+
|
52
|
+
def on_any_event(
|
53
|
+
self: 'FileSystemEventHandlerToQueue',
|
54
|
+
event: watchdog.events.FileSystemEvent,
|
55
|
+
) -> None:
|
56
|
+
if event.event_type in (
|
57
|
+
watchdog.events.EVENT_TYPE_OPENED,
|
58
|
+
watchdog.events.EVENT_TYPE_CLOSED,
|
59
|
+
):
|
60
|
+
return
|
61
|
+
if event.event_type == watchdog.events.EVENT_TYPE_MODIFIED and event.is_directory:
|
62
|
+
return
|
63
|
+
if '/.git' in event.src_path:
|
64
|
+
return
|
65
|
+
if hasattr(event, 'dest_path') and '/.git' in event.dest_path:
|
66
|
+
return
|
67
|
+
self._loop.call_soon_threadsafe(self._queue.put_nowait, event)
|
68
|
+
|
69
|
+
|
70
|
+
async def run_subprocess(
|
71
|
+
program: str,
|
72
|
+
args: str,
|
73
|
+
*,
|
74
|
+
name: str,
|
75
|
+
cwd: typing.Optional[pathlib.Path] = None,
|
76
|
+
env: typing.Optional[dict] = None,
|
77
|
+
capture_stdout: bool = True,
|
78
|
+
print_stdout: bool = True,
|
79
|
+
capture_stderr: bool = True,
|
80
|
+
print_stderr: bool = True,
|
81
|
+
) -> None:
|
82
|
+
args_str = ' '.join(args)
|
83
|
+
process = await asyncio.create_subprocess_exec(
|
84
|
+
program,
|
85
|
+
*args,
|
86
|
+
cwd=cwd,
|
87
|
+
stdin=asyncio.subprocess.DEVNULL,
|
88
|
+
stdout=asyncio.subprocess.PIPE if capture_stdout else asyncio.subprocess.DEVNULL,
|
89
|
+
stderr=asyncio.subprocess.PIPE if capture_stderr else asyncio.subprocess.DEVNULL,
|
90
|
+
env=env,
|
91
|
+
)
|
92
|
+
stdout = b''
|
93
|
+
stderr = b''
|
94
|
+
try:
|
95
|
+
if not capture_stdout and not capture_stderr:
|
96
|
+
await process.wait()
|
97
|
+
else:
|
98
|
+
tasks = set()
|
99
|
+
if capture_stdout:
|
100
|
+
stdout_readline = asyncio.create_task(process.stdout.readline())
|
101
|
+
tasks.add(stdout_readline)
|
102
|
+
if capture_stderr:
|
103
|
+
stderr_readline = asyncio.create_task(process.stderr.readline())
|
104
|
+
tasks.add(stderr_readline)
|
105
|
+
while process.returncode is None:
|
106
|
+
done, pending = await asyncio.wait(
|
107
|
+
tasks,
|
108
|
+
return_when=asyncio.FIRST_COMPLETED,
|
109
|
+
)
|
110
|
+
if capture_stdout and stdout_readline in done:
|
111
|
+
stdout_line = await stdout_readline
|
112
|
+
if print_stdout and stdout_line.decode().strip():
|
113
|
+
logger.info('%s: %s', name, stdout_line.decode().strip())
|
114
|
+
stdout += stdout_line + b'\n'
|
115
|
+
stdout_readline = asyncio.create_task(process.stdout.readline())
|
116
|
+
pending.add(stdout_readline)
|
117
|
+
if capture_stderr and stderr_readline in done:
|
118
|
+
stderr_line = await stderr_readline
|
119
|
+
if print_stderr and stderr_line.decode().strip():
|
120
|
+
logger.warning('%s: %s', name, stderr_line.decode().strip())
|
121
|
+
stderr += stderr_line + b'\n'
|
122
|
+
stderr_readline = asyncio.create_task(process.stderr.readline())
|
123
|
+
pending.add(stderr_readline)
|
124
|
+
tasks = pending
|
125
|
+
finally:
|
126
|
+
if process.returncode is None:
|
127
|
+
logger.debug('Terminating "%s %s"', program, args_str)
|
128
|
+
process.terminate()
|
129
|
+
try:
|
130
|
+
await asyncio.wait_for(process.wait(), timeout=3)
|
131
|
+
except asyncio.TimeoutError:
|
132
|
+
logger.info('Killing "%s %s"', program, args_str)
|
133
|
+
process.kill()
|
134
|
+
await asyncio.wait_for(process.wait(), timeout=3)
|
135
|
+
if process.returncode:
|
136
|
+
if cwd is None:
|
137
|
+
msg = f'Command "{program} {args_str}" failed with returncode {process.returncode}.'
|
138
|
+
else:
|
139
|
+
msg = f'Command "{program} {args_str}" in "{cwd}" failed with returncode {process.returncode}.'
|
140
|
+
raise SubprocessError(msg)
|
141
|
+
logger.debug(
|
142
|
+
'Command "%s %s" succeeded.',
|
143
|
+
program,
|
144
|
+
args_str,
|
145
|
+
)
|
146
|
+
return stdout, stderr
|
147
|
+
|
148
|
+
|
149
|
+
def _get_event_significant_path(event: watchdog.events.FileSystemEvent) -> str:
|
150
|
+
if hasattr(event, 'dest_path'):
|
151
|
+
return event.dest_path
|
152
|
+
return event.src_path
|
153
|
+
|
154
|
+
|
155
|
+
def is_relative_to(self: pathlib.Path, other: pathlib.Path) -> bool:
|
156
|
+
return other == self or other in self.parents
|
157
|
+
|
158
|
+
|
159
|
+
class Cli:
|
160
|
+
def __init__(self: 'Cli', args: argparse.Namespace) -> None:
|
161
|
+
rich.print(f'Cloudomation devstack-cli {version.MAJOR}+{version.BRANCH_NAME}.{version.BUILD_DATE}.{version.SHORT_SHA}')
|
162
|
+
self.hostname = args.hostname
|
163
|
+
self.local_source_directory = pathlib.Path(args.source_directory)
|
164
|
+
|
165
|
+
self.local_output_directory = pathlib.Path(args.output_directory) if args.output_directory else None
|
166
|
+
if (
|
167
|
+
self.local_output_directory is not None
|
168
|
+
and (
|
169
|
+
is_relative_to(self.local_source_directory, self.local_output_directory)
|
170
|
+
or is_relative_to(self.local_output_directory, self.local_source_directory)
|
171
|
+
)
|
172
|
+
):
|
173
|
+
logger.error('Source-directory and output-directory must not overlap!')
|
174
|
+
sys.exit(1)
|
175
|
+
self.ssh_client = None
|
176
|
+
self.sftp_client = None
|
177
|
+
self.filesystem_watch_task = None
|
178
|
+
self.known_hosts_file = None
|
179
|
+
self.console = rich.console.Console()
|
180
|
+
if args.verbose:
|
181
|
+
logger.setLevel(logging.DEBUG)
|
182
|
+
|
183
|
+
async def run(self: 'Cli') -> None:
|
184
|
+
self.loop = asyncio.get_running_loop()
|
185
|
+
key_queue = asyncio.Queue()
|
186
|
+
await self._prepare_known_hosts()
|
187
|
+
try:
|
188
|
+
await self._connect_to_rde()
|
189
|
+
await self._init_local_cache()
|
190
|
+
sync_task = asyncio.create_task(self._start_sync())
|
191
|
+
port_forwarding_task = asyncio.create_task(self._start_port_forwarding())
|
192
|
+
logs_task = None
|
193
|
+
await self._setup_keyboard(key_queue)
|
194
|
+
try:
|
195
|
+
logger.info('Ready!')
|
196
|
+
key_queue.put_nowait('h')
|
197
|
+
while True:
|
198
|
+
key_press = await key_queue.get()
|
199
|
+
# check status
|
200
|
+
if sync_task is not None and sync_task.done():
|
201
|
+
sync_task = None
|
202
|
+
if port_forwarding_task is not None and port_forwarding_task.done():
|
203
|
+
port_forwarding_task = None
|
204
|
+
if logs_task is not None and logs_task.done():
|
205
|
+
logs_task = None
|
206
|
+
|
207
|
+
if key_press == 'h':
|
208
|
+
table = rich.table.Table(title='Help')
|
209
|
+
table.add_column('Key', style='cyan')
|
210
|
+
table.add_column('Function')
|
211
|
+
table.add_column('Status')
|
212
|
+
table.add_row('h', 'Print help')
|
213
|
+
table.add_row('v', 'Toggle debug logs', '[green]on' if logger.getEffectiveLevel() == logging.DEBUG else '[red]off')
|
214
|
+
table.add_row('s', 'Toggle file sync', '[red]off' if sync_task is None else '[green]on')
|
215
|
+
table.add_row('p', 'Toggle port forwarding', '[red]off' if port_forwarding_task is None else '[green]on')
|
216
|
+
table.add_row('l', 'Toggle following logs', '[red]off' if logs_task is None else '[green]on')
|
217
|
+
table.add_row('q', 'Quit')
|
218
|
+
rich.print(table)
|
219
|
+
elif key_press == 'v':
|
220
|
+
if logger.getEffectiveLevel() == logging.INFO:
|
221
|
+
logger.info('Enabling debug logs')
|
222
|
+
logger.setLevel(logging.DEBUG)
|
223
|
+
else:
|
224
|
+
logger.info('Disabling debug logs')
|
225
|
+
logger.setLevel(logging.INFO)
|
226
|
+
elif key_press == 's':
|
227
|
+
if sync_task is None:
|
228
|
+
sync_task = asyncio.create_task(self._start_sync())
|
229
|
+
else:
|
230
|
+
sync_task.cancel()
|
231
|
+
try:
|
232
|
+
await sync_task
|
233
|
+
except asyncio.CancelledError:
|
234
|
+
pass
|
235
|
+
except Exception:
|
236
|
+
logger.exception('Error during file sync')
|
237
|
+
sync_task = None
|
238
|
+
elif key_press == 'p':
|
239
|
+
if port_forwarding_task is None:
|
240
|
+
port_forwarding_task = asyncio.create_task(self._start_port_forwarding())
|
241
|
+
else:
|
242
|
+
port_forwarding_task.cancel()
|
243
|
+
try:
|
244
|
+
await port_forwarding_task
|
245
|
+
except asyncio.CancelledError:
|
246
|
+
pass
|
247
|
+
except Exception:
|
248
|
+
logger.exception('Error during port forwarding')
|
249
|
+
port_forwarding_task = None
|
250
|
+
elif key_press == 'l':
|
251
|
+
if logs_task is None:
|
252
|
+
logs_task = asyncio.create_task(self._start_logs())
|
253
|
+
else:
|
254
|
+
logs_task.cancel()
|
255
|
+
try:
|
256
|
+
await logs_task
|
257
|
+
except asyncio.CancelledError:
|
258
|
+
pass
|
259
|
+
except Exception:
|
260
|
+
logger.exception('Error during logs')
|
261
|
+
logs_task = None
|
262
|
+
elif key_press == 'q':
|
263
|
+
break
|
264
|
+
elif ord(key_press) == 10: # return
|
265
|
+
rich.print('')
|
266
|
+
else:
|
267
|
+
logger.debug('Unknown keypress "%s" (%d)', key_press if key_press in string.printable else '?', ord(key_press))
|
268
|
+
finally:
|
269
|
+
await self._reset_keyboard()
|
270
|
+
if port_forwarding_task is not None:
|
271
|
+
port_forwarding_task.cancel()
|
272
|
+
with contextlib.suppress(asyncio.CancelledError):
|
273
|
+
await port_forwarding_task
|
274
|
+
if sync_task is not None:
|
275
|
+
sync_task.cancel()
|
276
|
+
with contextlib.suppress(asyncio.CancelledError):
|
277
|
+
await sync_task
|
278
|
+
if logs_task is not None:
|
279
|
+
logs_task.cancel()
|
280
|
+
with contextlib.suppress(asyncio.CancelledError):
|
281
|
+
await logs_task
|
282
|
+
await self._disconnect_from_rde()
|
283
|
+
finally:
|
284
|
+
await self._cleanup_known_hosts_file()
|
285
|
+
|
286
|
+
async def _setup_keyboard(self: 'Cli', queue: asyncio.Queue) -> None:
|
287
|
+
self._fd = sys.stdin.fileno()
|
288
|
+
self._tcattr = termios.tcgetattr(self._fd)
|
289
|
+
tty.setcbreak(self._fd)
|
290
|
+
def on_stdin() -> None:
|
291
|
+
self.loop.call_soon_threadsafe(queue.put_nowait, sys.stdin.read(1))
|
292
|
+
self.loop.add_reader(sys.stdin, on_stdin)
|
293
|
+
|
294
|
+
async def _reset_keyboard(self: 'Cli') -> None:
|
295
|
+
termios.tcsetattr(self._fd, termios.TCSADRAIN, self._tcattr)
|
296
|
+
self.loop.remove_reader(sys.stdin)
|
297
|
+
|
298
|
+
async def _prepare_known_hosts(self: 'Cli') -> None:
|
299
|
+
self.known_hosts_file = tempfile.NamedTemporaryFile(delete=False)
|
300
|
+
logger.info('Writing temporary known_hosts file "%s"', self.known_hosts_file.name)
|
301
|
+
logger.debug('Scanning hostkeys of "%s"', self.hostname)
|
302
|
+
try:
|
303
|
+
stdout, stderr = await run_subprocess(
|
304
|
+
'ssh-keyscan',
|
305
|
+
[
|
306
|
+
self.hostname,
|
307
|
+
],
|
308
|
+
name='ssh-keyscan',
|
309
|
+
print_stdout=False,
|
310
|
+
print_stderr=False,
|
311
|
+
)
|
312
|
+
except SubprocessError as ex:
|
313
|
+
logger.error('%s Failed to fetch hostkeys. Is you RDE running?', ex) # noqa: TRY400
|
314
|
+
sys.exit(1)
|
315
|
+
self.known_hosts_file.write(stdout)
|
316
|
+
with contextlib.suppress(FileNotFoundError):
|
317
|
+
self.known_hosts_file.write(pathlib.Path('~/.ssh/known_hosts').expanduser().read_bytes())
|
318
|
+
self.known_hosts_file.close()
|
319
|
+
|
320
|
+
async def _cleanup_known_hosts_file(self: 'Cli') -> None:
|
321
|
+
if self.known_hosts_file is None:
|
322
|
+
return
|
323
|
+
pathlib.Path(self.known_hosts_file.name).unlink()
|
324
|
+
|
325
|
+
async def _connect_to_rde(self: 'Cli') -> None:
|
326
|
+
self.ssh_client = paramiko.SSHClient()
|
327
|
+
self.ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
328
|
+
logger.info('Connecting to RDE')
|
329
|
+
try:
|
330
|
+
self.ssh_client.connect(
|
331
|
+
hostname=self.hostname,
|
332
|
+
username=REMOTE_USERNAME,
|
333
|
+
timeout=30,
|
334
|
+
)
|
335
|
+
except TimeoutError:
|
336
|
+
logger.exception('Timeout while connecting to RDE. Is your RDE running?')
|
337
|
+
sys.exit(1)
|
338
|
+
return
|
339
|
+
transport = self.ssh_client.get_transport()
|
340
|
+
self.sftp_client = paramiko.sftp_client.SFTPClient.from_transport(transport)
|
341
|
+
|
342
|
+
async def _disconnect_from_rde(self: 'Cli') -> None:
|
343
|
+
if self.sftp_client is not None:
|
344
|
+
self.sftp_client.close()
|
345
|
+
self.sftp_client = None
|
346
|
+
if self.ssh_client is not None:
|
347
|
+
self.ssh_client.close()
|
348
|
+
self.ssh_client = None
|
349
|
+
|
350
|
+
async def _init_local_cache(self: 'Cli') -> None:
|
351
|
+
self.local_source_directory.mkdir(parents=True, exist_ok=True)
|
352
|
+
logger.debug('Listing remote items')
|
353
|
+
listing = self.sftp_client.listdir_attr(REMOTE_SOURCE_DIRECTORY)
|
354
|
+
|
355
|
+
logger.info('Processing %d remote items...', len(listing))
|
356
|
+
for file_info in rich.progress.track(
|
357
|
+
sequence=sorted(
|
358
|
+
listing,
|
359
|
+
key=lambda file_info: file_info.filename.casefold(),
|
360
|
+
),
|
361
|
+
description='Processing remote items',
|
362
|
+
):
|
363
|
+
logger.info('Processing "%s"', file_info.filename)
|
364
|
+
try:
|
365
|
+
result = await self._process_remote_item(file_info)
|
366
|
+
except SubprocessError:
|
367
|
+
logger.exception('Failed')
|
368
|
+
else:
|
369
|
+
logger.info(result)
|
370
|
+
|
371
|
+
async def _process_remote_item(self: 'Cli', file_info: paramiko.sftp_attr.SFTPAttributes) -> str:
|
372
|
+
filename = file_info.filename
|
373
|
+
|
374
|
+
if file_info.st_mode & stat.S_IFDIR:
|
375
|
+
# check if .git exists
|
376
|
+
try:
|
377
|
+
git_stat = self.sftp_client.stat(f'{REMOTE_SOURCE_DIRECTORY}/{filename}/.git')
|
378
|
+
except FileNotFoundError:
|
379
|
+
pass
|
380
|
+
else:
|
381
|
+
if git_stat.st_mode & stat.S_IFDIR:
|
382
|
+
repo_dir = self.local_source_directory / filename
|
383
|
+
if not repo_dir.exists():
|
384
|
+
return await self._process_remote_item_clone(file_info.filename)
|
385
|
+
return f'Repository "{filename}" already exists'
|
386
|
+
return await self._process_remote_item_copy_dir(file_info.filename)
|
387
|
+
return await self._process_remote_item_copy_file(file_info.filename)
|
388
|
+
|
389
|
+
async def _process_remote_item_copy_dir(self: 'Cli', filename: str) -> str:
|
390
|
+
await run_subprocess(
|
391
|
+
'rsync',
|
392
|
+
[
|
393
|
+
'-e', f'ssh -o ConnectTimeout=10 -o UserKnownHostsFile={self.known_hosts_file.name}',
|
394
|
+
'--archive',
|
395
|
+
'--checksum',
|
396
|
+
f'{REMOTE_USERNAME}@{self.hostname}:{REMOTE_SOURCE_DIRECTORY}/{filename}/',
|
397
|
+
str(self.local_source_directory / filename),
|
398
|
+
],
|
399
|
+
name='Copy remote directory',
|
400
|
+
)
|
401
|
+
return f'Copied directory "{filename}"'
|
402
|
+
|
403
|
+
async def _process_remote_item_copy_file(self: 'Cli', filename: str) -> str:
|
404
|
+
await self.loop.run_in_executor(
|
405
|
+
executor=None,
|
406
|
+
func=functools.partial(
|
407
|
+
self.sftp_client.get,
|
408
|
+
remotepath=f'{REMOTE_SOURCE_DIRECTORY}/{filename}',
|
409
|
+
localpath=str(self.local_source_directory / filename),
|
410
|
+
),
|
411
|
+
)
|
412
|
+
return f'Copied file "{filename}"'
|
413
|
+
|
414
|
+
async def _process_remote_item_clone(self: 'Cli', filename: str) -> str:
|
415
|
+
await run_subprocess(
|
416
|
+
'git',
|
417
|
+
[
|
418
|
+
'clone',
|
419
|
+
'-q',
|
420
|
+
f'{REMOTE_USERNAME}@{self.hostname}:{REMOTE_SOURCE_DIRECTORY}/{filename}',
|
421
|
+
],
|
422
|
+
name='Git clone',
|
423
|
+
cwd=self.local_source_directory,
|
424
|
+
env={
|
425
|
+
'GIT_SSH_COMMAND': f'ssh -o ConnectTimeout=10 -o UserKnownHostsFile={self.known_hosts_file.name}',
|
426
|
+
},
|
427
|
+
)
|
428
|
+
ssh_stdin, ssh_stdout, ssh_stderr = await self.loop.run_in_executor(
|
429
|
+
executor=None,
|
430
|
+
func=functools.partial(
|
431
|
+
self.ssh_client.exec_command,
|
432
|
+
shlex.join([
|
433
|
+
'git',
|
434
|
+
'-C',
|
435
|
+
f'{REMOTE_SOURCE_DIRECTORY}/{filename}',
|
436
|
+
'config',
|
437
|
+
'--get',
|
438
|
+
'remote.origin.url',
|
439
|
+
]),
|
440
|
+
),
|
441
|
+
)
|
442
|
+
upstream = ssh_stdout.readline().strip()
|
443
|
+
await run_subprocess(
|
444
|
+
'git',
|
445
|
+
[
|
446
|
+
'remote',
|
447
|
+
'set-url',
|
448
|
+
'origin',
|
449
|
+
upstream,
|
450
|
+
],
|
451
|
+
name='Git remote set-url',
|
452
|
+
cwd=self.local_source_directory / filename,
|
453
|
+
env={
|
454
|
+
'GIT_SSH_COMMAND': f'ssh -o ConnectTimeout=10 -o UserKnownHostsFile={self.known_hosts_file.name}',
|
455
|
+
},
|
456
|
+
)
|
457
|
+
return f'Cloned repository "{filename}"'
|
458
|
+
|
459
|
+
async def _start_sync(self: 'Cli') -> None:
|
460
|
+
logger.info('Starting file sync')
|
461
|
+
filesystem_event_queue = asyncio.Queue()
|
462
|
+
filesystem_watch_task = asyncio.create_task(
|
463
|
+
self._watch_filesystem(
|
464
|
+
queue=filesystem_event_queue,
|
465
|
+
),
|
466
|
+
)
|
467
|
+
if self.local_output_directory:
|
468
|
+
remote_sync_task = asyncio.create_task(
|
469
|
+
self._remote_sync(),
|
470
|
+
)
|
471
|
+
else:
|
472
|
+
remote_sync_task = None
|
473
|
+
background_sync_task = None
|
474
|
+
try:
|
475
|
+
while True:
|
476
|
+
filesystem_events = []
|
477
|
+
if background_sync_task is not None:
|
478
|
+
background_sync_task.cancel()
|
479
|
+
with contextlib.suppress(asyncio.CancelledError):
|
480
|
+
await background_sync_task
|
481
|
+
background_sync_task = asyncio.create_task(self._background_sync())
|
482
|
+
filesystem_events.append(await filesystem_event_queue.get())
|
483
|
+
logger.debug('first event, debouncing...')
|
484
|
+
# debounce
|
485
|
+
await asyncio.sleep(EVENT_DEBOUNCE_SECONDS)
|
486
|
+
logger.debug('collecting changes')
|
487
|
+
while not filesystem_event_queue.empty():
|
488
|
+
filesystem_events.append(filesystem_event_queue.get_nowait())
|
489
|
+
for event in filesystem_events:
|
490
|
+
logger.debug('non-unique event: %s', event)
|
491
|
+
# remove duplicates
|
492
|
+
events = [
|
493
|
+
event
|
494
|
+
for i, event
|
495
|
+
in enumerate(filesystem_events)
|
496
|
+
if _get_event_significant_path(event) not in (
|
497
|
+
_get_event_significant_path(later_event)
|
498
|
+
for later_event
|
499
|
+
in filesystem_events[i+1:]
|
500
|
+
)
|
501
|
+
]
|
502
|
+
for i, event in enumerate(events, start=1):
|
503
|
+
logger.debug('unique event [%d/%d]: %s', i, len(events), event)
|
504
|
+
await self._process_sync_event(event)
|
505
|
+
except asyncio.CancelledError:
|
506
|
+
logger.info('File sync interrupted')
|
507
|
+
raise
|
508
|
+
except Exception:
|
509
|
+
logger.exception('File sync failed')
|
510
|
+
else:
|
511
|
+
logger.info('File sync stopped')
|
512
|
+
finally:
|
513
|
+
filesystem_watch_task.cancel()
|
514
|
+
with contextlib.suppress(asyncio.CancelledError):
|
515
|
+
await filesystem_watch_task
|
516
|
+
if remote_sync_task is not None:
|
517
|
+
remote_sync_task.cancel()
|
518
|
+
with contextlib.suppress(asyncio.CancelledError):
|
519
|
+
await remote_sync_task
|
520
|
+
if background_sync_task is not None:
|
521
|
+
background_sync_task.cancel()
|
522
|
+
with contextlib.suppress(asyncio.CancelledError):
|
523
|
+
await background_sync_task
|
524
|
+
|
525
|
+
async def _background_sync(self: 'Cli') -> None:
|
526
|
+
logger.debug('Starting background sync')
|
527
|
+
self.local_source_directory.mkdir(parents=True, exist_ok=True)
|
528
|
+
with contextlib.suppress(OSError):
|
529
|
+
self.sftp_client.mkdir(REMOTE_SOURCE_DIRECTORY)
|
530
|
+
try:
|
531
|
+
await run_subprocess(
|
532
|
+
'rsync',
|
533
|
+
[
|
534
|
+
'-e', f'ssh -o ConnectTimeout=10 -o UserKnownHostsFile={self.known_hosts_file.name}',
|
535
|
+
'--archive',
|
536
|
+
'--delete',
|
537
|
+
'--exclude', 'build-cache-*', # TODO: make exclusions configurable
|
538
|
+
'--exclude', 'dev-tool/config',
|
539
|
+
'--exclude', 'alembic.ini',
|
540
|
+
'--exclude', 'cypress/screenshots',
|
541
|
+
'--exclude', 'cypress/videos',
|
542
|
+
'--exclude', 'flow_api',
|
543
|
+
'--exclude', '.git',
|
544
|
+
'--exclude', '__pycache__',
|
545
|
+
'--exclude', '.cache',
|
546
|
+
'--exclude', 'node_modules',
|
547
|
+
'--exclude', '.venv',
|
548
|
+
'--exclude', 'bundle-content', # until https://app.clickup.com/t/86bxn0exx
|
549
|
+
'--exclude', 'cloudomation-fe/build',
|
550
|
+
'--exclude', 'devstack-self-service-portal/vite-cache',
|
551
|
+
'--exclude', 'devstack-self-service-portal/dist',
|
552
|
+
'--exclude', 'documentation/generator/generated',
|
553
|
+
'--exclude', 'version.py',
|
554
|
+
'--exclude', 'instantclient-basic-linux.x64.zip',
|
555
|
+
'--exclude', 'msodbcsql.deb',
|
556
|
+
'--exclude', 'auth/report',
|
557
|
+
'--exclude', 'cloudomation-fe/.env',
|
558
|
+
'--exclude', 'cloudomation/tmp_git_task',
|
559
|
+
'--exclude', 'cloudomation/tmp',
|
560
|
+
'--exclude', 'cloudomation/notifications',
|
561
|
+
'--exclude', 'documentation/versioned_docs',
|
562
|
+
'--human-readable',
|
563
|
+
'--info=name1',
|
564
|
+
f'{self.local_source_directory}/',
|
565
|
+
f'{REMOTE_USERNAME}@{self.hostname}:{REMOTE_SOURCE_DIRECTORY}',
|
566
|
+
],
|
567
|
+
name='Background sync',
|
568
|
+
)
|
569
|
+
except asyncio.CancelledError:
|
570
|
+
logger.debug('Background sync interrupted')
|
571
|
+
raise
|
572
|
+
except SubprocessError as ex:
|
573
|
+
logger.error('Background sync failed: %s', ex) # noqa: TRY400
|
574
|
+
except Exception:
|
575
|
+
logger.exception('Background sync failed')
|
576
|
+
else:
|
577
|
+
logger.info('Background sync done')
|
578
|
+
|
579
|
+
async def _reverse_background_sync(self: 'Cli') -> None:
|
580
|
+
logger.debug('Starting reverse background sync')
|
581
|
+
with contextlib.suppress(OSError):
|
582
|
+
self.sftp_client.mkdir(REMOTE_OUTPUT_DIRECTORY)
|
583
|
+
self.local_output_directory.mkdir(parents=True, exist_ok=True)
|
584
|
+
try:
|
585
|
+
stdout, stderr = await run_subprocess(
|
586
|
+
'rsync',
|
587
|
+
[
|
588
|
+
'-e', f'ssh -o ConnectTimeout=10 -o UserKnownHostsFile={self.known_hosts_file.name}',
|
589
|
+
'--archive',
|
590
|
+
'--exclude', '__pycache__',
|
591
|
+
'--human-readable',
|
592
|
+
'--info=name1',
|
593
|
+
f'{REMOTE_USERNAME}@{self.hostname}:{REMOTE_OUTPUT_DIRECTORY}/',
|
594
|
+
str(self.local_output_directory),
|
595
|
+
],
|
596
|
+
name='Reverse background sync',
|
597
|
+
)
|
598
|
+
except asyncio.CancelledError:
|
599
|
+
logger.debug('Reverse background sync interrupted')
|
600
|
+
raise
|
601
|
+
except SubprocessError:
|
602
|
+
logger.exception('Reverse background sync failed')
|
603
|
+
else:
|
604
|
+
logger.debug('Reverse background sync done')
|
605
|
+
|
606
|
+
async def _watch_filesystem(
|
607
|
+
self: 'Cli',
|
608
|
+
queue: asyncio.Queue,
|
609
|
+
) -> None:
|
610
|
+
handler = FileSystemEventHandlerToQueue(queue, self.loop)
|
611
|
+
filesystem_observer = watchdog.observers.Observer()
|
612
|
+
filesystem_observer.schedule(
|
613
|
+
event_handler=handler,
|
614
|
+
path=str(self.local_source_directory),
|
615
|
+
recursive=True,
|
616
|
+
)
|
617
|
+
filesystem_observer.start()
|
618
|
+
logger.info('Filesystem watches established')
|
619
|
+
try:
|
620
|
+
await self.loop.run_in_executor(
|
621
|
+
executor=None,
|
622
|
+
func=filesystem_observer.join,
|
623
|
+
)
|
624
|
+
finally:
|
625
|
+
filesystem_observer.stop()
|
626
|
+
filesystem_observer.join(3)
|
627
|
+
|
628
|
+
async def _remote_sync(self: 'Cli') -> None:
|
629
|
+
while True:
|
630
|
+
await self._reverse_background_sync()
|
631
|
+
await asyncio.sleep(10)
|
632
|
+
|
633
|
+
async def _process_sync_event(self: 'Cli', event: watchdog.events.FileSystemEvent) -> None:
|
634
|
+
local_path = pathlib.Path(event.src_path)
|
635
|
+
relative_path = local_path.relative_to(self.local_source_directory)
|
636
|
+
remote_path = f'{REMOTE_SOURCE_DIRECTORY}/{relative_path}'
|
637
|
+
if isinstance(event, watchdog.events.DirCreatedEvent):
|
638
|
+
await self._remote_directory_create(remote_path)
|
639
|
+
elif isinstance(event, watchdog.events.DirDeletedEvent):
|
640
|
+
await self._remote_directory_delete(remote_path)
|
641
|
+
elif isinstance(event, watchdog.events.FileCreatedEvent):
|
642
|
+
await self._remote_file_create(remote_path)
|
643
|
+
elif isinstance(event, watchdog.events.FileModifiedEvent):
|
644
|
+
stat = local_path.stat()
|
645
|
+
times = (stat.st_atime, stat.st_mtime)
|
646
|
+
await self._remote_file_copy(event.src_path, remote_path, times)
|
647
|
+
elif isinstance(event, watchdog.events.FileDeletedEvent):
|
648
|
+
await self._remote_file_delete(remote_path)
|
649
|
+
elif isinstance(event, watchdog.events.FileMovedEvent):
|
650
|
+
dest_local_path = pathlib.Path(event.dest_path)
|
651
|
+
dest_relative_path = dest_local_path.relative_to(self.local_source_directory)
|
652
|
+
dest_remote_path = f'{REMOTE_SOURCE_DIRECTORY}/{dest_relative_path}'
|
653
|
+
stat = dest_local_path.stat()
|
654
|
+
times = (stat.st_atime, stat.st_mtime)
|
655
|
+
await self._remote_file_move(remote_path, dest_remote_path, times)
|
656
|
+
|
657
|
+
async def _remote_directory_create(self: 'Cli', remote_path: str) -> None:
|
658
|
+
logger.info('Create directory: "%s" (remote)', remote_path)
|
659
|
+
try:
|
660
|
+
self.sftp_client.mkdir(remote_path)
|
661
|
+
except OSError:
|
662
|
+
logger.exception('-> failed')
|
663
|
+
try:
|
664
|
+
stat = self.sftp_client.stat(remote_path)
|
665
|
+
except FileNotFoundError:
|
666
|
+
logger.info('-> remote directory does not exist')
|
667
|
+
else:
|
668
|
+
logger.info('-> remote directory already exists:\n%s', stat)
|
669
|
+
|
670
|
+
async def _remote_directory_delete(self: 'Cli', remote_path: str) -> None:
|
671
|
+
logger.info('Delete directory: "%s" (remote)', remote_path)
|
672
|
+
try:
|
673
|
+
self.sftp_client.rmdir(remote_path)
|
674
|
+
except FileNotFoundError:
|
675
|
+
logger.exception('-> remote directory does not exist')
|
676
|
+
except OSError:
|
677
|
+
logger.exception('-> failed')
|
678
|
+
|
679
|
+
async def _remote_file_create(self: 'Cli', remote_path: str) -> None:
|
680
|
+
logger.info('Create file: "%s" (remote)', remote_path)
|
681
|
+
self.sftp_client.putfo(io.BytesIO(), remote_path)
|
682
|
+
|
683
|
+
async def _remote_file_copy(self: 'Cli', local_path: str, remote_path: str, times: typing.Tuple[int, int]) -> None:
|
684
|
+
logger.info('Copy file: "%s" (local) -> "%s" (remote)', local_path, remote_path)
|
685
|
+
self.sftp_client.put(local_path, remote_path)
|
686
|
+
self.sftp_client.utime(remote_path, times)
|
687
|
+
|
688
|
+
async def _remote_file_delete(self: 'Cli', remote_path: str) -> None:
|
689
|
+
logger.info('Delete file: "%s" (remote)', remote_path)
|
690
|
+
try:
|
691
|
+
self.sftp_client.remove(remote_path)
|
692
|
+
except FileNotFoundError:
|
693
|
+
logger.info('-> remote file does not exist')
|
694
|
+
|
695
|
+
async def _remote_file_move(self: 'Cli', remote_path: str, dest_remote_path: str, times: typing.Tuple[int, int]) -> None:
|
696
|
+
logger.info('Move file: "%s" (remote) -> "%s" (remote)', remote_path, dest_remote_path)
|
697
|
+
self.sftp_client.rename(remote_path, dest_remote_path)
|
698
|
+
self.sftp_client.utime(dest_remote_path, times)
|
699
|
+
|
700
|
+
async def _start_port_forwarding(self: 'Cli') -> None:
|
701
|
+
logger.info('Starting port forwarding of ports 8443, 5678, 6678, 7678, 8678, 3000, 2022')
|
702
|
+
try:
|
703
|
+
await run_subprocess(
|
704
|
+
'ssh',
|
705
|
+
[
|
706
|
+
'-o', 'ConnectTimeout=10',
|
707
|
+
'-o', f'UserKnownHostsFile={self.known_hosts_file.name}',
|
708
|
+
'-NT',
|
709
|
+
f'{REMOTE_USERNAME}@{self.hostname}',
|
710
|
+
'-L', '8443:localhost:443', # TODO: make ports configurable
|
711
|
+
'-L', '5678:localhost:5678',
|
712
|
+
'-L', '6678:localhost:6678',
|
713
|
+
'-L', '7678:localhost:7678',
|
714
|
+
'-L', '8678:localhost:8678',
|
715
|
+
'-L', '3000:localhost:3000',
|
716
|
+
'-L', '2022:localhost:2022',
|
717
|
+
],
|
718
|
+
name='Port forwarding',
|
719
|
+
capture_stdout=False,
|
720
|
+
)
|
721
|
+
except asyncio.CancelledError:
|
722
|
+
logger.info('Port forwarding interrupted')
|
723
|
+
raise
|
724
|
+
except SubprocessError:
|
725
|
+
logger.exception('Port forwarding failed')
|
726
|
+
else:
|
727
|
+
logger.info('Port forwarding done')
|
728
|
+
|
729
|
+
|
730
|
+
async def _start_logs(self: 'Cli') -> None:
|
731
|
+
logger.info('Following logs')
|
732
|
+
stdout_queue = asyncio.Queue()
|
733
|
+
stderr_queue = asyncio.Queue()
|
734
|
+
stream_task = self.loop.run_in_executor(
|
735
|
+
executor=None,
|
736
|
+
func=functools.partial(
|
737
|
+
self._stream_logs,
|
738
|
+
stdout_queue=stdout_queue,
|
739
|
+
stderr_queue=stderr_queue,
|
740
|
+
),
|
741
|
+
)
|
742
|
+
try:
|
743
|
+
stdout_get = asyncio.create_task(stdout_queue.get())
|
744
|
+
stderr_get = asyncio.create_task(stderr_queue.get())
|
745
|
+
while True:
|
746
|
+
done, pending = await asyncio.wait(
|
747
|
+
{stdout_get, stderr_get},
|
748
|
+
return_when=asyncio.FIRST_COMPLETED,
|
749
|
+
)
|
750
|
+
if stdout_get in done:
|
751
|
+
stdout = await stdout_get
|
752
|
+
if stdout is not None:
|
753
|
+
self.console.print(rich.markup.escape(stdout.strip()), style='default on grey23', justify='left')
|
754
|
+
stdout_get = asyncio.create_task(stdout_queue.get())
|
755
|
+
if stderr_get in done:
|
756
|
+
stderr = await stderr_get
|
757
|
+
if stderr is not None:
|
758
|
+
self.console.print(rich.markup.escape(stderr.strip()), style='default on red', justify='left')
|
759
|
+
stderr_get = asyncio.create_task(stderr_queue.get())
|
760
|
+
except asyncio.CancelledError:
|
761
|
+
logger.info('Following logs interrupted')
|
762
|
+
raise
|
763
|
+
except Exception:
|
764
|
+
logger.exception('Following logs failed')
|
765
|
+
else:
|
766
|
+
logger.info('Stopped following logs')
|
767
|
+
finally:
|
768
|
+
stream_task.cancel()
|
769
|
+
with contextlib.suppress(asyncio.CancelledError):
|
770
|
+
await stream_task
|
771
|
+
|
772
|
+
|
773
|
+
def _stream_logs(
|
774
|
+
self: 'Cli',
|
775
|
+
stdout_queue: asyncio.Queue,
|
776
|
+
stderr_queue: asyncio.Queue,
|
777
|
+
) -> None:
|
778
|
+
ssh_stdin, ssh_stdout, ssh_stderr = self.ssh_client.exec_command(
|
779
|
+
'cd /home/devstack-user/starflows/research/dev-tool && . dev.sh logs',
|
780
|
+
get_pty=False,
|
781
|
+
timeout=0,
|
782
|
+
)
|
783
|
+
ssh_stdin.close()
|
784
|
+
have_stdout = False
|
785
|
+
have_stderr = False
|
786
|
+
while True:
|
787
|
+
try:
|
788
|
+
stdout = ssh_stdout.readline(1024)
|
789
|
+
except TimeoutError:
|
790
|
+
have_stdout = False
|
791
|
+
else:
|
792
|
+
have_stdout = True
|
793
|
+
try:
|
794
|
+
stderr = ssh_stderr.readline(1024)
|
795
|
+
except TimeoutError:
|
796
|
+
have_stderr = False
|
797
|
+
else:
|
798
|
+
have_stderr = True
|
799
|
+
if have_stdout and stdout:
|
800
|
+
self.loop.call_soon_threadsafe(stdout_queue.put_nowait, stdout)
|
801
|
+
if have_stderr and stderr:
|
802
|
+
self.loop.call_soon_threadsafe(stderr_queue.put_nowait, stderr)
|
803
|
+
if have_stdout and not stdout and have_stderr and not stderr:
|
804
|
+
break
|
805
|
+
if not have_stdout and not have_stderr:
|
806
|
+
time.sleep(.5)
|
807
|
+
self.loop.call_soon_threadsafe(stdout_queue.put_nowait, None)
|
808
|
+
self.loop.call_soon_threadsafe(stderr_queue.put_nowait, None)
|
809
|
+
|
810
|
+
|
811
|
+
def main() -> None:
|
812
|
+
parser = argparse.ArgumentParser()
|
813
|
+
parser.add_argument(
|
814
|
+
'-H', '--hostname',
|
815
|
+
required=True,
|
816
|
+
help='the IP or hostname of the RDE',
|
817
|
+
)
|
818
|
+
parser.add_argument(
|
819
|
+
'-s', '--source-directory',
|
820
|
+
required=True,
|
821
|
+
help='a local directory where the sources from the RDE are cached',
|
822
|
+
)
|
823
|
+
parser.add_argument(
|
824
|
+
'-o', '--output-directory',
|
825
|
+
help='a local directory where artifacts created on the RDE are stored',
|
826
|
+
)
|
827
|
+
parser.add_argument(
|
828
|
+
'-v', '--verbose',
|
829
|
+
action='store_true',
|
830
|
+
help='enable debug logging',
|
831
|
+
)
|
832
|
+
parser.add_argument(
|
833
|
+
'-V', '--version',
|
834
|
+
action='version',
|
835
|
+
version=f'Cloudomation devstack-cli {version.MAJOR}+{version.BRANCH_NAME}.{version.BUILD_DATE}.{version.SHORT_SHA}',
|
836
|
+
)
|
837
|
+
args = parser.parse_args()
|
838
|
+
|
839
|
+
cli = Cli(args)
|
840
|
+
with contextlib.suppress(KeyboardInterrupt):
|
841
|
+
asyncio.run(cli.run())
|
842
|
+
logger.info('Bye!')
|
843
|
+
|
844
|
+
|
845
|
+
if __name__ == '__main__':
|
846
|
+
main()
|
@@ -0,0 +1,80 @@
|
|
1
|
+
Metadata-Version: 2.1
|
2
|
+
Name: devstack-cli
|
3
|
+
Version: 9.0.0
|
4
|
+
Summary: Command-line access to Remote Development Environments (RDEs) created by Cloudomation DevStack
|
5
|
+
Author-email: Stefan Mückstein <stefan@cloudomation.com>
|
6
|
+
Project-URL: Homepage, https://cloudomation.com/
|
7
|
+
Project-URL: Documentation, https://docs.cloudomation.com/devstack/
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
10
|
+
Classifier: Operating System :: OS Independent
|
11
|
+
Requires-Python: >=3.8
|
12
|
+
Description-Content-Type: text/markdown
|
13
|
+
License-File: LICENSE
|
14
|
+
Requires-Dist: paramiko
|
15
|
+
Requires-Dist: watchdog
|
16
|
+
Requires-Dist: rich
|
17
|
+
|
18
|
+
# DevStack CLI
|
19
|
+
|
20
|
+
The `devstack-cli` provides command-line access to Remote Development Environments (RDEs) created by Cloudomation DevStack. Learn more about [Cloudomation DevStack](https://docs.cloudomation.com/devstack/docs/overview-and-concept).
|
21
|
+
|
22
|
+
## Installation
|
23
|
+
|
24
|
+
The following binaries must be installed to be able to use `devstack-cli`:
|
25
|
+
|
26
|
+
* `ssh`
|
27
|
+
* `ssh-keyscan`
|
28
|
+
* `rsync`
|
29
|
+
* `git`
|
30
|
+
|
31
|
+
On Debian/Ubuntu the packages can be installed with
|
32
|
+
|
33
|
+
```
|
34
|
+
apt install openssh-client rsync git
|
35
|
+
```
|
36
|
+
|
37
|
+
Then you can install `devstack-cli` by running
|
38
|
+
|
39
|
+
```
|
40
|
+
python -m pip install devstack-cli
|
41
|
+
```
|
42
|
+
|
43
|
+
## Usage
|
44
|
+
|
45
|
+
```
|
46
|
+
usage: devstack-cli [-h] -H HOSTNAME -s SOURCE_DIRECTORY [-o OUTPUT_DIRECTORY] [-v]
|
47
|
+
```
|
48
|
+
|
49
|
+
You need to specify the hostname/IP of the RDE created by Cloudomation DevStack as well as the path to a directory where the sources will be cached locally. Optionally you can specify an output directory where artifacts created on the RDE will be stored locally.
|
50
|
+
The `-v` switch enables debug logging.
|
51
|
+
|
52
|
+
## Support
|
53
|
+
|
54
|
+
`devstack-cli` is part of Cloudomation DevStack and support is provided to you with an active subscription.
|
55
|
+
|
56
|
+
## Authors and acknowledgment
|
57
|
+
|
58
|
+
Cloudomation actively maintains the `devstack-cli` command line tool as part of Cloudomation DevStack
|
59
|
+
|
60
|
+
## License
|
61
|
+
|
62
|
+
Copyright (c) 2024 Starflows OG
|
63
|
+
|
64
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
65
|
+
of this software and associated documentation files (the "Software"), to deal
|
66
|
+
in the Software without restriction, including without limitation the rights
|
67
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
68
|
+
copies of the Software, and to permit persons to whom the Software is
|
69
|
+
furnished to do so, subject to the following conditions:
|
70
|
+
|
71
|
+
The above copyright notice and this permission notice shall be included in all
|
72
|
+
copies or substantial portions of the Software.
|
73
|
+
|
74
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
75
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
76
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
77
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
78
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
79
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
80
|
+
SOFTWARE.
|
@@ -0,0 +1,12 @@
|
|
1
|
+
LICENSE
|
2
|
+
README.md
|
3
|
+
pyproject.toml
|
4
|
+
src/__init__.py
|
5
|
+
src/cli.py
|
6
|
+
src/version.py
|
7
|
+
src/devstack_cli.egg-info/PKG-INFO
|
8
|
+
src/devstack_cli.egg-info/SOURCES.txt
|
9
|
+
src/devstack_cli.egg-info/dependency_links.txt
|
10
|
+
src/devstack_cli.egg-info/entry_points.txt
|
11
|
+
src/devstack_cli.egg-info/requires.txt
|
12
|
+
src/devstack_cli.egg-info/top_level.txt
|
@@ -0,0 +1 @@
|
|
1
|
+
|