batchfetch 1.2.7__tar.gz → 1.2.9__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {batchfetch-1.2.7 → batchfetch-1.2.9}/PKG-INFO +42 -6
- {batchfetch-1.2.7 → batchfetch-1.2.9}/README.md +41 -5
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch/batchfetch_base.py +3 -3
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch/batchfetch_cli.py +55 -38
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch/batchfetch_git.py +81 -106
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch.egg-info/PKG-INFO +42 -6
- {batchfetch-1.2.7 → batchfetch-1.2.9}/setup.py +1 -1
- {batchfetch-1.2.7 → batchfetch-1.2.9}/LICENSE +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch/__init__.py +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch/helpers.py +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch.egg-info/SOURCES.txt +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch.egg-info/dependency_links.txt +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch.egg-info/entry_points.txt +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch.egg-info/requires.txt +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/batchfetch.egg-info/top_level.txt +0 -0
- {batchfetch-1.2.7 → batchfetch-1.2.9}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: batchfetch
|
|
3
|
-
Version: 1.2.
|
|
3
|
+
Version: 1.2.9
|
|
4
4
|
Summary: Efficiently clone and pull multiple Git repositories.
|
|
5
5
|
Home-page: https://github.com/jamescherti/batchfetch
|
|
6
6
|
Author: James Cherti
|
|
@@ -73,22 +73,33 @@ Execute the `batchfetch` command from the same directory as `batchfetch.yml` to
|
|
|
73
73
|
Here are the various options that `batchfetch` provides, along with descriptions of their usage:
|
|
74
74
|
|
|
75
75
|
```
|
|
76
|
-
usage: batchfetch [--option]
|
|
76
|
+
usage: batchfetch [--option] [TARGET]
|
|
77
77
|
|
|
78
78
|
Efficiently clone/pull multiple Git repositories in parallel.
|
|
79
79
|
|
|
80
|
+
positional arguments:
|
|
81
|
+
target This is a target path that batchfetch is supposed to
|
|
82
|
+
handle. When no target is specified, execute the tasks
|
|
83
|
+
of all target paths defined in the batchfetch.yml list
|
|
84
|
+
of tasks.
|
|
85
|
+
|
|
80
86
|
options:
|
|
81
87
|
-h, --help show this help message and exit
|
|
82
88
|
-f FILE, --file FILE Specify the batchfetch YAML file (default:
|
|
83
89
|
'./batchfetch.yaml').
|
|
84
90
|
-C DIRECTORY, --directory DIRECTORY
|
|
85
91
|
Change the working directory before reading the
|
|
86
|
-
batchfetch.yaml file. If not specified, the directory
|
|
87
|
-
set to the parent directory of the batchfetch.yaml
|
|
92
|
+
batchfetch.yaml file. If not specified, the directory
|
|
93
|
+
is set to the parent directory of the batchfetch.yaml
|
|
94
|
+
file.
|
|
88
95
|
-j JOBS, --jobs JOBS Run up to N parallel processes (default: 5).
|
|
89
|
-
Alternatively, the BATCHFETCH_JOBS environment
|
|
90
|
-
can be used to configure the number of jobs.
|
|
96
|
+
Alternatively, the BATCHFETCH_JOBS environment
|
|
97
|
+
variable can be used to configure the number of jobs.
|
|
91
98
|
-v, --verbose Enable verbose mode.
|
|
99
|
+
-u, --check-untracked
|
|
100
|
+
Abort if untracked files or directories exist.
|
|
101
|
+
Alternatively, set the BATCHFETCH_CHECK_UNTRACKED=1
|
|
102
|
+
environment variable to enable this check.
|
|
92
103
|
```
|
|
93
104
|
|
|
94
105
|
## Features
|
|
@@ -160,6 +171,31 @@ tasks:
|
|
|
160
171
|
exec_after: ["sh", "-c", "echo exec_after_task"]
|
|
161
172
|
```
|
|
162
173
|
|
|
174
|
+
### How to make batchfetch handle only one path?
|
|
175
|
+
|
|
176
|
+
To configure `batchfetch` to handle a specific path, you can define your tasks in a `batchfetch.yml` file and pass the desired path as an argument to the `batchfetch` command.
|
|
177
|
+
|
|
178
|
+
#### Example `batchfetch.yml` file:
|
|
179
|
+
|
|
180
|
+
In the following example, the `easysession` task clones two Git repositories:
|
|
181
|
+
```yaml
|
|
182
|
+
---
|
|
183
|
+
tasks:
|
|
184
|
+
- git: https://github.com/jamescherti/easysession.el
|
|
185
|
+
path: easysession
|
|
186
|
+
|
|
187
|
+
- git: https://github.com/jamescherti/outline-indent.el
|
|
188
|
+
revision: "1.1.0"
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
To make `batchfetch` clone only `easysession`, pass its path as an argument:
|
|
192
|
+
|
|
193
|
+
```bash
|
|
194
|
+
batchfetch easysession
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
This will execute only the task corresponding to the `easysession` path, skipping all others in the `batchfetch.yml` file.
|
|
198
|
+
|
|
163
199
|
## License
|
|
164
200
|
|
|
165
201
|
Copyright (C) 2024 [James Cherti](https://www.jamescherti.com)
|
|
@@ -53,22 +53,33 @@ Execute the `batchfetch` command from the same directory as `batchfetch.yml` to
|
|
|
53
53
|
Here are the various options that `batchfetch` provides, along with descriptions of their usage:
|
|
54
54
|
|
|
55
55
|
```
|
|
56
|
-
usage: batchfetch [--option]
|
|
56
|
+
usage: batchfetch [--option] [TARGET]
|
|
57
57
|
|
|
58
58
|
Efficiently clone/pull multiple Git repositories in parallel.
|
|
59
59
|
|
|
60
|
+
positional arguments:
|
|
61
|
+
target This is a target path that batchfetch is supposed to
|
|
62
|
+
handle. When no target is specified, execute the tasks
|
|
63
|
+
of all target paths defined in the batchfetch.yml list
|
|
64
|
+
of tasks.
|
|
65
|
+
|
|
60
66
|
options:
|
|
61
67
|
-h, --help show this help message and exit
|
|
62
68
|
-f FILE, --file FILE Specify the batchfetch YAML file (default:
|
|
63
69
|
'./batchfetch.yaml').
|
|
64
70
|
-C DIRECTORY, --directory DIRECTORY
|
|
65
71
|
Change the working directory before reading the
|
|
66
|
-
batchfetch.yaml file. If not specified, the directory
|
|
67
|
-
set to the parent directory of the batchfetch.yaml
|
|
72
|
+
batchfetch.yaml file. If not specified, the directory
|
|
73
|
+
is set to the parent directory of the batchfetch.yaml
|
|
74
|
+
file.
|
|
68
75
|
-j JOBS, --jobs JOBS Run up to N parallel processes (default: 5).
|
|
69
|
-
Alternatively, the BATCHFETCH_JOBS environment
|
|
70
|
-
can be used to configure the number of jobs.
|
|
76
|
+
Alternatively, the BATCHFETCH_JOBS environment
|
|
77
|
+
variable can be used to configure the number of jobs.
|
|
71
78
|
-v, --verbose Enable verbose mode.
|
|
79
|
+
-u, --check-untracked
|
|
80
|
+
Abort if untracked files or directories exist.
|
|
81
|
+
Alternatively, set the BATCHFETCH_CHECK_UNTRACKED=1
|
|
82
|
+
environment variable to enable this check.
|
|
72
83
|
```
|
|
73
84
|
|
|
74
85
|
## Features
|
|
@@ -140,6 +151,31 @@ tasks:
|
|
|
140
151
|
exec_after: ["sh", "-c", "echo exec_after_task"]
|
|
141
152
|
```
|
|
142
153
|
|
|
154
|
+
### How to make batchfetch handle only one path?
|
|
155
|
+
|
|
156
|
+
To configure `batchfetch` to handle a specific path, you can define your tasks in a `batchfetch.yml` file and pass the desired path as an argument to the `batchfetch` command.
|
|
157
|
+
|
|
158
|
+
#### Example `batchfetch.yml` file:
|
|
159
|
+
|
|
160
|
+
In the following example, the `easysession` task clones two Git repositories:
|
|
161
|
+
```yaml
|
|
162
|
+
---
|
|
163
|
+
tasks:
|
|
164
|
+
- git: https://github.com/jamescherti/easysession.el
|
|
165
|
+
path: easysession
|
|
166
|
+
|
|
167
|
+
- git: https://github.com/jamescherti/outline-indent.el
|
|
168
|
+
revision: "1.1.0"
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
To make `batchfetch` clone only `easysession`, pass its path as an argument:
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
batchfetch easysession
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
This will execute only the task corresponding to the `easysession` path, skipping all others in the `batchfetch.yml` file.
|
|
178
|
+
|
|
143
179
|
## License
|
|
144
180
|
|
|
145
181
|
Copyright (C) 2024 [James Cherti](https://www.jamescherti.com)
|
|
@@ -138,7 +138,7 @@ class TaskBatchFetch(TaskBase):
|
|
|
138
138
|
# Mark values as initialized
|
|
139
139
|
self._values_initialized = True
|
|
140
140
|
|
|
141
|
-
def
|
|
141
|
+
def _pre_run(self, *args, **kwargs):
|
|
142
142
|
stdout = run_indent_str(*args, **kwargs)
|
|
143
143
|
|
|
144
144
|
if not stdout.endswith("\n"):
|
|
@@ -151,7 +151,7 @@ class TaskBatchFetch(TaskBase):
|
|
|
151
151
|
if self["delete"] or not cmd:
|
|
152
152
|
return
|
|
153
153
|
|
|
154
|
-
self.
|
|
154
|
+
self._pre_run(cmd, cwd=str(cwd), env=self.env, spaces=self.indent)
|
|
155
155
|
|
|
156
156
|
def _run_post_exec(self, cwd: os.PathLike = Path(".")):
|
|
157
157
|
self._initialize_data()
|
|
@@ -159,7 +159,7 @@ class TaskBatchFetch(TaskBase):
|
|
|
159
159
|
if not cmd or self["delete"] or not self.is_changed():
|
|
160
160
|
return
|
|
161
161
|
|
|
162
|
-
self.
|
|
162
|
+
self._pre_run(cmd, cwd=str(cwd), env=self.env, spaces=self.indent)
|
|
163
163
|
|
|
164
164
|
def is_changed(self) -> bool:
|
|
165
165
|
self._initialize_data()
|
|
@@ -25,7 +25,7 @@ import subprocess
|
|
|
25
25
|
import sys
|
|
26
26
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
|
27
27
|
from pathlib import Path
|
|
28
|
-
from typing import Any, Dict, Set, Union
|
|
28
|
+
from typing import Any, Dict, List, Set, Union
|
|
29
29
|
|
|
30
30
|
import colorama
|
|
31
31
|
import yaml # type: ignore
|
|
@@ -40,15 +40,16 @@ from .batchfetch_git import BatchFetchGit
|
|
|
40
40
|
class BatchFetchCli:
|
|
41
41
|
"""Command-line-interface that downloads."""
|
|
42
42
|
|
|
43
|
-
def __init__(self, max_workers: int, verbose: bool, check_untracked: bool
|
|
43
|
+
def __init__(self, max_workers: int, verbose: bool, check_untracked: bool,
|
|
44
|
+
targets: List[str]):
|
|
45
|
+
self._logger = logging.getLogger(self.__class__.__name__)
|
|
46
|
+
self.targets = {Path(item).absolute() for item in targets}
|
|
44
47
|
self.cfg: dict = {}
|
|
45
|
-
self.folder = Path(".")
|
|
46
48
|
self.check_untracked = check_untracked
|
|
47
49
|
self.tracked_paths: Dict[Path, Set[str]] = {}
|
|
48
50
|
self.ignore_untracked_paths: Set[Path] = set()
|
|
49
51
|
self.verbose = verbose
|
|
50
52
|
self.max_workers = max_workers
|
|
51
|
-
self._logger = logging.getLogger(self.__class__.__name__)
|
|
52
53
|
self.dirs_relative_to_batchfetch: Set[str] = set()
|
|
53
54
|
|
|
54
55
|
# Plugin
|
|
@@ -183,10 +184,31 @@ class BatchFetchCli:
|
|
|
183
184
|
self.dirs_relative_to_batchfetch = set()
|
|
184
185
|
|
|
185
186
|
all_tasks = self.cfg["tasks"]
|
|
187
|
+
|
|
188
|
+
# Check targets
|
|
189
|
+
if self.targets:
|
|
190
|
+
targets_executed = set()
|
|
191
|
+
for task in all_tasks:
|
|
192
|
+
full_path = Path(task["path"]).absolute()
|
|
193
|
+
if full_path not in self.targets:
|
|
194
|
+
continue
|
|
195
|
+
|
|
196
|
+
targets_executed.add(full_path)
|
|
197
|
+
|
|
198
|
+
targets_not_found = self.targets - targets_executed
|
|
199
|
+
if targets_not_found:
|
|
200
|
+
err_str = \
|
|
201
|
+
f"Error: Target(s) not found: "
|
|
202
|
+
err_str += ", ".join([str(item)
|
|
203
|
+
for item in targets_not_found])
|
|
204
|
+
print(f"Error: {err_str}", file=sys.stderr)
|
|
205
|
+
sys.exit(1)
|
|
206
|
+
|
|
186
207
|
for task in all_tasks:
|
|
208
|
+
full_path = Path(task["path"]).absolute()
|
|
209
|
+
|
|
187
210
|
self.dirs_relative_to_batchfetch.add(str(task["path"]))
|
|
188
211
|
if not task["delete"]:
|
|
189
|
-
full_path = Path(task["path"]).absolute()
|
|
190
212
|
base_path = full_path.parent
|
|
191
213
|
try:
|
|
192
214
|
self.tracked_paths[base_path]
|
|
@@ -195,6 +217,8 @@ class BatchFetchCli:
|
|
|
195
217
|
|
|
196
218
|
self.tracked_paths[base_path].add(full_path.name)
|
|
197
219
|
|
|
220
|
+
if self.targets and full_path not in self.targets:
|
|
221
|
+
continue
|
|
198
222
|
threads.append(executor_update.submit(task.update))
|
|
199
223
|
|
|
200
224
|
for future in as_completed(threads):
|
|
@@ -292,9 +316,16 @@ def parse_args():
|
|
|
292
316
|
check_untracked = bool(check_untracked)
|
|
293
317
|
|
|
294
318
|
desc = "Efficiently clone/pull multiple Git repositories in parallel."
|
|
295
|
-
usage = "%(prog)s [--option]"
|
|
319
|
+
usage = "%(prog)s [--option] [TARGET]"
|
|
296
320
|
parser = argparse.ArgumentParser(description=desc, usage=usage)
|
|
297
321
|
|
|
322
|
+
parser.add_argument(
|
|
323
|
+
"targets", metavar="target", type=str, nargs="*",
|
|
324
|
+
help=("This is a target path that batchfetch is supposed to handle. "
|
|
325
|
+
"When no target is specified, execute the tasks of all target "
|
|
326
|
+
"paths defined in the batchfetch.yml list of tasks."),
|
|
327
|
+
)
|
|
328
|
+
|
|
298
329
|
parser.add_argument("-f",
|
|
299
330
|
"--file",
|
|
300
331
|
default=None,
|
|
@@ -350,31 +381,6 @@ def parse_args():
|
|
|
350
381
|
return args
|
|
351
382
|
|
|
352
383
|
|
|
353
|
-
def run_batchfetch_procedure(file: Path,
|
|
354
|
-
directory: Union[None, Path],
|
|
355
|
-
verbose: bool,
|
|
356
|
-
jobs: int,
|
|
357
|
-
check_untracked: bool) -> int:
|
|
358
|
-
errno = 0
|
|
359
|
-
batchfetch_cli = BatchFetchCli(verbose=verbose,
|
|
360
|
-
max_workers=int(jobs),
|
|
361
|
-
check_untracked=check_untracked)
|
|
362
|
-
os.chdir(directory)
|
|
363
|
-
batchfetch_cli.load(file)
|
|
364
|
-
|
|
365
|
-
try:
|
|
366
|
-
if not batchfetch_cli.run_tasks():
|
|
367
|
-
errno = 1
|
|
368
|
-
except KeyboardInterrupt:
|
|
369
|
-
print("Interrupted.", file=sys.stderr)
|
|
370
|
-
errno = 1
|
|
371
|
-
except BatchFetchError as err:
|
|
372
|
-
print(f"Error: {err}.", file=sys.stderr)
|
|
373
|
-
errno = 1
|
|
374
|
-
|
|
375
|
-
return errno
|
|
376
|
-
|
|
377
|
-
|
|
378
384
|
def command_line_interface():
|
|
379
385
|
"""Command line interface."""
|
|
380
386
|
try:
|
|
@@ -388,7 +394,7 @@ def command_line_interface():
|
|
|
388
394
|
|
|
389
395
|
args = parse_args()
|
|
390
396
|
done = []
|
|
391
|
-
file = Path(args.file)
|
|
397
|
+
file = Path(args.file).absolute()
|
|
392
398
|
file_resolved = file.absolute()
|
|
393
399
|
if not file_resolved:
|
|
394
400
|
print(f"Error: cannot resolve the path {file}",
|
|
@@ -405,12 +411,23 @@ def command_line_interface():
|
|
|
405
411
|
print(f"[CHECK UNTRACKED] {args.check_untracked}")
|
|
406
412
|
print()
|
|
407
413
|
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
414
|
+
os.chdir(args.directory)
|
|
415
|
+
batchfetch_cli = BatchFetchCli(verbose=args.verbose,
|
|
416
|
+
max_workers=args.jobs,
|
|
417
|
+
check_untracked=args.check_untracked,
|
|
418
|
+
targets=args.targets)
|
|
419
|
+
batchfetch_cli.load(file)
|
|
420
|
+
|
|
421
|
+
try:
|
|
422
|
+
if not batchfetch_cli.run_tasks():
|
|
423
|
+
errno = 1
|
|
424
|
+
except KeyboardInterrupt:
|
|
425
|
+
print("Interrupted.", file=sys.stderr)
|
|
426
|
+
errno = 1
|
|
427
|
+
except BatchFetchError as err:
|
|
428
|
+
print(f"Error: {err}.", file=sys.stderr)
|
|
429
|
+
errno = 1
|
|
413
430
|
|
|
414
431
|
sys.exit(errno)
|
|
415
432
|
except BrokenPipeError:
|
|
416
|
-
|
|
433
|
+
sys.exit(1)
|
|
@@ -18,13 +18,13 @@
|
|
|
18
18
|
#
|
|
19
19
|
"Clone and update Git repositories."
|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
import os
|
|
22
22
|
import posixpath
|
|
23
23
|
import shutil
|
|
24
24
|
import subprocess
|
|
25
25
|
import textwrap
|
|
26
26
|
from pathlib import Path
|
|
27
|
-
from typing import List, Union
|
|
27
|
+
from typing import List, Tuple, Union
|
|
28
28
|
|
|
29
29
|
from schema import Optional
|
|
30
30
|
|
|
@@ -59,7 +59,6 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
59
59
|
# Local options
|
|
60
60
|
self.main_key: str,
|
|
61
61
|
Optional("revision"): str,
|
|
62
|
-
Optional("reference"): str,
|
|
63
62
|
|
|
64
63
|
# Same as global options
|
|
65
64
|
Optional("git_clone_args"): [str],
|
|
@@ -104,11 +103,6 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
104
103
|
"""Clone or update a Git repository."""
|
|
105
104
|
super().update()
|
|
106
105
|
|
|
107
|
-
# Backward compatibility
|
|
108
|
-
if "reference" in self._item_values \
|
|
109
|
-
and "revision" not in self._item_values:
|
|
110
|
-
self._item_values["revision"] = "reference"
|
|
111
|
-
|
|
112
106
|
is_clone = False
|
|
113
107
|
if not self.git_local_dir.exists():
|
|
114
108
|
is_clone = True
|
|
@@ -141,7 +135,18 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
141
135
|
self._repo_fix_remote_origin()
|
|
142
136
|
self._run_pre_exec(cwd=self.git_local_dir)
|
|
143
137
|
|
|
144
|
-
|
|
138
|
+
if not self["revision"]:
|
|
139
|
+
self.values["revision"] = self._run_get_firstline(
|
|
140
|
+
"git symbolic-ref refs/remotes/origin/HEAD"
|
|
141
|
+
).split("/")[-1]
|
|
142
|
+
if not self["revision"]:
|
|
143
|
+
raise BatchFetchError(
|
|
144
|
+
"Unable to determine the default origin branch"
|
|
145
|
+
)
|
|
146
|
+
|
|
147
|
+
self.add_output(self.indent_spaces +
|
|
148
|
+
f"[INFO] Update revision to: '" +
|
|
149
|
+
self["revision"] + "'\n")
|
|
145
150
|
|
|
146
151
|
git_fetch_done = self._repo_fetch()
|
|
147
152
|
|
|
@@ -152,9 +157,6 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
152
157
|
if git_fetch_done:
|
|
153
158
|
git_merge_done = self._git_merge()
|
|
154
159
|
|
|
155
|
-
# if (git_fetch_done and git_merge_done) or git_branch_changed:
|
|
156
|
-
# self._repo_update_submodules()
|
|
157
|
-
|
|
158
160
|
if self.get_changed():
|
|
159
161
|
self._run_post_exec(cwd=self.git_local_dir)
|
|
160
162
|
except BatchFetchError as err:
|
|
@@ -173,6 +175,31 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
173
175
|
|
|
174
176
|
return self.values
|
|
175
177
|
|
|
178
|
+
def _run_get_firstline(self, *args, **kwargs):
|
|
179
|
+
stdout, _ = self._run(*args, **kwargs)
|
|
180
|
+
try:
|
|
181
|
+
return stdout[0]
|
|
182
|
+
except IndexError:
|
|
183
|
+
return ""
|
|
184
|
+
|
|
185
|
+
def _run(self, cmd: Union[List[str], str],
|
|
186
|
+
cwd: Union[None, os.PathLike] = None,
|
|
187
|
+
env: Union[None, dict] = None,
|
|
188
|
+
**kwargs) -> Tuple[List[str], List[str]]:
|
|
189
|
+
"""
|
|
190
|
+
Executes a command and returns stdout and stderr as separate lists of
|
|
191
|
+
strings.
|
|
192
|
+
|
|
193
|
+
:param cmd: Command to be executed. Can be a list or a string.
|
|
194
|
+
:param kwargs: Additional keyword arguments for Popen.
|
|
195
|
+
:return: Tuple containing two lists: stdout lines and stderr lines.
|
|
196
|
+
"""
|
|
197
|
+
if not cwd:
|
|
198
|
+
cwd = self.git_local_dir
|
|
199
|
+
if not env:
|
|
200
|
+
env = self.env
|
|
201
|
+
return run_simple(cmd=cmd, env=env, cwd=cwd, **kwargs)
|
|
202
|
+
|
|
176
203
|
def _git_ref(self, cwd: Union[None, Path] = None) -> str:
|
|
177
204
|
"""Get the commit revision of HEAD.
|
|
178
205
|
|
|
@@ -180,7 +207,7 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
180
207
|
"""
|
|
181
208
|
cmd = ["git", "show-ref", "--head", "--verify", "HEAD"]
|
|
182
209
|
try:
|
|
183
|
-
stdout, _ =
|
|
210
|
+
stdout, _ = self._run(cmd, cwd=cwd)
|
|
184
211
|
output = stdout[0].split(" ")[0]
|
|
185
212
|
except IndexError:
|
|
186
213
|
return ""
|
|
@@ -189,11 +216,7 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
189
216
|
def _update_current_branch_name(self):
|
|
190
217
|
try:
|
|
191
218
|
# This returns the branch name
|
|
192
|
-
stdout, _ =
|
|
193
|
-
["git", "symbolic-ref", "--short", "HEAD"],
|
|
194
|
-
env=self.env,
|
|
195
|
-
cwd=self.git_local_dir,
|
|
196
|
-
)
|
|
219
|
+
stdout, _ = self._run(["git", "symbolic-ref", "--short", "HEAD"])
|
|
197
220
|
self.current_branch = stdout[0]
|
|
198
221
|
self.is_branch = True
|
|
199
222
|
except (IndexError, subprocess.CalledProcessError):
|
|
@@ -230,58 +253,46 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
230
253
|
|
|
231
254
|
def _repo_clone(self):
|
|
232
255
|
git_clone_args = self["git_clone_args"]
|
|
233
|
-
git_clone_args += ["--recurse-submodules"]
|
|
256
|
+
# git_clone_args += ["--recurse-submodules"]
|
|
234
257
|
|
|
235
258
|
cmd = ["git", "clone"] + git_clone_args + \
|
|
236
259
|
[self[self.main_key], str(self.git_local_dir)]
|
|
237
|
-
self._run(cmd,
|
|
260
|
+
self._run(cmd, cwd=".")
|
|
238
261
|
self.set_changed(True)
|
|
239
262
|
|
|
240
|
-
def _repo_reset(self):
|
|
241
|
-
# Remove local changes
|
|
242
|
-
cmd = ["git", "reset", "--hard", "HEAD"]
|
|
243
|
-
self._run(cmd, cwd=str(self.git_local_dir), env=self.env)
|
|
244
|
-
|
|
245
263
|
def _repo_fetch(self):
|
|
246
264
|
# Merge
|
|
247
265
|
do_git_fetch = self["git_pull"]
|
|
248
|
-
|
|
266
|
+
do_git_fetch = False
|
|
267
|
+
commit_ref = None
|
|
268
|
+
|
|
269
|
+
try:
|
|
270
|
+
# Check if the revision such as
|
|
271
|
+
stdout, _ = self._run(["git", "cat-file", "-e",
|
|
272
|
+
self["revision"]])
|
|
273
|
+
except subprocess.CalledProcessError:
|
|
249
274
|
do_git_fetch = True
|
|
250
|
-
self.add_output(
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
commit_ref = None
|
|
275
|
+
self.add_output(
|
|
276
|
+
self.indent_spaces
|
|
277
|
+
+ "[INFO] Git fetch origin reason: "
|
|
278
|
+
+ f"The revision does not exist: {self['revision']}"
|
|
279
|
+
+ "\n")
|
|
256
280
|
|
|
281
|
+
# The revision exists, but if it a branch, git pull anyway
|
|
282
|
+
if not do_git_fetch:
|
|
257
283
|
try:
|
|
258
|
-
# Check if the
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
except GitRevisionDoesNotExist:
|
|
263
|
-
do_git_fetch = True
|
|
284
|
+
# Check if the branch is a tag or a branch
|
|
285
|
+
cmd = ["git", "show-ref", "--verify", "--quiet",
|
|
286
|
+
f"refs/heads/{self['revision']}"]
|
|
287
|
+
self._run(cmd)
|
|
264
288
|
self.add_output(
|
|
265
289
|
self.indent_spaces
|
|
266
290
|
+ "[INFO] Git fetch origin reason: "
|
|
267
|
-
+ f"
|
|
291
|
+
+ f"{self['revision']} is a branch, not a tag"
|
|
268
292
|
+ "\n")
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
try:
|
|
273
|
-
# Check if the branch is a tag or a branch
|
|
274
|
-
cmd = ["git", "show-ref", "--verify", "--quiet",
|
|
275
|
-
f"refs/heads/{self['revision']}"]
|
|
276
|
-
run_simple(cmd, env=self.env, cwd=self.git_local_dir)
|
|
277
|
-
self.add_output(
|
|
278
|
-
self.indent_spaces
|
|
279
|
-
+ "[INFO] Git fetch origin reason: "
|
|
280
|
-
+ f"{self['revision']} is a branch, not a tag"
|
|
281
|
-
+ "\n")
|
|
282
|
-
do_git_fetch = True
|
|
283
|
-
except subprocess.CalledProcessError:
|
|
284
|
-
pass
|
|
293
|
+
do_git_fetch = True
|
|
294
|
+
except subprocess.CalledProcessError:
|
|
295
|
+
pass
|
|
285
296
|
|
|
286
297
|
if not do_git_fetch:
|
|
287
298
|
self.add_output(self.indent_spaces + "[INFO] git fetch ignored\n")
|
|
@@ -300,8 +311,7 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
300
311
|
# TODO: only merge when difference from upstream
|
|
301
312
|
commit_ref_head = self._git_ref(cwd=self.git_local_dir)
|
|
302
313
|
self._run(["git", "merge", "--ff-only",
|
|
303
|
-
f"origin/{self.current_branch}"]
|
|
304
|
-
cwd=str(self.git_local_dir), env=self.env)
|
|
314
|
+
f"origin/{self.current_branch}"])
|
|
305
315
|
git_ref_after_merge = self._git_ref(cwd=self.git_local_dir)
|
|
306
316
|
if commit_ref_head != git_ref_after_merge:
|
|
307
317
|
git_merge = True
|
|
@@ -309,19 +319,15 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
309
319
|
self._run(["git", "log",
|
|
310
320
|
'--pretty=format:"%h %ad %s [%cn]"',
|
|
311
321
|
"--decorate", "--date=short",
|
|
312
|
-
f"{commit_ref_head}..{git_ref_after_merge}"]
|
|
313
|
-
cwd=str(self.git_local_dir),
|
|
314
|
-
env=self.env)
|
|
322
|
+
f"{commit_ref_head}..{git_ref_after_merge}"])
|
|
315
323
|
|
|
316
324
|
return git_merge
|
|
317
325
|
|
|
318
326
|
def _git_get_remote_url(self, remote_name: str = "origin") -> str:
|
|
319
327
|
origin_url = ""
|
|
320
328
|
try:
|
|
321
|
-
stdout, _ =
|
|
322
|
-
|
|
323
|
-
env=self.env,
|
|
324
|
-
cwd=self.git_local_dir)
|
|
329
|
+
stdout, _ = self._run(["git", "config",
|
|
330
|
+
f"remote.{remote_name}.url"])
|
|
325
331
|
origin_url = stdout[0]
|
|
326
332
|
except (subprocess.CalledProcessError, IndexError) as err:
|
|
327
333
|
raise GitRemoteError(
|
|
@@ -333,18 +339,13 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
333
339
|
remote_name: str = "origin") -> str:
|
|
334
340
|
origin_url = ""
|
|
335
341
|
try:
|
|
336
|
-
|
|
337
|
-
env=self.env, cwd=self.git_local_dir)
|
|
342
|
+
self._run(["git", "remote", "remove", remote_name])
|
|
338
343
|
except subprocess.CalledProcessError:
|
|
339
344
|
# Ignore when it cannot be removed when it does not exist
|
|
340
345
|
pass
|
|
341
346
|
|
|
342
347
|
try:
|
|
343
|
-
stdout, _ =
|
|
344
|
-
["git", "remote", "add", remote_name, url],
|
|
345
|
-
env=self.env,
|
|
346
|
-
cwd=self.git_local_dir,
|
|
347
|
-
)
|
|
348
|
+
stdout, _ = self._run(["git", "remote", "add", remote_name, url])
|
|
348
349
|
origin_url = stdout[0]
|
|
349
350
|
except (subprocess.CalledProcessError, IndexError) as err:
|
|
350
351
|
raise GitRemoteError(
|
|
@@ -355,9 +356,8 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
355
356
|
def _git_is_local_branch(self, branch: str) -> bool:
|
|
356
357
|
"Return True if it is a local branch that exists."
|
|
357
358
|
try:
|
|
358
|
-
stdout, _ =
|
|
359
|
-
|
|
360
|
-
cwd=self.git_local_dir)
|
|
359
|
+
stdout, _ = self._run(["git", "rev-parse", "--symbolic-full-name",
|
|
360
|
+
branch])
|
|
361
361
|
if not stdout:
|
|
362
362
|
return False
|
|
363
363
|
|
|
@@ -374,9 +374,7 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
374
374
|
stdout: List[str] = []
|
|
375
375
|
error = False
|
|
376
376
|
try:
|
|
377
|
-
stdout, _ =
|
|
378
|
-
env=self.env,
|
|
379
|
-
cwd=self.git_local_dir)
|
|
377
|
+
stdout, _ = self._run(["git", "rev-parse", "--verify", revision])
|
|
380
378
|
if not stdout:
|
|
381
379
|
error = True
|
|
382
380
|
except subprocess.CalledProcessError:
|
|
@@ -394,11 +392,7 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
394
392
|
if self["revision"]:
|
|
395
393
|
# We also need tags because sometimes, a branch
|
|
396
394
|
# returns a different commit revision
|
|
397
|
-
git_tags, _ =
|
|
398
|
-
["git", "tag", "--points-at", "HEAD"],
|
|
399
|
-
env=self.env,
|
|
400
|
-
cwd=self.git_local_dir,
|
|
401
|
-
)
|
|
395
|
+
git_tags, _ = self._run(["git", "tag", "--points-at", "HEAD"])
|
|
402
396
|
|
|
403
397
|
# Also check the commit revision in case
|
|
404
398
|
# branch is a commit revision instead of a tag
|
|
@@ -410,8 +404,8 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
410
404
|
except GitRevisionDoesNotExist:
|
|
411
405
|
# Check if the commit ref exists
|
|
412
406
|
try:
|
|
413
|
-
git_ref_branch =
|
|
414
|
-
0]
|
|
407
|
+
git_ref_branch = \
|
|
408
|
+
self._git_rev_parse_verify(self["revision"])[0]
|
|
415
409
|
except GitRevisionDoesNotExist as err:
|
|
416
410
|
raise BatchFetchError(f"The branch '{self['revision']}' "
|
|
417
411
|
"does not exist.") from err
|
|
@@ -419,8 +413,7 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
419
413
|
if git_ref_after_merge != git_ref_branch and \
|
|
420
414
|
self["revision"] not in git_tags:
|
|
421
415
|
# Update the branch
|
|
422
|
-
self._run(["git", "checkout"] + [self["revision"]]
|
|
423
|
-
cwd=str(self.git_local_dir), env=self.env)
|
|
416
|
+
self._run(["git", "checkout"] + [self["revision"]])
|
|
424
417
|
self.add_output(self.indent_spaces
|
|
425
418
|
+ "[INFO] Branch changed to "
|
|
426
419
|
+ self["revision"] + "\n")
|
|
@@ -436,26 +429,9 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
436
429
|
# Fetch
|
|
437
430
|
if not self._git_fetch_origin_done:
|
|
438
431
|
cmd = ["git", "fetch", "origin"]
|
|
439
|
-
self._run(cmd
|
|
432
|
+
self._run(cmd)
|
|
440
433
|
self._git_fetch_origin_done = True
|
|
441
434
|
|
|
442
|
-
# def _repo_update_submodules(self):
|
|
443
|
-
# # This parameter instructs Git to initiate the update
|
|
444
|
-
# # process for submodules:
|
|
445
|
-
# # 1. Git fetches the commits specified in the parent
|
|
446
|
-
# # repository's configuration for each submodule.
|
|
447
|
-
# # 2. Updates are based solely on the commit pointers stored
|
|
448
|
-
# # within the parent repository's submodule configuration.
|
|
449
|
-
# # 3. It does not directly consult the upstream repositories
|
|
450
|
-
# # of the submodules.
|
|
451
|
-
# # 4. Submodules are updated to reflect the exact commits
|
|
452
|
-
# # revision in the parent repository's configuration,
|
|
453
|
-
# # potentially lagging behind the latest changes made in the
|
|
454
|
-
# # upstream repositories.
|
|
455
|
-
# if self.git_local_dir.joinpath(".gitmodules").is_file():
|
|
456
|
-
# cmd = ["git", "submodule", "update", "--recursive"]
|
|
457
|
-
# self._run(cmd, cwd=str(self.git_local_dir), env=self.env)
|
|
458
|
-
|
|
459
435
|
def _repo_fix_remote_origin(self):
|
|
460
436
|
correct_origin_url = self[self.main_key]
|
|
461
437
|
update_remote_origin = False
|
|
@@ -489,8 +465,7 @@ class BatchFetchGit(TaskBatchFetch):
|
|
|
489
465
|
|
|
490
466
|
cmd = ["git", "branch",
|
|
491
467
|
f"--set-upstream-to=origin/{self.current_branch}"]
|
|
492
|
-
_, _ =
|
|
493
|
-
cwd=self.git_local_dir)
|
|
468
|
+
_, _ = self._run(cmd)
|
|
494
469
|
# TODO: handle errors
|
|
495
470
|
except subprocess.CalledProcessError as err:
|
|
496
471
|
raise BatchFetchError(str(err)) from err
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: batchfetch
|
|
3
|
-
Version: 1.2.
|
|
3
|
+
Version: 1.2.9
|
|
4
4
|
Summary: Efficiently clone and pull multiple Git repositories.
|
|
5
5
|
Home-page: https://github.com/jamescherti/batchfetch
|
|
6
6
|
Author: James Cherti
|
|
@@ -73,22 +73,33 @@ Execute the `batchfetch` command from the same directory as `batchfetch.yml` to
|
|
|
73
73
|
Here are the various options that `batchfetch` provides, along with descriptions of their usage:
|
|
74
74
|
|
|
75
75
|
```
|
|
76
|
-
usage: batchfetch [--option]
|
|
76
|
+
usage: batchfetch [--option] [TARGET]
|
|
77
77
|
|
|
78
78
|
Efficiently clone/pull multiple Git repositories in parallel.
|
|
79
79
|
|
|
80
|
+
positional arguments:
|
|
81
|
+
target This is a target path that batchfetch is supposed to
|
|
82
|
+
handle. When no target is specified, execute the tasks
|
|
83
|
+
of all target paths defined in the batchfetch.yml list
|
|
84
|
+
of tasks.
|
|
85
|
+
|
|
80
86
|
options:
|
|
81
87
|
-h, --help show this help message and exit
|
|
82
88
|
-f FILE, --file FILE Specify the batchfetch YAML file (default:
|
|
83
89
|
'./batchfetch.yaml').
|
|
84
90
|
-C DIRECTORY, --directory DIRECTORY
|
|
85
91
|
Change the working directory before reading the
|
|
86
|
-
batchfetch.yaml file. If not specified, the directory
|
|
87
|
-
set to the parent directory of the batchfetch.yaml
|
|
92
|
+
batchfetch.yaml file. If not specified, the directory
|
|
93
|
+
is set to the parent directory of the batchfetch.yaml
|
|
94
|
+
file.
|
|
88
95
|
-j JOBS, --jobs JOBS Run up to N parallel processes (default: 5).
|
|
89
|
-
Alternatively, the BATCHFETCH_JOBS environment
|
|
90
|
-
can be used to configure the number of jobs.
|
|
96
|
+
Alternatively, the BATCHFETCH_JOBS environment
|
|
97
|
+
variable can be used to configure the number of jobs.
|
|
91
98
|
-v, --verbose Enable verbose mode.
|
|
99
|
+
-u, --check-untracked
|
|
100
|
+
Abort if untracked files or directories exist.
|
|
101
|
+
Alternatively, set the BATCHFETCH_CHECK_UNTRACKED=1
|
|
102
|
+
environment variable to enable this check.
|
|
92
103
|
```
|
|
93
104
|
|
|
94
105
|
## Features
|
|
@@ -160,6 +171,31 @@ tasks:
|
|
|
160
171
|
exec_after: ["sh", "-c", "echo exec_after_task"]
|
|
161
172
|
```
|
|
162
173
|
|
|
174
|
+
### How to make batchfetch handle only one path?
|
|
175
|
+
|
|
176
|
+
To configure `batchfetch` to handle a specific path, you can define your tasks in a `batchfetch.yml` file and pass the desired path as an argument to the `batchfetch` command.
|
|
177
|
+
|
|
178
|
+
#### Example `batchfetch.yml` file:
|
|
179
|
+
|
|
180
|
+
In the following example, the `easysession` task clones two Git repositories:
|
|
181
|
+
```yaml
|
|
182
|
+
---
|
|
183
|
+
tasks:
|
|
184
|
+
- git: https://github.com/jamescherti/easysession.el
|
|
185
|
+
path: easysession
|
|
186
|
+
|
|
187
|
+
- git: https://github.com/jamescherti/outline-indent.el
|
|
188
|
+
revision: "1.1.0"
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
To make `batchfetch` clone only `easysession`, pass its path as an argument:
|
|
192
|
+
|
|
193
|
+
```bash
|
|
194
|
+
batchfetch easysession
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
This will execute only the task corresponding to the `easysession` path, skipping all others in the `batchfetch.yml` file.
|
|
198
|
+
|
|
163
199
|
## License
|
|
164
200
|
|
|
165
201
|
Copyright (C) 2024 [James Cherti](https://www.jamescherti.com)
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|