nbdevAuto 0.3.66__tar.gz → 0.3.68__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {nbdevauto-0.3.66/nbdevAuto.egg-info → nbdevauto-0.3.68}/PKG-INFO +5 -3
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/README.md +4 -2
- nbdevauto-0.3.68/nbdevAuto/__init__.py +1 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto/_modidx.py +19 -1
- nbdevauto-0.3.68/nbdevAuto/fleet.py +442 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68/nbdevAuto.egg-info}/PKG-INFO +5 -3
- nbdevauto-0.3.66/nbdevAuto/__init__.py +0 -1
- nbdevauto-0.3.66/nbdevAuto/fleet.py +0 -201
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/LICENSE +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/MANIFEST.in +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto/automate.py +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto/core.py +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto/functions.py +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto/github.py +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto/pdf.py +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto.egg-info/SOURCES.txt +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto.egg-info/dependency_links.txt +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto.egg-info/entry_points.txt +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto.egg-info/requires.txt +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/nbdevAuto.egg-info/top_level.txt +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/pyproject.toml +0 -0
- {nbdevauto-0.3.66 → nbdevauto-0.3.68}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: nbdevAuto
|
|
3
|
-
Version: 0.3.
|
|
3
|
+
Version: 0.3.68
|
|
4
4
|
Summary: automating nbdev
|
|
5
5
|
Author-email: Benedict Thekkel <bthekkel1@gmail.com>
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -73,23 +73,25 @@ from nbdevAuto.pdf import PDFreader
|
|
|
73
73
|
| [Automate](01_Automate.ipynb) | The command line surface: the prep/commit/push pipeline, the GitHub and PyPI release halves, and the help output that lists them all |
|
|
74
74
|
| [PDF Reader](02_PDF_reader.ipynb) | [`PDFreader`](https://bthek1.github.io/nbdevAuto/pdf_reader.html#pdfreader), a class that converts a PDF through `pdf2image` and renders page ranges inline when you slice it (`pdf[0:5]`) |
|
|
75
75
|
| [GitHub](03_GitHub.ipynb) | [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus): reading `.gitmodules`, resolving a token, and asking the GitHub API about every submodule at once - CI runs, Pages deploys, Pages builds, remote `main` shas - plus the `--report` and `--audit` views and the exit codes that keep “absent” apart from “cannot ask” |
|
|
76
|
+
| [Fleet](04_Fleet.ipynb) | [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet): running one command across every submodule of a superproject. A walker that reports only what did work and counts the rest, plus the verbs built on it - upload, push, copy, status, update and sync |
|
|
76
77
|
|
|
77
78
|
------------------------------------------------------------------------
|
|
78
79
|
|
|
79
80
|
## The Command Line Surface
|
|
80
81
|
|
|
81
|
-
Installing the package puts these on your `PATH`. Most are ordinary functions in [Automate](01_Automate.ipynb), exposed as scripts through `[project.scripts]`; [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) comes from [GitHub](03_GitHub.ipynb).
|
|
82
|
+
Installing the package puts these on your `PATH`. Most are ordinary functions in [Automate](01_Automate.ipynb), exposed as scripts through `[project.scripts]`; [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) comes from [GitHub](03_GitHub.ipynb) and [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet) from [Fleet](04_Fleet.ipynb).
|
|
82
83
|
|
|
83
84
|
| Command | Does |
|
|
84
85
|
|----|----|
|
|
85
86
|
| [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) | Bump the version, then export, test and clean the notebooks, refreshing `_quarto.yml` and the README |
|
|
86
|
-
| [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp) | git add, commit, and push. Without `-m` the
|
|
87
|
+
| [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp) | git add, commit, and push. Without `-m` the message is built from the staged paths |
|
|
87
88
|
| [`status`](https://bthek1.github.io/nbdevAuto/automate.html#status) | Show the working tree state |
|
|
88
89
|
| [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) | [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) then [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp), the one you actually type |
|
|
89
90
|
| `gitrelease` | Bump the **minor** version, then tag and create a GitHub release |
|
|
90
91
|
| `piprelease` | Build the sdist and wheel, then upload to PyPI with twine |
|
|
91
92
|
| [`release`](https://bthek1.github.io/nbdevAuto/automate.html#release) | `gitrelease` then `piprelease` |
|
|
92
93
|
| [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) | Remote status for every submodule of an nbdev superproject, from the GitHub API. `--report` for the sync view, `--audit` to check `.gitmodules` against the repos the account owns, `--json` for machine output |
|
|
94
|
+
| [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet) | Run one command across every submodule: `fleet upload|push|copy|status|update|sync`. Every verb takes `--help`, and the destructive ones take `--dry-run` |
|
|
93
95
|
| `h` | Print the list of available commands |
|
|
94
96
|
|
|
95
97
|
Both [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) and [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) take `-p` to choose which version part to increment; it defaults to 2, the patch, so every [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) ships a new version.
|
|
@@ -43,23 +43,25 @@ from nbdevAuto.pdf import PDFreader
|
|
|
43
43
|
| [Automate](01_Automate.ipynb) | The command line surface: the prep/commit/push pipeline, the GitHub and PyPI release halves, and the help output that lists them all |
|
|
44
44
|
| [PDF Reader](02_PDF_reader.ipynb) | [`PDFreader`](https://bthek1.github.io/nbdevAuto/pdf_reader.html#pdfreader), a class that converts a PDF through `pdf2image` and renders page ranges inline when you slice it (`pdf[0:5]`) |
|
|
45
45
|
| [GitHub](03_GitHub.ipynb) | [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus): reading `.gitmodules`, resolving a token, and asking the GitHub API about every submodule at once - CI runs, Pages deploys, Pages builds, remote `main` shas - plus the `--report` and `--audit` views and the exit codes that keep “absent” apart from “cannot ask” |
|
|
46
|
+
| [Fleet](04_Fleet.ipynb) | [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet): running one command across every submodule of a superproject. A walker that reports only what did work and counts the rest, plus the verbs built on it - upload, push, copy, status, update and sync |
|
|
46
47
|
|
|
47
48
|
------------------------------------------------------------------------
|
|
48
49
|
|
|
49
50
|
## The Command Line Surface
|
|
50
51
|
|
|
51
|
-
Installing the package puts these on your `PATH`. Most are ordinary functions in [Automate](01_Automate.ipynb), exposed as scripts through `[project.scripts]`; [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) comes from [GitHub](03_GitHub.ipynb).
|
|
52
|
+
Installing the package puts these on your `PATH`. Most are ordinary functions in [Automate](01_Automate.ipynb), exposed as scripts through `[project.scripts]`; [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) comes from [GitHub](03_GitHub.ipynb) and [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet) from [Fleet](04_Fleet.ipynb).
|
|
52
53
|
|
|
53
54
|
| Command | Does |
|
|
54
55
|
|----|----|
|
|
55
56
|
| [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) | Bump the version, then export, test and clean the notebooks, refreshing `_quarto.yml` and the README |
|
|
56
|
-
| [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp) | git add, commit, and push. Without `-m` the
|
|
57
|
+
| [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp) | git add, commit, and push. Without `-m` the message is built from the staged paths |
|
|
57
58
|
| [`status`](https://bthek1.github.io/nbdevAuto/automate.html#status) | Show the working tree state |
|
|
58
59
|
| [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) | [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) then [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp), the one you actually type |
|
|
59
60
|
| `gitrelease` | Bump the **minor** version, then tag and create a GitHub release |
|
|
60
61
|
| `piprelease` | Build the sdist and wheel, then upload to PyPI with twine |
|
|
61
62
|
| [`release`](https://bthek1.github.io/nbdevAuto/automate.html#release) | `gitrelease` then `piprelease` |
|
|
62
63
|
| [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) | Remote status for every submodule of an nbdev superproject, from the GitHub API. `--report` for the sync view, `--audit` to check `.gitmodules` against the repos the account owns, `--json` for machine output |
|
|
64
|
+
| [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet) | Run one command across every submodule: `fleet upload|push|copy|status|update|sync`. Every verb takes `--help`, and the destructive ones take `--dry-run` |
|
|
63
65
|
| `h` | Print the list of available commands |
|
|
64
66
|
|
|
65
67
|
Both [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) and [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) take `-p` to choose which version part to increment; it defaults to 2, the patch, so every [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) ships a new version.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "0.3.68"
|
|
@@ -36,16 +36,34 @@ d = { 'settings': { 'branch': 'main',
|
|
|
36
36
|
'nbdevAuto.fleet.Report.failed': ('fleet.html#report.failed', 'nbdevAuto/fleet.py'),
|
|
37
37
|
'nbdevAuto.fleet.Report.ran': ('fleet.html#report.ran', 'nbdevAuto/fleet.py'),
|
|
38
38
|
'nbdevAuto.fleet.Report.skip_counts': ('fleet.html#report.skip_counts', 'nbdevAuto/fleet.py'),
|
|
39
|
+
'nbdevAuto.fleet.Skip': ('fleet.html#skip', 'nbdevAuto/fleet.py'),
|
|
39
40
|
'nbdevAuto.fleet._classify': ('fleet.html#_classify', 'nbdevAuto/fleet.py'),
|
|
40
|
-
'nbdevAuto.fleet.
|
|
41
|
+
'nbdevAuto.fleet._count': ('fleet.html#_count', 'nbdevAuto/fleet.py'),
|
|
42
|
+
'nbdevAuto.fleet._fail_parts': ('fleet.html#_fail_parts', 'nbdevAuto/fleet.py'),
|
|
43
|
+
'nbdevAuto.fleet._git_ok': ('fleet.html#_git_ok', 'nbdevAuto/fleet.py'),
|
|
44
|
+
'nbdevAuto.fleet._porcelain': ('fleet.html#_porcelain', 'nbdevAuto/fleet.py'),
|
|
41
45
|
'nbdevAuto.fleet._render': ('fleet.html#_render', 'nbdevAuto/fleet.py'),
|
|
46
|
+
'nbdevAuto.fleet.ahead_of_remote': ('fleet.html#ahead_of_remote', 'nbdevAuto/fleet.py'),
|
|
47
|
+
'nbdevAuto.fleet.copy_one': ('fleet.html#copy_one', 'nbdevAuto/fleet.py'),
|
|
48
|
+
'nbdevAuto.fleet.dirty_submodules': ('fleet.html#dirty_submodules', 'nbdevAuto/fleet.py'),
|
|
49
|
+
'nbdevAuto.fleet.fetch_all': ('fleet.html#fetch_all', 'nbdevAuto/fleet.py'),
|
|
42
50
|
'nbdevAuto.fleet.finish': ('fleet.html#finish', 'nbdevAuto/fleet.py'),
|
|
43
51
|
'nbdevAuto.fleet.fleet': ('fleet.html#fleet', 'nbdevAuto/fleet.py'),
|
|
52
|
+
'nbdevAuto.fleet.fleet_copy': ('fleet.html#fleet_copy', 'nbdevAuto/fleet.py'),
|
|
53
|
+
'nbdevAuto.fleet.fleet_push': ('fleet.html#fleet_push', 'nbdevAuto/fleet.py'),
|
|
54
|
+
'nbdevAuto.fleet.fleet_status': ('fleet.html#fleet_status', 'nbdevAuto/fleet.py'),
|
|
55
|
+
'nbdevAuto.fleet.fleet_sync': ('fleet.html#fleet_sync', 'nbdevAuto/fleet.py'),
|
|
56
|
+
'nbdevAuto.fleet.fleet_update': ('fleet.html#fleet_update', 'nbdevAuto/fleet.py'),
|
|
44
57
|
'nbdevAuto.fleet.fleet_upload': ('fleet.html#fleet_upload', 'nbdevAuto/fleet.py'),
|
|
45
58
|
'nbdevAuto.fleet.format_summary': ('fleet.html#format_summary', 'nbdevAuto/fleet.py'),
|
|
59
|
+
'nbdevAuto.fleet.init_new': ('fleet.html#init_new', 'nbdevAuto/fleet.py'),
|
|
46
60
|
'nbdevAuto.fleet.parse_verb': ('fleet.html#parse_verb', 'nbdevAuto/fleet.py'),
|
|
61
|
+
'nbdevAuto.fleet.pull_root': ('fleet.html#pull_root', 'nbdevAuto/fleet.py'),
|
|
62
|
+
'nbdevAuto.fleet.push_one': ('fleet.html#push_one', 'nbdevAuto/fleet.py'),
|
|
47
63
|
'nbdevAuto.fleet.run_verb': ('fleet.html#run_verb', 'nbdevAuto/fleet.py'),
|
|
48
64
|
'nbdevAuto.fleet.script_path': ('fleet.html#script_path', 'nbdevAuto/fleet.py'),
|
|
65
|
+
'nbdevAuto.fleet.status_one': ('fleet.html#status_one', 'nbdevAuto/fleet.py'),
|
|
66
|
+
'nbdevAuto.fleet.update_one': ('fleet.html#update_one', 'nbdevAuto/fleet.py'),
|
|
49
67
|
'nbdevAuto.fleet.upload_one': ('fleet.html#upload_one', 'nbdevAuto/fleet.py'),
|
|
50
68
|
'nbdevAuto.fleet.usage': ('fleet.html#usage', 'nbdevAuto/fleet.py'),
|
|
51
69
|
'nbdevAuto.fleet.walk': ('fleet.html#walk', 'nbdevAuto/fleet.py')},
|
|
@@ -0,0 +1,442 @@
|
|
|
1
|
+
"""Run one command across every submodule of a superproject
|
|
2
|
+
|
|
3
|
+
Docs: https://bthek1.github.io/nbdevAuto/fleet.html.md"""
|
|
4
|
+
|
|
5
|
+
# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/04_Fleet.ipynb.
|
|
6
|
+
|
|
7
|
+
# %% auto #0
|
|
8
|
+
__all__ = ['VERBS', 'Skip', 'Outcome', 'Report', 'format_summary', 'walk', 'finish', 'script_path', 'upload_one', 'fleet_upload',
|
|
9
|
+
'ahead_of_remote', 'push_one', 'copy_one', 'fleet_push', 'fleet_copy', 'status_one', 'fleet_status',
|
|
10
|
+
'fetch_all', 'update_one', 'dirty_submodules', 'init_new', 'pull_root', 'fleet_update', 'fleet_sync',
|
|
11
|
+
'parse_verb', 'usage', 'run_verb', 'fleet']
|
|
12
|
+
|
|
13
|
+
# %% ../nbs/04_Fleet.ipynb #d361ee5f-9f09-464f-aec6-0e27cec90fe9
|
|
14
|
+
import re
|
|
15
|
+
import shutil
|
|
16
|
+
import subprocess
|
|
17
|
+
import sys
|
|
18
|
+
from concurrent.futures import ThreadPoolExecutor
|
|
19
|
+
from dataclasses import dataclass, field
|
|
20
|
+
from pathlib import Path
|
|
21
|
+
|
|
22
|
+
from fastcore.script import *
|
|
23
|
+
|
|
24
|
+
# The rendering layer lives in `automate`, the git/discovery layer in `github`. This
|
|
25
|
+
# module is the only one that knows about MANY repos; those two stay single-repo.
|
|
26
|
+
from .automate import StepFailed, _push_summary, _run, _step, console, escape
|
|
27
|
+
from .github import find_root, git, local_state, parse_gitmodules
|
|
28
|
+
|
|
29
|
+
# %% ../nbs/04_Fleet.ipynb #4e6b24c5-cd71-434c-84e6-e6994d5959c7
|
|
30
|
+
class Skip(Exception):
|
|
31
|
+
"Raised by a walked callable that has decided this submodule has nothing to do"
|
|
32
|
+
|
|
33
|
+
@dataclass
|
|
34
|
+
class Outcome:
|
|
35
|
+
"What happened in one submodule"
|
|
36
|
+
path: str
|
|
37
|
+
state: str # ran | skipped | failed
|
|
38
|
+
reason: str = "" # why it was skipped, or how it failed
|
|
39
|
+
detail: list = field(default_factory=list) # short lines to show under the path
|
|
40
|
+
|
|
41
|
+
@dataclass
|
|
42
|
+
class Report:
|
|
43
|
+
"Everything a `walk` did, and the exit code it implies"
|
|
44
|
+
outcomes: list = field(default_factory=list)
|
|
45
|
+
|
|
46
|
+
@property
|
|
47
|
+
def ran(self): return [o for o in self.outcomes if o.state == "ran"]
|
|
48
|
+
@property
|
|
49
|
+
def failed(self): return [o for o in self.outcomes if o.state == "failed"]
|
|
50
|
+
@property
|
|
51
|
+
def code(self): return 1 if self.failed else 0
|
|
52
|
+
|
|
53
|
+
def skip_counts(self):
|
|
54
|
+
"Skip reason -> count, in the order the reasons were first seen"
|
|
55
|
+
counts = {}
|
|
56
|
+
for o in self.outcomes:
|
|
57
|
+
if o.state == "skipped": counts[o.reason] = counts.get(o.reason, 0) + 1
|
|
58
|
+
return counts
|
|
59
|
+
|
|
60
|
+
def format_summary(rep, verb="ran"):
|
|
61
|
+
"The one line that replaces a message per submodule. Quiet submodules are counted, not narrated"
|
|
62
|
+
n = len(rep.outcomes)
|
|
63
|
+
parts = [f"{len(rep.ran)} {verb}"]
|
|
64
|
+
parts += [f"{c} {reason}" for reason, c in rep.skip_counts().items()]
|
|
65
|
+
if rep.failed: parts.append(f"{len(rep.failed)} FAILED")
|
|
66
|
+
return f"{n} submodule{'' if n == 1 else 's'}: " + ", ".join(parts)
|
|
67
|
+
|
|
68
|
+
# %% ../nbs/04_Fleet.ipynb #099242e8-c9ad-4b48-8ebc-7a818e5f8f81
|
|
69
|
+
def _classify(base, sub, need_nbs, only_dirty):
|
|
70
|
+
"The reason to skip this submodule before running anything, or None to run it"
|
|
71
|
+
p = base/sub.path
|
|
72
|
+
# An uninitialised submodule is an empty directory: present, but with no .git
|
|
73
|
+
if not (p/".git").exists(): return "not initialised"
|
|
74
|
+
if need_nbs and not (p/"nbs").is_dir(): return "without nbs"
|
|
75
|
+
if only_dirty and not git(p, "status", "--porcelain"): return "clean"
|
|
76
|
+
return None
|
|
77
|
+
|
|
78
|
+
def _fail_parts(e):
|
|
79
|
+
"""(headline, remaining lines) for a callable that did not finish.
|
|
80
|
+
|
|
81
|
+
A `StepFailed` carries the command's own output, and its first line is the message
|
|
82
|
+
worth putting on the summary line - "uncommitted changes" says far more than
|
|
83
|
+
"update exited 1". The rest is kept for the block renderer."""
|
|
84
|
+
if isinstance(e, StepFailed):
|
|
85
|
+
out = (e.output or "").strip().splitlines()
|
|
86
|
+
if out: return out[0], out[1:]
|
|
87
|
+
return f"{e.cmd} exited {e.code}", []
|
|
88
|
+
return f"{type(e).__name__}: {e}", []
|
|
89
|
+
|
|
90
|
+
def _render(o, style="block", width=0, header=True):
|
|
91
|
+
"One submodule's result. A skip prints nothing at all; it is counted instead"
|
|
92
|
+
if o.state == "skipped": return
|
|
93
|
+
if style == "line":
|
|
94
|
+
pad = f"{o.path:<{width}}"
|
|
95
|
+
if o.state == "failed":
|
|
96
|
+
console.print(f"[bold red]{escape(pad)}[/bold red] [red]{escape(o.reason)}[/red]")
|
|
97
|
+
return
|
|
98
|
+
console.print(f"[cyan]{escape(pad)}[/cyan] [dim]{escape(o.detail[0] if o.detail else '')}[/dim]")
|
|
99
|
+
for line in o.detail[1:]: console.print(f"{' ' * width} [dim]{escape(line)}[/dim]")
|
|
100
|
+
return
|
|
101
|
+
if header: console.print(f"\n[bold cyan]==> {escape(o.path)}[/bold cyan]")
|
|
102
|
+
if o.state == "failed":
|
|
103
|
+
console.print(f" [bold red]FAIL[/bold red] [red]{escape(o.reason)}[/red]")
|
|
104
|
+
for line in o.detail: console.print(f" [dim]{escape(line)}[/dim]")
|
|
105
|
+
|
|
106
|
+
def walk(fn, *, root=None, only_dirty=False, need_nbs=False, workers=1, style="block"):
|
|
107
|
+
"""Run `fn(path, sub)` in each submodule, returning a `Report`.
|
|
108
|
+
|
|
109
|
+
`fn` returns a list of short strings to show under the submodule's name, or raises
|
|
110
|
+
`Skip` once it knows there is nothing to do. Only submodules that do something print;
|
|
111
|
+
the rest are counted by `format_summary`.
|
|
112
|
+
|
|
113
|
+
`style="block"` gives each submodule a `==> path` heading with its own output beneath,
|
|
114
|
+
which is what a long command like `upload` needs. `style="line"` gives one padded line
|
|
115
|
+
per submodule, which suits verbs that report a single fact.
|
|
116
|
+
|
|
117
|
+
Sequential block runs print the heading BEFORE calling `fn`, so its output streams
|
|
118
|
+
underneath. Everything else prints after, in submodule order, so parallel work cannot
|
|
119
|
+
interleave and two runs of the same fleet read the same way."""
|
|
120
|
+
base = Path(root) if root else find_root()
|
|
121
|
+
rep, todo = Report(), []
|
|
122
|
+
for s in parse_gitmodules(base):
|
|
123
|
+
why = _classify(base, s, need_nbs, only_dirty)
|
|
124
|
+
if why: rep.outcomes.append(Outcome(s.path, "skipped", why))
|
|
125
|
+
else: todo.append(s)
|
|
126
|
+
width = max((len(s.path) for s in todo), default=0) + 2
|
|
127
|
+
|
|
128
|
+
def _one(s):
|
|
129
|
+
try: return Outcome(s.path, "ran", detail=list(fn(base/s.path, s) or []))
|
|
130
|
+
except Skip as e: return Outcome(s.path, "skipped", str(e))
|
|
131
|
+
except BaseException as e:
|
|
132
|
+
head, rest = _fail_parts(e)
|
|
133
|
+
return Outcome(s.path, "failed", head, rest)
|
|
134
|
+
|
|
135
|
+
if workers > 1 and len(todo) > 1:
|
|
136
|
+
with ThreadPoolExecutor(max_workers=workers) as ex: done = list(ex.map(_one, todo))
|
|
137
|
+
for o in done: _render(o, style, width)
|
|
138
|
+
else:
|
|
139
|
+
done = []
|
|
140
|
+
live = style == "block"
|
|
141
|
+
for s in todo:
|
|
142
|
+
if live: console.print(f"\n[bold cyan]==> {escape(s.path)}[/bold cyan]")
|
|
143
|
+
o = _one(s)
|
|
144
|
+
done.append(o)
|
|
145
|
+
_render(o, style, width, header=not live)
|
|
146
|
+
rep.outcomes += done
|
|
147
|
+
rep.outcomes.sort(key=lambda o: o.path)
|
|
148
|
+
return rep
|
|
149
|
+
|
|
150
|
+
def finish(rep, verb):
|
|
151
|
+
"Print the counted summary, name any failures, and return the exit code"
|
|
152
|
+
console.print(f"\n[bold]{escape(format_summary(rep, verb))}[/bold]")
|
|
153
|
+
for o in rep.failed:
|
|
154
|
+
console.print(f"[bold red]FAILED[/bold red] {escape(o.path)}: [dim]{escape(o.reason)}[/dim]")
|
|
155
|
+
return rep.code
|
|
156
|
+
|
|
157
|
+
# %% ../nbs/04_Fleet.ipynb #99e2c4fe-c885-4880-b9df-31da51e61f3a
|
|
158
|
+
def script_path(name):
|
|
159
|
+
"""The console script `name` from the environment this is running in.
|
|
160
|
+
|
|
161
|
+
Taking it from `sys.executable`'s directory is what makes this work under a
|
|
162
|
+
non-interactive shell that never put the venv on PATH."""
|
|
163
|
+
cand = Path(sys.executable).parent/name
|
|
164
|
+
if cand.is_file(): return str(cand)
|
|
165
|
+
found = shutil.which(name)
|
|
166
|
+
if found: return found
|
|
167
|
+
raise RuntimeError(f"Cannot find the '{name}' console script. Run `uv sync` first")
|
|
168
|
+
|
|
169
|
+
def upload_one(path, sub=None, args=()):
|
|
170
|
+
"""Run the single-repo `upload` in `path`, letting its own rich output through.
|
|
171
|
+
|
|
172
|
+
A subprocess rather than an in-process call: `prep` reads nbdev's config from the cwd
|
|
173
|
+
and caches it, and `nbdev_test` spawns its own process pool, so nineteen repos in one
|
|
174
|
+
interpreter is a trap. The subprocess keeps this tty, so colour survives."""
|
|
175
|
+
r = subprocess.run([script_path("upload"), *args], cwd=str(path), check=False)
|
|
176
|
+
if r.returncode: raise StepFailed("upload", r.returncode)
|
|
177
|
+
return []
|
|
178
|
+
|
|
179
|
+
# %% ../nbs/04_Fleet.ipynb #2ef9f1e5-239a-4790-8cec-43fffb8bc422
|
|
180
|
+
def fleet_upload(
|
|
181
|
+
all:bool = False, # Upload every submodule with an nbs/ folder, not just the changed ones
|
|
182
|
+
root:str = None, # Superproject root (default: nearest ancestor with .gitmodules)
|
|
183
|
+
):
|
|
184
|
+
"Run `upload` in each submodule that has local changes"
|
|
185
|
+
# Sequential on purpose: `upload` runs nbdev_test with 8 workers, and knowledge-lab
|
|
186
|
+
# has 4 vCPU. Parallel uploads would oversubscribe the box and interleave the output.
|
|
187
|
+
rep = walk(upload_one, root=root, need_nbs=True, only_dirty=not all, workers=1)
|
|
188
|
+
sys.exit(finish(rep, "uploaded"))
|
|
189
|
+
|
|
190
|
+
# %% ../nbs/04_Fleet.ipynb #cee8b478-470e-46fb-9d4a-e77bf6718c23
|
|
191
|
+
def ahead_of_remote(path):
|
|
192
|
+
"(commits origin has not seen, branch name). Branch is None when HEAD is detached"
|
|
193
|
+
br = git(path, "rev-parse", "--abbrev-ref", "HEAD")
|
|
194
|
+
if not br or br == "HEAD": return 0, None
|
|
195
|
+
n = git(path, "rev-list", "--count", f"origin/{br}..HEAD")
|
|
196
|
+
return (int(n) if n.isdigit() else 0), br
|
|
197
|
+
|
|
198
|
+
def push_one(path, sub=None, dry_run=False):
|
|
199
|
+
"Push `path` when it has local commits origin has not seen"
|
|
200
|
+
n, br = ahead_of_remote(path)
|
|
201
|
+
if br is None: raise Skip("detached HEAD")
|
|
202
|
+
if not git(path, "remote"): raise Skip("no remote")
|
|
203
|
+
if not n: raise Skip("nothing to push")
|
|
204
|
+
if dry_run: return [f"would push {n} commit{'' if n == 1 else 's'} on {br}"]
|
|
205
|
+
return _push_summary(_run("git", "-C", str(path), "push"))
|
|
206
|
+
|
|
207
|
+
def copy_one(path, sub=None, files=(), dest="nbs", dry_run=False):
|
|
208
|
+
"""Copy `files` into `path`/`dest`.
|
|
209
|
+
|
|
210
|
+
The file list belongs to the caller, so this stays a general-purpose copy rather than
|
|
211
|
+
learning about any particular superproject's shared chrome."""
|
|
212
|
+
out = Path(path)/dest
|
|
213
|
+
if not out.is_dir(): raise Skip(f"no {dest}/")
|
|
214
|
+
names = []
|
|
215
|
+
for f in files:
|
|
216
|
+
src = Path(f)
|
|
217
|
+
if not src.is_file(): raise StepFailed(f"copy {src.name}", 1, f"no such file: {src}")
|
|
218
|
+
if not dry_run: shutil.copy2(src, out/src.name)
|
|
219
|
+
names.append(src.name)
|
|
220
|
+
return [("would copy " if dry_run else "copied ") + ", ".join(names) + f" -> {dest}/"]
|
|
221
|
+
|
|
222
|
+
# %% ../nbs/04_Fleet.ipynb #93610aa9-b47c-47f0-af54-f6293984e066
|
|
223
|
+
def fleet_push(
|
|
224
|
+
dry_run:bool = False, # Report what would be pushed without pushing it
|
|
225
|
+
root:str = None, # Superproject root (default: nearest ancestor with .gitmodules)
|
|
226
|
+
):
|
|
227
|
+
"Push every submodule that has local commits not yet on its remote"
|
|
228
|
+
# Network-bound and independent per remote, so this one does parallelise.
|
|
229
|
+
rep = walk(lambda p, s: push_one(p, s, dry_run=dry_run),
|
|
230
|
+
root=root, workers=8, style="line")
|
|
231
|
+
sys.exit(finish(rep, "would push" if dry_run else "pushed"))
|
|
232
|
+
|
|
233
|
+
def fleet_copy(
|
|
234
|
+
files:str, # Files to copy into each submodule, comma or space separated
|
|
235
|
+
dest:str = "nbs", # Directory inside each submodule to copy into
|
|
236
|
+
dry_run:bool = False, # Report what would be copied without copying it
|
|
237
|
+
root:str = None, # Superproject root (default: nearest ancestor with .gitmodules)
|
|
238
|
+
):
|
|
239
|
+
"Copy shared files into every submodule that has the destination directory"
|
|
240
|
+
fs = [f for f in re.split(r"[,\s]+", files) if f]
|
|
241
|
+
if not fs: sys.exit("fleet copy: no files given")
|
|
242
|
+
rep = walk(lambda p, s: copy_one(p, s, files=fs, dest=dest, dry_run=dry_run),
|
|
243
|
+
root=root, workers=8, style="line")
|
|
244
|
+
sys.exit(finish(rep, "would update" if dry_run else "updated"))
|
|
245
|
+
|
|
246
|
+
# %% ../nbs/04_Fleet.ipynb #5e870c46-a03b-4be9-a7a5-0a30e9af8830
|
|
247
|
+
def _porcelain(path):
|
|
248
|
+
"""`git status -s` for `path`, NOT stripped.
|
|
249
|
+
|
|
250
|
+
`github.git` strips its output, and `git status -s` pads the code to two columns, so
|
|
251
|
+
stripping eats the leading space of the FIRST line and shifts that one path by a
|
|
252
|
+
character. `automate.status` documents the same trap."""
|
|
253
|
+
return subprocess.run(["git", "-C", str(path), "status", "-s"],
|
|
254
|
+
capture_output=True, text=True, check=False).stdout
|
|
255
|
+
|
|
256
|
+
def status_one(path, sub=None, short=False):
|
|
257
|
+
"""Branch, dirty flag and short HEAD for `path`.
|
|
258
|
+
|
|
259
|
+
With `short`, clean submodules are skipped and the dirty ones list their changed
|
|
260
|
+
paths - the fleet-wide "what have I got outstanding" view."""
|
|
261
|
+
st = local_state(path)
|
|
262
|
+
br = st["branch"] or "?"
|
|
263
|
+
if br == "HEAD": br = "DETACHED"
|
|
264
|
+
line = f"{br:<10} {'dirty' if st['dirty'] else 'clean':<6} {st['head'] or '-'}"
|
|
265
|
+
if not short: return [line]
|
|
266
|
+
if not st["dirty"]: raise Skip("clean")
|
|
267
|
+
return [line] + [f" {l}" for l in _porcelain(path).splitlines()]
|
|
268
|
+
|
|
269
|
+
def fleet_status(
|
|
270
|
+
short:bool = False, # Only the submodules with changes, listing the changed paths
|
|
271
|
+
root:str = None, # Superproject root (default: nearest ancestor with .gitmodules)
|
|
272
|
+
):
|
|
273
|
+
"Branch, dirty flag and HEAD for every submodule"
|
|
274
|
+
rep = walk(lambda p, s: status_one(p, s, short=short), root=root, workers=8, style="line")
|
|
275
|
+
sys.exit(finish(rep, "changed" if short else "reported"))
|
|
276
|
+
|
|
277
|
+
# %% ../nbs/04_Fleet.ipynb #68d6b97c-8e6c-4bd5-86bd-a0e0ca83469c
|
|
278
|
+
def _git_ok(path, *args):
|
|
279
|
+
"True when git succeeded in `path`. `git()` hides the exit code, and here it matters"
|
|
280
|
+
return subprocess.run(["git", "-C", str(path), *args],
|
|
281
|
+
capture_output=True, check=False).returncode == 0
|
|
282
|
+
|
|
283
|
+
def _count(path, rng):
|
|
284
|
+
"Commit count for an `a..b` range, 0 when the range does not resolve"
|
|
285
|
+
n = git(path, "rev-list", "--count", rng)
|
|
286
|
+
return int(n) if n.isdigit() else 0
|
|
287
|
+
|
|
288
|
+
def fetch_all(root=None, workers=8):
|
|
289
|
+
"Fetch every initialised submodule concurrently. This is the slow half of an update"
|
|
290
|
+
base = Path(root) if root else find_root()
|
|
291
|
+
subs = [s for s in parse_gitmodules(base) if (base/s.path/".git").exists()]
|
|
292
|
+
with ThreadPoolExecutor(max_workers=workers) as ex:
|
|
293
|
+
list(ex.map(lambda s: git(base/s.path, "fetch", "-q", "origin"), subs))
|
|
294
|
+
return len(subs)
|
|
295
|
+
|
|
296
|
+
def update_one(path, sub=None, dry_run=False, branch="main"):
|
|
297
|
+
"""Reattach `path` to `branch` and fast-forward it to origin/<branch>.
|
|
298
|
+
|
|
299
|
+
Assumes `fetch_all` already ran: fetching is the half that parallelises, and this is
|
|
300
|
+
the half that must not, because a checkout rewrites the worktree."""
|
|
301
|
+
if git(path, "status", "--porcelain"):
|
|
302
|
+
raise StepFailed("update", 1, "uncommitted changes; commit or stash first")
|
|
303
|
+
if not git(path, "remote"): raise Skip("no remote")
|
|
304
|
+
target = f"origin/{branch}"
|
|
305
|
+
if not git(path, "rev-parse", "--verify", "--quiet", target): raise Skip(f"no {target}")
|
|
306
|
+
|
|
307
|
+
br = git(path, "rev-parse", "--abbrev-ref", "HEAD")
|
|
308
|
+
move = br != branch
|
|
309
|
+
# Compare the branch we are ABOUT to be on, not the one we happen to be on now
|
|
310
|
+
ref = branch if _git_ok(path, "rev-parse", "--verify", "--quiet", branch) else "HEAD"
|
|
311
|
+
behind = _count(path, f"{ref}..{target}")
|
|
312
|
+
ahead = _count(path, f"{target}..{ref}")
|
|
313
|
+
# Diverged needs a human. The bash version echoed a line and moved on, which is easy
|
|
314
|
+
# to scroll past; this fails the run so the exit code carries it.
|
|
315
|
+
if ahead and behind:
|
|
316
|
+
raise StepFailed("merge --ff-only", 1,
|
|
317
|
+
f"diverged from {target}: {ahead} ahead, {behind} behind - resolve by hand")
|
|
318
|
+
was = "DETACHED" if br == "HEAD" else br
|
|
319
|
+
if not move and not behind: raise Skip("up to date")
|
|
320
|
+
if dry_run:
|
|
321
|
+
bits = ([f"checkout {branch} (from {was})"] if move else []) + \
|
|
322
|
+
([f"fast-forward {behind}"] if behind else [])
|
|
323
|
+
return ["would " + ", ".join(bits)]
|
|
324
|
+
|
|
325
|
+
done = []
|
|
326
|
+
if move:
|
|
327
|
+
if not _git_ok(path, "checkout", "-q", branch):
|
|
328
|
+
_run("git", "-C", str(path), "checkout", "-q", "-b", branch, target)
|
|
329
|
+
done.append(f"checked out {branch} (was {was})")
|
|
330
|
+
behind = _count(path, f"HEAD..{target}") # the checkout may already have moved us
|
|
331
|
+
if behind:
|
|
332
|
+
_run("git", "-C", str(path), "merge", "--ff-only", target)
|
|
333
|
+
done.append(f"fast-forwarded {behind} commit{'' if behind == 1 else 's'}")
|
|
334
|
+
return done
|
|
335
|
+
|
|
336
|
+
def dirty_submodules(root=None):
|
|
337
|
+
"Submodule paths holding uncommitted changes"
|
|
338
|
+
base = Path(root) if root else find_root()
|
|
339
|
+
return [s.path for s in parse_gitmodules(base)
|
|
340
|
+
if (base/s.path/".git").exists() and git(base/s.path, "status", "--porcelain")]
|
|
341
|
+
|
|
342
|
+
def init_new(root=None, dry_run=False):
|
|
343
|
+
"""Initialise only the submodules git reports as uninitialised.
|
|
344
|
+
|
|
345
|
+
A blanket `submodule update --init` would reset every initialised submodule back to
|
|
346
|
+
its recorded gitlink and detach its HEAD, which is the one thing this setup avoids."""
|
|
347
|
+
base = Path(root) if root else find_root()
|
|
348
|
+
new = [l.split()[1] for l in git(base, "submodule", "status").splitlines() if l.startswith("-")]
|
|
349
|
+
if new and not dry_run:
|
|
350
|
+
_run("git", "-C", str(base), "submodule", "update", "--init", "--", *new)
|
|
351
|
+
return new
|
|
352
|
+
|
|
353
|
+
def pull_root(base):
|
|
354
|
+
"Fast-forward the superproject itself, with the advice its usual failure needs"
|
|
355
|
+
try: return _run("git", "-C", str(base), "pull", "--ff-only").strip().splitlines()[-1:]
|
|
356
|
+
except StepFailed as e:
|
|
357
|
+
raise StepFailed(e.cmd, e.code, f"{e.output}\n"
|
|
358
|
+
"If it complains about submodule paths, the recorded gitlinks drifted.\n"
|
|
359
|
+
"Discard them (the update step re-advances them anyway):\n"
|
|
360
|
+
" git checkout -- $(git diff --name-only)") from None
|
|
361
|
+
|
|
362
|
+
# %% ../nbs/04_Fleet.ipynb #8fb7bc91-d79a-4c1a-bb81-e28b3a77b118
|
|
363
|
+
def fleet_update(
|
|
364
|
+
dry_run:bool = False, # Report what would move without moving it
|
|
365
|
+
root:str = None, # Superproject root (default: nearest ancestor with .gitmodules)
|
|
366
|
+
):
|
|
367
|
+
"Fetch every submodule, then fast-forward each one's main to origin/main"
|
|
368
|
+
base = Path(root) if root else find_root()
|
|
369
|
+
with _step(1, 2, "fetch every submodule") as d:
|
|
370
|
+
d += [f"{fetch_all(base)} fetched"]
|
|
371
|
+
console.print("[dim]2/2[/dim] [bold]fast-forward each main[/bold]")
|
|
372
|
+
rep = walk(lambda p, s: update_one(p, s, dry_run=dry_run), root=base, workers=1, style="line")
|
|
373
|
+
sys.exit(finish(rep, "would update" if dry_run else "updated"))
|
|
374
|
+
|
|
375
|
+
def fleet_sync(
|
|
376
|
+
dry_run:bool = False, # Report what would move without moving it
|
|
377
|
+
root:str = None, # Superproject root (default: nearest ancestor with .gitmodules)
|
|
378
|
+
):
|
|
379
|
+
"Bring this checkout level with every remote: pull, init new submodules, fast-forward each"
|
|
380
|
+
base = Path(root) if root else find_root()
|
|
381
|
+
# Pre-flight FIRST: a dirty submodule turns the checkout/merge below into a conflicted
|
|
382
|
+
# tree, so nothing at all should happen until this passes.
|
|
383
|
+
dirty = dirty_submodules(base)
|
|
384
|
+
if dirty:
|
|
385
|
+
console.print("[bold red]Uncommitted changes in:[/bold red]")
|
|
386
|
+
for d in dirty: console.print(f" {escape(d)}")
|
|
387
|
+
console.print("Commit or stash them first ([bold]just upload[/bold]), then re-run.")
|
|
388
|
+
sys.exit(1)
|
|
389
|
+
|
|
390
|
+
with _step(1, 3, "pull the superproject") as d:
|
|
391
|
+
d += ["skipped (dry run)"] if dry_run else pull_root(base)
|
|
392
|
+
with _step(2, 3, "initialise new submodules") as d:
|
|
393
|
+
new = init_new(base, dry_run=dry_run)
|
|
394
|
+
d += [f"{'would init ' if dry_run else 'initialised '}{', '.join(new)}"] if new else ["none"]
|
|
395
|
+
with _step(3, 3, "fetch every submodule") as d:
|
|
396
|
+
d += [f"{fetch_all(base)} fetched"]
|
|
397
|
+
rep = walk(lambda p, s: update_one(p, s, dry_run=dry_run), root=base, workers=1, style="line")
|
|
398
|
+
sys.exit(finish(rep, "would update" if dry_run else "updated"))
|
|
399
|
+
|
|
400
|
+
# %% ../nbs/04_Fleet.ipynb #0281bd2b-91ba-4e49-90dc-d1a95d54d5a6
|
|
401
|
+
VERBS = {"upload": fleet_upload, "push": fleet_push, "copy": fleet_copy,
|
|
402
|
+
"status": fleet_status, "update": fleet_update, "sync": fleet_sync}
|
|
403
|
+
|
|
404
|
+
def parse_verb(argv):
|
|
405
|
+
"(verb, remaining args) for `fleet ...`, or None when `argv` does not name a verb"
|
|
406
|
+
if not argv or argv[0].startswith("-"): return None
|
|
407
|
+
return (argv[0], argv[1:]) if argv[0] in VERBS else None
|
|
408
|
+
|
|
409
|
+
def usage():
|
|
410
|
+
"The verb list, for --help and for an unknown verb"
|
|
411
|
+
lines = ["Usage: fleet <verb> [options]", "", "Verbs:"]
|
|
412
|
+
lines += [f" {name:<8} {(fn.__doc__ or '').strip()}" for name, fn in sorted(VERBS.items())]
|
|
413
|
+
lines += ["", "Any verb takes --help for its own options."]
|
|
414
|
+
return "\n".join(lines)
|
|
415
|
+
|
|
416
|
+
def run_verb(name, argv):
|
|
417
|
+
"""Parse `argv` against the verb's own annotations, then call it.
|
|
418
|
+
|
|
419
|
+
Deliberately NOT `@call_parse`: fastcore runs the first decorated function in a module
|
|
420
|
+
the moment that module is `__main__`, which is exactly what makes `python -m
|
|
421
|
+
nbdevAuto.github` work with its single verb. With several verbs it would run whichever
|
|
422
|
+
happened to be defined first, so the verbs stay plain functions and the parser is built
|
|
423
|
+
here. `anno_parser` is the same one `call_parse` uses, so `--help` is unchanged."""
|
|
424
|
+
fn = VERBS[name]
|
|
425
|
+
args = vars(anno_parser(fn, prog=f"fleet {name}").parse_args(argv))
|
|
426
|
+
for k in ("pdb", "xtra"): args.pop(k, None)
|
|
427
|
+
return fn(**args)
|
|
428
|
+
|
|
429
|
+
def fleet():
|
|
430
|
+
"""Dispatch `fleet <verb>` to the matching command.
|
|
431
|
+
|
|
432
|
+
`upload` and `status` are already console scripts bound to the single-repo versions in
|
|
433
|
+
`automate`, so the fleet-wide ones cannot claim those names. One script with verbs
|
|
434
|
+
sidesteps the collision and keeps `--help` working per verb."""
|
|
435
|
+
picked = parse_verb(sys.argv[1:])
|
|
436
|
+
if not picked:
|
|
437
|
+
print(usage())
|
|
438
|
+
sys.exit(0 if sys.argv[1:2] in ([], ["-h"], ["--help"]) else 2)
|
|
439
|
+
return run_verb(*picked)
|
|
440
|
+
|
|
441
|
+
# %% ../nbs/04_Fleet.ipynb #7adb89c6-6788-4241-acff-53cb4566b5ce
|
|
442
|
+
if __name__ == "__main__": fleet()
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: nbdevAuto
|
|
3
|
-
Version: 0.3.
|
|
3
|
+
Version: 0.3.68
|
|
4
4
|
Summary: automating nbdev
|
|
5
5
|
Author-email: Benedict Thekkel <bthekkel1@gmail.com>
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -73,23 +73,25 @@ from nbdevAuto.pdf import PDFreader
|
|
|
73
73
|
| [Automate](01_Automate.ipynb) | The command line surface: the prep/commit/push pipeline, the GitHub and PyPI release halves, and the help output that lists them all |
|
|
74
74
|
| [PDF Reader](02_PDF_reader.ipynb) | [`PDFreader`](https://bthek1.github.io/nbdevAuto/pdf_reader.html#pdfreader), a class that converts a PDF through `pdf2image` and renders page ranges inline when you slice it (`pdf[0:5]`) |
|
|
75
75
|
| [GitHub](03_GitHub.ipynb) | [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus): reading `.gitmodules`, resolving a token, and asking the GitHub API about every submodule at once - CI runs, Pages deploys, Pages builds, remote `main` shas - plus the `--report` and `--audit` views and the exit codes that keep “absent” apart from “cannot ask” |
|
|
76
|
+
| [Fleet](04_Fleet.ipynb) | [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet): running one command across every submodule of a superproject. A walker that reports only what did work and counts the rest, plus the verbs built on it - upload, push, copy, status, update and sync |
|
|
76
77
|
|
|
77
78
|
------------------------------------------------------------------------
|
|
78
79
|
|
|
79
80
|
## The Command Line Surface
|
|
80
81
|
|
|
81
|
-
Installing the package puts these on your `PATH`. Most are ordinary functions in [Automate](01_Automate.ipynb), exposed as scripts through `[project.scripts]`; [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) comes from [GitHub](03_GitHub.ipynb).
|
|
82
|
+
Installing the package puts these on your `PATH`. Most are ordinary functions in [Automate](01_Automate.ipynb), exposed as scripts through `[project.scripts]`; [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) comes from [GitHub](03_GitHub.ipynb) and [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet) from [Fleet](04_Fleet.ipynb).
|
|
82
83
|
|
|
83
84
|
| Command | Does |
|
|
84
85
|
|----|----|
|
|
85
86
|
| [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) | Bump the version, then export, test and clean the notebooks, refreshing `_quarto.yml` and the README |
|
|
86
|
-
| [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp) | git add, commit, and push. Without `-m` the
|
|
87
|
+
| [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp) | git add, commit, and push. Without `-m` the message is built from the staged paths |
|
|
87
88
|
| [`status`](https://bthek1.github.io/nbdevAuto/automate.html#status) | Show the working tree state |
|
|
88
89
|
| [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) | [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) then [`gacp`](https://bthek1.github.io/nbdevAuto/automate.html#gacp), the one you actually type |
|
|
89
90
|
| `gitrelease` | Bump the **minor** version, then tag and create a GitHub release |
|
|
90
91
|
| `piprelease` | Build the sdist and wheel, then upload to PyPI with twine |
|
|
91
92
|
| [`release`](https://bthek1.github.io/nbdevAuto/automate.html#release) | `gitrelease` then `piprelease` |
|
|
92
93
|
| [`ghstatus`](https://bthek1.github.io/nbdevAuto/github.html#ghstatus) | Remote status for every submodule of an nbdev superproject, from the GitHub API. `--report` for the sync view, `--audit` to check `.gitmodules` against the repos the account owns, `--json` for machine output |
|
|
94
|
+
| [`fleet`](https://bthek1.github.io/nbdevAuto/fleet.html#fleet) | Run one command across every submodule: `fleet upload|push|copy|status|update|sync`. Every verb takes `--help`, and the destructive ones take `--dry-run` |
|
|
93
95
|
| `h` | Print the list of available commands |
|
|
94
96
|
|
|
95
97
|
Both [`prep`](https://bthek1.github.io/nbdevAuto/automate.html#prep) and [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) take `-p` to choose which version part to increment; it defaults to 2, the patch, so every [`upload`](https://bthek1.github.io/nbdevAuto/automate.html#upload) ships a new version.
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
__version__ = "0.3.66"
|
|
@@ -1,201 +0,0 @@
|
|
|
1
|
-
"""Run one command across every submodule of a superproject
|
|
2
|
-
|
|
3
|
-
Docs: https://bthek1.github.io/nbdevAuto/fleet.html.md"""
|
|
4
|
-
|
|
5
|
-
# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/04_Fleet.ipynb.
|
|
6
|
-
|
|
7
|
-
# %% auto #0
|
|
8
|
-
__all__ = ['VERBS', 'Outcome', 'Report', 'format_summary', 'walk', 'finish', 'script_path', 'upload_one', 'fleet_upload',
|
|
9
|
-
'parse_verb', 'usage', 'run_verb', 'fleet']
|
|
10
|
-
|
|
11
|
-
# %% ../nbs/04_Fleet.ipynb #d361ee5f-9f09-464f-aec6-0e27cec90fe9
|
|
12
|
-
import shutil
|
|
13
|
-
import subprocess
|
|
14
|
-
import sys
|
|
15
|
-
from concurrent.futures import ThreadPoolExecutor
|
|
16
|
-
from dataclasses import dataclass, field
|
|
17
|
-
from pathlib import Path
|
|
18
|
-
|
|
19
|
-
from fastcore.script import *
|
|
20
|
-
|
|
21
|
-
# The rendering layer lives in `automate`, the git/discovery layer in `github`. This
|
|
22
|
-
# module is the only one that knows about MANY repos; those two stay single-repo.
|
|
23
|
-
from .automate import StepFailed, console, escape
|
|
24
|
-
from .github import find_root, git, parse_gitmodules
|
|
25
|
-
|
|
26
|
-
# %% ../nbs/04_Fleet.ipynb #4e6b24c5-cd71-434c-84e6-e6994d5959c7
|
|
27
|
-
@dataclass
|
|
28
|
-
class Outcome:
|
|
29
|
-
"What happened in one submodule"
|
|
30
|
-
path: str
|
|
31
|
-
state: str # ran | skipped | failed
|
|
32
|
-
reason: str = "" # why it was skipped, or how it failed
|
|
33
|
-
detail: list = field(default_factory=list) # short lines to show under the path
|
|
34
|
-
|
|
35
|
-
@dataclass
|
|
36
|
-
class Report:
|
|
37
|
-
"Everything a `walk` did, and the exit code it implies"
|
|
38
|
-
outcomes: list = field(default_factory=list)
|
|
39
|
-
|
|
40
|
-
@property
|
|
41
|
-
def ran(self): return [o for o in self.outcomes if o.state == "ran"]
|
|
42
|
-
@property
|
|
43
|
-
def failed(self): return [o for o in self.outcomes if o.state == "failed"]
|
|
44
|
-
@property
|
|
45
|
-
def code(self): return 1 if self.failed else 0
|
|
46
|
-
|
|
47
|
-
def skip_counts(self):
|
|
48
|
-
"Skip reason -> count, in the order the reasons were first seen"
|
|
49
|
-
counts = {}
|
|
50
|
-
for o in self.outcomes:
|
|
51
|
-
if o.state == "skipped": counts[o.reason] = counts.get(o.reason, 0) + 1
|
|
52
|
-
return counts
|
|
53
|
-
|
|
54
|
-
def format_summary(rep, verb="ran"):
|
|
55
|
-
"The one line that replaces a message per submodule. Quiet submodules are counted, not narrated"
|
|
56
|
-
n = len(rep.outcomes)
|
|
57
|
-
parts = [f"{len(rep.ran)} {verb}"]
|
|
58
|
-
parts += [f"{c} {reason}" for reason, c in rep.skip_counts().items()]
|
|
59
|
-
if rep.failed: parts.append(f"{len(rep.failed)} FAILED")
|
|
60
|
-
return f"{n} submodule{'' if n == 1 else 's'}: " + ", ".join(parts)
|
|
61
|
-
|
|
62
|
-
# %% ../nbs/04_Fleet.ipynb #099242e8-c9ad-4b48-8ebc-7a818e5f8f81
|
|
63
|
-
def _classify(base, sub, need_nbs, only_dirty):
|
|
64
|
-
"The reason to skip this submodule, or None to run it"
|
|
65
|
-
p = base/sub.path
|
|
66
|
-
# An uninitialised submodule is an empty directory: present, but with no .git
|
|
67
|
-
if not (p/".git").exists(): return "not initialised"
|
|
68
|
-
if need_nbs and not (p/"nbs").is_dir(): return "without nbs"
|
|
69
|
-
if only_dirty and not git(p, "status", "--porcelain"): return "clean"
|
|
70
|
-
return None
|
|
71
|
-
|
|
72
|
-
def _fail_reason(e):
|
|
73
|
-
"One line describing why a submodule's callable did not finish"
|
|
74
|
-
if isinstance(e, StepFailed): return f"{e.cmd} exited {e.code}"
|
|
75
|
-
return f"{type(e).__name__}: {e}"
|
|
76
|
-
|
|
77
|
-
def walk(fn, *, root=None, only_dirty=False, need_nbs=False, workers=1, header=True):
|
|
78
|
-
"""Run `fn(path, sub)` in each submodule, returning a `Report`.
|
|
79
|
-
|
|
80
|
-
`fn` may return a list of short strings to show under the submodule's name. Only
|
|
81
|
-
submodules that actually do something get printed; the rest are counted by the caller
|
|
82
|
-
through `format_summary`. Sequential runs print the header BEFORE the callable so its
|
|
83
|
-
own output streams under it; parallel runs print after, so lines cannot interleave."""
|
|
84
|
-
base = Path(root) if root else find_root()
|
|
85
|
-
rep, todo = Report(), []
|
|
86
|
-
for s in parse_gitmodules(base):
|
|
87
|
-
why = _classify(base, s, need_nbs, only_dirty)
|
|
88
|
-
if why: rep.outcomes.append(Outcome(s.path, "skipped", why))
|
|
89
|
-
else: todo.append(s)
|
|
90
|
-
|
|
91
|
-
def _one(s):
|
|
92
|
-
try: return Outcome(s.path, "ran", detail=list(fn(base/s.path, s) or []))
|
|
93
|
-
except BaseException as e: return Outcome(s.path, "failed", _fail_reason(e))
|
|
94
|
-
|
|
95
|
-
def _head(path): console.print(f"\n[bold cyan]==> {escape(path)}[/bold cyan]")
|
|
96
|
-
|
|
97
|
-
if workers > 1 and len(todo) > 1:
|
|
98
|
-
# Results are rendered in submodule order, not completion order, so two runs of
|
|
99
|
-
# the same fleet read the same way.
|
|
100
|
-
with ThreadPoolExecutor(max_workers=workers) as ex: done = list(ex.map(_one, todo))
|
|
101
|
-
for o in done:
|
|
102
|
-
if header: _head(o.path)
|
|
103
|
-
_render(o)
|
|
104
|
-
rep.outcomes.append(o)
|
|
105
|
-
else:
|
|
106
|
-
for s in todo:
|
|
107
|
-
if header: _head(s.path)
|
|
108
|
-
o = _one(s)
|
|
109
|
-
_render(o)
|
|
110
|
-
rep.outcomes.append(o)
|
|
111
|
-
rep.outcomes.sort(key=lambda o: o.path)
|
|
112
|
-
return rep
|
|
113
|
-
|
|
114
|
-
def _render(o):
|
|
115
|
-
"A submodule's own lines, or why it failed"
|
|
116
|
-
if o.state == "failed":
|
|
117
|
-
console.print(f" [bold red]FAIL[/bold red] [dim]{escape(o.reason)}[/dim]")
|
|
118
|
-
for line in o.detail: console.print(f" [dim]{escape(line)}[/dim]")
|
|
119
|
-
|
|
120
|
-
def finish(rep, verb):
|
|
121
|
-
"Print the counted summary, name any failures, and return the exit code"
|
|
122
|
-
console.print(f"\n[bold]{escape(format_summary(rep, verb))}[/bold]")
|
|
123
|
-
for o in rep.failed:
|
|
124
|
-
console.print(f"[bold red]FAILED[/bold red] {escape(o.path)}: [dim]{escape(o.reason)}[/dim]")
|
|
125
|
-
return rep.code
|
|
126
|
-
|
|
127
|
-
# %% ../nbs/04_Fleet.ipynb #99e2c4fe-c885-4880-b9df-31da51e61f3a
|
|
128
|
-
def script_path(name):
|
|
129
|
-
"""The console script `name` from the environment this is running in.
|
|
130
|
-
|
|
131
|
-
Taking it from `sys.executable`'s directory is what makes this work under a
|
|
132
|
-
non-interactive shell that never put the venv on PATH."""
|
|
133
|
-
cand = Path(sys.executable).parent/name
|
|
134
|
-
if cand.is_file(): return str(cand)
|
|
135
|
-
found = shutil.which(name)
|
|
136
|
-
if found: return found
|
|
137
|
-
raise RuntimeError(f"Cannot find the '{name}' console script. Run `uv sync` first")
|
|
138
|
-
|
|
139
|
-
def upload_one(path, sub=None, args=()):
|
|
140
|
-
"""Run the single-repo `upload` in `path`, letting its own rich output through.
|
|
141
|
-
|
|
142
|
-
A subprocess rather than an in-process call: `prep` reads nbdev's config from the cwd
|
|
143
|
-
and caches it, and `nbdev_test` spawns its own process pool, so nineteen repos in one
|
|
144
|
-
interpreter is a trap. The subprocess keeps this tty, so colour survives."""
|
|
145
|
-
r = subprocess.run([script_path("upload"), *args], cwd=str(path))
|
|
146
|
-
if r.returncode: raise StepFailed("upload", r.returncode)
|
|
147
|
-
return []
|
|
148
|
-
|
|
149
|
-
# %% ../nbs/04_Fleet.ipynb #2ef9f1e5-239a-4790-8cec-43fffb8bc422
|
|
150
|
-
def fleet_upload(
|
|
151
|
-
all:bool = False, # Upload every submodule with an nbs/ folder, not just the changed ones
|
|
152
|
-
root:str = None, # Superproject root (default: nearest ancestor with .gitmodules)
|
|
153
|
-
):
|
|
154
|
-
"Run `upload` in each submodule that has local changes"
|
|
155
|
-
# Sequential on purpose: `upload` runs nbdev_test with 8 workers, and knowledge-lab
|
|
156
|
-
# has 4 vCPU. Parallel uploads would oversubscribe the box and interleave the output.
|
|
157
|
-
rep = walk(upload_one, root=root, need_nbs=True, only_dirty=not all, workers=1)
|
|
158
|
-
sys.exit(finish(rep, "uploaded"))
|
|
159
|
-
|
|
160
|
-
# %% ../nbs/04_Fleet.ipynb #0281bd2b-91ba-4e49-90dc-d1a95d54d5a6
|
|
161
|
-
VERBS = {"upload": fleet_upload}
|
|
162
|
-
|
|
163
|
-
def parse_verb(argv):
|
|
164
|
-
"(verb, remaining args) for `fleet ...`, or None when `argv` does not name a verb"
|
|
165
|
-
if not argv or argv[0].startswith("-"): return None
|
|
166
|
-
return (argv[0], argv[1:]) if argv[0] in VERBS else None
|
|
167
|
-
|
|
168
|
-
def usage():
|
|
169
|
-
"The verb list, for --help and for an unknown verb"
|
|
170
|
-
lines = ["Usage: fleet <verb> [options]", "", "Verbs:"]
|
|
171
|
-
lines += [f" {name:<10} {(fn.__doc__ or '').strip()}" for name, fn in sorted(VERBS.items())]
|
|
172
|
-
lines += ["", "Any verb takes --help for its own options."]
|
|
173
|
-
return "\n".join(lines)
|
|
174
|
-
|
|
175
|
-
def run_verb(name, argv):
|
|
176
|
-
"""Parse `argv` against the verb's own annotations, then call it.
|
|
177
|
-
|
|
178
|
-
Deliberately NOT `@call_parse`: fastcore runs the first decorated function in a module
|
|
179
|
-
the moment that module is `__main__`, which is exactly what makes `python -m
|
|
180
|
-
nbdevAuto.github` work with its single verb. With several verbs it would run whichever
|
|
181
|
-
happened to be defined first, so the verbs stay plain functions and the parser is built
|
|
182
|
-
here. `anno_parser` is the same one `call_parse` uses, so `--help` is unchanged."""
|
|
183
|
-
fn = VERBS[name]
|
|
184
|
-
args = vars(anno_parser(fn, prog=f"fleet {name}").parse_args(argv))
|
|
185
|
-
for k in ("pdb", "xtra"): args.pop(k, None)
|
|
186
|
-
return fn(**args)
|
|
187
|
-
|
|
188
|
-
def fleet():
|
|
189
|
-
"""Dispatch `fleet <verb>` to the matching command.
|
|
190
|
-
|
|
191
|
-
`upload` and `status` are already console scripts bound to the single-repo versions in
|
|
192
|
-
`automate`, so the fleet-wide ones cannot claim those names. One script with verbs
|
|
193
|
-
sidesteps the collision and keeps `--help` working per verb."""
|
|
194
|
-
picked = parse_verb(sys.argv[1:])
|
|
195
|
-
if not picked:
|
|
196
|
-
print(usage())
|
|
197
|
-
sys.exit(0 if sys.argv[1:2] in ([], ["-h"], ["--help"]) else 2)
|
|
198
|
-
return run_verb(*picked)
|
|
199
|
-
|
|
200
|
-
# %% ../nbs/04_Fleet.ipynb #7adb89c6-6788-4241-acff-53cb4566b5ce
|
|
201
|
-
if __name__ == "__main__": fleet()
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|