@kova1/pullr 0.5.0 → 0.7.0

package/README.md

@@ -5,7 +5,7 @@
 [![Homebrew](https://img.shields.io/github/v/release/kova1max/pullr?logo=homebrew&label=homebrew)](https://github.com/kova1max/homebrew-tap)
 [![License: MIT](https://img.shields.io/github/license/kova1max/pullr)](LICENSE)
 
-Fast-forward pull every git repository under a directory.
+Safe, fast-forward pull every git repository under a directory.
 
 Built for workspaces made of many independent repositories, where keeping
 everything current means `cd`-ing into each one.
@@ -40,9 +40,6 @@ To try it without installing anything:
 npx @kova1/pullr --dry-run ~/work
 ```
 
-Or copy [`bin/pullr`](bin/pullr) anywhere on your `PATH` - it is a single bash
-script with no dependencies beyond `git`.
-
 Every release is also mirrored to
 [GitHub Packages](https://github.com/kova1max/pullr/pkgs/npm/pullr) as
 `@kova1max/pullr` (GitHub requires the repo owner as scope). Installing from
@@ -64,8 +61,12 @@ first. It will never:
   incoming files don't stop the update.
 - **Touch a repository on a detached `HEAD`**, or a branch with no upstream.
   Both are skipped.
-- **Enter a nested repository.** Repositories inside another repository,
-  including submodules, are left to their parent.
+- **Enter a nested repository.** Repositories inside another repository are
+  left to their parent.
+- **Move a submodule off its branch.** With `--submodules`, a submodule is
+  only ever fast-forwarded on the branch it is on. (If you set git's
+  `submodule.recurse`, pullr does what `git pull` does with it: checks
+  submodules out at the recorded commit.)
 - **Follow symlinked directories.**
 
 ## Usage
@@ -74,13 +75,14 @@ first. It will never:
 pullr [options] [DIR]
 ```
 
-| Option | Default | Meaning |
-| --- | --- | --- |
-| `DIR` | current directory | Where to look for repositories |
-| `-j`, `--jobs N` | `4` | Work on up to `N` repositories in parallel. Output is buffered per repository and printed in discovery order, so it reads the same as a sequential run. With `N > 1` git never prompts for credentials; use `--jobs 1` for repositories that need an interactive login. |
+| Option                       | Default | Meaning |
+| :--- | :--- | :--- |
+| `DIR` | current directory | Where to look for repositories |
+| `-j`, `--jobs N` | `8` | Work on up to `N` repositories in parallel. Output is buffered per repository and printed in discovery order, so it reads the same as a sequential run. With `N > 1` git never prompts for credentials; use `--jobs 1` for repositories that need an interactive login. |
 | `--max-depth N` | `2` | How many directory levels below `DIR` to search. `0` = only `DIR` itself, `1` = `DIR` and its direct children, ... |
 | `-n`, `--dry-run` | | Show what a run would do without pulling, see below |
 | `-r`, `--rebase` | off | Rebase local commits onto the upstream instead of refusing a diverged branch (`git pull --rebase`). A rebase that hits a conflict is aborted and the repository is left as it was. |
+| `--submodules` | off | Also fast-forward each submodule that is checked out on a branch, see below |
 | `--no-color` | | Disable colored output. Also disabled when the [`NO_COLOR`](https://no-color.org) environment variable is set, or when output is not a terminal. |
 | `-V`, `--version` | | Print the version |
 | `-h`, `--help` | | Show help |
@@ -117,6 +119,47 @@ remote as it is now, then prints the same lines a real run would (with
 only updates remote-tracking refs: the working tree and current branch are
 never modified. Exit codes match a real run.
 
+### Submodules
+
+By default pullr updates a repository the way `git pull` does: submodules stay
+where they are, unless you set git's `submodule.recurse`, in which case they
+are checked out at the commits the parent now records, as `git pull` would.
+
+`--submodules` is for working inside submodules. Each submodule that is
+checked out on a branch is treated as a repository of its own - fetched,
+fast-forwarded with the same rules, and listed under its parent:
+
+```console
+$ pullr --submodules ~/work
++ app (main, 2 new commits)
+  + app/lib/core (main, 5 new commits)
+  - app/vendor/sdk (detached HEAD)
+```
+
+Submodules on a detached HEAD (git's default after `git submodule update`) are
+skipped, and uninitialized ones are left out.
+
+## How it compares
+
+| | pullr | [gita](https://github.com/nosarthur/gita) | [gitup](https://github.com/earwig/git-repo-updater) | [myrepos](https://myrepos.branchable.com) (`mr`) | [mani](https://github.com/alajmo/mani) |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| Needs | bash and git | Python 3.8+ | Python 3.10+ | Perl | Go binary |
+| Finds repos | scans a directory | register first (`gita add`) | scans a directory | register first (`~/.mrconfig`) | register first (`mani.yaml`) |
+| Pulls in parallel | 8 at a time by default | all at once, no limit | no | opt-in | opt-in |
+| Update | fetch + fast‑forward | `git pull` | fetch + fast‑forward | `git pull` | no built-in pull |
+| Merge commits | never | depends on your git config | never | depends on your git config | depends on your command |
+| Local changes in the way | reported as dirty | git's error | skipped | git's error | - |
+| Dry run | yes | no | no | no | prints commands |
+
+**Why "8 at a time" and not "all at once".** Many self-hosted git servers
+limit how many SSH connections can be opening at the same time; OpenSSH's
+default (`MaxStartups 10:30:100`) starts refusing them past 10. Fetching
+every repository at once then fails for reasons that have nothing to do with
+your repositories. On a self-hosted GitLab with 61 repositories, 16 parallel
+fetches produced 6-9 of those spurious failures and 32 produced 10-14, while
+8 produced none. GitHub itself handled 64 at once without errors, so raise
+`-j` freely there.
+
 ## License
 
 [MIT](LICENSE)

package/bin/pullr

@@ -1,9 +1,9 @@
 #!/usr/bin/env bash
 # Recursively find git repos under a directory and fast-forward pull each one.
-# Usage: pullr [--jobs N] [--max-depth N] [--dry-run] [--rebase] [--no-color] [DIR]
+# Usage: pullr [--jobs N] [--max-depth N] [--dry-run] [--rebase] [--submodules] [--no-color] [DIR]
 set -euo pipefail
 
-VERSION="0.5.0" # set by the release workflow
+VERSION="0.7.0" # set by the release workflow
 
 usage() {
   cat <<'EOF'
@@ -13,7 +13,7 @@ Finds git repositories under DIR (default: current directory) and runs
 `git pull --ff-only` in each. Does not descend into a repository once found.
 
 Options:
-  -j, --jobs N       Work on up to N repositories in parallel (default: 4).
+  -j, --jobs N       Work on up to N repositories in parallel (default: 8).
                      Output is buffered per repository and printed in order.
                      With N > 1 git never prompts for credentials.
       --max-depth N  How many directory levels below DIR to search (default: 2).
@@ -24,6 +24,9 @@ Options:
                      refusing a branch that has diverged (git pull --rebase).
                      A rebase that hits a conflict is aborted, leaving the
                      repository as it was.
+      --submodules   Also fast-forward each submodule that is checked out on a
+                     branch, with the same rules as any other repository.
+                     Submodules on a detached HEAD are skipped.
       --no-color     Disable colored output. Also disabled when NO_COLOR is
                      set (https://no-color.org) or output is not a terminal.
   -V, --version      Print the version.
@@ -32,8 +35,9 @@ EOF
 }
 
 mode=pull
-jobs=4
+jobs=8
 rebase=0
+submodules=0
 color=1
 max_depth=2
 root=""
@@ -46,6 +50,7 @@ while (($#)); do
     --max-depth=*) max_depth="${1#*=}"; shift ;;
     -n|--dry-run) mode=dryrun; shift ;;
     -r|--rebase) rebase=1; shift ;;
+    --submodules) submodules=1; shift ;;
     --no-color) color=0; shift ;;
     -h|--help) usage; exit 0 ;;
     -V|--version) echo "pullr $VERSION"; exit 0 ;;
@@ -142,9 +147,24 @@ commits() {
   printf '%s commit%s' "$1" "$([[ "$1" == 1 ]] || printf s)"
 }
 
-# Uncommitted changes to tracked files, staged or not.
+# Uncommitted changes to tracked files, staged or not. A submodule checked out
+# at a different commit than the one recorded is not one: git neither refuses
+# to update over it nor touches it.
 has_tracked_changes() {
-  ! git -C "$1" diff --quiet HEAD --
+  ! git -C "$1" diff --quiet --ignore-submodules HEAD --
+}
+
+has_local_changes() {
+  [[ -n "$(git -C "$1" status --porcelain --ignore-submodules=all)" ]]
+}
+
+# Mirror `git pull` for people who set submodule.recurse: after the update,
+# check submodules out at the commits the parent now records. With
+# --submodules each one is fast-forwarded on its own branch instead.
+sync_submodules() {
+  [[ "$submodules" == 0 ]] || return 0
+  [[ "$(git -C "$1" config --type=bool --get submodule.recurse 2>/dev/null)" == true ]] || return 0
+  git -C "$1" submodule update --recursive 2>&1
 }
 
 pull_repo() {
@@ -157,8 +177,8 @@ pull_repo() {
   elif [[ "$ahead" == 0 ]]; then
     # git refuses, touching nothing, when local changes are in the way
     if out="$(git -C "$repo" merge --ff-only --quiet '@{upstream}' 2>&1)"; then
-      report_updated "$rel" "$branch" "$behind" 0 ""
-    elif [[ -n "$(git -C "$repo" status --porcelain)" ]]; then
+      report_synced "$repo" "$rel" "$branch" "$behind" 0 ""
+    elif has_local_changes "$repo"; then
       report_dirty "$rel" "$branch" "$behind"
     else
       report_failed "$rel" "$branch" "$out"
@@ -168,13 +188,13 @@ pull_repo() {
   elif has_tracked_changes "$repo"; then
     report_dirty "$rel" "$branch" "$behind"
   elif out="$(git -C "$repo" rebase --quiet '@{upstream}' 2>&1)"; then
-    report_updated "$rel" "$branch" "$behind" "$ahead" rebased
+    report_synced "$repo" "$rel" "$branch" "$behind" "$ahead" rebased
   else
     # A rebase that stopped on a conflict must not leave the repository
     # mid-rebase; --abort succeeds only when one is in progress.
     if git -C "$repo" rebase --abort >/dev/null 2>&1; then
       report_failed "$rel" "$branch" "$out"
-    elif [[ -n "$(git -C "$repo" status --porcelain)" ]]; then
+    elif has_local_changes "$repo"; then
       report_dirty "$rel" "$branch" "$behind"
     else
       report_failed "$rel" "$branch" "$out"
@@ -182,12 +202,22 @@ pull_repo() {
   fi
 }
 
+# The repository was updated; report it, unless syncing submodules failed.
+report_synced() {
+  local out
+  if out="$(sync_submodules "$1")"; then
+    report_updated "$2" "$3" "$4" "$5" "$6"
+  else
+    report_failed "$2" "$3" "updated, but updating submodules failed:"$'\n'"$out"
+  fi
+}
+
 # Would local changes stop a fast-forward? Git refuses when a file it has to
 # update is modified locally or exists untracked, so compare the two lists.
 changes_block_update() {
   local repo="$1" incoming local_paths
   incoming="$(git -C "$repo" diff --name-only HEAD '@{upstream}')"
-  local_paths="$(git -C "$repo" status --porcelain --untracked-files=all | cut -c4-)"
+  local_paths="$(git -C "$repo" status --porcelain --untracked-files=all --ignore-submodules=all | cut -c4-)"
   [[ -n "$incoming" && -n "$local_paths" ]] || return 1
   printf '%s\n' "$local_paths" | grep -qFxf <(printf '%s\n' "$incoming")
 }
@@ -225,11 +255,12 @@ worker() {
 
 # --- discovery ---------------------------------------------------------------
 
-repos=()
+repos=() nested=()
 walk() {
   local dir="$1" depth="$2"
   if [[ -e "$dir/.git" ]]; then
-    repos+=("$dir")
+    repos+=("$dir") nested+=(0)
+    [[ "$submodules" == 0 ]] || add_submodules "$dir"
     return
   fi
   ((depth < max_depth)) || return 0
@@ -239,6 +270,17 @@ walk() {
     walk "${child%/}" $((depth + 1))
   done
 }
+# Checked-out submodules, recursively, in git's order; uninitialized ones
+# have no checkout to update and are left out.
+add_submodules() {
+  local sub
+  # shellcheck disable=SC2016  # $displaypath is expanded by git submodule foreach
+  while IFS= read -r sub; do
+    [[ -n "$sub" ]] || continue
+    repos+=("$1/$sub") nested+=(1)
+  done < <(git -C "$1" submodule foreach --quiet --recursive 'printf "%s\n" "$displaypath"' 2>/dev/null)
+}
+
 walk "$root" 0
 
 count=${#repos[@]}
@@ -256,6 +298,8 @@ failed_repos=() dirty_repos=()
 
 consume() {
   local text="$1" head="${1%%$'\n'*}" body="${1#*$'\n'}"
+  # a submodule's lines sit under its parent's
+  [[ "${2:-0}" == 0 ]] || body="  ${body//$'\n'/$'\n'  }"
   local category="${head%%$'\t'*}" label="${head#*$'\t'}"
   case "$category" in
     updated) updated=$((updated + 1)) ;;
@@ -272,8 +316,8 @@ if ((jobs > 1 && count > 1)); then
   scratch="$(mktemp -d)"
   trap 'rm -rf "$scratch"' EXIT
   export GIT_TERMINAL_PROMPT=0
-  export mode rebase root green yellow red dim reset
-  export -f worker pull_repo dryrun_repo repo_label indented prepare report_failed report_diverged report_dirty report_current report_updated commits has_tracked_changes changes_block_update
+  export mode rebase submodules root green yellow red dim reset
+  export -f worker pull_repo dryrun_repo repo_label indented prepare report_failed report_diverged report_dirty report_current report_updated report_synced commits has_tracked_changes has_local_changes sync_submodules changes_block_update
   # shellcheck disable=SC2016  # the single-quoted body runs inside the child shell
   for ((i = 0; i < count; i++)); do printf '%s\0%s\0' "$i" "${repos[i]}"; done |
     xargs -0 -n 2 -P "$jobs" "$BASH" -c 'worker "$@"' _ "$scratch" &
@@ -287,12 +331,12 @@ if ((jobs > 1 && count > 1)); then
       fi
       sleep 0.1
     done
-    consume "$(cat "$scratch/$i.done")"
+    consume "$(cat "$scratch/$i.done")" "${nested[i]}"
   done
   wait "$pool" || true
 else
-  for repo in "${repos[@]}"; do
-    consume "$("${mode}_repo" "$repo")"
+  for ((i = 0; i < count; i++)); do
+    consume "$("${mode}_repo" "${repos[i]}")" "${nested[i]}"
   done
 fi
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@kova1/pullr",
-  "version": "0.5.0",
-  "description": "Fast-forward pull every git repository under a directory",
+  "version": "0.7.0",
+  "description": "Safe, fast-forward pull every git repository under a directory",
   "keywords": [
     "git",
     "pull",