@kova1/pullr 0.6.0 → 0.8.0

package/README.md

@@ -13,13 +13,13 @@ everything current means `cd`-ing into each one.
 ```console
 $ pullr ~/work
 + api (main, 3 new commits)
-= web (main)
-~ docs (main, 2 commits held back by local changes)
++ web (main, 1 new commit, local changes reapplied)
+~ docs (main, 2 commits held back because local changes conflict with them)
 - scratch (main has no upstream)
 x infra (main)
     diverged from upstream (behind 2, ahead 1): cannot fast-forward
 
-Repos: 5  updated: 1  up to date: 1  dirty: 1  skipped: 1  failed: 1
+Repos: 5  updated: 2  up to date: 0  dirty: 1  skipped: 1  failed: 1
 Dirty: docs
 Failed: infra
 ```
@@ -55,14 +55,21 @@ first. It will never:
   failed and left exactly as it was. Rebasing it is opt-in with `--rebase`,
   and a rebase that hits a conflict is aborted, leaving the repository as it
   was.
-- **Stash, overwrite or discard your changes.** There is no auto-stash. If
-  local edits or untracked files are in the way of incoming changes, the
-  repository is reported as dirty and left as it was; edits that don't touch
-  incoming files don't stop the update.
+- **Leave your changes half-merged, or lose them.** When local edits or
+  untracked files are in the way of an update, pullr stashes them, updates,
+  and re-applies them on top. If they conflict with the incoming commits, it
+  puts the repository back exactly as it was - same commit, same files, same
+  staged and unstaged changes, no stash left behind - and reports it as
+  dirty. Edits that don't touch incoming files are never stashed at all.
+  `--no-autostash` skips the stash and just reports the repository as dirty.
 - **Touch a repository on a detached `HEAD`**, or a branch with no upstream.
   Both are skipped.
-- **Enter a nested repository.** Repositories inside another repository,
-  including submodules, are left to their parent.
+- **Enter a nested repository.** Repositories inside another repository are
+  left to their parent.
+- **Move a submodule off its branch.** With `--submodules`, a submodule is
+  only ever fast-forwarded on the branch it is on. (If you set git's
+  `submodule.recurse`, pullr does what `git pull` does with it: checks
+  submodules out at the recorded commit.)
 - **Follow symlinked directories.**
 
 ## Usage
@@ -78,6 +85,8 @@ pullr [options] [DIR]
 | `--max-depth N` | `2` | How many directory levels below `DIR` to search. `0` = only `DIR` itself, `1` = `DIR` and its direct children, ... |
 | `-n`, `--dry-run` | | Show what a run would do without pulling, see below |
 | `-r`, `--rebase` | off | Rebase local commits onto the upstream instead of refusing a diverged branch (`git pull --rebase`). A rebase that hits a conflict is aborted and the repository is left as it was. |
+| `--no-autostash` | | Don't stash local changes that are in the way of an update; report the repository as dirty instead. Autostash is on by default, see "What it will never do". |
+| `--submodules` | off | Also fast-forward each submodule that is checked out on a branch, see below |
 | `--no-color` | | Disable colored output. Also disabled when the [`NO_COLOR`](https://no-color.org) environment variable is set, or when output is not a terminal. |
 | `-V`, `--version` | | Print the version |
 | `-h`, `--help` | | Show help |
@@ -85,7 +94,7 @@ pullr [options] [DIR]
 ### Output
 
 Each repository gets one line: `+` updated, `=` already up to date, `~`
-dirty (local changes are in the way of the update, which is held back), `-`
+dirty (local changes conflict with the update, which is held back), `-`
 skipped, `x` failed (with the reason indented below it). Exits `1` if any
 pull failed, `2` on invalid arguments, `0` otherwise; a dirty repository is
 not a failure.
@@ -108,12 +117,34 @@ Failed: infra
 ```
 
 It runs `git fetch` in each repository first, so the answer reflects the
-remote as it is now, then prints the same lines a real run would (with
-`--rebase`, a diverged repository shows as `+ ... , 2 to rebase` instead of
-`x`; whether the rebase would conflict is only known by running it). Fetching
+remote as it is now, then prints the same lines a real run would. An update
+that would need to stash local changes shows as `+ ... , local changes to
+autostash`, and with `--rebase` a diverged repository shows as
+`+ ... , 2 to rebase` instead of `x`; whether re-applying or rebasing would
+conflict is only known by running it. Fetching
 only updates remote-tracking refs: the working tree and current branch are
 never modified. Exit codes match a real run.
 
+### Submodules
+
+By default pullr updates a repository the way `git pull` does: submodules stay
+where they are, unless you set git's `submodule.recurse`, in which case they
+are checked out at the commits the parent now records, as `git pull` would.
+
+`--submodules` is for working inside submodules. Each submodule that is
+checked out on a branch is treated as a repository of its own - fetched,
+fast-forwarded with the same rules, and listed under its parent:
+
+```console
+$ pullr --submodules ~/work
++ app (main, 2 new commits)
+  + app/lib/core (main, 5 new commits)
+  - app/vendor/sdk (detached HEAD)
+```
+
+Submodules on a detached HEAD (git's default after `git submodule update`) are
+skipped, and uninitialized ones are left out.
+
 ## How it compares
 
 | | pullr | [gita](https://github.com/nosarthur/gita) | [gitup](https://github.com/earwig/git-repo-updater) | [myrepos](https://myrepos.branchable.com) (`mr`) | [mani](https://github.com/alajmo/mani) |
@@ -123,7 +154,7 @@ never modified. Exit codes match a real run.
 | Pulls in parallel | 8 at a time by default | all at once, no limit | no | opt-in | opt-in |
 | Update | fetch + fast‑forward | `git pull` | fetch + fast‑forward | `git pull` | no built-in pull |
 | Merge commits | never | depends on your git config | never | depends on your git config | depends on your command |
-| Local changes in the way | reported as dirty | git's error | skipped | git's error | - |
+| Local changes in the way | stashed and reapplied; rolled back on conflict | git's error | skipped | git's error | - |
 | Dry run | yes | no | no | no | prints commands |
 
 **Why "8 at a time" and not "all at once".** Many self-hosted git servers

package/bin/pullr

@@ -1,9 +1,9 @@
 #!/usr/bin/env bash
 # Recursively find git repos under a directory and fast-forward pull each one.
-# Usage: pullr [--jobs N] [--max-depth N] [--dry-run] [--rebase] [--no-color] [DIR]
+# Usage: pullr [--jobs N] [--max-depth N] [--dry-run] [--rebase] [--no-autostash] [--submodules] [--no-color] [DIR]
 set -euo pipefail
 
-VERSION="0.6.0" # set by the release workflow
+VERSION="0.8.0" # set by the release workflow
 
 usage() {
   cat <<'EOF'
@@ -24,6 +24,14 @@ Options:
                      refusing a branch that has diverged (git pull --rebase).
                      A rebase that hits a conflict is aborted, leaving the
                      repository as it was.
+      --no-autostash Don't stash local changes that are in the way of an
+                     update. By default they are stashed, the update is
+                     made and they are re-applied; if re-applying conflicts,
+                     the repository is put back exactly as it was.
+      --autostash    The default; accepted for scripts that want to be explicit.
+      --submodules   Also fast-forward each submodule that is checked out on a
+                     branch, with the same rules as any other repository.
+                     Submodules on a detached HEAD are skipped.
       --no-color     Disable colored output. Also disabled when NO_COLOR is
                      set (https://no-color.org) or output is not a terminal.
   -V, --version      Print the version.
@@ -34,6 +42,8 @@ EOF
 mode=pull
 jobs=8
 rebase=0
+autostash=1
+submodules=0
 color=1
 max_depth=2
 root=""
@@ -46,6 +56,9 @@ while (($#)); do
     --max-depth=*) max_depth="${1#*=}"; shift ;;
     -n|--dry-run) mode=dryrun; shift ;;
     -r|--rebase) rebase=1; shift ;;
+    --autostash) autostash=1; shift ;;
+    --no-autostash) autostash=0; shift ;;
+    --submodules) submodules=1; shift ;;
     --no-color) color=0; shift ;;
     -h|--help) usage; exit 0 ;;
     -V|--version) echo "pullr $VERSION"; exit 0 ;;
@@ -120,8 +133,8 @@ report_diverged() {
 
 report_dirty() {
   printf 'dirty\t%s\n' "$1"
-  printf '%s~%s %s %s(%s, %s held back by local changes)%s\n' \
-    "$yellow" "$reset" "$1" "$dim" "$2" "$(commits "$3")" "$reset"
+  printf '%s~%s %s %s(%s, %s held back %s)%s\n' \
+    "$yellow" "$reset" "$1" "$dim" "$2" "$(commits "$3")" "${4:-by local changes}" "$reset"
 }
 
 report_current() {
@@ -133,6 +146,7 @@ report_updated() {
   local note
   note="$3 new commit$([[ "$3" == 1 ]] || printf s)"
   [[ "$4" == 0 ]] || note+=", $4 $5"
+  [[ -z "${6:-}" ]] || note+=", $6"
   echo updated
   printf '%s+%s %s %s(%s, %s)%s\n' "$green" "$reset" "$1" "$dim" "$2" "$note" "$reset"
 }
@@ -142,9 +156,24 @@ commits() {
   printf '%s commit%s' "$1" "$([[ "$1" == 1 ]] || printf s)"
 }
 
-# Uncommitted changes to tracked files, staged or not.
+# Uncommitted changes to tracked files, staged or not. A submodule checked out
+# at a different commit than the one recorded is not one: git neither refuses
+# to update over it nor touches it.
 has_tracked_changes() {
-  ! git -C "$1" diff --quiet HEAD --
+  ! git -C "$1" diff --quiet --ignore-submodules HEAD --
+}
+
+has_local_changes() {
+  [[ -n "$(git -C "$1" status --porcelain --ignore-submodules=all)" ]]
+}
+
+# Mirror `git pull` for people who set submodule.recurse: after the update,
+# check submodules out at the commits the parent now records. With
+# --submodules each one is fast-forwarded on its own branch instead.
+sync_submodules() {
+  [[ "$submodules" == 0 ]] || return 0
+  [[ "$(git -C "$1" config --type=bool --get submodule.recurse 2>/dev/null)" == true ]] || return 0
+  git -C "$1" submodule update --recursive 2>&1
 }
 
 pull_repo() {
@@ -157,24 +186,28 @@ pull_repo() {
   elif [[ "$ahead" == 0 ]]; then
     # git refuses, touching nothing, when local changes are in the way
     if out="$(git -C "$repo" merge --ff-only --quiet '@{upstream}' 2>&1)"; then
-      report_updated "$rel" "$branch" "$behind" 0 ""
-    elif [[ -n "$(git -C "$repo" status --porcelain)" ]]; then
+      report_synced "$repo" "$rel" "$branch" "$behind" 0 ""
+    elif has_local_changes "$repo" && [[ "$autostash" == 1 ]]; then
+      autostash_update "$repo" "$rel" "$branch" "$behind" 0 ff
+    elif has_local_changes "$repo"; then
       report_dirty "$rel" "$branch" "$behind"
     else
       report_failed "$rel" "$branch" "$out"
     fi
   elif [[ "$rebase" == 0 ]]; then
     report_diverged "$rel" "$branch" "$behind" "$ahead"
+  elif has_tracked_changes "$repo" && [[ "$autostash" == 1 ]]; then
+    autostash_update "$repo" "$rel" "$branch" "$behind" "$ahead" rebase
   elif has_tracked_changes "$repo"; then
     report_dirty "$rel" "$branch" "$behind"
   elif out="$(git -C "$repo" rebase --quiet '@{upstream}' 2>&1)"; then
-    report_updated "$rel" "$branch" "$behind" "$ahead" rebased
+    report_synced "$repo" "$rel" "$branch" "$behind" "$ahead" rebased
   else
     # A rebase that stopped on a conflict must not leave the repository
     # mid-rebase; --abort succeeds only when one is in progress.
     if git -C "$repo" rebase --abort >/dev/null 2>&1; then
       report_failed "$rel" "$branch" "$out"
-    elif [[ -n "$(git -C "$repo" status --porcelain)" ]]; then
+    elif has_local_changes "$repo"; then
       report_dirty "$rel" "$branch" "$behind"
     else
       report_failed "$rel" "$branch" "$out"
@@ -182,12 +215,98 @@ pull_repo() {
   fi
 }
 
+# Local changes are in the way of the update ($6: ff or rebase). Stash them,
+# update, and re-apply them. If re-applying is anything but clean, put the
+# repository back exactly as it was and report it as dirty. The stash is only
+# dropped once its changes are back in the working tree.
+autostash_update() {
+  local repo="$1" rel="$2" branch="$3" behind="$4" ahead="$5" how="$6"
+  local before stash out
+  before="$(git -C "$repo" rev-parse HEAD)"
+  if ! out="$(git -C "$repo" stash push --include-untracked --quiet -m "pullr autostash" 2>&1)" ||
+    ! stash="$(git -C "$repo" rev-parse --verify --quiet refs/stash)"; then
+    report_failed "$rel" "$branch" "could not stash local changes:"$'\n'"$out"
+    return
+  fi
+
+  if [[ "$how" == ff ]]; then
+    out="$(git -C "$repo" merge --ff-only --quiet '@{upstream}' 2>&1)"
+  else
+    out="$(git -C "$repo" rebase --quiet '@{upstream}' 2>&1)"
+  fi || {
+    # Not caused by local changes - they were stashed. Undo, then restore
+    # them onto the commit they came from, where they apply cleanly.
+    git -C "$repo" rebase --abort >/dev/null 2>&1 || true
+    if restore_stash "$repo" "$stash"; then
+      report_failed "$rel" "$branch" "$out"
+    else
+      report_failed "$rel" "$branch" "$out"$'\n'"$(stash_kept "$repo")"
+    fi
+    return
+  }
+
+  if git -C "$repo" stash apply --index --quiet "$stash" >/dev/null 2>&1; then
+    drop_stash "$repo" "$stash"
+    local verb=""
+    [[ "$how" == ff ]] || verb=rebased
+    report_synced "$repo" "$rel" "$branch" "$behind" "$([[ "$how" == ff ]] && echo 0 || echo "$ahead")" "$verb" "local changes reapplied"
+    return
+  fi
+
+  # Conflict: go back to where we started.
+  git -C "$repo" reset --hard --quiet "$before"
+  clear_restored_untracked "$repo" "$stash"
+  if restore_stash "$repo" "$stash"; then
+    report_dirty "$rel" "$branch" "$behind" "because local changes conflict with them"
+  else
+    report_failed "$rel" "$branch" "could not re-apply local changes after an update conflict."$'\n'"$(stash_kept "$repo")"
+  fi
+}
+
+# Re-apply a stash onto the commit it was made from, then drop it.
+restore_stash() {
+  git -C "$1" stash apply --index --quiet "$2" >/dev/null 2>&1 || return 1
+  drop_stash "$1" "$2"
+}
+
+# `git stash drop` takes stash@{n}, not a commit: find the entry that is ours.
+drop_stash() {
+  local ref
+  ref="$(git -C "$1" stash list --format='%gd %H' | awk -v c="$2" '$2 == c { print $1; exit }')"
+  [[ -z "$ref" ]] || git -C "$1" stash drop --quiet "$ref" >/dev/null 2>&1 || true
+}
+
+# A failed `stash apply` may already have written the stash's untracked files
+# back. They were moved into the stash, so anything at those paths now came
+# from that apply; remove them so the stash can be applied again.
+clear_restored_untracked() {
+  local path
+  git -C "$1" rev-parse --verify --quiet "$2^3" >/dev/null || return 0
+  while IFS= read -r path; do
+    [[ -z "$path" ]] || rm -f -- "$1/$path"
+  done < <(git -C "$1" ls-tree -r --name-only "$2^3")
+}
+
+stash_kept() {
+  printf 'Your local changes are safe in the stash (git -C %q stash list).' "$1"
+}
+
+# The repository was updated; report it, unless syncing submodules failed.
+report_synced() {
+  local out
+  if out="$(sync_submodules "$1")"; then
+    report_updated "$2" "$3" "$4" "$5" "$6" "${7:-}"
+  else
+    report_failed "$2" "$3" "updated, but updating submodules failed:"$'\n'"$out"
+  fi
+}
+
 # Would local changes stop a fast-forward? Git refuses when a file it has to
 # update is modified locally or exists untracked, so compare the two lists.
 changes_block_update() {
   local repo="$1" incoming local_paths
   incoming="$(git -C "$repo" diff --name-only HEAD '@{upstream}')"
-  local_paths="$(git -C "$repo" status --porcelain --untracked-files=all | cut -c4-)"
+  local_paths="$(git -C "$repo" status --porcelain --untracked-files=all --ignore-submodules=all | cut -c4-)"
   [[ -n "$incoming" && -n "$local_paths" ]] || return 1
   printf '%s\n' "$local_paths" | grep -qFxf <(printf '%s\n' "$incoming")
 }
@@ -200,13 +319,17 @@ dryrun_repo() {
   if [[ "$behind" == 0 ]]; then
     report_current "$rel" "$branch"
   elif [[ "$ahead" == 0 ]]; then
-    if changes_block_update "$repo"; then
+    if changes_block_update "$repo" && [[ "$autostash" == 1 ]]; then
+      report_updated "$rel" "$branch" "$behind" 0 "" "local changes to autostash"
+    elif changes_block_update "$repo"; then
       report_dirty "$rel" "$branch" "$behind"
     else
       report_updated "$rel" "$branch" "$behind" 0 ""
     fi
   elif [[ "$rebase" == 0 ]]; then
     report_diverged "$rel" "$branch" "$behind" "$ahead"
+  elif has_tracked_changes "$repo" && [[ "$autostash" == 1 ]]; then
+    report_updated "$rel" "$branch" "$behind" "$ahead" "to rebase" "local changes to autostash"
   elif has_tracked_changes "$repo"; then
     report_dirty "$rel" "$branch" "$behind"
   else
@@ -225,11 +348,12 @@ worker() {
 
 # --- discovery ---------------------------------------------------------------
 
-repos=()
+repos=() nested=()
 walk() {
   local dir="$1" depth="$2"
   if [[ -e "$dir/.git" ]]; then
-    repos+=("$dir")
+    repos+=("$dir") nested+=(0)
+    [[ "$submodules" == 0 ]] || add_submodules "$dir"
     return
   fi
   ((depth < max_depth)) || return 0
@@ -239,6 +363,17 @@ walk() {
     walk "${child%/}" $((depth + 1))
   done
 }
+# Checked-out submodules, recursively, in git's order; uninitialized ones
+# have no checkout to update and are left out.
+add_submodules() {
+  local sub
+  # shellcheck disable=SC2016  # $displaypath is expanded by git submodule foreach
+  while IFS= read -r sub; do
+    [[ -n "$sub" ]] || continue
+    repos+=("$1/$sub") nested+=(1)
+  done < <(git -C "$1" submodule foreach --quiet --recursive 'printf "%s\n" "$displaypath"' 2>/dev/null)
+}
+
 walk "$root" 0
 
 count=${#repos[@]}
@@ -256,6 +391,8 @@ failed_repos=() dirty_repos=()
 
 consume() {
   local text="$1" head="${1%%$'\n'*}" body="${1#*$'\n'}"
+  # a submodule's lines sit under its parent's
+  [[ "${2:-0}" == 0 ]] || body="  ${body//$'\n'/$'\n'  }"
   local category="${head%%$'\t'*}" label="${head#*$'\t'}"
   case "$category" in
     updated) updated=$((updated + 1)) ;;
@@ -272,8 +409,8 @@ if ((jobs > 1 && count > 1)); then
   scratch="$(mktemp -d)"
   trap 'rm -rf "$scratch"' EXIT
   export GIT_TERMINAL_PROMPT=0
-  export mode rebase root green yellow red dim reset
-  export -f worker pull_repo dryrun_repo repo_label indented prepare report_failed report_diverged report_dirty report_current report_updated commits has_tracked_changes changes_block_update
+  export mode rebase autostash submodules root green yellow red dim reset
+  export -f worker pull_repo dryrun_repo repo_label indented prepare report_failed report_diverged report_dirty report_current report_updated report_synced autostash_update restore_stash drop_stash clear_restored_untracked stash_kept commits has_tracked_changes has_local_changes sync_submodules changes_block_update
   # shellcheck disable=SC2016  # the single-quoted body runs inside the child shell
   for ((i = 0; i < count; i++)); do printf '%s\0%s\0' "$i" "${repos[i]}"; done |
     xargs -0 -n 2 -P "$jobs" "$BASH" -c 'worker "$@"' _ "$scratch" &
@@ -287,12 +424,12 @@ if ((jobs > 1 && count > 1)); then
       fi
       sleep 0.1
     done
-    consume "$(cat "$scratch/$i.done")"
+    consume "$(cat "$scratch/$i.done")" "${nested[i]}"
   done
   wait "$pool" || true
 else
-  for repo in "${repos[@]}"; do
-    consume "$("${mode}_repo" "$repo")"
+  for ((i = 0; i < count; i++)); do
+    consume "$("${mode}_repo" "${repos[i]}")" "${nested[i]}"
   done
 fi
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kova1/pullr",
-  "version": "0.6.0",
+  "version": "0.8.0",
   "description": "Safe, fast-forward pull every git repository under a directory",
   "keywords": [
     "git",