@kova1/pullr 0.4.0 → 0.6.0

package/README.md

@@ -5,7 +5,7 @@
 [![Homebrew](https://img.shields.io/github/v/release/kova1max/pullr?logo=homebrew&label=homebrew)](https://github.com/kova1max/homebrew-tap)
 [![License: MIT](https://img.shields.io/github/license/kova1max/pullr)](LICENSE)
 
-Fast-forward pull every git repository under a directory.
+Safe, fast-forward pull every git repository under a directory.
 
 Built for workspaces made of many independent repositories, where keeping
 everything current means `cd`-ing into each one.
@@ -14,11 +14,13 @@ everything current means `cd`-ing into each one.
 $ pullr ~/work
 + api (main, 3 new commits)
 = web (main)
+~ docs (main, 2 commits held back by local changes)
 - scratch (main has no upstream)
 x infra (main)
-    fatal: Not possible to fast-forward, aborting.
+    diverged from upstream (behind 2, ahead 1): cannot fast-forward
 
-Repos: 4  updated: 1  up to date: 1  skipped: 1  failed: 1
+Repos: 5  updated: 1  up to date: 1  dirty: 1  skipped: 1  failed: 1
+Dirty: docs
 Failed: infra
 ```
 
@@ -38,9 +40,6 @@ To try it without installing anything:
 npx @kova1/pullr --dry-run ~/work
 ```
 
-Or copy [`bin/pullr`](bin/pullr) anywhere on your `PATH` - it is a single bash
-script with no dependencies beyond `git`.
-
 Every release is also mirrored to
 [GitHub Packages](https://github.com/kova1max/pullr/pkgs/npm/pullr) as
 `@kova1max/pullr` (GitHub requires the repo owner as scope). Installing from
@@ -56,8 +55,10 @@ first. It will never:
   failed and left exactly as it was. Rebasing it is opt-in with `--rebase`,
   and a rebase that hits a conflict is aborted, leaving the repository as it
   was.
-- **Stash or discard your changes.** There is no auto-stash; if git refuses
-  to update a dirty working tree, the repository is reported as failed.
+- **Stash, overwrite or discard your changes.** There is no auto-stash. If
+  local edits or untracked files are in the way of incoming changes, the
+  repository is reported as dirty and left as it was; edits that don't touch
+  incoming files don't stop the update.
 - **Touch a repository on a detached `HEAD`**, or a branch with no upstream.
   Both are skipped.
 - **Enter a nested repository.** Repositories inside another repository,
@@ -70,10 +71,10 @@ first. It will never:
 pullr [options] [DIR]
 ```
 
-| Option | Default | Meaning |
-| --- | --- | --- |
-| `DIR` | current directory | Where to look for repositories |
-| `-j`, `--jobs N` | `4` | Work on up to `N` repositories in parallel. Output is buffered per repository and printed in discovery order, so it reads the same as a sequential run. With `N > 1` git never prompts for credentials; use `--jobs 1` for repositories that need an interactive login. |
+| Option                       | Default | Meaning |
+| :--- | :--- | :--- |
+| `DIR` | current directory | Where to look for repositories |
+| `-j`, `--jobs N` | `8` | Work on up to `N` repositories in parallel. Output is buffered per repository and printed in discovery order, so it reads the same as a sequential run. With `N > 1` git never prompts for credentials; use `--jobs 1` for repositories that need an interactive login. |
 | `--max-depth N` | `2` | How many directory levels below `DIR` to search. `0` = only `DIR` itself, `1` = `DIR` and its direct children, ... |
 | `-n`, `--dry-run` | | Show what a run would do without pulling, see below |
 | `-r`, `--rebase` | off | Rebase local commits onto the upstream instead of refusing a diverged branch (`git pull --rebase`). A rebase that hits a conflict is aborted and the repository is left as it was. |
@@ -83,9 +84,11 @@ pullr [options] [DIR]
 
 ### Output
 
-Each repository gets one line: `+` updated, `=` already up to date, `-`
-skipped, `x` failed (with git's output indented below it). Exits `1` if any
-pull failed, `2` on invalid arguments, `0` otherwise.
+Each repository gets one line: `+` updated, `=` already up to date, `~`
+dirty (local changes are in the way of the update, which is held back), `-`
+skipped, `x` failed (with the reason indented below it). Exits `1` if any
+pull failed, `2` on invalid arguments, `0` otherwise; a dirty repository is
+not a failure.
 
 ### Dry run
 
@@ -100,7 +103,7 @@ x infra (main)
     diverged from upstream (behind 2, ahead 1): cannot fast-forward
 - scratch (main has no upstream)
 
-Repos: 4  would update: 1  up to date: 1  skipped: 1  would fail: 1
+Repos: 4  would update: 1  up to date: 1  dirty: 0  skipped: 1  would fail: 1
 Failed: infra
 ```
 
@@ -111,6 +114,27 @@ remote as it is now, then prints the same lines a real run would (with
 only updates remote-tracking refs: the working tree and current branch are
 never modified. Exit codes match a real run.
 
+## How it compares
+
+| | pullr | [gita](https://github.com/nosarthur/gita) | [gitup](https://github.com/earwig/git-repo-updater) | [myrepos](https://myrepos.branchable.com) (`mr`) | [mani](https://github.com/alajmo/mani) |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| Needs | bash and git | Python 3.8+ | Python 3.10+ | Perl | Go binary |
+| Finds repos | scans a directory | register first (`gita add`) | scans a directory | register first (`~/.mrconfig`) | register first (`mani.yaml`) |
+| Pulls in parallel | 8 at a time by default | all at once, no limit | no | opt-in | opt-in |
+| Update | fetch + fast‑forward | `git pull` | fetch + fast‑forward | `git pull` | no built-in pull |
+| Merge commits | never | depends on your git config | never | depends on your git config | depends on your command |
+| Local changes in the way | reported as dirty | git's error | skipped | git's error | - |
+| Dry run | yes | no | no | no | prints commands |
+
+**Why "8 at a time" and not "all at once".** Many self-hosted git servers
+limit how many SSH connections can be opening at the same time; OpenSSH's
+default (`MaxStartups 10:30:100`) starts refusing them past 10. Fetching
+every repository at once then fails for reasons that have nothing to do with
+your repositories. On a self-hosted GitLab with 61 repositories, 16 parallel
+fetches produced 6-9 of those spurious failures and 32 produced 10-14, while
+8 produced none. GitHub itself handled 64 at once without errors, so raise
+`-j` freely there.
+
 ## License
 
 [MIT](LICENSE)

package/bin/pullr

@@ -3,7 +3,7 @@
 # Usage: pullr [--jobs N] [--max-depth N] [--dry-run] [--rebase] [--no-color] [DIR]
 set -euo pipefail
 
-VERSION="0.4.0" # set by the release workflow
+VERSION="0.6.0" # set by the release workflow
 
 usage() {
   cat <<'EOF'
@@ -13,7 +13,7 @@ Finds git repositories under DIR (default: current directory) and runs
 `git pull --ff-only` in each. Does not descend into a repository once found.
 
 Options:
-  -j, --jobs N       Work on up to N repositories in parallel (default: 4).
+  -j, --jobs N       Work on up to N repositories in parallel (default: 8).
                      Output is buffered per repository and printed in order.
                      With N > 1 git never prompts for credentials.
       --max-depth N  How many directory levels below DIR to search (default: 2).
@@ -32,7 +32,7 @@ EOF
 }
 
 mode=pull
-jobs=4
+jobs=8
 rebase=0
 color=1
 max_depth=2
@@ -84,96 +84,133 @@ indented() {
   printf '    %s\n' "${1//$'\n'/$'\n'    }"
 }
 
-pull_repo() {
-  local repo="$1" rel branch
-  rel="$(repo_label "$repo")"
-
+# Shared first half of a pull and a dry run: check the branch, fetch, and
+# count commits. On success it sets the caller's branch, behind and ahead;
+# otherwise it prints the result line and returns 1.
+prepare() {
+  local repo="$1" rel="$2" out counts
   if ! branch="$(git -C "$repo" symbolic-ref --quiet --short HEAD)"; then
     echo skipped
     printf '%s-%s %s %s(detached HEAD)%s\n' "$yellow" "$reset" "$rel" "$dim" "$reset"
-    return
+    return 1
   fi
   if ! git -C "$repo" rev-parse --quiet --verify '@{upstream}' >/dev/null 2>&1; then
     echo skipped
     printf '%s-%s %s %s(%s has no upstream)%s\n' "$yellow" "$reset" "$rel" "$dim" "$branch" "$reset"
-    return
+    return 1
   fi
-
-  local before after out how=--ff-only
-  [[ "$rebase" == 0 ]] || how=--rebase
-  before="$(git -C "$repo" rev-parse HEAD)"
-  if ! out="$(git -C "$repo" pull "$how" 2>&1)"; then
-    # a conflicting rebase must not leave the repository mid-rebase
-    [[ "$rebase" == 0 ]] || git -C "$repo" rebase --abort >/dev/null 2>&1 || true
-    printf 'failed\t%s\n' "$rel"
-    printf '%sx%s %s %s(%s)%s\n' "$red" "$reset" "$rel" "$dim" "$branch" "$reset"
-    indented "$out"
-    return
+  if ! out="$(git -C "$repo" fetch --quiet 2>&1)"; then
+    report_failed "$rel" "$branch" "$out"
+    return 1
   fi
-  after="$(git -C "$repo" rev-parse HEAD)"
+  counts="$(git -C "$repo" rev-list --left-right --count '@{upstream}...HEAD')"
+  behind="${counts%%[[:space:]]*}"
+  ahead="${counts##*[[:space:]]}"
+}
 
-  if [[ "$before" == "$after" ]]; then
-    echo current
-    printf '%s=%s %s %s(%s)%s\n' "$dim" "$reset" "$rel" "$dim" "$branch" "$reset"
-    return
-  fi
-  # New upstream commits, and (after a rebase) local commits now on top of them.
-  local n local_n=0
-  n="$(git -C "$repo" rev-list --count "$before..@{upstream}")"
-  [[ "$rebase" == 0 ]] || local_n="$(git -C "$repo" rev-list --count '@{upstream}..HEAD')"
+report_failed() {
+  printf 'failed\t%s\n' "$1"
+  printf '%sx%s %s %s(%s)%s\n' "$red" "$reset" "$1" "$dim" "$2" "$reset"
+  indented "$3"
+}
+
+report_diverged() {
+  report_failed "$1" "$2" "diverged from upstream (behind $3, ahead $4): cannot fast-forward"
+}
+
+report_dirty() {
+  printf 'dirty\t%s\n' "$1"
+  printf '%s~%s %s %s(%s, %s held back by local changes)%s\n' \
+    "$yellow" "$reset" "$1" "$dim" "$2" "$(commits "$3")" "$reset"
+}
+
+report_current() {
+  echo current
+  printf '%s=%s %s %s(%s)%s\n' "$dim" "$reset" "$1" "$dim" "$2" "$reset"
+}
+
+report_updated() {
+  local note
+  note="$3 new commit$([[ "$3" == 1 ]] || printf s)"
+  [[ "$4" == 0 ]] || note+=", $4 $5"
   echo updated
-  printf '%s+%s %s %s(%s, %s)%s\n' "$green" "$reset" "$rel" "$dim" "$branch" \
-    "$(update_note "$n" "$local_n" rebased)" "$reset"
+  printf '%s+%s %s %s(%s, %s)%s\n' "$green" "$reset" "$1" "$dim" "$2" "$note" "$reset"
 }
 
-# "N new commits[, M rebased|to rebase]", singular when N is 1
-update_note() {
-  local n="$1" local_n="$2" verb="$3" note
-  note="$n new commit$([[ "$n" == 1 ]] || printf s)"
-  [[ "$local_n" == 0 ]] || note+=", $local_n $verb"
-  printf '%s' "$note"
+# "1 commit", "3 commits"
+commits() {
+  printf '%s commit%s' "$1" "$([[ "$1" == 1 ]] || printf s)"
 }
 
-dryrun_repo() {
-  local repo="$1" rel branch
+# Uncommitted changes to tracked files, staged or not.
+has_tracked_changes() {
+  ! git -C "$1" diff --quiet HEAD --
+}
+
+pull_repo() {
+  local repo="$1" rel branch behind ahead out
   rel="$(repo_label "$repo")"
+  prepare "$repo" "$rel" || return 0
 
-  if ! branch="$(git -C "$repo" symbolic-ref --quiet --short HEAD)"; then
-    echo skipped
-    printf '%s-%s %s %s(detached HEAD)%s\n' "$yellow" "$reset" "$rel" "$dim" "$reset"
-    return
-  fi
-  if ! git -C "$repo" rev-parse --quiet --verify '@{upstream}' >/dev/null 2>&1; then
-    echo skipped
-    printf '%s-%s %s %s(%s has no upstream)%s\n' "$yellow" "$reset" "$rel" "$dim" "$branch" "$reset"
-    return
+  if [[ "$behind" == 0 ]]; then
+    report_current "$rel" "$branch"
+  elif [[ "$ahead" == 0 ]]; then
+    # git refuses, touching nothing, when local changes are in the way
+    if out="$(git -C "$repo" merge --ff-only --quiet '@{upstream}' 2>&1)"; then
+      report_updated "$rel" "$branch" "$behind" 0 ""
+    elif [[ -n "$(git -C "$repo" status --porcelain)" ]]; then
+      report_dirty "$rel" "$branch" "$behind"
+    else
+      report_failed "$rel" "$branch" "$out"
+    fi
+  elif [[ "$rebase" == 0 ]]; then
+    report_diverged "$rel" "$branch" "$behind" "$ahead"
+  elif has_tracked_changes "$repo"; then
+    report_dirty "$rel" "$branch" "$behind"
+  elif out="$(git -C "$repo" rebase --quiet '@{upstream}' 2>&1)"; then
+    report_updated "$rel" "$branch" "$behind" "$ahead" rebased
+  else
+    # A rebase that stopped on a conflict must not leave the repository
+    # mid-rebase; --abort succeeds only when one is in progress.
+    if git -C "$repo" rebase --abort >/dev/null 2>&1; then
+      report_failed "$rel" "$branch" "$out"
+    elif [[ -n "$(git -C "$repo" status --porcelain)" ]]; then
+      report_dirty "$rel" "$branch" "$behind"
+    else
+      report_failed "$rel" "$branch" "$out"
+    fi
   fi
+}
 
-  local out
-  if ! out="$(git -C "$repo" fetch --quiet 2>&1)"; then
-    printf 'failed\t%s\n' "$rel"
-    printf '%sx%s %s %s(%s)%s\n' "$red" "$reset" "$rel" "$dim" "$branch" "$reset"
-    indented "$out"
-    return
-  fi
+# Would local changes stop a fast-forward? Git refuses when a file it has to
+# update is modified locally or exists untracked, so compare the two lists.
+changes_block_update() {
+  local repo="$1" incoming local_paths
+  incoming="$(git -C "$repo" diff --name-only HEAD '@{upstream}')"
+  local_paths="$(git -C "$repo" status --porcelain --untracked-files=all | cut -c4-)"
+  [[ -n "$incoming" && -n "$local_paths" ]] || return 1
+  printf '%s\n' "$local_paths" | grep -qFxf <(printf '%s\n' "$incoming")
+}
 
-  local counts behind ahead
-  counts="$(git -C "$repo" rev-list --left-right --count '@{upstream}...HEAD')"
-  behind="${counts%%[[:space:]]*}"
-  ahead="${counts##*[[:space:]]}"
+dryrun_repo() {
+  local repo="$1" rel branch behind ahead
+  rel="$(repo_label "$repo")"
+  prepare "$repo" "$rel" || return 0
 
-  if [[ "$behind" != 0 && "$ahead" != 0 && "$rebase" == 0 ]]; then
-    printf 'failed\t%s\n' "$rel"
-    printf '%sx%s %s %s(%s)%s\n' "$red" "$reset" "$rel" "$dim" "$branch" "$reset"
-    indented "diverged from upstream (behind $behind, ahead $ahead): cannot fast-forward"
-  elif [[ "$behind" != 0 ]]; then
-    # without --rebase this branch is only reached when ahead is 0
-    echo updated
-    printf '%s+%s %s %s(%s, %s)%s\n' "$green" "$reset" "$rel" "$dim" "$branch" \
-      "$(update_note "$behind" "$ahead" "to rebase")" "$reset"
+  if [[ "$behind" == 0 ]]; then
+    report_current "$rel" "$branch"
+  elif [[ "$ahead" == 0 ]]; then
+    if changes_block_update "$repo"; then
+      report_dirty "$rel" "$branch" "$behind"
+    else
+      report_updated "$rel" "$branch" "$behind" 0 ""
+    fi
+  elif [[ "$rebase" == 0 ]]; then
+    report_diverged "$rel" "$branch" "$behind" "$ahead"
+  elif has_tracked_changes "$repo"; then
+    report_dirty "$rel" "$branch" "$behind"
   else
-    echo current
-    printf '%s=%s %s %s(%s)%s\n' "$dim" "$reset" "$rel" "$dim" "$branch" "$reset"
+    report_updated "$rel" "$branch" "$behind" "$ahead" "to rebase"
   fi
 }
 
@@ -214,8 +251,8 @@ fi
 
 [[ "$mode" != dryrun ]] || printf '%sDry run: nothing will be pulled.%s\n' "$dim" "$reset"
 
-updated=0 current=0 skipped=0 failed=0
-failed_repos=()
+updated=0 current=0 skipped=0 failed=0 dirty=0
+failed_repos=() dirty_repos=()
 
 consume() {
   local text="$1" head="${1%%$'\n'*}" body="${1#*$'\n'}"
@@ -224,6 +261,7 @@ consume() {
     updated) updated=$((updated + 1)) ;;
     current) current=$((current + 1)) ;;
     skipped) skipped=$((skipped + 1)) ;;
+    dirty) dirty=$((dirty + 1)); dirty_repos+=("$label") ;;
     failed) failed=$((failed + 1)); failed_repos+=("$label") ;;
     *) echo "pullr: unexpected worker result: $text" >&2; exit 1 ;;
   esac
@@ -235,7 +273,7 @@ if ((jobs > 1 && count > 1)); then
   trap 'rm -rf "$scratch"' EXIT
   export GIT_TERMINAL_PROMPT=0
   export mode rebase root green yellow red dim reset
-  export -f worker pull_repo dryrun_repo repo_label indented update_note
+  export -f worker pull_repo dryrun_repo repo_label indented prepare report_failed report_diverged report_dirty report_current report_updated commits has_tracked_changes changes_block_update
   # shellcheck disable=SC2016  # the single-quoted body runs inside the child shell
   for ((i = 0; i < count; i++)); do printf '%s\0%s\0' "$i" "${repos[i]}"; done |
     xargs -0 -n 2 -P "$jobs" "$BASH" -c 'worker "$@"' _ "$scratch" &
@@ -260,12 +298,13 @@ fi
 
 echo
 if [[ "$mode" == dryrun ]]; then
-  printf 'Repos: %d  %swould update: %d%s  up to date: %d  %sskipped: %d%s  %swould fail: %d%s\n' \
-    "$count" "$green" "$updated" "$reset" "$current" "$yellow" "$skipped" "$reset" "$red" "$failed" "$reset"
+  printf 'Repos: %d  %swould update: %d%s  up to date: %d  %sdirty: %d  skipped: %d%s  %swould fail: %d%s\n' \
+    "$count" "$green" "$updated" "$reset" "$current" "$yellow" "$dirty" "$skipped" "$reset" "$red" "$failed" "$reset"
 else
-  printf 'Repos: %d  %supdated: %d%s  up to date: %d  %sskipped: %d%s  %sfailed: %d%s\n' \
-    "$count" "$green" "$updated" "$reset" "$current" "$yellow" "$skipped" "$reset" "$red" "$failed" "$reset"
+  printf 'Repos: %d  %supdated: %d%s  up to date: %d  %sdirty: %d  skipped: %d%s  %sfailed: %d%s\n' \
+    "$count" "$green" "$updated" "$reset" "$current" "$yellow" "$dirty" "$skipped" "$reset" "$red" "$failed" "$reset"
 fi
+((dirty == 0)) || printf 'Dirty: %s\n' "${dirty_repos[*]}"
 if ((failed)); then
   printf 'Failed: %s\n' "${failed_repos[*]}"
   exit 1

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@kova1/pullr",
-  "version": "0.4.0",
-  "description": "Fast-forward pull every git repository under a directory",
+  "version": "0.6.0",
+  "description": "Safe, fast-forward pull every git repository under a directory",
   "keywords": [
     "git",
     "pull",