fun_ci 2.3.0 → 2.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b55c9059ad2ffd38aa87dd4dd9958631a14390bcae0e0e7577e78bd83d4c530f
-  data.tar.gz: 6c40537e0a4eb9425dc9eab279e22dec5fdfe0142bab07b49c47f290cdaceeb3
+  metadata.gz: eb6518d38e67aecd470fe2c65227af6467e70cf60ef26ece496d0bb91301e80f
+  data.tar.gz: a57fc48c3b2a1799baabf0f4477da9085c01d5406aa2762df8f5af603d49f203
 SHA512:
-  metadata.gz: 82576ff9731c282102710ad24d83b2ff3430a71bab671b33d6823dd507a47ec3855ccf912afbff706b9514d2f045313bd69461aa3076dbf85673b17e2db32920
-  data.tar.gz: 28c718c40751cf17bdc74bba6db5f8b07d4d9ecf889409b7861563302c22c10590f54f82680facbaf5d0000552d5ed8344831eff856fb9927acaf6f113383242
+  metadata.gz: ae8a1ce41d3fbb54a807509a001c9ffe72020fe586973dc74cd4b643708938920ec1cd920266c4a52c0e62c1a7f6d1e6e2c40e181b50d4b0de6925e94e65810b
+  data.tar.gz: ebfc96602d16d8c89101e1dc016537366a3055b7d16aca410c9e1a3700126a59cc6015143e39559ea75935e00403f614c71eb11e34738fcc5c72a13e78af3e8c

data/CHANGELOG.md

@@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.4.0] - 2026-10-19
+
+### Added
+- `fun-ci COMMAND --help` (or `-h`) prints that command's usage and the
+  options it takes. Before, `init`, `check`, `prune` and `console` ignored
+  the flag and went ahead, `install-hooks` took it for a hook type, and the
+  agent commands called it an invalid option.
+- `fun-ci init --stack NAME` writes the scripts for the stack you name
+  (`go`, `node_npm`, ...) instead of the one it detects.
+- In a project whose stack `fun-ci init` doesn't recognise, such as a new
+  one with no code yet, it writes four placeholder scripts that pass, instead
+  of stopping with nothing written. `fun-ci check` warns about each one still
+  in place, since that stage tests nothing.
+- `clean: [public/assets, dist]` in `.fun-ci/config` names build outputs that
+  git ignores and that must not carry over from another branch's run. When a
+  worktree moves to a commit that doesn't descend from the one it last
+  tested, fun-ci removes the ignored files under those paths first, and that
+  run's lint and build get the longer budget a new worktree gets. Other
+  ignored files, such as `node_modules`, stay. `fun-ci check` names a `clean`
+  it can't use.
+- A failed stage's evidence names the stages of the project's other runs
+  that ran at the same time, as `other runs alongside: 7ed514c's slow` in
+  `fun-ci why`, so a fast suite over its budget beside another commit's slow
+  suite says so.
+
+### Changed
+- fun-ci's worktrees, the runs' slots and the jobs', move out of `.git/` into
+  fun-ci's state directory (`~/.local/state/fun-ci/projects/`). Under
+  `.git/`, Vite's default `fs.deny` refused to serve test files, so a Vitest
+  suite with a browser-like environment failed in fun-ci and nowhere else.
+  The first run after upgrading makes new worktrees, with the longer budget
+  a new worktree gets; `fun-ci prune` removes the old ones from `.git/fun-ci/`.
+- For Node, pnpm, Yarn and Bun projects, `fun-ci init` writes a `lint.sh`
+  that checks the syntax of the source without the packages (`node --check`
+  on `.js`, `.mjs` and `.cjs` files; `bun build --no-bundle`, TypeScript
+  included, for Bun), and runs ESLint in `build.sh` after the install, when
+  the project configures ESLint. ESLint in `lint.sh` ran beside the install:
+  in a new worktree it ran a registry ESLint and failed on a plugin it
+  couldn't find, and in a project without ESLint it always failed. A
+  TypeScript project on Node gets no check of its `.ts` files in `lint.sh`;
+  ESLint or `tsc` in `build.sh` covers them.
+- A worktree slot nobody took for a day counts as cold, as a new one does, so
+  lint and build get the longer budget there once. A second slot, used only
+  when runs overlap, kept caches days old and ran a build over its budget.
+  `fun-ci status` now says `(cold worktree, budget 300s)`, not `new worktree`,
+  and a run in the foreground says `a cold worktree (new, just cleaned, or
+  unused for a day)`.
+- The console holds at most the twenty branches whose newest run changed
+  last, before putting them in order, so a failure on a branch nobody has
+  touched since leaves the board once twenty others have moved on, instead
+  of staying at the top for good.
+- `fun-ci init` refuses an option or argument it doesn't take (exit 64)
+  instead of ignoring it, so a misspelt `--everything` no longer sets up
+  only half.
+
+### Fixed
+- `fun-ci console` no longer dies with a `TypeError` while a project's first
+  fetch of its trunk is under way.
+- A run the post-commit hook starts leads a session of its own, so Ctrl-C in
+  the terminal that committed, closing it, or an agent harness ending the
+  `git commit` command no longer kills the run's slow suite.
+- A slow suite whose process died is no longer shown as a plain failure:
+  `fun-ci why` says its process stopped before it said how the stage ended,
+  as it already did for a job.
+- Cancelling a run, or killing a stage over its budget, also stops what the
+  stage started that moved to a process group or session of its own, as
+  Node's detached children do: a superseded run left Playwright's web server
+  holding its port, and the next run failed on it. A child forked just as
+  the group was killed is killed too.
+
 ## [2.3.0] - 2026-10-02
 
 ### Added

data/README.md

@@ -23,7 +23,7 @@ fun-ci init --everything
 
 This detects your project type, writes four stage scripts into `.fun-ci/`, installs a `post-commit` and a `pre-push` git hook, and checks the setup. It also adds a short section to your `AGENTS.md` (or `CLAUDE.md`, if that is the only one) telling coding agents what to do after a commit; run again after an upgrade, it brings that section up to date.
 
-`fun-ci init` has templates for Ruby, Gradle, Maven, Rust, Go, Elixir, Dart, Swift, PHP, .NET, Python, Deno, Bun, Node, Perl, and C or C++ built with CMake or make. [docs/stacks.md](docs/stacks.md) shows how it recognises each one and the commands it writes. Fun-CI works with any project: the stages are shell scripts, so edit them to run whatever your project uses.
+`fun-ci init` has templates for Ruby, Gradle, Maven, Rust, Go, Elixir, Dart, Swift, PHP, .NET, Python, Deno, Bun, Node, Perl, and C or C++ built with CMake or make. [docs/stacks.md](docs/stacks.md) shows how it recognises each one and the commands it writes; `--stack go` picks one by name. In a project it doesn't recognise, such as a new one with no code yet, it writes placeholder scripts that pass, and `fun-ci check` warns about each one until you replace it. Fun-CI works with any project: the stages are shell scripts, so edit them to run whatever your project uses.
 
 - `lint.sh`: linter and static analysis
 - `build.sh`: compile or build
@@ -51,7 +51,7 @@ After each commit, the `post-commit` hook starts the pipeline in the background
 
 `install-hooks` leaves a hook another tool wrote alone, and `fun-ci check` warns that no commit is tested until that hook calls fun-ci. Git LFS's own hooks are the exception: fun-ci writes its hooks in their place, and they run git-lfs's after fun-ci's, so LFS objects still reach the remote. After that, `git lfs install` says the hooks exist and exits 2, since it accepts no hook but its own; there is nothing for it to do. `git lfs update --force` would put git-lfs's back, leaving fun-ci out, until you run `fun-ci install-hooks` again.
 
-Each run happens in a git worktree of its own under `.git/fun-ci/worktrees/`, checked out at the commit it tests, so you can keep editing while it runs. Ignored files such as `vendor/bundle` or `node_modules` stay in a worktree from one run to the next, which keeps builds quick. A worktree fun-ci has just made has none of them yet, so in its first run lint and build get the slow suite's five minutes rather than 30 seconds, and `fun-ci status` says so on their lines. Two runs can go at once; set `worktree_slots: 3` in `.fun-ci/config` for more, and `fun-ci prune` removes the worktrees when you want the space back. A newer commit on a branch cancels the older run that is still going, unless an agent is waiting on it without `--follow-branch`.
+Each run happens in a git worktree of its own in fun-ci's state directory (`~/.local/state/fun-ci/projects/`, outside `.git/`, which tools such as Vite won't serve files from), checked out at the commit it tests, so you can keep editing while it runs. Ignored files such as `vendor/bundle` or `node_modules` stay in a worktree from one run to the next, which keeps builds quick. A worktree fun-ci has just made has none of them yet, and one nobody used for a day has them as they were then, so in such a run lint and build get the slow suite's five minutes rather than 30 seconds, and `fun-ci status` says so on their lines. A run's build outputs that git ignores stay too, and are only as right as the commit that made them, so a worktree that last tested another branch can hand a run outputs its commit wouldn't make. List those outputs in `.fun-ci/config`, such as `clean: [public/assets, dist]`, and fun-ci removes their ignored files when a worktree moves to a commit that doesn't descend from the one it last tested; that run's lint and build get the longer budget too. Two runs can go at once; set `worktree_slots: 3` in `.fun-ci/config` for more, and `fun-ci prune` removes the worktrees when you want the space back. A newer commit on a branch cancels the older run that is still going, unless an agent is waiting on it without `--follow-branch`.
 
 Results are kept in SQLite under `$XDG_STATE_HOME/fun-ci/` (`~/.local/state/fun-ci/` by default), shared by every project on the machine.
 
@@ -68,7 +68,7 @@ The script's name is the job's name. Like a stage, it gets the commit's hash as
 
 Commits start jobs. When the `post-commit` hook runs a pipeline, it also starts the project's due jobs on the commit just made: a job is due when it has never run, when its last run was cancelled, or a day (daily) or a week (weekly) after its last run started, so a project nobody commits to runs none. The jobs take turns, spread evenly over the day: with four jobs, the first starts at once and each next one six hours later. `job_spacing: 30m` in `.fun-ci/config` sets the gap instead, and `0` starts them together. A job waiting its turn says when it starts.
 
-Each job runs once at a time, in a worktree of its own under `.git/fun-ci/jobs/`, at a lower priority than the stages, so it takes only the cores they leave idle and never holds up a commit's run. It has 24 hours. A newer commit never cancels a job, and a job's result changes no run, no streak and no exit code. The [design](docs/design.md#daily-and-weekly-jobs) has the rest of the rules.
+Each job runs once at a time, in a worktree of its own beside them, at a lower priority than the stages, so it takes only the cores they leave idle and never holds up a commit's run. It has 24 hours. A newer commit never cancels a job, and a job's result changes no run, no streak and no exit code. The [design](docs/design.md#daily-and-weekly-jobs) has the rest of the rules.
 
 A failed job keeps its evidence as a failed stage does; add entries under `evidence: jobs: <name>:` in `.fun-ci/config` to keep more. `fun-ci check` lists the jobs it found, and says which scripts it can't run.
 
@@ -80,7 +80,7 @@ A failed job keeps its evidence as a failed stage does; add entries under `evide
 fun-ci console
 ```
 
-The console shows where each branch stands, one row per branch under its project's name: a mark for each stage (lint, the build, the fast suite and the slow suite, named in a legend at the top of the table), what happened (`the fast suite failed after 1.4s`) and when. What needs you comes first, in one deep block; passed rows stay pale. Above them is a picture that tells you what is happening without reading a word.
+The console shows where each branch stands, one row per branch under its project's name: a mark for each stage (lint, the build, the fast suite and the slow suite, named in a legend at the top of the table), what happened (`the fast suite failed after 1.4s`) and when. What needs you comes first, in one deep block; passed rows stay pale. It holds the twenty branches whose newest run changed last, so a failure on a branch nobody touches any more leaves it in time. Above them is a picture that tells you what is happening without reading a word.
 
 ![The console's table on a busy board: a legend naming lint, the build, the fast suite and the slow suite over their columns of marks, three projects, a failed branch that conflicts with main in a deep red card, a timeout, a running branch and two pale passed ones](docs/screenshots/table.png)
 
@@ -246,6 +246,7 @@ Secrets in the stage's environment are masked before anything is kept: the value
 ```
 fun-ci init                                     Scaffold .fun-ci/ for the detected project type
 fun-ci init --everything                        init + install-hooks + check in one step
+fun-ci init --stack NAME                        Scaffold .fun-ci/ for the stack named, such as go
 fun-ci install-hooks                            Install post-commit and pre-push hooks
 fun-ci install-hooks post-commit                Install a single hook type
 fun-ci check                                    Verify .fun-ci/ setup
@@ -264,7 +265,7 @@ fun-ci trigger --background <commit> <branch>   Start the pipeline in the backgr
 fun-ci prune                                    Remove fun-ci's worktrees when no pipeline is running
 ```
 
-`fun-ci --help` lists every option.
+`fun-ci --help` lists every option, and `fun-ci COMMAND --help` a command's own.
 
 ## Upgrading to 2.0
 

data/lib/fun_ci/agent/status_text.rb

@@ -54,11 +54,11 @@ module FunCi
         "  #{stage.name.ljust(6)} #{WORDS.fetch(stage.state, stage.state).ljust(12)}#{seconds}#{note}".rstrip
       end
 
-      # Only a worktree just made gives a stage more than its usual budget
+      # Only a cold worktree (WorktreePool) gives a stage more than its usual budget
       # (AT-1.14), which a run in the background says nowhere else.
       def self.longer(stage)
         usual = Pipeline::DEFAULT_BUDGETS[stage.name]
-        stage.budget && usual && stage.budget > usual ? "  (new worktree, budget #{stage.budget}s)" : ""
+        stage.budget && usual && stage.budget > usual ? "  (cold worktree, budget #{stage.budget}s)" : ""
       end
 
       def self.footer(report)

data/lib/fun_ci/cli.rb

@@ -2,6 +2,7 @@
 
 require_relative "../fun_ci"
 require_relative "cli_help"
+require_relative "cli_command_help"
 require_relative "pipeline/trigger_params"
 require_relative "persistence/state_dir"
 require_relative "setup/commands"
@@ -37,6 +38,7 @@ module FunCi
       return CliHelp.version(@io.stdout) if subcommand == "--version"
 
       return unknown_command(subcommand) unless ROUTES.key?(subcommand) || AGENT_COMMANDS.include?(subcommand)
+      return CliCommandHelp.help(@io.stdout, subcommand) if CliCommandHelp.asked?(args.drop(1))
 
       dispatch(subcommand, args.drop(1))
     end

data/lib/fun_ci/cli_command_help.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+require_relative "cli_help"
+
+module FunCi
+  # What `fun-ci COMMAND --help` prints in place of running the command: its
+  # usage, what it is for and the options it takes, the last two read from
+  # `fun-ci --help`, whose options name in brackets the commands taking them.
+  module CliCommandHelp
+    FLAGS = %w[-h --help].freeze
+    USAGES = {
+      "trigger" => "fun-ci trigger [--background] <commit-hash> <branch>",
+      "console" => "fun-ci console",
+      "init" => "fun-ci init [--everything] [--stack NAME]",
+      "install-hooks" => "fun-ci install-hooks [post-commit | pre-push]",
+      "check" => "fun-ci check",
+      "prune" => "fun-ci prune",
+      "status" => "fun-ci status [REV] [--need LEVEL] [--trunk] [--json]",
+      "wait" => "fun-ci wait [REV] [--need LEVEL] [--within TIME] [--follow-branch] [--trunk] [--json]",
+      "runs" => "fun-ci runs [-n N] [--branch NAME] [--json]",
+      "events" => "fun-ci events [--follow] [--only failures]",
+      "why" => "fun-ci why [REV] [STAGE | trunk] [--need LEVEL] [--raw] [--json]\n       " \
+               "fun-ci why --job NAME [--raw] [--json]",
+      "jobs" => "fun-ci jobs [--json]",
+      "cancel" => "fun-ci cancel --job NAME",
+      "extract" => "fun-ci extract STAGE --output FILE [--exit N | --timed-out] [--json]"
+    }.freeze
+
+    def self.asked?(args) = args.any? { |arg| FLAGS.include?(arg) }
+
+    def self.help(out, command)
+      out.puts "Usage: #{USAGES.fetch(command)}", "", purpose(command)
+      options = options_of(command)
+      out.puts "", "Options:", *options unless options.empty?
+      0
+    end
+
+    def self.purpose(command) = CliHelp::TEXT[/^  #{Regexp.escape(command)} +(.+)$/, 1]
+
+    def self.options_of(command)
+      lines = CliHelp::TEXT.lines.select { |line| takes?(line, command) }
+      lines.map { |line| line.sub(/\([^)]+\) /, "") }
+    end
+
+    def self.takes?(option_line, command)
+      commands = option_line[/^\s+-\S.*?\(([^)]+)\)/, 1]
+      !commands.nil? && commands.split(", ").include?(command)
+    end
+    private_class_method :purpose, :options_of, :takes?
+  end
+end

data/lib/fun_ci/cli_help.rb

@@ -36,13 +36,13 @@ module FunCi
         extract        Try a stage's evidence extractors on a saved output
 
       Options:
-        -h, --help     Show this help message
+        -h, --help     Show this help message; after a command, that command's usage
         --version      Show version
 
-      Trigger options:
-        --background   Run the pipeline in the background and return at once
-
-      Agent options:
+      Command options:
+        --background   (trigger) Run the pipeline in the background and return at once
+        --everything   (init) Run init + install-hooks + check in one step
+        --stack NAME   (init) Write the scripts for this stack, such as go, instead of the one detected
         --need LEVEL   (status, wait, why) What must pass: build, fast (default) or all
         --within TIME  (wait) Give up undecided after 30, 30s or 5m
         --follow-branch  (wait) Move on to the newer commit that superseded the run
@@ -53,13 +53,13 @@ module FunCi
         --branch NAME  (runs) Only runs on this branch
         --follow       (events) Keep printing events as they happen, until stopped
         --only failures  (events) Print only failed stages, superseded runs and failed jobs
-        --json         Print JSON instead of text (events always print JSON lines)
+        --output FILE  (extract) The saved output of the stage to read
+        --exit N       (extract) The exit status the stage ended with (default 1)
+        --timed-out    (extract) The stage ran out of time instead of exiting
+        --json         (status, wait, runs, why, jobs, extract) Print JSON instead of text; events always print JSON lines
         Exit codes: 0 passed, 1 failed, 2 over budget, 3 undecided,
         4 superseded, 5 no run, 6 conflicts with the trunk (--trunk), 64 usage error
 
-      Init options:
-        --everything   Run init + install-hooks + check in one step
-
       Examples:
         fun-ci init --everything
         fun-ci trigger abc1234 main

data/lib/fun_ci/console/board_data.rb

@@ -19,6 +19,9 @@ module FunCi
       # How many of a branch's runs are read to fold its cancelled ones and
       # find its standing against the trunk.
       BRANCH_WINDOW = 100
+      # How many branches the board holds at most: those whose newest run
+      # changed most recently, so old failures leave it in time.
+      RECENT_BRANCHES = 20
 
       # `clock` answers the time now, which decides which jobs are due.
       def initialize(db, page_size: 15, run_canceller: Pipeline::RunCanceller.new, clock: -> { Time.now })
@@ -34,7 +37,7 @@ module FunCi
       def resize(page_size) = @paging.resize(page_size)
 
       # Whether there are branches beyond those `runs` loads.
-      def more? = Persistence::PipelineRun.branch_heads(@db, limit: @paging.limit + 1).size > @paging.limit
+      def more? = heads(@paging.limit + 1).size > @paging.limit
 
       # Records failed each slow suite whose process died (acceptance-tests.md, AT-8.3).
       def record_dead_slow_suites = @run_canceller.record_dead(@db)
@@ -55,7 +58,7 @@ module FunCi
       # and then those run most recently, in RowOrder: a run of cancelled
       # runs folded into one, with its stages and the branch's standing against the trunk.
       def runs
-        RowOrder.of(Persistence::PipelineRun.branch_heads(@db, limit: @paging.limit).map { |head| branch_row(head) })
+        RowOrder.of(heads(@paging.limit).map { |head| branch_row(head) })
       end
 
       # The projects among `runs` whose trunk is stale.
@@ -74,6 +77,8 @@ module FunCi
 
       private
 
+      def heads(limit) = Persistence::PipelineRun.branch_heads(@db, limit: limit, among: RECENT_BRANCHES)
+
       def branch_row(head)
         branch_runs = Persistence::PipelineRun.of_branch(@db, head[:project_path], head[:branch], limit: BRANCH_WINDOW)
         row = CancelledFolding.fold(branch_runs).first

data/lib/fun_ci/console/trunk_marks.rb

@@ -44,10 +44,12 @@ module FunCi
 
       def stale_since(project, now)
         last = Persistence::TrunkFetches.new(@db, project).last
-        return nil unless last && (last.error || now - last.fetched_at > Trunk::Shown::FRESH_FOR)
+        return nil unless last && (last.error || old?(last.fetched_at, now))
 
         { project: project, since: last.fetched_at&.to_i }
       end
+
+      def old?(fetched_at, now) = !fetched_at.nil? && now - fetched_at > Trunk::Shown::FRESH_FOR
     end
   end
 end

data/lib/fun_ci/evidence/about_stage.rb

@@ -10,7 +10,7 @@ module FunCi
     # command to read.
     module AboutStage
       def self.of(sources, outcome, output)
-        About.new(**outcome.to_h.except(:overrun), **started(sources), **where(sources, output),
+        About.new(**outcome.to_h.except(:overrun, :elsewhere), **started(sources), **where(sources, output),
                   stage: sources.stage, budget: sources.budget, commit: sources.commit)
       end
 

data/lib/fun_ci/evidence/collector.rb

@@ -37,7 +37,7 @@ module FunCi
       def collect(output, outcome = FAILED)
         result = extracted(output, outcome)
         overrun = outcome.overrun || NOTHING
-        parts = [["fun-ci", own_facts(outcome.alongside)], *overrun.parts, *result.parts, ["output-tail", tail(output)]]
+        parts = [["fun-ci", own_facts(outcome)], *overrun.parts, *result.parts, ["output-tail", tail(output)]]
         Caps.new.apply(masking.document(assemble(parts, [overrun, result])))
       end
 
@@ -76,10 +76,13 @@ module FunCi
 
       def on_of(raw) = raw.is_a?(Hash) ? raw["on"] : nil
 
-      def own_facts(alongside)
-        Findings.new(facts: alongside.empty? ? [] : [{ name: "alongside", value: alongside.join(", ") }])
+      def own_facts(outcome)
+        named = { "alongside" => outcome.alongside, "other runs alongside" => outcome.elsewhere }
+        Findings.new(facts: named.reject { |_, stages| stages.empty? }.map { |name, stages| fact(name, stages) })
       end
 
+      def fact(name, stages) = { name: name, value: stages.join(", ") }
+
       def tail(output)
         text = Persistence::OutputTail.of(output, mask: masking.method(:mask))
         Findings.new(excerpts: [{ title: Document::TAIL_TITLE, location: "output", lines: text.lines(chomp: true) }])

data/lib/fun_ci/evidence/outcome.rb

@@ -4,12 +4,15 @@ module FunCi
   module Evidence
     # How a failed stage ended, known once it has: its state in the words
     # `status --json` uses, its exit status or signal, and the other stages
-    # that shared its slot while it ran, and for an overrun, what was found
+    # that shared its slot while it ran, the stages of the project's other runs
+    # that ran meanwhile (elsewhere), and for an overrun, what was found
     # before the kill (an Extraction::Result).
-    Outcome = Data.define(:state, :exit_status, :signal, :alongside, :overrun)
+    Outcome = Data.define(:state, :exit_status, :signal, :alongside, :elsewhere, :overrun)
 
     class Outcome
-      def initialize(**given) = super(exit_status: nil, signal: nil, alongside: [], overrun: nil, **given)
+      def initialize(**given)
+        super(exit_status: nil, signal: nil, alongside: [], elsewhere: [], overrun: nil, **given)
+      end
     end
   end
 end

data/lib/fun_ci/jobs/locks.rb

@@ -6,7 +6,7 @@ require_relative "../pipeline/slot"
 module FunCi
   module Jobs
     # Each job's worktree and the lock beside it, under a project's
-    # <git-common-dir>/fun-ci/jobs: `<name>` and `<name>.lock`. Whoever holds
+    # Worktrees#jobs_root: `<name>` and `<name>.lock`. Whoever holds
     # a job's lock runs it, so it runs once at a time, and a lock dies with
     # the process that held it.
     class Locks

data/lib/fun_ci/persistence/active_jobs.rb

@@ -2,8 +2,7 @@
 
 require "time"
 require_relative "active_runs"
-require "json"
-require_relative "../evidence/document"
+require_relative "died"
 
 module FunCi
   module Persistence
@@ -25,14 +24,6 @@ module FunCi
       # [id, lock file] of each job run still running.
       def self.running(db) = db.execute("SELECT id, lock_file FROM job_runs WHERE #{ACTIVE}")
 
-      # What a run whose process died keeps: that it has no result, and why
-      # that may be. It keeps no end, since when it was found dead says
-      # nothing of how long it ran.
-      DIED_FACT = { name: "ended", extractor: "fun-ci",
-                    value: "its process stopped before it said how the run ended (the machine slept or " \
-                           "restarted, or something killed it), so this run has no result" }.freeze
-      DIED = Evidence::Document.new(chosen: [], failures: [], excerpts: [], problems: [], facts: [DIED_FACT])
-
       def self.cancelled(db, id) = ended(db, id, "cancelled", ACTIVE)
 
       # A run whose process is gone without recording how it ended: failed,
@@ -41,7 +32,7 @@ module FunCi
       def self.died(db, id)
         ended(db, id, "cancelled", "status = 'scheduled'")
         db.execute("UPDATE job_runs SET status = 'failed', evidence = ? WHERE id = ? AND status = 'running'",
-                   [JSON.generate(DIED.to_h), id])
+                   [Died.evidence("run"), id])
       end
 
       # The ids of a project's job's runs still running.

data/lib/fun_ci/persistence/active_runs.rb

@@ -3,6 +3,7 @@
 require "time"
 require_relative "pipeline_run"
 require_relative "stage_job"
+require_relative "died"
 
 module FunCi
   module Persistence
@@ -39,10 +40,11 @@ module FunCi
                    "AND pipeline_runs.pid IS NOT NULL")
       end
 
-      # Records the slow stage failed, unless its result was recorded meanwhile.
+      # Records the slow stage failed, and that its process died, unless its
+      # result was recorded meanwhile.
       def self.slow_suite_died(db, job_id)
-        db.execute("UPDATE stage_jobs SET status = 'failed', completed_at = ? WHERE id = ? AND status = 'running'",
-                   [Time.now.utc.iso8601(3), job_id])
+        db.execute("UPDATE stage_jobs SET status = 'failed', completed_at = ?, evidence = ? " \
+                   "WHERE id = ? AND status = 'running'", [Time.now.utc.iso8601(3), Died.evidence("stage"), job_id])
       end
 
       def self.cancelled(db, run)

data/lib/fun_ci/persistence/died.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+require "json"
+require_relative "../evidence/document"
+
+module FunCi
+  module Persistence
+    # The evidence kept of a stage or job run whose process died: that it has
+    # no result, and why that may be, so it reads apart from a failure of the
+    # code. It keeps no end, since when it was found dead says nothing of how
+    # long it ran.
+    module Died
+      def self.evidence(what)
+        fact = { name: "ended", extractor: "fun-ci",
+                 value: "its process stopped before it said how the #{what} ended (the machine slept or " \
+                        "restarted, or something killed it), so this #{what} has no result" }
+        JSON.generate(Evidence::Document.new(chosen: [], failures: [], excerpts: [], problems: [], facts: [fact]).to_h)
+      end
+    end
+  end
+end

data/lib/fun_ci/persistence/job_recorder.rb

@@ -35,6 +35,7 @@ module FunCi
 
       # No stage shares a job's worktree.
       def alongside(_id) = []
+      def alongside_elsewhere(_id) = []
 
       private
 

data/lib/fun_ci/persistence/null_recorder.rb

@@ -19,6 +19,7 @@ module FunCi
       def trunk_heads(**) = {}
       def trunk_fetches = TrunkRecording::NoFetches.new
       def alongside(_job_id) = []
+      def alongside_elsewhere(_job_id) = []
       def stage_process(_job_id, _pid); end
       def slot_taken(_lock_file); end
       def foreground_done; end

data/lib/fun_ci/persistence/pipeline_recorder.rb

@@ -97,6 +97,7 @@ module FunCi
 
       # The stages that shared the slot with this one (architecture.md, "Evidence of a failed stage").
       def alongside(job_id) = tolerating { StageJob.alongside(@db, job_id) } || []
+      def alongside_elsewhere(job_id) = tolerating { StageJob.alongside_elsewhere(@db, job_id) } || []
 
       def keep_exit(job_id, exit_status, signal)
         tolerating { StageJob.keep_exit(@db, job_id, exit_status, signal) }

data/lib/fun_ci/persistence/pipeline_run.rb

@@ -56,9 +56,8 @@ module FunCi
         db.execute("UPDATE pipeline_runs SET trunk_started_at = ? WHERE id = ?", [at.utc.iso8601, id])
       end
 
-      def self.update_status(db, id, new_status)
-        now = Time.now.utc.iso8601
-        db.execute("UPDATE pipeline_runs SET status = ?, updated_at = ? WHERE id = ?", [new_status, now, id])
+      def self.update_status(db, id, new_status, at: Time.now)
+        db.execute("UPDATE pipeline_runs SET status = ?, updated_at = ? WHERE id = ?", [new_status, at.utc.iso8601, id])
       end
 
       def self.recent(db, limit: 20)
@@ -69,11 +68,15 @@ module FunCi
       NEEDING = "CASE status WHEN 'failed' THEN 0 WHEN 'timed_out' THEN 1 WHEN 'running' THEN 2 " \
                 "WHEN 'scheduled' THEN 3 WHEN 'completed' THEN 4 ELSE 5 END"
 
-      # The newest run of each of `limit` branches: those that most need you,
-      # then those run most recently, in that order.
-      def self.branch_heads(db, limit:)
-        query(db, "WHERE id IN (SELECT MAX(id) FROM pipeline_runs GROUP BY project_path, branch) " \
-                  "ORDER BY #{NEEDING}, id DESC LIMIT ?", limit)
+      # Each branch's newest run.
+      HEADS = "SELECT MAX(id) FROM pipeline_runs GROUP BY project_path, branch"
+
+      # The newest run of each of `limit` branches, of the `among` whose newest
+      # run changed most recently: those that most need you, then those run
+      # most recently, in that order.
+      def self.branch_heads(db, limit:, among:)
+        query(db, "WHERE id IN (SELECT id FROM pipeline_runs WHERE id IN (#{HEADS}) " \
+                  "ORDER BY updated_at DESC, id DESC LIMIT ?) ORDER BY #{NEEDING}, id DESC LIMIT ?", among, limit)
       end
 
       # A project's branch's runs, newest first.

data/lib/fun_ci/persistence/stage_job.rb

@@ -51,6 +51,18 @@ module FunCi
                    [id]).flatten
       end
 
+      # Stages of the project's other runs running at any moment between this
+      # stage's start and now, as "<short commit>'s <stage>".
+      def self.alongside_elsewhere(db, id)
+        db.execute("SELECT runs.commit_hash, others.stage FROM stage_jobs AS others " \
+                   "JOIN pipeline_runs AS runs ON runs.id = others.pipeline_run_id " \
+                   "JOIN stage_jobs AS me ON me.id = ? JOIN pipeline_runs AS mine ON mine.id = me.pipeline_run_id " \
+                   "WHERE runs.project_path IS mine.project_path AND runs.id != mine.id " \
+                   "AND others.started_at IS NOT NULL " \
+                   "AND (others.completed_at IS NULL OR others.completed_at > me.started_at) ORDER BY others.id",
+                   [id]).map { |sha, stage| "#{sha[0, 7]}'s #{stage}" }
+      end
+
       def self.ids(db) = db.execute("SELECT id FROM stage_jobs").flatten
 
       def self.find(db, id)

data/lib/fun_ci/pipeline/group_kill.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module FunCi
+  module Pipeline
+    # Kills a process group and what left it (ProcessTree): TERM, then KILL.
+    # A member forking as the KILL arrives leaves a child in the group that
+    # the signal missed, so the group is killed again while any member lives,
+    # a bounded number of times, since a mutant may stop it ever going.
+    module GroupKill
+      ROUNDS = 10
+
+      # escaped: further targets, -pgid for a group, pid for a process.
+      def self.kill(pid, escaped, signal: Process.method(:kill))
+        %w[TERM KILL].product([-pid, *escaped]).each { |name, target| send_to(signal, name, target) }
+        ROUNDS.times do
+          return unless lives?(pid, signal)
+
+          send_to(signal, "KILL", -pid)
+        end
+      end
+
+      # On macOS a group left with only zombies answers EPERM, not ESRCH.
+      def self.lives?(pid, signal)
+        signal.call(0, -pid)
+        true
+      rescue Errno::ESRCH, Errno::EPERM
+        false
+      end
+
+      def self.send_to(signal, name, target)
+        signal.call(name, target)
+      rescue Errno::ESRCH, Errno::EPERM
+        # Gone already.
+      end
+      private_class_method :lives?, :send_to
+    end
+  end
+end

data/lib/fun_ci/pipeline/pipeline_forker.rb

@@ -66,8 +66,10 @@ module FunCi
       end
 
       # Trigger requires this file, through TriggerCommand, so it is loaded
-      # here, where it is used, rather than above.
+      # here, where it is used, rather than above. The run leads a session of
+      # its own, so that ending the shell that committed doesn't end it.
       def self.run_in_child(commit_hash:, branch:, db_path:)
+        Process.setsid
         require_relative "trigger"
         recorder = Persistence::DbRecorder.new(Persistence::Database.connection(db_path))
         trigger = trigger(commit_hash, branch, recorder)

data/lib/fun_ci/pipeline/process_runner.rb

@@ -3,6 +3,9 @@
 require_relative "gate"
 require_relative "git_environment"
 require_relative "output_window"
+require_relative "process_tree"
+require_relative "group_kill"
+require_relative "../evidence/process_table"
 
 module FunCi
   module Pipeline
@@ -112,8 +115,10 @@ module FunCi
         [output, status, false]
       end
 
+      # With the group go what it started that left it (ProcessTree), found
+      # before the kill orphans them (GroupKill).
       def kill_process_group(pid)
-        %w[TERM KILL].each { |signal| ignoring_errors { Process.kill(signal, -pid) } }
+        GroupKill.kill(pid, ProcessTree.escaped([pid], Evidence::ProcessTable.now))
         ignoring_errors { Process.waitpid(pid) }
       end