ppkit 0.2.1__tar.gz → 0.2.2__tar.gz

{ppkit-0.2.1 → ppkit-0.2.2}/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## v0.2.2
+
+`pp queue` records a kill as a kill when the worker sending it dies before
+it records the send. Such a job no longer ends as its own failure, and its
+`on_fail` hook does not run. A stop's new `going` field says when pp began
+sending it.
+
+`pp queue`: a kill whose worker dies while sending it is still a kill.
+
+- A worker that died between sending a kill's stop and recording it left the
+  next worker to read the job's end as the job's own. That job was recorded
+  as its own failure, with exit code 137 or reason `lost`, and its `on_fail`
+  hook ran. It is now recorded `killed`, and no hook runs. A timeout's stop
+  in the same window now ends as a timeout.
+- A stop gains `going` in `show --json` and `status --json`: when pp began
+  sending it, recorded before anything goes out. An exit code the attempt
+  wrote after that time is the stop's outcome, with the code kept, unless pp
+  was refused the stop. A job that ends on its own just as its worker dies
+  before sending is recorded the same way.
+- `show` says "sending began" for a stop pp began sending and has nothing
+  marked sent for. A stop pp was refused still says "nothing signalled".
+- `kill` reports a job that ended in that window as killed, "ended as its
+  stop was being sent", instead of saying the job had already ended.
+
 ## v0.2.1
 
 pp finds the session that runs it with nothing set, so `--mine` works for

{ppkit-0.2.1 → ppkit-0.2.2}/CONTRIBUTING.md

@@ -351,8 +351,9 @@ A kill, pass by pass:
 
 ```
 kill:   ask_stop(merge_stop) -> worker_up(wake=True) -> SIGUSR1 to the worker
-worker: carry_out: the facts -> decide = TERM -> the project's stop command,
-        or the SIGTERM to the group, then txn marks `sent`
+worker: carry_out: the facts -> decide = TERM, and its txn records `going`
+     -> the project's stop command, or the SIGTERM to the group,
+        then txn marks `sent`
      -> (the grace) -> decide = KILL9 -> signal
      -> the wrapper ends, so the job's run claim is free
      -> decide = ENDED_AS -> _finish records the outcome and the hook it owes
@@ -367,11 +368,14 @@ own exit time and not the queue's poll period.
 
 The rules the worker follows are the ones `pp help queue` states under "How a
 job ends, whatever reached it", and that help text is their one statement.
-Two more are for contributors: which of the two a stop is comes from the
+Three more are for contributors. Which of the two a stop is comes from the
 `timeout` field a worker writes on the stop it wrote itself, never from the
-name of whoever asked; and an exit code the attempt wrote after its stop went
-out is that stop's outcome, with the code kept, since a container runner writes
-the code of a container the stop command ended.
+name of whoever asked. An exit code the attempt wrote after its stop began
+going out is that stop's outcome, with the code kept, since a container runner
+writes the code of a container the stop command ended. And that moment is the
+stop's `going`, which the decision's transaction records with its own clock
+before anything goes out: a worker can die between a signal and its record,
+and a job can end between its facts and its decision.
 
 `TestTheDecision` in `tests/test_queue.py` is a table over every combination
 of those facts. Each row carries the rule from that help as its own label,

{ppkit-0.2.1 → ppkit-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ppkit
-Version: 0.2.1
+Version: 0.2.2
 Summary: A CLI kit for user & agents
 Keywords: cli,job queue,gpu,agents,toolkit
 Author: The pp Authors

{ppkit-0.2.1 → ppkit-0.2.2}/pp/__init__.py

@@ -1,4 +1,4 @@
 """pp: a CLI kit for user & agents. One file per subcommand, discovered at runtime."""
 
-__version__ = "0.2.1"
+__version__ = "0.2.2"
 TAGLINE = "a CLI kit for user & agents"      # pp -h's header; README and pyproject say it too (tests/test_cli.py)

{ppkit-0.2.1 → ppkit-0.2.2}/pp/commands/queue.py

@@ -256,25 +256,27 @@ PP_QUEUE_RCFILE itself: what is done about such a job is what its stop asks for.
     Once the run is over the exit code the attempt wrote is the command's last word, and the job ends here. A stop
     someone asked for is never retried.
 
-    A job a kill's signal stopped is recorded "killed" and fires no hook; an exit code the attempt wrote after that
-    signal went out is kept as the attempt's code.
-    A job that wrote its own exit code before a kill's signal went out, or that a kill wrote a stop for that no signal
-    of pp's ever reached, ends with its own outcome, the exit code it wrote or the signal it died of, and fires the
-    hook that outcome owes.
+    A job still running when a kill's stop began going out is recorded "killed" and fires no hook; an exit code the
+    attempt wrote after that is kept as the attempt's code.
+    A job that wrote its own exit code before a kill's stop began going out, or whose kill pp never began sending,
+    ends with its own outcome, the exit code it wrote or the signal it died of, and fires the hook that outcome owes.
     An exit code of 0 is done, and on_done fires.
-    A timeout's stop pp signalled or was refused is a timeout, which may be retried; an exit code the attempt wrote
-    after that stop went out is kept as the attempt's code, and on a final timeout on_fail fires.
+    A timeout's stop pp began sending or was refused is a timeout, which may be retried; an exit code the attempt
+    wrote after that stop began going out is kept as the attempt's code, and on a final timeout on_fail fires.
     Anything else is the job's own failure, the exit code it wrote or the signal it died of, retried while an attempt
     is left and --retry-if matches it, and on_fail fires on the last.
 
 One run of a job carries one stop: a kill's ask on a timeout's makes it a kill, a second ask keeps what has gone out
 and only shortens the grace, a refused stop is asked for again, and `show` and --json say whose stop it is, what has
-gone out for it, its command's exit code where it had one, and whether pp was refused. Where a job's work is reached
-by the project's stop command, that command goes out in place of the SIGTERM, and `stop.rc: null` means pp has not
-seen it end, not that it worked. Anything left in the job's group when its run is over goes with it, and a run that
-ended with no exit code of its own, or that a stop was asked of, runs that stop command once more, so nothing the
-runner started outlives its attempt. Nobody asked for that last run, so its output goes where the worker's own lines
-go (its log, or the output of a `pp queue work`), and the job's log holds the job and the stops asked of it.
+gone out for it, its command's exit code where it had one, and whether pp was refused. A stop begins going out when
+the worker records its `going` time, before it sends anything. A stop pp was refused before anything went out counts
+as one it never began sending. A worker that dies before recording what it sent leaves that time, so the next worker
+counts the stop as gone out and sends it again while the run is live. Where a job's work is reached by the project's
+stop command, that command goes out in place of the SIGTERM, and `stop.rc: null` means pp has not seen it end, not
+that it worked. Anything left in the job's group when its run is over goes with it, and a run that ended with no
+exit code of its own, or that a stop was asked of, runs that stop command once more, so nothing the runner started
+outlives its attempt. Nobody asked for that last run, so its output goes where the worker's own lines go (its log,
+or the output of a `pp queue work`), and the job's log holds the job and the stops asked of it.
 
 Workers, and after a reboot
     One worker per queue starts what fits, reaps what ended, carries out stops, re-queues retries and fires hooks.
@@ -331,7 +333,7 @@ Agents
     runner (each an argv), stop_cmd ("group", or the command that reaches what the runner started), prio, needs,
     cards, after, grace, timeout, retries, attempt, rc, reason, not_retried, added, started, ended, seconds, log,
     log_at, container, envfile, rcfile, owner, history (one entry per attempt that ended, with its own files and
-    cards), notes, reported, hooks, stop (by, t, grace, timeout, sent, kill9, denied, was, cmd, rc), env and
+    cards), notes, reported, hooks, stop (by, t, grace, timeout, going, sent, kill9, denied, was, cmd, rc), env and
     shell_env (their names, every value ***). Each is on a job in every state, null or empty where the record holds
     none yet, but `after` only on a job that names what it waits for and `stop` only on one asked to stop. A field is
     added and never renamed or retyped inside a minor version.

{ppkit-0.2.1 → ppkit-0.2.2}/pp/lib/jobcore.py

@@ -35,6 +35,7 @@ import signal
 import time
 import traceback
 from contextlib import contextmanager, nullcontext
+from dataclasses import replace
 from datetime import datetime
 
 from pp.lib import config, jobpolicy as policy, jobstore, lock, paths, proc, session
@@ -203,14 +204,13 @@ class Worker:
         A timeout, a kill and a reap are not three passes over the queue with three views of a job. They are
         facts about one job and one decision made from all of them, so no part of it can be wrong about what
         another part did."""
-        now = time.time()
         with jobstore.listing() as s:    # one pass: the jobs to decide about, and the hooks the queue still owes
             jobs = list(s.jobs(("running",) + jobstore.ENDED, self.queue))
         owed = [j for j in jobs if j["state"] in jobstore.ENDED and j.get("hook_due")]
         # while a stop is in flight, and a verb is waking this worker for it, the loop's wait is STOP_PERIOD
         # and not the poll (work()); `policy.in_flight` is what one of them is, over the record this pass
         # listed and the action it came to
-        self.serving = sum(policy.in_flight(j, self._step(j, self._facts(j, now)))
+        self.serving = sum(policy.in_flight(j, self._step(j, self._facts(j)))
                            for j in jobs if j["state"] == "running")
         for j in owed:
             self.fire(j, j["hook_due"])     # owed by a worker that did not get to fire it
@@ -220,7 +220,7 @@ class Worker:
             if p.poll() is not None and (jobstore.find(jid) or {}).get("state") != "running":
                 del self.children[jid]
 
-    def _facts(self, j: dict, now: float) -> policy.Facts:
+    def _facts(self, j: dict) -> policy.Facts:
         """What is true of this running job besides its record, all of it read here and nothing decided:
         whether a run of it is live, the exit code that run wrote and when, the signal its own child died of,
         and whether --retry-if matches the tail of its log.
@@ -236,10 +236,10 @@ class Worker:
             # them, before its parent can read how it ended.
             if held is not None:
                 self._adopt(j, held)
-            return policy.facts(True, now=now)
+            return policy.facts(True)
         self.children.pop(j["id"], None)
         rc, ended = ended_on_its_own(j) or (None, None)
-        return policy.facts(False, p.poll() if p is not None else None, rc, ended, now=now,
+        return policy.facts(False, p.poll() if p is not None else None, rc, ended,
                             hit=policy.wants_tail(j, rc) and policy.tail_hit(j, self.tail(j, 200)))
 
     def _step(self, j: dict, facts: policy.Facts) -> policy.Action:
@@ -249,20 +249,21 @@ class Worker:
 
         The decision is made inside the transaction, on the record the store holds and the facts read before
         it: a stop another actor wrote since those facts were read is part of it, so a job a kill has touched
-        is never retried past its stop."""
+        is never retried past its stop. Its clock is read in there too, after the facts, since that is the
+        time a stop's `going` records: a code the job wrote while its stale facts said it ran is its own."""
         jid = j["id"]
         with jobstore.txn() as s:
             cur = s.job(jid, must_be=("running",))
             if cur is None:
                 return policy.Action(policy.NOTHING)      # nothing else records a running job: there is nothing to record
-            a = policy.decide(cur, facts)
+            a = policy.decide(cur, replace(facts, now=time.time()))
             was = dict(cur)     # this attempt's own names, which a requeue below takes off the record
             if a.sweep:
                 # the leader of this job's group is gone and its run is over, so anything left in that group
                 # goes too: a command that held the signal back does not outlive the job
                 self._signal_group(cur, signal.SIGKILL)
-            if a.do == policy.TIME_UP:
-                cur["stop"] = {"by": "timeout", "timeout": True, "t": a.t, "grace": a.grace}
+            if a.stop:          # a timeout's stop, or the time a stop began going out, before anything does
+                cur["stop"] = a.stop
                 s.save(cur)
             elif a.do == policy.AGAIN:
                 jobstore._requeue(s, cur, a)
@@ -292,10 +293,10 @@ class Worker:
         job's processes, so a kill, a timeout and a reap never cross.
 
         The signal goes out first and is recorded after, so a record that says a signal went out is one that
-        did, which is all the decision about this job has to go on. The time recorded with it is read before
-        it goes out, because that time is what an exit code is late against (I10): a code the attempt wrote
-        while this stop was on its way out is that stop's outcome, and a time read afterwards would leave the
-        window the stop command takes to start reading as the job's own end.
+        did. What an exit code is late against (I10) is the stop's `going`, which the decision's transaction
+        recorded before this was called, so a worker that dies between the signal and its record leaves a stop
+        the next decision still counts as gone out. The time recorded here is read before the signal goes out
+        too, and the grace runs from it.
 
         Which of the two went out is recorded with it, as `cmd`: the command where the project's one started,
         and nothing where pp's own signal went. So a reader of the record says what went out for that stop

{ppkit-0.2.1 → ppkit-0.2.2}/pp/lib/jobpolicy.py

@@ -37,7 +37,7 @@ GROUP = "group"             # the stop of a job whose work is in its own process
 
 NOTHING = "nothing"         # leave it as it is
 TIME_UP = "time-up"         # write a timeout's stop on it; from there it is a stop like any other
-TERM = "term"               # SIGTERM to its group, recorded as the stop's `sent`
+TERM = "term"               # SIGTERM to its group, recorded as the stop's `sent`, after its `going`
 KILL9 = "kill9"             # SIGKILL to its group, recorded as the stop's `kill9`
 ENDED_AS = "ended-as"       # record how it ended, and the hook that is owed for it
 AGAIN = "again"             # back to its queue for another attempt
@@ -53,7 +53,9 @@ class Facts:
             is in its environment); a code is the command's last word only once the run is over.
     rc, rc_at   the exit code this attempt wrote and when it wrote it (`ended_on_its_own`), or None for both.
     sig     the signal the attempt's process died of, where this worker started it and so holds the child.
-    now     the time the facts were read at, which is the only clock the decision has.
+    now     the time the decision is made at, which is the only clock the decision has. The core reads it
+            inside the decision's transaction (`Worker._step`), after the rest, since a stop's `going` is this
+            time: an exit code the attempt wrote before it is the job's own, however stale the rest is.
     hit     whether --retry-if matches the tail of this attempt's log (`tail_hit`), matched where the facts
             were read: a pattern of the caller's own takes as long as it likes over a tail, and the decision
             runs inside the store's transaction (I12).
@@ -72,16 +74,18 @@ class Action:
     """The one thing to be done about a job. `do` is which, and a field a kind does not use is None.
 
         nothing                                                      leave it alone
-        time-up   t, grace                                  write {by: timeout, timeout: True, t, grace} on it
-        term                                                         SIGTERM to its group
+        time-up   stop                                      write {by: timeout, timeout: True, t, grace} on it
+        term      stop                                               SIGTERM to its group
         kill9                                                        SIGKILL to its group
         ended-as  state, rc, reason, hook, ended, sweep, not_retried  record how it ended
         again     attempt, not_before, history, clear, ended, sweep   back to its queue for another attempt
 
-    hook is the event whose hook is owed ("failed", "done"), or None where the outcome owes none or the job
-    carries no such hook. sweep says anything left in the job's process group goes with it. history is the
-    entry a requeue writes for the attempt that ended, and clear the keys it takes off the record, so the
-    store writes what this says and works nothing out again.
+    stop is the job's stop as the decision's transaction writes it, or None where it writes none: a timeout's,
+    and a stop's `going` the first time a TERM is decided for it. hook is the event whose hook is owed
+    ("failed", "done"), or None where the outcome owes none or the job carries no such hook. sweep says
+    anything left in the job's process group goes with it. history is the entry a requeue writes for the
+    attempt that ended, and clear the keys it takes off the record, so the store writes what this says and
+    works nothing out again.
     """
 
     do: str
@@ -94,8 +98,7 @@ class Action:
     attempt: int | None = None
     not_before: float | None = None
     not_retried: str | None = None
-    t: float | None = None
-    grace: float | None = None
+    stop: dict | None = None
     history: dict | None = None
     clear: tuple = ()
 
@@ -126,8 +129,8 @@ def attempt_of(job: dict, rc: int | None, reason: str | None, ended: float) -> d
 def facts(held: bool, code: int | None = None, rc: int | None = None, rc_at: float | None = None,
           now: float = 0.0, hit: bool = False) -> Facts:
     """The facts about one running job, from the readings the core took: whether a run of it is live, the exit
-    code it wrote and when, the status of a child this worker holds, the time those were read at, and whether
-    --retry-if matched the tail of its log.
+    code it wrote and when, the status of a child this worker holds, and whether --retry-if matched the tail of
+    its log. `now` is the decision's clock, which the core puts on inside the decision's transaction (`Facts`).
 
     While the run is live none of the rest counts for anything, since the command itself can write the exit
     code file, so those facts are dropped here rather than read by the decision. A child's status below zero
@@ -194,7 +197,7 @@ def decide(job: dict, facts: Facts) -> Action:
     """
     stop = job.get("stop") or {}
     asked = bool(stop) and not stop.get("timeout")      # a kill: someone asked for this job to stop
-    out = bool(stop.get("sent")) or bool(stop.get("kill9"))     # a signal of pp's went out for this stop
+    out = went(stop) is not None        # pp began sending this stop, and was not refused it
     if facts.held:
         # a run of it is live, so nothing is recorded about how it ended and its exit code says nothing yet:
         # the one thing to do about it is to carry out the stop it carries, or to write a timeout's
@@ -203,9 +206,12 @@ def decide(job: dict, facts: Facts) -> Action:
         if not stop:
             due = (job.get("timeout") and job.get("started")
                    and facts.now - float(job["started"]) >= float(job["timeout"]))
-            return Action(TIME_UP, t=facts.now, grace=grace(job)) if due else Action(NOTHING)
+            return (Action(TIME_UP, stop={"by": "timeout", "timeout": True, "t": facts.now, "grace": grace(job)})
+                    if due else Action(NOTHING))
         if stop.get("sent") is None:
-            return Action(TERM)
+            # recorded before anything goes out, so a worker that dies before recording `sent` leaves it, and
+            # the next one sends the stop again with its first time kept
+            return Action(TERM, stop=None if stop.get("going") else {**stop, "going": facts.now})
         if not stop.get("kill9") and facts.now - float(stop["sent"]) >= float(stop.get("grace") or 0):
             return Action(KILL9)
         return Action(NOTHING)      # signalled, and inside the grace it was given
@@ -215,11 +221,11 @@ def decide(job: dict, facts: Facts) -> Action:
     own = None if facts.rc is not None else (f"signal {facts.sig}" if facts.sig else "lost")
     late = facts.rc is None or after(stop, facts.rc_at)
     if asked and out and late:
-        # a kill's signal stopped it: "killed", and no hook. A code the attempt wrote after that signal went
-        # out is that stop's outcome, and the code is kept
+        # a kill's stop began going out while it ran: "killed", and no hook. A code the attempt wrote after
+        # that is that stop's outcome, and the code is kept
         return _ends(job, facts.rc, "killed", facts.rc_at, sweep, None)
     if asked:
-        # it wrote its own code before the stop went out, or no signal of pp's ever went out for that stop:
+        # it wrote its own code before the stop began going out, or pp never began sending that stop:
         # the job ended as it ended, and the hook that outcome owes fires. A job a kill touched is never retried
         return _ends(job, facts.rc, own, facts.rc_at, sweep, "done" if facts.rc == 0 else "failed")
     if facts.rc == 0 and not late:
@@ -231,18 +237,28 @@ def decide(job: dict, facts: Facts) -> Action:
     return _or_again(job, facts, facts.rc, own, facts.rc_at, sweep)
 
 
+def went(stop: dict) -> float | None:
+    """When pp began sending this stop, or None where it never did or was refused before anything went out.
+
+    `going` is that time. The decision's transaction records it before anything goes out (`decide`), so a
+    worker that dies between sending the stop and recording `sent` leaves it, and the job ends as that stop's
+    outcome rather than as its own. A record whose stop carries no `going` (an older worker's) has `sent` or
+    `kill9` instead, each read before its signal went out."""
+    if stop.get("denied") and not (stop.get("sent") or stop.get("kill9")):
+        return None
+    return stop.get("going") or stop.get("sent") or stop.get("kill9")
+
+
 def after(stop: dict, rc_at: float | None) -> bool:
-    """Whether the attempt wrote its exit code after the first thing went out for this stop.
+    """Whether the attempt wrote its exit code after this stop began going out (`went`).
 
     A stop decides the reason: a container runner writes the code of a container the stop command ended (137),
     so without this a kill or a timeout of a container job would read as the job's own failure. The code is
-    kept either way, and what it decides is the reason (I10). A stop nothing has gone out for has no time to
-    compare against, so no code is ever late for one.
-
-    The time to compare against is read before the stop goes out and recorded once it has (`Worker._send`),
-    so the window in which a stop command is starting belongs to that stop and not to the job."""
-    went = stop.get("sent") or stop.get("kill9")
-    return went is not None and rc_at is not None and rc_at >= went
+    kept either way, and what it decides is the reason (I10). A stop pp never began sending has no time to
+    compare against, so no code is ever late for one. So the window in which a stop command is starting, or a
+    signal is on its way, belongs to that stop and not to the job."""
+    w = went(stop)
+    return w is not None and rc_at is not None and rc_at >= w
 
 
 def hook_due(job: dict, event: str | None) -> str | None:
@@ -319,8 +335,9 @@ def merge_stop(existing: dict | None, ask: dict) -> dict:
     asked and when.
 
     One stop per run: what went out is all the decision and the waiting verb have to go on, so a second ask
-    never writes over `t`, `sent` or `kill9`, and keeps only the shorter of the two graces, since a second ask
-    hastens the SIGKILL and never delays it. It clears `denied`, so a signal pp was refused is asked for again.
+    never writes over `t`, `going`, `sent` or `kill9`, and keeps only the shorter of the two graces, since a
+    second ask hastens the SIGKILL and never delays it. It clears `denied`, so a signal pp was refused is asked
+    for again, and with it the `going` of a stop nothing went out for, which begins again when it is next sent.
 
     A kill's ask on the stop a worker wrote for a timeout becomes the kill's: `by` is the asker, `timeout`
     goes, and the timeout stays in `was`. So a job a kill has touched is ended as a kill, never retried,
@@ -332,7 +349,8 @@ def merge_stop(existing: dict | None, ask: dict) -> dict:
         return dict(ask)
     out = dict(existing)
     out["grace"] = min(float(existing.get("grace") or 0), float(ask.get("grace") or 0))
-    out.pop("denied", None)
+    if out.pop("denied", None) is not None and went(existing) is None:
+        out.pop("going", None)
     if existing.get("timeout") and not ask.get("timeout"):
         out["was"], out["by"] = existing["by"], ask["by"]
         out.pop("timeout")

{ppkit-0.2.1 → ppkit-0.2.2}/pp/lib/jobq.py

@@ -993,9 +993,9 @@ def public(j: dict) -> dict:
     attempt's own output begins), cards (the card indices it reserved), container (the name whatever its
     runner started elsewhere carries), envfile (the file that runner reads its environment from),
     notes [{t, by, text}], reported [{t, by, session}], history (one entry per attempt that ended, each with
-    that attempt's own rcfile, envfile, container, log_at and cards), stop {by, t, grace, timeout, sent,
-    kill9, denied, was, cmd, rc} on a job asked to stop, after [id] on a job that names the jobs it waits
-    for, and from `show` also jobcore.watchers [{pid, verb, session}].
+    that attempt's own rcfile, envfile, container, log_at and cards), stop {by, t, grace, timeout, going,
+    sent, kill9, denied, was, cmd, rc} on a job asked to stop, after [id] on a job that names the jobs it
+    waits for, and from `show` also jobcore.watchers [{pid, verb, session}].
 
     `env` and `shell_env` keep their keys and give every value as "***". A token or a secret is what the help
     says to pass with --env, and an agent reads `--json` to see how its jobs are getting on, so the values
@@ -1003,11 +1003,13 @@ def public(j: dict) -> dict:
     disk, which is the job's own, holds what the job runs with.
 
     A stop's `timeout` is set on the one a worker wrote because the job's own time ran out, and on no other,
-    so what a stop is never rests on the name of whoever asked. `sent` and `kill9` are when each signal went
-    out, read just before it did, so an exit code the attempt wrote while that stop was going out is late
-    against it and is that stop's outcome. `denied` is when pp was refused the job's process group. A stop with `denied` set is not in
-    flight: nothing more goes out for it, and the job runs on until it ends on its own. `was` is who asked
-    for the stop this one landed on, where a kill landed on a timeout's."""
+    so what a stop is never rests on the name of whoever asked. `going` is when pp began sending it, recorded
+    before anything went out, so an exit code the attempt wrote after it is late against it and is that stop's
+    outcome (`jobpolicy.went`). A stop pp was refused before anything went out counts as never begun, whatever
+    its `going` says. `sent` and `kill9` are when each signal went out, read just before it did.
+    `denied` is when pp was refused the job's process group. A stop with `denied` set is not in flight:
+    nothing more goes out for it, and the job runs on until it ends on its own. `was` is who asked for the
+    stop this one landed on, where a kill landed on a timeout's."""
     out = {**NOT_YET, **{k: v for k, v in j.items() if not k.startswith("_")}}
     for k in ("env", "shell_env"):
         if isinstance(out.get(k), dict):
@@ -1456,7 +1458,8 @@ def cmd_show(a) -> int:
                      # what went out for it, off the record: the project's command where that is what went
                      # out, and pp's own signal where it is
                      + ((f", its stop command went out ({_text(st['cmd'])})" if st.get("cmd")
-                         else ", SIGTERM sent") if st.get("sent") else ", nothing signalled")
+                         else ", SIGTERM sent") if st.get("sent")
+                        else ", sending began" if policy.went(st) is not None else ", nothing signalled")
                      + (", SIGKILL sent" if st.get("kill9") else "")
                      + (f", its command exited {st['rc']}" if st.get("rc") is not None else "")
                      # a denied stop is not in flight: say so, or it reads as one still being carried out
@@ -1627,7 +1630,8 @@ def _await_stop(j: dict, grace: float, t: float) -> int:
             q, waited = j["queue"], ui.human_duration(time.time() - started_at)
             went = ("the SIGKILL went out" if stop.get("kill9") else
                     "the project's stop command went out" if stop.get("cmd") else
-                    "the SIGTERM went out" if stop.get("sent") else "no signal of pp's has gone out")
+                    "the SIGTERM went out" if stop.get("sent") else
+                    "sending it began" if policy.went(stop) is not None else "no signal of pp's has gone out")
             said = (f"{_PROG}: job {jid} is still running {waited} after it was asked to stop; {went}. "
                     f"`pp {_PROG} show {jid}` says what went out for it.")
             if jobcore.worker_now(q).up:
@@ -1644,7 +1648,7 @@ def _await_stop(j: dict, grace: float, t: float) -> int:
             continue        # a round of starts took this pass past the deadline: look once more and report
         time.sleep(KILL_POLL)
     stop = cur.get("stop") or {}
-    if not stop.get("sent"):
+    if policy.went(stop) is None:
         print(f"job {jid} had already ended: {_ended_line(cur, jid)}")
     elif not cur.get("reason"):
         # the reason on the record says whose outcome this is: an exit code the attempt wrote after the stop
@@ -1656,8 +1660,10 @@ def _await_stop(j: dict, grace: float, t: float) -> int:
         # and what went out for it, off the record and not off the command the job carries: a command that
         # could not start ran nothing, and pp's own signal went out for that stop instead
         went = ("the project's stop command" if stop.get("cmd") else "SIGTERM")
+        # a stop with nothing marked sent is one whose worker ended while sending it, or whose send found the
+        # run already gone: the record cannot say what went out
         how = (f"SIGKILL after {float(stop.get('grace') or 0):g}s" if stop.get("kill9")
-               else f"stopped on {went}")
+               else f"stopped on {went}" if stop.get("sent") else "ended as its stop was being sent")
         # a stop command that failed is said with its code: pp will not report a stop it did not achieve
         said = "" if not stop.get("rc") else f"; the project's stop command failed (exit {stop['rc']})"
         print(f"killed {jid} ({how}){said}  {_name(cur)}")

{ppkit-0.2.1 → ppkit-0.2.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ppkit"
-version = "0.2.1"           # the same as pp/__init__.py's __version__; tests/test_cli.py checks
+version = "0.2.2"           # the same as pp/__init__.py's __version__; tests/test_cli.py checks
 description = "A CLI kit for user & agents"
 readme = "README.md"
 license = "Apache-2.0"

{ppkit-0.2.1 → ppkit-0.2.2}/tests/test_queue.py

@@ -3093,21 +3093,22 @@ class TestTwoAtOnce(QueueCase):
         self.assertIsNone(rec.get("reason"))        # the code it wrote itself, not a kill's
         self.assertIsNone(rec.get("history"))       # and no second attempt of a job a kill has touched
 
-    def test_a_stop_whose_worker_died_before_it_recorded_the_signal_is_not_retried(self):
-        """A worker killed between signalling a job's group and recording that it did leaves a stop with
-        nothing marked on it and a run that is over. The next worker ends the job there: a stop is never
-        retried, however little of it was recorded.
-
-        What the record says went out is all the decision has, and it says nothing did, so the job is
-        recorded as its own run ended rather than as a kill pp carried out."""
+    def test_a_kill_whose_run_ended_before_any_send_is_not_retried(self):
+        """A kill's stop lands, and the run ends before any worker began sending it: something else ended the
+        group. The next worker ends the job there, and a job a kill has touched is never retried, however
+        little of its stop was carried out.
+
+        No worker recorded a `going` for that stop, so the job is recorded as its own run ended (`lost`)
+        rather than as a kill pp carried out. A worker that dies after it began sending is the other case
+        (test_a_kill_whose_worker_died_before_marking_its_signal_is_still_a_kill)."""
         j = self.add("--retries", "3", cmd="sleep 30")
         w = jobq.worker_for("default")
         w.tick()
         with jobstore.txn() as s:
             cur = s.job(j, must_be=("running",))
-            cur["stop"] = {"by": "user", "t": time.time(), "grace": 10.0}   # written; nothing recorded yet
+            cur["stop"] = {"by": "user", "t": time.time(), "grace": 10.0}   # asked; no worker began sending it
             s.save(cur)
-        proc.signal_group(jobcore._group_handle(cur), signal.SIGKILL)          # the signal went out ...
+        proc.signal_group(jobcore._group_handle(cur), signal.SIGKILL)          # something else ends the run ...
         journey.until(lambda: jobq.lock.Claim.holder(jobstore.run_claim("default", j)) is None, timeout=10,
                       what="the job's run to be over")
         jobq.worker_for("default").carry_out()                                       # ... and the next worker reaps it
@@ -3477,8 +3478,10 @@ class TestTheWorkersWait(QueueCase):
         self.assertEqual(self.state(j), "running")
 
 
-# what has gone out for a stop, and whether pp was refused it: every state `_send` can leave one in
-SENT = ("unsent", "sent", "sent-kill9", "denied", "sent-denied")
+# what has gone out for a stop, and whether pp was refused it: every state `_send` can leave one in, with the
+# `going` the decision's transaction records before it and without it, as a record an older worker wrote is
+SENT = tuple(f"{going}{out}" for going in ("", "going-")
+             for out in ("unsent", "sent", "sent-kill9", "denied", "sent-denied"))
 # and the stop states a record can hold: who asked crossed with those, and no stop at all
 STOPS = ("none",) + tuple(f"{by}-{out}" for by in ("kill", "timeout") for out in SENT)
 
@@ -3488,23 +3491,24 @@ class TestTheDecision(unittest.TestCase):
     the one place that says what a second stop on it comes to. Both are pure, so this table needs no process,
     no thread and no clock.
 
-    Every combination of the facts is a row: the run's claim held or free, no exit code or one that is 0 or
-    not (written while the run was live, or after it), the eleven states a stop on the record can be in (who
-    asked for it crossed with what has gone out for it, and no stop), the grace up or not, the attempt number
-    (its first, and one far enough along that the delay would pass RETRY_DELAY_CAP), attempts left or none,
-    the job's time up or not, a signal its child died of or none, and a --retry-if that matches or one that
-    cannot. Each row's expected action is written once below, against the rule `pp help queue` states it as,
-    in the order that help states them; a row no rule holds for is a failure, so a new fact or a new action
-    cannot go in without saying what it means everywhere.
+    Every combination of the facts is a row: the run's claim held or free, no exit code or one that is 0 or not
+    (written while the run was live, or after it), the twenty-one states a stop on the record can be in (who asked
+    for it crossed with whether pp began sending it and what has gone out for it, and no stop), the grace up or not,
+    the attempt number (its first, and one far enough along that the delay would pass RETRY_DELAY_CAP), attempts
+    left or none, the job's time up or not, a signal its child died of or none, and a --retry-if that matches or one
+    that cannot. Each row's expected action is written once below, against the rule `pp help queue` states it as, in
+    the order that help states them; a row no rule holds for is a failure, so a new fact or a new action cannot go
+    in without saying what it means everywhere.
 
-    Each rule is written on the facts a stop carries, namely whether a kill asked for it, whether a signal
-    of pp's went out and whether pp was refused, and never on the name of a state, so a state nobody thought
-    of cannot slip past a rule that matched its name."""
+    Each rule is written on the facts a stop carries, namely whether a kill asked for it, whether pp began
+    sending it, what went out for it and whether pp was refused, and never on the name of a state, so a state
+    nobody thought of cannot slip past a rule that matched its name."""
 
     NOW = 2000.0
     GRACE = 10.0
     JOB_GRACE = 7.0         # the job's own, which a timeout's stop takes and no constant of pp's does
     RC_AT = 1990.0          # when the code was written, which is when the job really ended
+    GOING = 1960.0          # when pp began sending a stop that records it: before anything of it went out
     DELAY = 20.0
 
     def stop(self, kind: str, grace_up: bool) -> dict | None:
@@ -3517,6 +3521,8 @@ class TestTheDecision(unittest.TestCase):
         st = {"by": "timeout" if by == "timeout" else "user", "t": self.NOW - 60, "grace": self.GRACE}
         if by == "timeout":
             st["timeout"] = True
+        if "going" in went:
+            st["going"] = self.GOING
         if "sent" in went:
             st["sent"] = self.NOW - (self.GRACE + 5 if grace_up else 1)
         if "kill9" in went:
@@ -3531,12 +3537,14 @@ class TestTheDecision(unittest.TestCase):
         whether a kill asked for it (and not a worker's own timeout), whether a signal of pp's went out, and
         whether pp was refused the job's group."""
         if kind == "none":
-            return {"any": False, "asked": False, "sent": False, "kill9": False, "denied": False, "out": False}
+            return {"any": False, "asked": False, "going": False, "sent": False, "kill9": False,
+                    "denied": False, "out": False}
         by, out = kind.split("-", 1)
         went = out.split("-")           # the field names, so "unsent" is not read as a sent one
-        sent, kill9 = "sent" in went, "kill9" in went
-        return {"any": True, "asked": by != "timeout", "sent": sent, "kill9": kill9,
-                "denied": "denied" in went, "out": sent or kill9}
+        going, sent, kill9, denied = "going" in went, "sent" in went, "kill9" in went, "denied" in went
+        # what went out, or what pp began to send and was not refused: a worker can die between the two
+        return {"any": True, "asked": by != "timeout", "going": going, "sent": sent, "kill9": kill9,
+                "denied": denied, "out": sent or kill9 or (going and not denied)}
 
     # the first attempt, and one far enough along that doubling the delay per attempt would pass the cap
     ATTEMPTS = (1, 20)
@@ -3571,9 +3579,10 @@ class TestTheDecision(unittest.TestCase):
 
     # -- what each row's action is -------------------------------------------------------------------------
     def went(self, r: dict):
-        """When the first thing went out for this row's stop, or None where nothing has."""
+        """When pp began sending this row's stop, or None where nothing has gone out for it: its `going`, and
+        for a record without one the first signal's time."""
         st = self.stop(r["stop"], r["grace_up"]) or {}
-        return st.get("sent") or st.get("kill9")
+        return (st.get("going") or st.get("sent") or st.get("kill9")) if r["out"] else None
 
     def rc_at(self, r: dict):
         """When the attempt wrote its exit code: a second after the first thing went out for its stop, or a
@@ -3583,6 +3592,13 @@ class TestTheDecision(unittest.TestCase):
         went = self.went(r)
         return self.RC_AT if went is None else (went + 1 if r["late"] else went - 1)
 
+    def term(self, r: dict) -> jobpolicy.Action:
+        """The SIGTERM of a stop nothing is marked sent for. The decision's transaction records `going` on it
+        the first time, and keeps the first one after that, since a stop already begun is sent again."""
+        if r["going"]:
+            return jobpolicy.Action(jobpolicy.TERM)
+        return jobpolicy.Action(jobpolicy.TERM, stop={**self.stop(r["stop"], r["grace_up"]), "going": self.NOW})
+
     def late(self, r: dict) -> bool:
         """Whether the code this attempt wrote came after its stop went out, which an attempt that wrote none
         counts as, since there is no code of its own to stand instead."""
@@ -3638,7 +3654,7 @@ class TestTheDecision(unittest.TestCase):
              lambda r: jobpolicy.Action(jobpolicy.NOTHING)),
             ("A stop nobody has signalled yet goes out as SIGTERM to the job's process group.",
              lambda r: r["held"] and r["any"] and not r["sent"],
-             lambda r: jobpolicy.Action(jobpolicy.TERM)),
+             self.term),
             ("A stop whose grace is up is escalated to SIGKILL, once.",
              lambda r: r["held"] and r["sent"] and not r["kill9"] and r["grace_up"],
              lambda r: jobpolicy.Action(jobpolicy.KILL9)),
@@ -3647,27 +3663,28 @@ class TestTheDecision(unittest.TestCase):
              lambda r: jobpolicy.Action(jobpolicy.NOTHING)),
             ("A job with no stop whose own timeout has run out gets a timeout's stop.",
              lambda r: r["held"] and not r["any"] and r["due"],
-             lambda r: jobpolicy.Action(jobpolicy.TIME_UP, t=self.NOW, grace=self.JOB_GRACE)),
+             lambda r: jobpolicy.Action(jobpolicy.TIME_UP, stop={"by": "timeout", "timeout": True, "t": self.NOW,
+                                                                  "grace": self.JOB_GRACE})),
             ("A job with no stop whose time is not up is left as it is.",
              lambda r: r["held"] and not r["any"],
              lambda r: jobpolicy.Action(jobpolicy.NOTHING)),
             # its run is over, so its exit code is the command's last word and this is where it ends
-            ('A job a kill\'s signal stopped is recorded "killed" and fires no hook; an exit code the attempt '
-             "wrote after that signal went out is kept as the attempt's code.",
+            ('A job still running when a kill\'s stop began going out is recorded "killed" and fires no hook; '
+             "an exit code the attempt wrote after that is kept as the attempt's code.",
              lambda r: not r["held"] and r["asked"] and r["out"] and self.late(r),
              lambda r: self.ends(r["rc"], "killed", self.rc_at(r), self.sweep(r), None)),
-            ("A job that wrote its own exit code before a kill's signal went out, or that a kill wrote a stop "
-             "for that no signal of pp's ever reached, ends with its own outcome, the exit code it wrote or "
-             "the signal it died of, and fires the hook that outcome owes.",
+            ("A job that wrote its own exit code before a kill's stop began going out, or whose kill pp never "
+             "began sending, ends with its own outcome, the exit code it wrote or the signal it died of, and "
+             "fires the hook that outcome owes.",
              lambda r: not r["held"] and r["asked"],
              lambda r: self.ends(r["rc"], self.own(r), self.rc_at(r), self.sweep(r),
                                  "done" if r["rc"] == 0 else "failed")),
             ("An exit code of 0 is done, and on_done fires.",
              lambda r: not r["held"] and r["rc"] == 0 and not self.late(r),
              lambda r: self.ends(0, None, self.rc_at(r), self.sweep(r), "done")),
-            ("A timeout's stop pp signalled or was refused is a timeout, which may be retried; an exit code "
-             "the attempt wrote after that stop went out is kept as the attempt's code, and on a final "
-             "timeout on_fail fires.",
+            ("A timeout's stop pp began sending or was refused is a timeout, which may be retried; an exit "
+             "code the attempt wrote after that stop began going out is kept as the attempt's code, and on a "
+             "final timeout on_fail fires.",
              lambda r: not r["held"] and self.late(r) and (r["out"] or r["denied"]),
              lambda r: self.failure(r, r["rc"], "timeout", self.rc_at(r), self.sweep(r))),
             ("Anything else is the job's own failure, the exit code it wrote or the signal it died of, "
@@ -3688,7 +3705,7 @@ class TestTheDecision(unittest.TestCase):
             seen.add(what)
             job, facts = self.case(r)
             self.assertEqual(jobpolicy.decide(job, facts), then(r), f"{what}: {r}")
-        self.assertEqual(n, 8448)                               # 2*3*11*2*2*2*2*2*2*2 rows
+        self.assertEqual(n, 16128)                              # 2*3*21*2*2*2*2*2*2*2 rows
         self.assertEqual(seen, {what for what, _w, _t in rules})    # and every rule is a row of its own
         # both sides of the delay's cap are a row here, so neither arm can go quiet under a later constant
         self.assertGreater(self.DELAY * 2 ** (max(self.ATTEMPTS) - 1), jobpolicy.RETRY_DELAY_CAP)
@@ -3780,10 +3797,13 @@ class TestTheDecision(unittest.TestCase):
 
     def test_a_second_stop_merges_into_the_one_on_the_record(self):
         """One stop per run of a job: what went out is all the decision and the waiting verb have to go on, so
-        a second ask keeps `t`, `sent` and `kill9` and only shortens the grace. A kill's ask on a timeout's
-        stop is a kill, so the job is ended as one and never retried."""
-        tout = {"by": "timeout", "timeout": True, "t": 100.0, "grace": 300.0, "sent": 101.0, "kill9": 102.0}
-        out = {"by": "ann", "t": 100.0, "grace": 300.0, "sent": 101.0, "kill9": 102.0}
+        a second ask keeps `t`, `going`, `sent` and `kill9` and only shortens the grace. A kill's ask on a
+        timeout's stop is a kill, so the job is ended as one and never retried. A refused stop is asked for
+        again, and where nothing of it went out it begins going out again, so its `going` goes too."""
+        tout = {"by": "timeout", "timeout": True, "t": 100.0, "grace": 300.0, "going": 100.5, "sent": 101.0,
+                "kill9": 102.0}
+        out = {"by": "ann", "t": 100.0, "grace": 300.0, "going": 100.5, "sent": 101.0, "kill9": 102.0}
+        refused = {"by": "ann", "t": 100.0, "grace": 300.0, "going": 100.5, "denied": 101.0}
         kill = {"by": "ann", "t": 500.0, "grace": 10.0}
         tmo = {"by": "timeout", "timeout": True, "t": 500.0, "grace": 10.0}
         rows = [
@@ -3794,6 +3814,8 @@ class TestTheDecision(unittest.TestCase):
              {**out, "was": "timeout", "grace": 10.0}),
             ("a timeout on a kill's stop changes nothing but the grace", out, tmo, {**out, "grace": 10.0}),
             ("a refused stop is asked for again", {**out, "denied": 103.0}, kill, {**out, "grace": 10.0}),
+            ("a stop refused before anything went out begins going out again", refused, kill,
+             {"by": "ann", "t": 100.0, "grace": 10.0}),
             ("a timeout on a timeout's stop is still the timeout's", tout, tmo, {**tout, "grace": 10.0}),
             ("an asker called timeout asks as a kill, and the stop stops being a timeout's", tout,
              {"by": "timeout", "t": 500.0, "grace": 10.0},
@@ -4998,46 +5020,6 @@ class TestWatchEvents(unittest.TestCase):
         self.assertIsNone(self.ev("queued", "queued"))
 
 
-# ---- one test per mechanism invariant, by fault injection -------------------------------------------
-#
-# What follows is one test per invariant the mechanism keeps, each by a fault at a named step: a worker that
-# ends where a SIGKILL would end it, two processes that act at once, a record naming a process id another
-# process holds, a reader of the store beside a writer of it. The processes the tests need come first, each a
-# program of a few lines, then the scope they share (InjectedCase), then the tests.
-
-
-# ---- one test per mechanism invariant, by fault injection -------------------------------------------
-#
-# What follows is one test per invariant the mechanism keeps, each by a fault at a named step: a worker that
-# ends where a SIGKILL would end it, two processes that act at once, a record naming a process id another
-# process holds, a reader of the store beside a writer of it. The processes the tests need come first, each a
-# program of a few lines, then the scope they share (InjectedCase), then the tests.
-
-
-# ---- one test per mechanism invariant, by fault injection -------------------------------------------
-#
-# What follows is one test per invariant the mechanism keeps, each by a fault at a named step: a worker that
-# ends where a SIGKILL would end it, two processes that act at once, a record naming a process id another
-# process holds, a reader of the store beside a writer of it. The processes the tests need come first, each a
-# program of a few lines, then the scope they share (InjectedCase), then the tests.
-
-
-# ---- one test per mechanism invariant, by fault injection -------------------------------------------
-#
-# What follows is one test per invariant the mechanism keeps, each by a fault at a named step: a worker that
-# ends where a SIGKILL would end it, two processes that act at once, a record naming a process id another
-# process holds, a reader of the store beside a writer of it. The processes the tests need come first, each a
-# program of a few lines, then the scope they share (InjectedCase), then the tests.
-
-
-# ---- one test per mechanism invariant, by fault injection -------------------------------------------
-#
-# What follows is one test per invariant the mechanism keeps, each by a fault at a named step: a worker that
-# ends where a SIGKILL would end it, two processes that act at once, a record naming a process id another
-# process holds, a reader of the store beside a writer of it. The processes the tests need come first, each a
-# program of a few lines, then the scope they share (InjectedCase), then the tests.
-
-
 # ---- one test per mechanism invariant, by fault injection -------------------------------------------
 #
 # What follows is one test per invariant the mechanism keeps, each by a fault at a named step: a worker that
@@ -5409,11 +5391,13 @@ class TestAWorkerKilledWhileItStops(InjectedCase):
         stop = jobstore.find(j)["stop"]
         self.assertEqual((stop["t"], stop.get("sent"), stop.get("kill9"), stop.get("denied")),
                          (asked["t"], None, None, None))
+        self.assertIsNotNone(stop["going"])         # the decision's transaction recorded it before the send
         self.assertEqual(self.state(j), "running")
         went = self.recording()
         w.carry_out()
         self.assertEqual(went, [signal.SIGTERM])
         self.assertIsNotNone(jobstore.find(j)["stop"]["sent"])
+        self.assertEqual(jobstore.find(j)["stop"]["going"], stop["going"])     # the time it first began
         for _ in range(3):              # inside the grace it was given, so nothing more goes out for it
             w.carry_out()
         self.assertEqual(went, [signal.SIGTERM])
@@ -5421,7 +5405,8 @@ class TestAWorkerKilledWhileItStops(InjectedCase):
     def test_a_signal_that_went_out_is_recorded_before_anything_else_goes_out(self):
         """What went out for a stop is recorded after it went out, so a record that says a signal went out is
         a record of one that did. A worker that ends between the signal and the record leaves a stop with
-        nothing marked on it, which is the one side of that order a decision can be wrong on."""
+        nothing marked sent on it. The `going` the decision recorded before the signal is still on it, which is
+        what keeps the next decision right about that stop (the two tests after this one)."""
         j, w = self.a_job_whose_run_is_live()
         asked = self.a_kill_asks(j, grace=10.0)
         self.assertEqual(self.a_worker_killed_at("the signal"), [f"sent {int(signal.SIGTERM)}"])
@@ -5487,10 +5472,11 @@ class TestAWorkerKilledWhileItStops(InjectedCase):
         self.assertNotIn("cmd", group)
 
     def test_a_code_written_while_the_stop_goes_out_is_that_stops_outcome(self):
-        """A stop decides the reason, and what an exit code is late against is the time that stop went out.
-        That time is read before anything goes out for it, so the window in which the project's stop command
-        is starting belongs to the stop: a container runner writing the code of the container that command
-        ended (137) inside it is that stop's outcome, with the code kept, and not the job's own failure.
+        """A stop decides the reason, and what an exit code is late against is the time that stop began going
+        out, its `going`. That time is recorded before anything goes out for it, so the window in which the
+        project's stop command is starting belongs to the stop: a container runner writing the code of the
+        container that command ended (137) inside it is that stop's outcome, with the code kept, and not the
+        job's own failure.
 
         The window is driven by a named step, the start of the project's stop command, so the test says
         which window it is about instead of hoping a sleep lands in it."""
@@ -5520,6 +5506,7 @@ class TestAWorkerKilledWhileItStops(InjectedCase):
         stop = jobstore.find(j)["stop"]
         self.assertEqual(len(wrote), 1)
         self.assertIsNotNone(stop["sent"])
+        self.assertLessEqual(stop["going"], wrote[0])   # recorded before the stop went out
         self.assertLessEqual(stop["sent"], wrote[0])    # read before the stop went out, recorded once it had
         # the run ends with no code of its own, so the code written inside the window is the one that is read
         proc.signal_group(jobcore._group_handle(jobstore.find(j)), signal.SIGKILL)
@@ -5529,6 +5516,79 @@ class TestAWorkerKilledWhileItStops(InjectedCase):
         self.assertEqual((rec["state"], rec["reason"], rec["rc"]), ("failed", "killed", 137))
         self.assertIsNone(rec.get("history"), "a job a kill has touched was retried")
 
+    def test_a_kill_whose_worker_died_before_marking_its_signal_is_still_a_kill(self):
+        """A worker that ends between a kill's SIGTERM and the record of it leaves a stop with nothing marked
+        sent, and the signal ended the job. The `going` the decision recorded before the signal is on that
+        stop, so the next worker ends the job as the kill's: "killed", and no hook (A11), rather than as the
+        signal the job died of with its on_fail."""
+        f = self.hook_file()
+        j, w = self.a_job_whose_run_is_live("--on-fail", f"echo x >> {f}")
+        self.a_kill_asks(j, grace=10.0)
+        self.assertEqual(self.a_worker_killed_at("the signal of the group"), [f"sent {int(signal.SIGTERM)} alive"])
+        stop = jobstore.find(j)["stop"]
+        self.assertEqual((stop.get("sent"), stop.get("going") is not None), (None, True))
+        self.assertIn("going", jobq.public.__doc__)     # it is named where the stop's fields are, and is there
+        self.assertEqual(json.loads(self.call("show", str(j), "--json")[1])["stop"]["going"], stop["going"])
+        self.the_run_is_over(j)
+        w.carry_out()
+        rec = jobstore.find(j)
+        self.assertEqual((rec["state"], rec["reason"], rec.get("rc")), ("failed", "killed", None))
+        self.assertNotIn("hook_due", rec)
+        self.assertEqual(rec.get("hooks") or [], [], "a kill fired the job's hook")
+
+    def test_a_kill_whose_worker_died_as_its_stop_command_started_is_still_a_kill(self):
+        """The same window where the project's stop command is what goes out: the worker ends once that command
+        has started, the command ends the work, and the runner writes the code the work got (137). That code
+        came after the stop began going out, so it is the kill's outcome with the code kept, and the job's
+        on_fail does not fire."""
+        f, stopped = self.hook_file(), self.gate("stopped")
+        j, w = self.a_job_whose_run_is_live("--on-fail", f"echo x >> {f}",
+                                            cmd=f"while [ ! -f {stopped} ]; do sleep 0.02; done; exit 137")
+        with jobstore.txn() as s:
+            cur = s.job(j, must_be=("running",))
+            cur["stop_cmd"] = f"touch {stopped}"
+            s.save(cur)
+        self.a_kill_asks(j, grace=10.0)
+        self.a_worker_killed_at("the start")        # the stop command started, and the worker ended
+        stop = jobstore.find(j)["stop"]
+        self.assertEqual((stop.get("sent"), stop.get("going") is not None), (None, True))
+        self.the_run_is_over(j)
+        w.carry_out()
+        rec = jobstore.find(j)
+        self.assertEqual((rec["state"], rec["reason"], rec["rc"]), ("failed", "killed", 137))
+        self.assertNotIn("hook_due", rec)
+        self.assertEqual(rec.get("hooks") or [], [], "a kill fired the job's hook")
+
+    def test_a_code_written_before_the_decision_is_the_jobs_own(self):
+        """The time a stop began going out is read inside the decision's transaction, never with the facts
+        before it. A job that ends on its own after its facts were read and before that transaction is sent
+        its stop on facts that say it runs. It still ends with its own outcome and fires the hook that outcome
+        owes, since its code was written before the stop's `going`.
+
+        The window is driven by a named step, the reading of the facts, so the test says which window it is
+        about instead of hoping a sleep lands in it."""
+        f = self.hook_file()
+        j, w = self.a_job_whose_run_is_live("--on-done", f"echo x >> {f}")
+        real = jobcore.Worker._facts
+
+        def the_job_ends_once_its_facts_are_read(wk, job):
+            facts = real(wk, job)
+            self.assertTrue(facts.held)
+            self.open_the_gate()
+            self.the_run_is_over(j)
+            return facts
+        self.a_kill_asks(j, grace=10.0)
+        with mock.patch.object(jobcore.Worker, "_facts", the_job_ends_once_its_facts_are_read):
+            w.carry_out()
+        rec = jobstore.find(j)
+        at = jobcore.ended_on_its_own(rec)[1]
+        self.assertEqual(rec["state"], "running")
+        self.assertLess(at, rec["stop"]["going"], "the stop began going out before the job's own end")
+        w.carry_out()
+        rec = jobstore.find(j)
+        self.assertEqual((rec["state"], rec.get("reason"), rec["rc"]), ("done", None, 0))
+        self.assertEqual([h["event"] for h in rec.get("hooks") or []], ["done"])
+
     def test_a_grace_that_is_up_escalates_once_and_only_once(self):
         """The SIGKILL of a stop whose grace is up goes out once. A worker that ends between the grace being
         up and that signal leaves a stop with no SIGKILL marked, the next worker sends it, and no pass after