ppkit 0.1.0__tar.gz → 0.2.0__tar.gz

ppkit-0.2.0/CHANGELOG.md

@@ -0,0 +1,234 @@
+# Changelog
+
+## v0.2.0
+
+The interface this minor version keeps. `pp help queue` says what is promised
+and what may change in a patch. `pp queue` stops what a container runner
+started, takes a campaign from a file with one job's flags per line, and runs
+a stage after the one it names.
+
+`pp queue`: a job is named and selected the same way by every verb.
+
+- A job added without `--tag` has no name. Views show its command, and a
+  selection by tag never matches it.
+- A `--tag` is at most 64 characters of letters, digits and `.`, `_`, `-`
+  and `:`. It may not hold `..`, be `.` on its own, or read as an id or a
+  range, because a tag can be part of a log file's name.
+- `counter` is not a queue name, beside `probes` and `watchers`: a queue of
+  that name took the place of the job id counter, and every later `add`
+  failed.
+- Every verb that takes jobs takes the same five forms: an id, a range
+  (`12-15`), a tag matched whole, `--prefix P` (repeatable), and `--all`
+  with `-q NAME`. `--all` without `-q` is a usage error, and a tag or a
+  prefix that names jobs in two queues is an error that lists the ids.
+  `pp queue note` is the exception, and takes one job by its id or by a
+  tag that names one.
+- `pp queue clean` takes a selection as well as `--older AGE`.
+
+`pp queue`: a project says how its work is launched and stopped, so a kill
+and a timeout reach work that is not in the job's process group.
+
+- `[queue] stop` in a project's `.pp.toml`, and `--stop CMD` on `add`:
+  `"group"` where the runner's work stays in the job's own process group,
+  else a shell command that ends it. `pp queue add` is refused in a project
+  that sets `[queue] runner` and no stop, naming both spellings.
+  `--runner CMD` on `add` sets the runner for one job.
+- In a project with a command stop, `pp queue kill` and a `--timeout` run
+  that command in place of the SIGTERM to the job's process group. The
+  grace, the SIGKILL and the sweep of the group are unchanged, so a project
+  with `stop = "group"` behaves as it did. The kill line says which ran, and
+  a stop command that failed is reported with its exit code.
+- A job added with `--no-runner` carries `stop = "group"` whatever its
+  project says.
+- The sweep that runs when a job's run is over also runs the project's stop
+  command, where there is one and the attempt wrote no exit code of its own,
+  and it takes the job's process group with it. So nothing the runner
+  started outlives the attempt that started it, and a leaked process no
+  longer holds a reservation the next job is told it may use.
+- A job and its runner are given `PP_QUEUE_CONTAINER`, a name no other run
+  on the machine has, one per attempt, beginning `pp-<queue>-<id>-`; and
+  `PP_QUEUE_ENVFILE`, the path of a file of `KEY=VALUE` lines readable by
+  its owner alone. That file holds the project's `[queue] env`, the job's
+  `--env`, `CUDA_VISIBLE_DEVICES` where a card was reserved, `PP_QUEUE`,
+  `PP_QUEUE_JOB` and `PP_QUEUE_CONTAINER`, and nothing else of the
+  machine's: no `PATH`, `VIRTUAL_ENV`, `CONDA_PREFIX`, `CONDA_DEFAULT_ENV`
+  or `PP_QUEUE_RCFILE`. The record names it, and it goes with the record.
+- An `--env` value, or a `[queue] env` value, holding a newline is refused,
+  and so is a name holding `=`, NUL or a newline.
+- An exit code an attempt wrote while a stop of it was going out is that
+  stop's outcome, killed or timeout, with the code kept as `rc`.
+- `pp queue kill --grace DUR` takes a duration and overrides the job's own
+  grace for the jobs that kill names.
+
+`pp queue`: a job's output goes where the project says.
+
+- `log` in a project's `[queue]`, and `--log PATH` on `add`: a path with
+  `{tag}` or `{id}` in it, relative to the project root or absolute. Any
+  other placeholder, a path holding `..`, and a relative path that leaves
+  the project root are refused. `{tag}` on a job with no tag is filled in
+  with the job's id.
+- Every attempt appends to that one file, after a `[pp queue]` line naming
+  the job and its command, and several jobs may name one file. The record
+  names the file; `log`, `show`, a failure's tail and `PP_QUEUE_LOG` all
+  read it from there. `drop` and `clean` remove the files pp put in its own
+  store, and never a log at a path you named.
+- `--retry-if` matches from the byte offset where the failed attempt's own
+  output begins, so it no longer matches an earlier attempt's lines or pp's
+  own line.
+
+`pp queue`: a campaign file carries one job's flags per line, and a job can
+wait for the jobs it names.
+
+- A line of a `--from` file is the rest of an `add` command line: its own
+  flags, then `--`, then the command. A flag on the `add` is the default for
+  every line and a line's own flag wins over it; `--env` and `--need` merge
+  key by key. A line that is only a command behaves as before.
+- A line is split into words the way a shell splits them, with quotes and
+  backslashes honoured. A line holding `|`, `&`, `;`, `<`, `>`, `(`, `)`,
+  `` ` `` or `$(` outside quotes is refused with its line number, and so is a
+  line with no `--` whose first word begins with `-`. A file one line of
+  which is refused queues nothing. A command that wants a shell operator
+  passes it quoted: `-- sh -c 'python x.py | grep -v noise'`.
+- `--from FILE` together with a command on the same command line is a usage
+  error, and so is a line that names the queue, the file, `--hold` or
+  `--no-start`, which are the batch's and no line's.
+- `--after SELECTION` on `add`: the job waits until every job that selection
+  named has ended, whatever the outcome. The value is one or more selections
+  separated by commas (`41,42`, `41-66`, `s1_eval,s1_report`), read as the
+  job is added and kept as ids on the record. An id no job has is a usage
+  error and nothing is queued. A job that is waiting is skipped rather than
+  waited for, so it never becomes the head of its queue, and a job whose
+  record is gone counts as ended.
+- `pp queue retry` over a range, a tag, a prefix or `--all` shows a plan and
+  asks, as `kill` and `drop` do; `-n` shows the plan and `-y` does not ask.
+  A job that succeeded has ended, so it suits `retry`. The new job is owned
+  by the session that asked for it, and a priority is set with
+  `pp queue reprio` on the job the retry queued.
+- A bare `pp queue wait` waits for every job that is queued or running
+  instead of returning at the first outcome. It names no held job; a
+  selection that names one waits for it and says so. `--timeout` is honoured
+  whatever `--poll` is, and ids together with `--idle` or `--mine` is a
+  usage error.
+- A worker that adopts a running job takes the offset its output begins at
+  from the job's own run claim, so `--retry-if` reads that attempt's lines
+  and never pp's own line.
+
+`pp queue`: every setting has one declaration, one default and one order of
+its sources.
+
+- Each setting of a job is a key of a project's `.pp.toml [queue]` as well
+  as of `[queue.NAME]`: `timeout`, `retries`, `retry_delay`, `retry_if`,
+  `on_fail`, `on_done`, `prio`, `gpu_gb`, `cpu`, `mem_gb`, `need`, `env`,
+  `log` and `grace`. The first source that gives a setting wins, in this
+  order: `add`'s flag, the project's `[queue]`, the queue's `[queue.NAME]`,
+  the built-in default. `env` and `need` merge key by key.
+- `grace` is one setting of the job: `--grace DUR` on `add`, the `grace`
+  key, the resolved value on the record, and 10s by default. A timeout's
+  stop and a kill of that job both take it.
+- Durations are read one way everywhere: `90s`, `30m`, `2h`, `1d`, a bare
+  number of seconds, `.5` and `1.5h`, and `none` where a setting offers it.
+  An empty value, a value that is not finite and one over 1e9 seconds are
+  refused, in a config key as on a command line.
+- Numbers are bounded: `retries` at most 1000, `retry_delay` at most the
+  one-hour cap, a queue's `poll` longer than 0. A need that is 0, negative
+  or nan is refused.
+- `pp queue work` takes no `--backfill` and no `--poll`: both are keys of
+  `[queue.NAME]`, which the queue's worker re-reads every loop, so they
+  change under running jobs with no restart.
+
+`pp queue`: the store and the machine surface.
+
+- A queue's `FORMAT` is 2. A pp that reads another format refuses the queue
+  where it opens it, naming the format the queue says and the one that pp
+  reads, rather than reading it.
+- The `--json` field `gpu`, an integer, becomes `cards`, a list of card
+  indices; a job that reserved no card has `cards: []`.
+- New `--json` fields: `stop_cmd`, `grace`, `after`, `container`, `envfile`,
+  `log_at`, and `cards`, `container`, `envfile` and `log_at` in each
+  `history` entry. `show --json` carries every promised field on a job in
+  each of the five states: a field the record does not hold yet is null, or
+  an empty list. The two exceptions are `after`, which is there only on a
+  job that names the jobs it waits for, and `stop`, only on a job asked to
+  stop.
+- `pp help queue` is one page: a first job, a project whose work runs in a
+  container in four steps, names and selection, campaign files, the settings
+  table, how a job ends, workers, and what is left to you.
+
+`pp kill`: `--grace DUR` takes a duration in the spelling every duration pp
+takes has. A bare number is still seconds.
+
+`pp promote`, `pp cmd` and `pp http`: plans under `-n`, help by flag, and
+links one target holds.
+
+- `pp -h NAME` and `pp --help NAME` print that command's help, as
+  `pp help NAME` does. `pp -h NAME VERB` prints the verb's.
+- `pp promote rollback`, `pp promote link`, `pp promote init` and
+  `pp cmd new` take `-n`: each one shows its plan and changes nothing.
+- `pp promote` does not take over a link path another target holds: the
+  refusal names that target, and `--force-links` moves the link.
+- `pp promote status --json` has `failed`: the current release's
+  activation failed, or nothing gated it.
+- `pp promote diff --json` has `source_gone`. A target whose project
+  directory is gone exits 1 there, in place of git's error.
+- `pp cmd ls --json` has `aliases`, each command's short names.
+- An unknown key in `[[http.roots]]` or `[[http.discover]]` is a config
+  error naming the file and the key, so a misspelt `marker` stops the
+  server instead of serving every folder under `under`.
+- `pp help http` documents every `[http]` key with its default.
+
+## v0.1.1
+
+Fixes. A few command lines and config values that did harm are refused.
+
+- `pp batch run` refuses a file-name field where a shell would read the
+  name again as code: inside `$( )`, backticks, `${ }`, arithmetic, a
+  `[[ ]]` test, a case pattern, a comment, `eval`, `trap`, a shell's `-c`
+  script, or a template with a heredoc or a function. The message names the
+  spelling that works: `sh -c 'cmd "$1"' sh {}`. Its plan lists every
+  command.
+- A `pp batch rename` that is interrupted leaves a journal of what landed,
+  and `pp batch undo` takes that journal and says so.
+- `pp promote WORD` releases nothing when the word is neither a verb nor a
+  path. It used to release the project in the current directory.
+- A wrong value in `[promote]` is refused, naming the file and the key.
+  `name` must be one path component: `name = ".."` let `gc` remove every
+  release and plugin.
+- `pp promote` keeps the release it flips away from, so the one you rolled
+  back to is still there after the next promote.
+- A socket or fifo in a tree project no longer hangs `pp promote`, and a
+  file pp cannot read is a blocker that offers `exclude`.
+- A hook that leaves a process running no longer holds `pp promote`. Each
+  hook's output is a file beside the run's log.
+- A release that `pp promote gc` cannot remove stays whole, and gc exits 1.
+- `pp queue`: a job that cannot start fails on its own and the rest start.
+  A job whose wrapper is still ending reads running, not lost.
+- `pp queue --json` and `pp queue show --json` give a job's variables by
+  name, with each value `***`.
+- An empty duration such as `--timeout ""` is a usage error, and so is an
+  empty `pp kill` target, which selected every process you own.
+- `pp http` never serves `.git`, serves a file whose name is not UTF-8, and
+  sorts by name when `?sort=` names anything else.
+- pp's messages and the `pp cmd add` and `update` screens escape the text
+  they name. TAB completion runs nothing in a command's name: run
+  `pp completion bash` or `zsh` again to replace a saved script.
+- A command file whose name is not UTF-8 no longer breaks `pp -h`.
+- pp needs Python 3.11.4 or later, which `pp promote` always needed.
+
+## v0.1.0
+
+The first release.
+
+- `pp queue` runs commands later, in named queues, when the GPU memory,
+  CPUs, RAM or counters they ask for are free. Jobs have timeouts, retries
+  for the failures you name, and hooks. Tags and ranges select groups of
+  jobs. A project's `.pp.toml` can run its jobs through a runner, such as a
+  container.
+- `pp queue wait`, `watch` and `wait --mine` tell a person or an agent how
+  jobs ended, with the end of the log for a failure.
+- `pp promote` releases a project from a git commit when its check passes
+  on that copy, and rolls back.
+- `pp kill`, `pp batch` and `pp http` stop processes, rename or convert
+  files, and show project folders in a browser.
+- `pp cmd` writes commands, and adds other people's from git, pinned to the
+  commit you read.
+- Completion for zsh and bash.

{ppkit-0.1.0 → ppkit-0.2.0}/CONTRIBUTING.md

@@ -28,19 +28,19 @@ If you installed pp as a tool, `uv tool uninstall ppkit` (or
 
 ## Building and testing
 
-You need Python 3.11 or later, git, sh and bash. The tests that need zsh
+You need Python 3.11.4 or later, git, sh and bash. The tests that need zsh
 skip without it. `ci/check` uses [uv](https://docs.astral.sh/uv/)
 for the linter and for other Python versions.
 
 ```
 ci/check                                 the linter, then every test, on python3
-ci/check 3.11                            the same on Python 3.11, which uv installs the first time
-ci/check --cpus 2 3.11                   the same on 2 CPUs, like a CI runner
+ci/check 3.11.4                          the same on Python 3.11.4, which uv installs the first time
+ci/check --cpus 2 3.11.4                 the same on 2 CPUs, like a CI runner
 python3 -m unittest discover -s tests    the tests alone
 ci/package                               build ppkit into dist/, install it fresh, and try it
 ```
 
-CI runs `ci/check 3.11`, `ci/check 3.14` and `ci/package` on every push to
+CI runs `ci/check 3.11.4`, `ci/check 3.14` and `ci/package` on every push to
 main and every pull request. The whole suite takes a few minutes. Most of
 that is the journeys (`tests/test_journey_*.py`), which run the real
 `bin/pp` once per step in a throwaway home directory.
@@ -55,7 +55,7 @@ on a developer's machine takes tens of milliseconds, and a runner's few
 hundred take a few. The test that needs another user's process is skipped
 there.
 
-A uv-managed CPython has no `os.pidfd_open`, so `ci/check 3.11`, `ci/check
+A uv-managed CPython has no `os.pidfd_open`, so `ci/check 3.11.4`, `ci/check
 3.14` and CI run `pp/lib/proc.py`'s ctypes syscall path, while the distro
 `python3` of a plain `ci/check` runs the wrapped calls. Both are tested only
 by running both.
@@ -149,6 +149,13 @@ These rules keep pp working as commands come and go:
   command, two workers, a signal and a write) go through a construct that
   is written once. "The constructs" below is the list of them, and a check
   added at one call site is not a fix.
+- A test checks a contract, not a wording. Every sentence pp prints may be
+  reworded in a patch, so a test asserts an exit status, a `--json` field, a
+  record field, a file's existence or its mode, an id, or a state, and nothing
+  else. Where a test has to know that pp said something, it asserts the field
+  the sentence was made from. Two exceptions are written down where they are
+  made: a journey types the examples of `pp help NAME`, and one journey step
+  holds the headings and the line width of `pp help queue`.
 - A test of a construct checks what the construct promises. A test that
   runs a second actor at an exact step keeps one fix from coming back
   (`TestTwoAtOnce` in `tests/test_queue.py`). That step goes through
@@ -185,8 +192,11 @@ list of the parts:
 
 - `pp/cli.py`: finding commands, the order of their origins, help and
   completion.
-- `pp/lib/jobq.py`: queues, workers, resources, and the files a job lives
-  in; `pp help queue` is the user's view of the same.
+- `pp/lib/jobstore.py`: a job's record, the files it lives in, and the
+  transaction that changes one. `pp/lib/jobcore.py`: the worker, the claims
+  and the processes of a queue. `pp/lib/jobpolicy.py`: every rule of a queue
+  as a value, so each one is a table. `pp/lib/jobq.py`: the verbs, the views,
+  the resources and the CLI; `pp help queue` is the user's view of the same.
 - `pp/commands/promote.py`: releases. `pp/commands/cmd.py`: plugins.
   `pp/commands/http.py`: serving a folder to a browser, with its cache and
   its server. `pp/commands/batch.py`: bulk file changes, and the journal
@@ -214,9 +224,9 @@ this table, and each rule's own docstring is its contract.
 | `alive` | judging a process alive by its pid | a claim's holder, or `Handle` | `pp/lib/proc.py` |
 | `lock` | taking a lock | `locked`, and `deferring` for a scope with no lock | `pp/lib/lock.py` |
 | `claim` | opening a claim's file | `Claim` | `pp/lib/lock.py` |
-| `job` | reading or writing a job's file | `txn`, `listing`, `find` | `pp/lib/jobq.py` |
-| `worker-claim` | taking a queue's worker claim, or reading its holder | `worker`, `worker_now`, `work` | `pp/lib/jobq.py` |
-| `job-process` | signalling or reaping a job's processes | the `Worker` class, `cmd_worker`, `work` | `pp/lib/jobq.py`, `pp/lib/proc.py`, `pp/commands/kill.py` |
+| `job` | reading or writing a job's file | `txn`, `listing`, `find` | `pp/lib/jobstore.py` |
+| `worker-claim` | taking a queue's worker claim, or reading its holder | `worker`, `worker_now`, `work` | `pp/lib/jobcore.py` |
+| `job-process` | signalling or reaping a job's processes | the `Worker` class, `worker_stop`, `work` | `pp/lib/jobcore.py`, `pp/lib/proc.py`, `pp/commands/kill.py` |
 | `by-name` | deciding on a served file by its name | act on the open file | `pp/commands/http.py` |
 | `sleep` | sleeping in a test | `until` | `tests/journey.py` |
 | `once` | a once-flag of a test's own | `Gate` | `tests/journey.py` |
@@ -246,15 +256,66 @@ replacing a whole file in one step goes through `paths.write_json`.
 `tests/test_cmd.py` and `ci/package` keep the installed case working, and
 the journeys in `tests/test_journey_cmd.py` the other two.
 
+### The queue's four modules, and what may go where
+
+The line between them is mechanism against policy, and a change goes to the
+module whose job it is:
+
+- `pp/lib/jobstore.py` is the record store and its transaction: one lock, no
+  timing, no process, no signal. A field a record gains and a write a verb
+  hands over go here.
+- `pp/lib/jobcore.py` is the worker and the claims: starting, stopping,
+  reaping and adopting a job's processes, the worker's loop, and the watchers.
+  A bug here loses a job, which is why it is the module held still.
+- `pp/lib/jobpolicy.py` is every rule that is values in and one value out: how
+  a job ends, what a selection names, what a setting comes to, what a job's
+  environment is. A rule goes here, and a table tests it.
+- `pp/lib/jobq.py` is the verbs, the views, the readings, the one settings
+  declaration and the CLI, and `pp/commands/queue.py` is the command and its
+  help. Every sentence a user reads is worded here.
+
+The store and the worker together are the core. These rules hold that line,
+and `tests/test_constructs.py` reads the source for each:
+
+- The boundary rule. Outside the core no function takes the store lock, and
+  none starts a job, a runner, a stop command, a hook or a worker, or signals
+  a job, a worker or a process group: a verb goes through the core's API, where
+  every write is one call and one whole transaction. Two sites outside it run a
+  process of their own, and both are named exemptions: `reading`, which runs a
+  user's probe under that probe's own claim, and `_cards`, which runs
+  nvidia-smi.
+- The purity rule. A function of `jobpolicy.py` reaches no clock, no store, no
+  process, no signal and no file. It is handed records, readings, settings and
+  `now`, and gives back a value, so a rule that could be raced is a rule no
+  table tests.
+- The import rule. `jobpolicy.py` imports nothing of pp's, the store imports
+  nothing of the worker, and neither imports the verbs above them. The
+  direction is a line, so no cycle can hide a rule that reads a mechanism.
+- The core reads no config file and takes no reading of its own: `read_settings`
+  hands a worker the settings of one pass of its loop, and `admit_for` the
+  readings of one launch pass. It writes no sentence either: it tells a sink
+  what it did, and `jobq` words every one of them.
+- The budgets, in code lines that are neither blank, comment nor docstring:
+  250 for the store, 490 for the worker, 730 for the two together. A feature
+  that cannot be built without going over one of them waits until something
+  leaves the core, and moving a rule out into `jobpolicy.py` is the first thing
+  to try.
+- Every function of the store and of the worker is named in one of the two
+  lists in that file, so a construct is placed when it lands, and a name that
+  has gone comes off the list in the same change.
+
 ### Where a queue's jobs live, and how one says it is alive
 
-The docstring of `pp/lib/jobq.py` is the reference: the files a queue is made
-of, how a worker and a job say they are alive, and the one actor that ends a
-job. Each mechanism inside it has one owner, and these are they:
+The docstring of `pp/lib/jobstore.py` is the reference for the files a queue
+is made of, and `pp/lib/jobcore.py`'s for how a worker and a job say they are
+alive and which actor ends a job. Each mechanism has one owner, and these are
+they:
 
-- the store: `Txn`, `txn`, `listing` and `find`. `open_queue` is where a
-  queue's `FORMAT` is checked, and `Txn.add` is where an id and the record
-  that holds it land together.
+- the store: `Txn`, `txn`, `listing` and `find`, in `pp/lib/jobstore.py`.
+  `open_queue` is where a queue's `FORMAT` is checked, and `Txn.add` is where
+  an id and the record that holds it land together. The writes a verb hands
+  over are the four at the end of that file, each one a whole transaction of
+  its own that is told the states it may act on.
 - being alive: `lock.Claim`. A claim is held for as long as its holder lives,
   so nothing is cleaned up after a holder that was killed and no record is
   ever read as a live holder. A queue's worker holds one on `worker.lock`,
@@ -262,9 +323,13 @@ job. Each mechanism inside it has one owner, and these are they:
 - whether a queue's worker is up: the block comment above `worker_up`, for
   the two things that can be true of that claim and the first moment that is
   neither.
-- how a job ends: `decide` and `merge_stop`, which are pure, and
-  `Worker.carry_out`, the one pass that gathers the facts and carries the
-  action out.
+- how a job ends: `decide` and `merge_stop` in `pp/lib/jobpolicy.py`, which
+  are pure, as every rule in that module is, and `Worker.carry_out`, the one
+  pass that gathers the facts and carries the action out.
+- what a worker is handed: it reads no config file and takes no reading of
+  its own, so `read_settings` gives it the settings of one pass of its loop
+  and `admit_for` the readings of one launch pass. It writes no sentence
+  either: it tells a sink what it did, and `jobq` words every one of them.
 - what a stop carries: `public()`, where every field of a job record that
   others rely on is named.
 
@@ -281,9 +346,10 @@ the same place.
 A kill, pass by pass:
 
 ```
-kill:   txn(merge_stop) -> worker_up(wake=True) -> SIGUSR1 to the worker
-worker: carry_out: the facts -> decide = TERM -> signal the group,
-        then txn marks `sent` -> (the grace) -> decide = KILL9 -> signal
+kill:   ask_stop(merge_stop) -> worker_up(wake=True) -> SIGUSR1 to the worker
+worker: carry_out: the facts -> decide = TERM -> the project's stop command,
+        or the SIGTERM to the group, then txn marks `sent`
+     -> (the grace) -> decide = KILL9 -> signal
      -> the wrapper ends, so the job's run claim is free
      -> decide = ENDED_AS -> _finish records the outcome and the hook it owes
      -> fire
@@ -297,9 +363,11 @@ own exit time and not the queue's poll period.
 
 The rules the worker follows are the ones `pp help queue` states under "How a
 job ends, whatever reached it", and that help text is their one statement.
-One more is for contributors: which of the two a stop is comes from the
+Two more are for contributors: which of the two a stop is comes from the
 `timeout` field a worker writes on the stop it wrote itself, never from the
-name of whoever asked.
+name of whoever asked; and an exit code the attempt wrote after its stop went
+out is that stop's outcome, with the code kept, since a container runner writes
+the code of a container the stop command ended.
 
 `TestTheDecision` in `tests/test_queue.py` is a table over every combination
 of those facts. Each row carries the rule from that help as its own label,

{ppkit-0.1.0 → ppkit-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ppkit
-Version: 0.1.0
+Version: 0.2.0
 Summary: A CLI kit for user & agents
 Keywords: cli,job queue,gpu,agents,toolkit
 Author: The pp Authors
@@ -14,7 +14,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Topic :: System :: Shells
 Classifier: Topic :: Utilities
-Requires-Python: >=3.11
+Requires-Python: >=3.11.4
 Project-URL: Changelog, https://github.com/xflash96/pp/blob/main/CHANGELOG.md
 Project-URL: Homepage, https://github.com/xflash96/pp
 Project-URL: Issues, https://github.com/xflash96/pp/issues
@@ -85,7 +85,7 @@ shell that added it.
 
 ## Install
 
-pp runs on Linux, with Python 3.11 or later. Install it as a tool, in an
+pp runs on Linux, with Python 3.11.4 or later. Install it as a tool, in an
 environment of its own:
 
 ```
@@ -132,8 +132,8 @@ pp queue add -q dl --need net=1 -- ./fetch.sh            the dl queue, taking on
 pp queue add --timeout 2h -- python x.py                 stopped after two hours
 pp queue add --retries 2 --retry-if 'Connection reset' -- ./sync.sh
 pp queue add --on-fail 'notify-send "job $PP_QUEUE_JOB failed"' -- ./nightly.sh
-pp queue add --tag sweep -- python x.py --lr 0.1         tagged, to act on the group later
-pp queue wait sweep                                      every job tagged sweep
+pp queue add --tag sweep.lr1 -- python x.py --lr 0.1     a name for the job, and for its family
+pp queue wait --prefix sweep.                            every job whose name starts with sweep.
 pp queue watch 12                                        a line for each start, retry and outcome
 pp queue drop 20-29                                      a range: it shows the jobs and asks first
 pp queue kill 12                                         waits while the worker stops it, then:
@@ -141,6 +141,11 @@ pp queue kill 12                                         waits while the worker
 pp queue retry 11        pp queue hold 15        pp queue release 15
 ```
 
+Every verb that takes jobs takes the same five forms: an id, a range of ids,
+a `--tag` name matched whole, `--prefix P`, and `--all` with `-q NAME`. `pp
+queue note` is the exception, and takes one job by its id or by a tag that
+names one. A job added without `--tag` has no name, and its id selects it.
+
 Limits go in `~/.config/pp/config.toml`. Every queue draws on the shared
 GPU memory, CPUs and RAM; a queue's own pool caps what it may take of them.
 A counter is a number, or a command that prints how much is free:
@@ -158,21 +163,36 @@ retry_if = "Connection reset|503"
 net = 2
 ```
 
-A project's `.pp.toml` can wrap every job queued from it, for example in a
-container, and add to the jobs' environment:
+A project's `.pp.toml` says how its jobs are launched and stopped, where
+their output goes, and what is in their environment:
 
 ```toml
 [queue]
-runner = "docker/run.sh"         # the job runs as docker/run.sh CMD...
+runner = "docker/run.sh"                   # the job runs as docker/run.sh CMD...
+stop = "docker rm -f $PP_QUEUE_CONTAINER"  # how a stop reaches the work the runner started
+log = "logs/{tag}.log"                     # under the project root; {tag} or {id}
 env = { HF_HUB_OFFLINE = "1" }
 ```
 
-`pp queue add --no-runner` skips the runner for one job. The runner gets the
-job's environment, including `CUDA_VISIBLE_DEVICES`, and a container runner
-has to pass it in. Give a job a token or a secret with `--env` or the
-project's `env`: a variable exported in your shell reaches it only through
-the worker. `pp help queue` has the examples, and `pp help queue add` every
-option of `add`.
+A runner puts the work where no signal of pp's reaches it, so a project that
+sets one says the stop as well, or the `add` is refused. pp hands the runner
+`PP_QUEUE_CONTAINER`, a name for whatever it starts elsewhere, and
+`PP_QUEUE_ENVFILE`, a file of `KEY=VALUE` lines for a container to read with
+`--env-file`. `pp queue add --no-runner` skips the runner for one job, and
+`--runner CMD` with `--stop CMD` gives one job both with no `.pp.toml`.
+
+A campaign is a file of one job per line, each line with its own flags, and a
+stage can wait for the one before it:
+
+```
+IDS=$(pp queue add --from stage1.txt)                   # one id per line on stdout
+pp queue add --from stage2.txt --after "$(echo "$IDS" | tr '\n' ',')"
+```
+
+Give a job a token or a secret with `--env` or the project's `env`: a variable
+exported in your shell reaches it only through the worker. `--json` gives a
+job's variables by name, with each value `***`. `pp help queue` is the whole
+page, with the runner to copy, and `pp help queue add` every option of `add`.
 
 ## Working with agents
 
@@ -231,6 +251,10 @@ pp promote rollback notes      back to the release before
 database migration that cannot be undone does not belong in it. pp keeps the
 last five releases, and always keeps the current one and the one before.
 
+Each hook's output goes to a file beside the run's log, and pp shows it as it
+comes. pp waits for the hook itself, not for what the hook leaves running, so
+redirect the output of a process an `after` starts.
+
 A systemd service runs the release by pointing at `current`:
 
 ```
@@ -262,7 +286,7 @@ pp http ~/notes .              a tab for each folder, at http://127.0.0.1:8000
 ```
 
 `pp http` listens on 127.0.0.1 unless you pass `--bind`, and does not serve
-files such as `.env` or private keys.
+`.git` or files such as `.env` or private keys.
 
 ## Your own commands
 
@@ -314,7 +338,7 @@ too. To add a command to pp itself, see [CONTRIBUTING.md](https://github.com/xfl
 pp is new, and its version is 0.x. A command or a config key may still
 change, and so may a field of `--json` output; the
 [changelog](https://github.com/xflash96/pp/blob/main/CHANGELOG.md) says when one does. CI runs the tests on
-Python 3.11 and 3.14, on Ubuntu 24.04.
+Python 3.11.4 and 3.14, on Ubuntu 24.04.
 
 ## Contributing
 

{ppkit-0.1.0 → ppkit-0.2.0}/README.md

@@ -63,7 +63,7 @@ shell that added it.
 
 ## Install
 
-pp runs on Linux, with Python 3.11 or later. Install it as a tool, in an
+pp runs on Linux, with Python 3.11.4 or later. Install it as a tool, in an
 environment of its own:
 
 ```
@@ -110,8 +110,8 @@ pp queue add -q dl --need net=1 -- ./fetch.sh            the dl queue, taking on
 pp queue add --timeout 2h -- python x.py                 stopped after two hours
 pp queue add --retries 2 --retry-if 'Connection reset' -- ./sync.sh
 pp queue add --on-fail 'notify-send "job $PP_QUEUE_JOB failed"' -- ./nightly.sh
-pp queue add --tag sweep -- python x.py --lr 0.1         tagged, to act on the group later
-pp queue wait sweep                                      every job tagged sweep
+pp queue add --tag sweep.lr1 -- python x.py --lr 0.1     a name for the job, and for its family
+pp queue wait --prefix sweep.                            every job whose name starts with sweep.
 pp queue watch 12                                        a line for each start, retry and outcome
 pp queue drop 20-29                                      a range: it shows the jobs and asks first
 pp queue kill 12                                         waits while the worker stops it, then:
@@ -119,6 +119,11 @@ pp queue kill 12                                         waits while the worker
 pp queue retry 11        pp queue hold 15        pp queue release 15
 ```
 
+Every verb that takes jobs takes the same five forms: an id, a range of ids,
+a `--tag` name matched whole, `--prefix P`, and `--all` with `-q NAME`. `pp
+queue note` is the exception, and takes one job by its id or by a tag that
+names one. A job added without `--tag` has no name, and its id selects it.
+
 Limits go in `~/.config/pp/config.toml`. Every queue draws on the shared
 GPU memory, CPUs and RAM; a queue's own pool caps what it may take of them.
 A counter is a number, or a command that prints how much is free:
@@ -136,21 +141,36 @@ retry_if = "Connection reset|503"
 net = 2
 ```
 
-A project's `.pp.toml` can wrap every job queued from it, for example in a
-container, and add to the jobs' environment:
+A project's `.pp.toml` says how its jobs are launched and stopped, where
+their output goes, and what is in their environment:
 
 ```toml
 [queue]
-runner = "docker/run.sh"         # the job runs as docker/run.sh CMD...
+runner = "docker/run.sh"                   # the job runs as docker/run.sh CMD...
+stop = "docker rm -f $PP_QUEUE_CONTAINER"  # how a stop reaches the work the runner started
+log = "logs/{tag}.log"                     # under the project root; {tag} or {id}
 env = { HF_HUB_OFFLINE = "1" }
 ```
 
-`pp queue add --no-runner` skips the runner for one job. The runner gets the
-job's environment, including `CUDA_VISIBLE_DEVICES`, and a container runner
-has to pass it in. Give a job a token or a secret with `--env` or the
-project's `env`: a variable exported in your shell reaches it only through
-the worker. `pp help queue` has the examples, and `pp help queue add` every
-option of `add`.
+A runner puts the work where no signal of pp's reaches it, so a project that
+sets one says the stop as well, or the `add` is refused. pp hands the runner
+`PP_QUEUE_CONTAINER`, a name for whatever it starts elsewhere, and
+`PP_QUEUE_ENVFILE`, a file of `KEY=VALUE` lines for a container to read with
+`--env-file`. `pp queue add --no-runner` skips the runner for one job, and
+`--runner CMD` with `--stop CMD` gives one job both with no `.pp.toml`.
+
+A campaign is a file of one job per line, each line with its own flags, and a
+stage can wait for the one before it:
+
+```
+IDS=$(pp queue add --from stage1.txt)                   # one id per line on stdout
+pp queue add --from stage2.txt --after "$(echo "$IDS" | tr '\n' ',')"
+```
+
+Give a job a token or a secret with `--env` or the project's `env`: a variable
+exported in your shell reaches it only through the worker. `--json` gives a
+job's variables by name, with each value `***`. `pp help queue` is the whole
+page, with the runner to copy, and `pp help queue add` every option of `add`.
 
 ## Working with agents
 
@@ -209,6 +229,10 @@ pp promote rollback notes      back to the release before
 database migration that cannot be undone does not belong in it. pp keeps the
 last five releases, and always keeps the current one and the one before.
 
+Each hook's output goes to a file beside the run's log, and pp shows it as it
+comes. pp waits for the hook itself, not for what the hook leaves running, so
+redirect the output of a process an `after` starts.
+
 A systemd service runs the release by pointing at `current`:
 
 ```
@@ -240,7 +264,7 @@ pp http ~/notes .              a tab for each folder, at http://127.0.0.1:8000
 ```
 
 `pp http` listens on 127.0.0.1 unless you pass `--bind`, and does not serve
-files such as `.env` or private keys.
+`.git` or files such as `.env` or private keys.
 
 ## Your own commands
 
@@ -292,7 +316,7 @@ too. To add a command to pp itself, see [CONTRIBUTING.md](https://github.com/xfl
 pp is new, and its version is 0.x. A command or a config key may still
 change, and so may a field of `--json` output; the
 [changelog](https://github.com/xflash96/pp/blob/main/CHANGELOG.md) says when one does. CI runs the tests on
-Python 3.11 and 3.14, on Ubuntu 24.04.
+Python 3.11.4 and 3.14, on Ubuntu 24.04.
 
 ## Contributing