pangea-flows 0.3.0__tar.gz

pangea_flows-0.3.0/.gitignore

@@ -0,0 +1,5 @@
+__pycache__/
+.venv/
+.pytest_cache/
+dist/
+out/

pangea_flows-0.3.0/PKG-INFO

@@ -0,0 +1,306 @@
+Metadata-Version: 2.5
+Name: pangea-flows
+Version: 0.3.0
+Summary: Write PANGEA processing workflows in Python
+Author-email: Thanasis Georgiou <ageorgiou@noa.gr>
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.6
+Provides-Extra: serve
+Requires-Dist: croniter>=2; extra == 'serve'
+Requires-Dist: pynacl>=1.5; extra == 'serve'
+Description-Content-Type: text/markdown
+
+# pangea-flows
+
+Write PANGEA processing workflows in Python: format conversions, plots, uploads to other databases. You write a function that turns input files into output files. The platform keeps track of what is done, what is waiting and what failed, so your code doesn't have to.
+
+## The quick way
+
+```python
+from pangea_flows import each_file
+
+
+@each_file(inputs="pollyxt/raw", pattern="*.nc", output="pollyxt/quicklooks")
+def quicklook(path, out_dir):
+    """A plot of each raw file"""
+    plot(path, out_dir / f"{path.stem}.png")  # Your code
+```
+
+That is a whole workflow. Every 15 minutes it looks for new files in `pollyxt/raw`, downloads each one, and calls `quicklook` with its path and an empty folder. Whatever files are in `out_dir` when it returns become one datum in `pollyxt/quicklooks`, with the input's time, linked to the input. Write nothing to produce nothing. Raise an exception and the file is tried again later.
+
+For one output per hour or day, from all the files in it:
+
+```python
+from datetime import timedelta
+from pangea_flows import each_window, hourly
+
+
+@each_window(hourly(settle=timedelta(hours=2)), inputs="pollyxt/raw", pattern="*.nc", output="pollyxt/hourly")
+def hourly_plot(paths, window, out_dir):
+    plot(paths, window.start, window.end, out_dir / f"{window.start:%Y%m%d_%H}.png")
+```
+
+`paths` are sorted by time. `settle` waits for files that arrive late. `daily()` or any `timedelta` work too.
+
+Both take `schedule=` and the other options of `@workflow`, below. To start a project:
+
+```bash
+pangea-flows new my-flows     # flows.py, pyproject.toml, a README and .env.example
+cd my-flows && uv sync
+pangea-flows try quicklook    # Runs it on the last day's data, into ./out/
+```
+
+## The full form
+
+When a workflow needs more, like several outputs, metadata or its own grouping of inputs, write it with `@workflow`. `each_file` above is this, written out:
+
+```python
+from pangea_flows import Context, Item, Output, per_datum, workflow
+
+
+@workflow(inputs="pollyxt/raw", outputs="pollyxt/quicklooks", keys=per_datum())
+def quicklook(item: Item, ctx: Context) -> list[Output]:
+    out_dir = ctx.tmpdir / "out"
+    out_dir.mkdir()
+    for path in ctx.fetch(item.inputs, pattern="*.nc"):
+        plot(path, out_dir / f"{path.stem}.png")
+    files = list(out_dir.iterdir())
+    return [Output("pollyxt/quicklooks", files, timestamp=item.inputs[0].timestamp)] if files else []
+```
+
+And with hours, two outputs and a setting of its own:
+
+```python
+from datetime import timedelta
+from pangea_flows import Context, Item, Output, hourly, workflow
+
+
+@workflow(
+    inputs="pollyxt/raw",
+    outputs="pollyxt/quicklooks",
+    keys=hourly(settle=timedelta(hours=2), span=timedelta(hours=1)),
+    schedule="0 * * * *",
+)
+def quicklooks(item: Item, ctx: Context) -> list[Output]:
+    """Hourly backscatter plots"""
+    files = ctx.fetch(item.inputs, pattern="*.nc")
+    png = ctx.tmpdir / f"{item.key}.png"
+    plot(files, item.window, png)  # Your code
+    return [Output("pollyxt/quicklooks", png, timestamp=item.window.start)]
+```
+
+## How it works
+
+Every workflow has a **ledger** on the platform: one **item** per unit of work, such as an hour of data. Each run:
+
+1. **Plans**: follows the uploads to the input datasets since the last run, and adds their datums to items. A datum only counts once all its files are uploaded.
+2. **Claims** items that are ready, one at a time. Two runs never get the same item.
+3. **Calls your function** with the item's inputs, in a temporary directory of its own.
+4. **Registers the outputs** your function returns, uploads their files, and records where they came from.
+5. **Marks the item** done, or failed with the error, to be retried later.
+
+Each run is recorded on the platform with its log, and each attempt at an item with what the function logged and printed (the last 64 KB of each). A run that fails outside any item, for example because a dataset name is wrong, notifies the instrument's subscribers, once until it works again. Members can also ask for a run from the platform instead of waiting for the schedule.
+
+A run that crashes or hangs loses its claim when the lease runs out, and another run takes the item over. When late data arrives for an item that is already done, the item runs again, and its outputs are updated in place rather than duplicated.
+
+## Setting up a workflow
+
+1. On the platform, a member of the instrument creates the workflow (`POST /workflows`). They choose which datasets it may write to, then issue its token (`POST /workflows/:id/token`).
+2. Give the token to the runner as `PANGEA_TOKEN_<NAME>`, where `<NAME>` is the function's name in capitals (`PANGEA_TOKEN_QUICKLOOKS` above), and set `PANGEA_URL`.
+3. Try it first with `pangea-flows try quicklooks`, which only reads (see Running). Then run it once for real: `pangea-flows run quicklooks`. The first run only records where to start from, so nothing happens until new data is uploaded. Pass `start=` to `@workflow` to also take data from earlier.
+
+## Writing the function
+
+It gets an `Item` and a `Context`, and returns a list of `Output`s, or nothing.
+
+| | |
+| --- | --- |
+| `item.key` | The item, e.g. `2026-11-04T10:00Z` |
+| `item.inputs` | Its datums (`.id`, `.timestamp`, `.metadata`, `.files`) |
+| `item.window` | The time it covers (`.start`, `.end`), for time-based keys |
+| `item.attempt` | 1 the first time, then 2, 3... |
+| `ctx.fetch(datums, pattern="*.nc")` | Downloads files into `ctx.tmpdir` and returns their paths |
+| `ctx.tmpdir` | A directory to work in, removed afterwards |
+| `ctx.log` | A logger |
+| `ctx.cancelled` | True if another run took the item over; you can stop early |
+| `ctx.settings` | The workflow's settings, read-only (see below) |
+| `ctx.datums(dataset, after=, before=, current=, ...)` | Datums of any dataset, with their files and metadata (see below) |
+| `ctx.datum(id)` | One datum |
+
+**Settings** are the extra keyword arguments of `@workflow`. They show on the platform, and the function reads them as `ctx.settings`:
+
+```python
+@workflow(inputs=..., outputs=..., keys=per_datum(), calibration_timeout_minutes=10, location="Antikythera")
+def scc_sync(item: Item, ctx: Context) -> list[Output]:
+    timeout = ctx.settings["calibration_timeout_minutes"]
+```
+
+They change with a commit, not on the platform. `each_file` and `each_window` functions don't get a `ctx`, so use `@workflow` when the function needs settings or other datums.
+
+**Other datums**: `ctx.datums(dataset, ...)` lists any dataset's datums, oldest first, with their files and metadata, so `ctx.fetch` works on them too. `dataset` is named like inputs. `after` and `before` (measurement time) and `created_after` and `created_before` (registered on the platform) leave out the bound itself. `current=True` leaves out outputs that a workflow no longer makes: when an item runs again and doesn't return an output it returned before, that output is superseded. Also `final=`, `order="desc"` and `limit=`. For example, to compare a calibration with the earlier ones that passed:
+
+```python
+history = [
+    d.metadata
+    for d in ctx.datums("pollyxt/scc-calibration", before=calibration_start, current=True)
+    if d.metadata
+]
+```
+
+`PangeaClient.datums(...)` does the same outside a workflow.
+
+`Output(dataset, files, timestamp, key=None, sources=None, metadata=None, final=False)`:
+
+- `dataset` must be one of the workflow's `outputs`.
+- `key` names the output within the item, and defaults to the dataset's slug. Give each output its own key when there are several in one dataset, like `key=f"calibration-{measurement_id}"`. When the item runs again, an output with the same key updates the same datum, so keep keys and filenames stable.
+- `sources` defaults to all of the item's inputs. Name the ones you actually used, if fewer.
+- `metadata` replaces the datum's metadata each time, and `None` clears it, so a rerun never leaves stale values.
+- An item with nothing to produce returns `[]`. It is done, and won't run again.
+
+**Errors**: any exception fails the attempt, and the item is retried later: after 10 minutes, then 20, and so on, up to `max_attempts` (3 by default). After that it is FAILED and the instrument's subscribers are notified. Raise `FailPermanently("why")` when retrying can't help, such as for a corrupt input. A member can reset FAILED items on the platform once the problem is fixed.
+
+**Things outside PANGEA**: when a workflow also uploads somewhere else, it may run again for the same item, so check whether that system already has the data first.
+
+## Keys: what an item is
+
+| | One item per... | Keys |
+| --- | --- | --- |
+| `per_datum()` | input datum | `datum-16584` |
+| `hourly()` | hour (UTC) | `2026-11-04T10:00Z` |
+| `daily()` | day (UTC) | `2026-11-04` |
+| `TimeWindows(timedelta(minutes=30))` | any window that divides a day | `2026-11-04T10:30Z` |
+
+Time-based schemes take:
+
+- `settle`: how long after a window ends to wait before running it, for data that arrives late.
+- `span`: how long each datum covers, so that it goes to every window it overlaps. A `timedelta` from its timestamp, or a function returning `(start, end)`, e.g. from `datum.metadata`. By default a datum belongs only to the window of its timestamp.
+
+For other rules, subclass `KeyScheme`: `keys_for(datum)` returns the item keys a datum belongs to, and `name` identifies the scheme. A workflow's scheme can never change, since its existing keys would mean something else; a different scheme needs a new workflow.
+
+## Datasets
+
+Name datasets as `"instrument/dataset"` by slug, by their slug alone if they belong to the workflow's own instrument, or by ID.
+
+Changing a workflow's inputs or outputs is refused, in case the code was changed by mistake, or it is someone else's code with this token. Run once with `--accept-changes` when it is intended.
+
+## Running
+
+```bash
+pangea-flows new my-flows                 # Start a project in a new folder
+pangea-flows list                         # The workflows in flows.py, and their token variables
+pangea-flows try quicklooks               # Try it on the last day's uploads
+pangea-flows try quicklooks --since 2026-11-01 --key 2026-11-01T10:00Z
+pangea-flows run quicklooks               # Plan and process up to max_items items, once
+```
+
+A workflow is named by its name in `flows.py`, or as `module`, `module:name` or `file.py:name`. `PANGEA_URL` and the tokens can also go in a `.env` file in the current folder.
+
+`try` (also `dry-run`) doesn't touch the ledger: it plans from uploads since `--since`, a day ago by default, runs your function, and copies the outputs into `./out/<key>/<output key>/`. It only reads, so your personal API token works too (`PANGEA_TOKEN`).
+
+### Running workflows from their repositories
+
+On the platform, a member gives the workflow a **source**: the git repository, the branch and the workflow in it, like `flows.py:quicklook`. For a private repository, the workflow's page then shows a public key: add it to the repository as a read-only deploy key (on GitHub: the repository's Settings → Deploy keys). From then on, every push to the branch is picked up within a minute or so. A run that fails to build or import shows on the workflow's page with the error.
+
+The repository needs a `uv.lock`, so the server installs exactly what you tested with: `pangea-flows new` sets that up. Runs get 4 GB of memory, 2 CPUs and 2 hours by default; ask for more in the decorator, up to what the server allows:
+
+```python
+@each_file(inputs="pollyxt/raw", output="pollyxt/quicklooks", memory="8g", cpus=4, timeout=timedelta(hours=6))
+```
+
+The server side is `pangea-flows serve` (`pip install "pangea-flows[serve]"`, Python 3.11+, git and podman). An admin creates a runner on the platform (`POST /runners`, then `POST /runners/:id/token`), and writes its settings:
+
+```toml
+api_url = "https://pangea.example.org/api"
+token_file = "token"                 # Or $PANGEA_RUNNER_TOKEN
+secrets_key_file = "secrets.key"     # Or $PANGEA_RUNNER_SECRETS_KEY; see below
+state_dir = "/var/lib/pangea-runner" # Clones, deploy keys, scratch space
+
+[container]
+api_url = "https://pangea.example.org/api" # What runs use; api_url by default
+network = ""                         # podman --network for runs; its default if empty
+# With podman's default network, a run can't reach this host by its own address.
+# If the API is on the same host, point its name at the host instead:
+add_hosts = ["pangea.example.org:host-gateway"]
+
+[limits]                             # For runs that don't ask
+memory = "4g"
+cpus = 2
+timeout_minutes = 120
+
+[max]                                # The most a run may ask for
+memory = "16g"
+cpus = 4
+timeout_minutes = 1440
+```
+
+`secrets_key_file` holds the private key, base64, that [secrets](#secrets) are sealed to; the API has its public key as `WORKFLOW_SECRETS_PUBLIC_KEY` (see `packages/pangea-api/docs/runners.md` for making the pair). The runner opens each run's secrets and passes them to its container by name, with the values only in podman's environment, and masks them in the output it keeps. Without the key, runs of workflows that have secrets fail with a message saying so.
+
+Then `pangea-flows serve --config runner.toml`. It builds an image for each new commit (from `ghcr.io/astral-sh/uv`, with `uv sync --frozen`), and runs each workflow in a container with a read-only filesystem, a scratch `/tmp` on the host, its memory and CPU limits, and a token for that run only. See `packages/pangea-api/docs/runners.md` for what it does on the platform.
+
+### From Python
+
+```python
+from pangea_flows import run_once
+
+report = run_once(quicklooks)  # Or run_once(quicklooks, client=PangeaClient(url, token))
+print(report.done, report.failed, report.run_id)
+```
+
+## Secrets
+
+Credentials for systems outside PANGEA, such as an SCC login, are **secrets**. Members set them on the workflow's page on the platform, and runs get each as an environment variable. Declare the ones the code needs:
+
+```python
+@workflow(..., secrets=["SCC_USERNAME", "SCC_PASSWORD"])
+def scc_sync(item: Item, ctx: Context) -> list[Output]:
+    login(os.environ["SCC_USERNAME"], os.environ["SCC_PASSWORD"])
+```
+
+- If one isn't set, the run fails at the start with `Missing secrets: SCC_PASSWORD`, instead of every item failing. The platform shows which are missing.
+- Names are like `SCC_PASSWORD`. `PANGEA_*`, `HOME`, `TMPDIR`, `PATH` and `PYTHONUNBUFFERED` are the runner's.
+- Values are masked as `***` in the logs and errors sent to the platform. Values shorter than 6 characters aren't masked.
+- `pangea-flows run` and `try` read them from the environment or `.env`, like the token. Keep `.env` out of git.
+- On the platform, nobody can read a value back, not even the workflow's own token.
+
+## Order
+
+Items with an input from a dataset in `first` are handled before the others, in the order of the list. Everything else goes oldest first.
+
+```python
+@workflow(
+    inputs=["pollyxt/scc", "pollyxt/scc-calibration-candidates"],
+    outputs="pollyxt/scc-calibration",
+    keys=per_datum(),
+    first=["scc-calibration-candidates"],
+)
+def scc_sync(item: Item, ctx: Context) -> list[Output]:
+    ...
+```
+
+Here every calibration is uploaded, checked, and deleted from SCC if it fails, before any data file that would use it. A run takes items one at a time, so it finishes everything ready in a `first` dataset before it starts on the rest. It is an order, not a dependency: an item waiting to be retried doesn't hold back the others. The platform works out each item's place from all its inputs, also those planned in earlier runs, and changing `first` reorders items that are still waiting. `pangea-flows try` doesn't use the ledger, so it goes by key.
+
+## Options
+
+| `@workflow(...)` | |
+| --- | --- |
+| `inputs`, `outputs` | Datasets, see above |
+| `keys` | See above |
+| `schedule`, `timezone` | A cron expression, every 15 minutes by default; `None` to only run when asked |
+| `max_items` | Items per run, 10 by default |
+| `max_attempts` | Tries per item, 3 by default |
+| `lease` | How long a claim lasts without a heartbeat, 1 hour by default. Heartbeats are sent while the function runs. |
+| `start` | On the first run, also take data uploaded since this time |
+| `first` | Input datasets whose items go first, see [Order](#order) |
+| `secrets` | Environment variables with credentials, see [Secrets](#secrets) |
+| `name`, `token_env` | Default to the function's name and `PANGEA_TOKEN_<NAME>` |
+| Anything else | Shown with the workflow on the platform |
+
+## Development
+
+```bash
+uv sync
+uv run pytest                     # Unit tests
+uv run pytest -m integration      # Against a running API, see tests/README.md
+uv run pytest -m e2e              # pangea-flows serve, with podman
+```

pangea_flows-0.3.0/README.md

@@ -0,0 +1,293 @@
+# pangea-flows
+
+Write PANGEA processing workflows in Python: format conversions, plots, uploads to other databases. You write a function that turns input files into output files. The platform keeps track of what is done, what is waiting and what failed, so your code doesn't have to.
+
+## The quick way
+
+```python
+from pangea_flows import each_file
+
+
+@each_file(inputs="pollyxt/raw", pattern="*.nc", output="pollyxt/quicklooks")
+def quicklook(path, out_dir):
+    """A plot of each raw file"""
+    plot(path, out_dir / f"{path.stem}.png")  # Your code
+```
+
+That is a whole workflow. Every 15 minutes it looks for new files in `pollyxt/raw`, downloads each one, and calls `quicklook` with its path and an empty folder. Whatever files are in `out_dir` when it returns become one datum in `pollyxt/quicklooks`, with the input's time, linked to the input. Write nothing to produce nothing. Raise an exception and the file is tried again later.
+
+For one output per hour or day, from all the files in it:
+
+```python
+from datetime import timedelta
+from pangea_flows import each_window, hourly
+
+
+@each_window(hourly(settle=timedelta(hours=2)), inputs="pollyxt/raw", pattern="*.nc", output="pollyxt/hourly")
+def hourly_plot(paths, window, out_dir):
+    plot(paths, window.start, window.end, out_dir / f"{window.start:%Y%m%d_%H}.png")
+```
+
+`paths` are sorted by time. `settle` waits for files that arrive late. `daily()` or any `timedelta` work too.
+
+Both take `schedule=` and the other options of `@workflow`, below. To start a project:
+
+```bash
+pangea-flows new my-flows     # flows.py, pyproject.toml, a README and .env.example
+cd my-flows && uv sync
+pangea-flows try quicklook    # Runs it on the last day's data, into ./out/
+```
+
+## The full form
+
+When a workflow needs more, like several outputs, metadata or its own grouping of inputs, write it with `@workflow`. `each_file` above is this, written out:
+
+```python
+from pangea_flows import Context, Item, Output, per_datum, workflow
+
+
+@workflow(inputs="pollyxt/raw", outputs="pollyxt/quicklooks", keys=per_datum())
+def quicklook(item: Item, ctx: Context) -> list[Output]:
+    out_dir = ctx.tmpdir / "out"
+    out_dir.mkdir()
+    for path in ctx.fetch(item.inputs, pattern="*.nc"):
+        plot(path, out_dir / f"{path.stem}.png")
+    files = list(out_dir.iterdir())
+    return [Output("pollyxt/quicklooks", files, timestamp=item.inputs[0].timestamp)] if files else []
+```
+
+And with hours, two outputs and a setting of its own:
+
+```python
+from datetime import timedelta
+from pangea_flows import Context, Item, Output, hourly, workflow
+
+
+@workflow(
+    inputs="pollyxt/raw",
+    outputs="pollyxt/quicklooks",
+    keys=hourly(settle=timedelta(hours=2), span=timedelta(hours=1)),
+    schedule="0 * * * *",
+)
+def quicklooks(item: Item, ctx: Context) -> list[Output]:
+    """Hourly backscatter plots"""
+    files = ctx.fetch(item.inputs, pattern="*.nc")
+    png = ctx.tmpdir / f"{item.key}.png"
+    plot(files, item.window, png)  # Your code
+    return [Output("pollyxt/quicklooks", png, timestamp=item.window.start)]
+```
+
+## How it works
+
+Every workflow has a **ledger** on the platform: one **item** per unit of work, such as an hour of data. Each run:
+
+1. **Plans**: follows the uploads to the input datasets since the last run, and adds their datums to items. A datum only counts once all its files are uploaded.
+2. **Claims** items that are ready, one at a time. Two runs never get the same item.
+3. **Calls your function** with the item's inputs, in a temporary directory of its own.
+4. **Registers the outputs** your function returns, uploads their files, and records where they came from.
+5. **Marks the item** done, or failed with the error, to be retried later.
+
+Each run is recorded on the platform with its log, and each attempt at an item with what the function logged and printed (the last 64 KB of each). A run that fails outside any item, for example because a dataset name is wrong, notifies the instrument's subscribers, once until it works again. Members can also ask for a run from the platform instead of waiting for the schedule.
+
+A run that crashes or hangs loses its claim when the lease runs out, and another run takes the item over. When late data arrives for an item that is already done, the item runs again, and its outputs are updated in place rather than duplicated.
+
+## Setting up a workflow
+
+1. On the platform, a member of the instrument creates the workflow (`POST /workflows`). They choose which datasets it may write to, then issue its token (`POST /workflows/:id/token`).
+2. Give the token to the runner as `PANGEA_TOKEN_<NAME>`, where `<NAME>` is the function's name in capitals (`PANGEA_TOKEN_QUICKLOOKS` above), and set `PANGEA_URL`.
+3. Try it first with `pangea-flows try quicklooks`, which only reads (see Running). Then run it once for real: `pangea-flows run quicklooks`. The first run only records where to start from, so nothing happens until new data is uploaded. Pass `start=` to `@workflow` to also take data from earlier.
+
+## Writing the function
+
+It gets an `Item` and a `Context`, and returns a list of `Output`s, or nothing.
+
+| | |
+| --- | --- |
+| `item.key` | The item, e.g. `2026-11-04T10:00Z` |
+| `item.inputs` | Its datums (`.id`, `.timestamp`, `.metadata`, `.files`) |
+| `item.window` | The time it covers (`.start`, `.end`), for time-based keys |
+| `item.attempt` | 1 the first time, then 2, 3... |
+| `ctx.fetch(datums, pattern="*.nc")` | Downloads files into `ctx.tmpdir` and returns their paths |
+| `ctx.tmpdir` | A directory to work in, removed afterwards |
+| `ctx.log` | A logger |
+| `ctx.cancelled` | True if another run took the item over; you can stop early |
+| `ctx.settings` | The workflow's settings, read-only (see below) |
+| `ctx.datums(dataset, after=, before=, current=, ...)` | Datums of any dataset, with their files and metadata (see below) |
+| `ctx.datum(id)` | One datum |
+
+**Settings** are the extra keyword arguments of `@workflow`. They show on the platform, and the function reads them as `ctx.settings`:
+
+```python
+@workflow(inputs=..., outputs=..., keys=per_datum(), calibration_timeout_minutes=10, location="Antikythera")
+def scc_sync(item: Item, ctx: Context) -> list[Output]:
+    timeout = ctx.settings["calibration_timeout_minutes"]
+```
+
+They change with a commit, not on the platform. `each_file` and `each_window` functions don't get a `ctx`, so use `@workflow` when the function needs settings or other datums.
+
+**Other datums**: `ctx.datums(dataset, ...)` lists any dataset's datums, oldest first, with their files and metadata, so `ctx.fetch` works on them too. `dataset` is named like inputs. `after` and `before` (measurement time) and `created_after` and `created_before` (registered on the platform) leave out the bound itself. `current=True` leaves out outputs that a workflow no longer makes: when an item runs again and doesn't return an output it returned before, that output is superseded. Also `final=`, `order="desc"` and `limit=`. For example, to compare a calibration with the earlier ones that passed:
+
+```python
+history = [
+    d.metadata
+    for d in ctx.datums("pollyxt/scc-calibration", before=calibration_start, current=True)
+    if d.metadata
+]
+```
+
+`PangeaClient.datums(...)` does the same outside a workflow.
+
+`Output(dataset, files, timestamp, key=None, sources=None, metadata=None, final=False)`:
+
+- `dataset` must be one of the workflow's `outputs`.
+- `key` names the output within the item, and defaults to the dataset's slug. Give each output its own key when there are several in one dataset, like `key=f"calibration-{measurement_id}"`. When the item runs again, an output with the same key updates the same datum, so keep keys and filenames stable.
+- `sources` defaults to all of the item's inputs. Name the ones you actually used, if fewer.
+- `metadata` replaces the datum's metadata each time, and `None` clears it, so a rerun never leaves stale values.
+- An item with nothing to produce returns `[]`. It is done, and won't run again.
+
+**Errors**: any exception fails the attempt, and the item is retried later: after 10 minutes, then 20, and so on, up to `max_attempts` (3 by default). After that it is FAILED and the instrument's subscribers are notified. Raise `FailPermanently("why")` when retrying can't help, such as for a corrupt input. A member can reset FAILED items on the platform once the problem is fixed.
+
+**Things outside PANGEA**: when a workflow also uploads somewhere else, it may run again for the same item, so check whether that system already has the data first.
+
+## Keys: what an item is
+
+| | One item per... | Keys |
+| --- | --- | --- |
+| `per_datum()` | input datum | `datum-16584` |
+| `hourly()` | hour (UTC) | `2026-11-04T10:00Z` |
+| `daily()` | day (UTC) | `2026-11-04` |
+| `TimeWindows(timedelta(minutes=30))` | any window that divides a day | `2026-11-04T10:30Z` |
+
+Time-based schemes take:
+
+- `settle`: how long after a window ends to wait before running it, for data that arrives late.
+- `span`: how long each datum covers, so that it goes to every window it overlaps. A `timedelta` from its timestamp, or a function returning `(start, end)`, e.g. from `datum.metadata`. By default a datum belongs only to the window of its timestamp.
+
+For other rules, subclass `KeyScheme`: `keys_for(datum)` returns the item keys a datum belongs to, and `name` identifies the scheme. A workflow's scheme can never change, since its existing keys would mean something else; a different scheme needs a new workflow.
+
+## Datasets
+
+Name datasets as `"instrument/dataset"` by slug, by their slug alone if they belong to the workflow's own instrument, or by ID.
+
+Changing a workflow's inputs or outputs is refused, in case the code was changed by mistake, or it is someone else's code with this token. Run once with `--accept-changes` when it is intended.
+
+## Running
+
+```bash
+pangea-flows new my-flows                 # Start a project in a new folder
+pangea-flows list                         # The workflows in flows.py, and their token variables
+pangea-flows try quicklooks               # Try it on the last day's uploads
+pangea-flows try quicklooks --since 2026-11-01 --key 2026-11-01T10:00Z
+pangea-flows run quicklooks               # Plan and process up to max_items items, once
+```
+
+A workflow is named by its name in `flows.py`, or as `module`, `module:name` or `file.py:name`. `PANGEA_URL` and the tokens can also go in a `.env` file in the current folder.
+
+`try` (also `dry-run`) doesn't touch the ledger: it plans from uploads since `--since`, a day ago by default, runs your function, and copies the outputs into `./out/<key>/<output key>/`. It only reads, so your personal API token works too (`PANGEA_TOKEN`).
+
+### Running workflows from their repositories
+
+On the platform, a member gives the workflow a **source**: the git repository, the branch and the workflow in it, like `flows.py:quicklook`. For a private repository, the workflow's page then shows a public key: add it to the repository as a read-only deploy key (on GitHub: the repository's Settings → Deploy keys). From then on, every push to the branch is picked up within a minute or so. A run that fails to build or import shows on the workflow's page with the error.
+
+The repository needs a `uv.lock`, so the server installs exactly what you tested with: `pangea-flows new` sets that up. Runs get 4 GB of memory, 2 CPUs and 2 hours by default; ask for more in the decorator, up to what the server allows:
+
+```python
+@each_file(inputs="pollyxt/raw", output="pollyxt/quicklooks", memory="8g", cpus=4, timeout=timedelta(hours=6))
+```
+
+The server side is `pangea-flows serve` (`pip install "pangea-flows[serve]"`, Python 3.11+, git and podman). An admin creates a runner on the platform (`POST /runners`, then `POST /runners/:id/token`), and writes its settings:
+
+```toml
+api_url = "https://pangea.example.org/api"
+token_file = "token"                 # Or $PANGEA_RUNNER_TOKEN
+secrets_key_file = "secrets.key"     # Or $PANGEA_RUNNER_SECRETS_KEY; see below
+state_dir = "/var/lib/pangea-runner" # Clones, deploy keys, scratch space
+
+[container]
+api_url = "https://pangea.example.org/api" # What runs use; api_url by default
+network = ""                         # podman --network for runs; its default if empty
+# With podman's default network, a run can't reach this host by its own address.
+# If the API is on the same host, point its name at the host instead:
+add_hosts = ["pangea.example.org:host-gateway"]
+
+[limits]                             # For runs that don't ask
+memory = "4g"
+cpus = 2
+timeout_minutes = 120
+
+[max]                                # The most a run may ask for
+memory = "16g"
+cpus = 4
+timeout_minutes = 1440
+```
+
+`secrets_key_file` holds the private key, base64, that [secrets](#secrets) are sealed to; the API has its public key as `WORKFLOW_SECRETS_PUBLIC_KEY` (see `packages/pangea-api/docs/runners.md` for making the pair). The runner opens each run's secrets and passes them to its container by name, with the values only in podman's environment, and masks them in the output it keeps. Without the key, runs of workflows that have secrets fail with a message saying so.
+
+Then `pangea-flows serve --config runner.toml`. It builds an image for each new commit (from `ghcr.io/astral-sh/uv`, with `uv sync --frozen`), and runs each workflow in a container with a read-only filesystem, a scratch `/tmp` on the host, its memory and CPU limits, and a token for that run only. See `packages/pangea-api/docs/runners.md` for what it does on the platform.
+
+### From Python
+
+```python
+from pangea_flows import run_once
+
+report = run_once(quicklooks)  # Or run_once(quicklooks, client=PangeaClient(url, token))
+print(report.done, report.failed, report.run_id)
+```
+
+## Secrets
+
+Credentials for systems outside PANGEA, such as an SCC login, are **secrets**. Members set them on the workflow's page on the platform, and runs get each as an environment variable. Declare the ones the code needs:
+
+```python
+@workflow(..., secrets=["SCC_USERNAME", "SCC_PASSWORD"])
+def scc_sync(item: Item, ctx: Context) -> list[Output]:
+    login(os.environ["SCC_USERNAME"], os.environ["SCC_PASSWORD"])
+```
+
+- If one isn't set, the run fails at the start with `Missing secrets: SCC_PASSWORD`, instead of every item failing. The platform shows which are missing.
+- Names are like `SCC_PASSWORD`. `PANGEA_*`, `HOME`, `TMPDIR`, `PATH` and `PYTHONUNBUFFERED` are the runner's.
+- Values are masked as `***` in the logs and errors sent to the platform. Values shorter than 6 characters aren't masked.
+- `pangea-flows run` and `try` read them from the environment or `.env`, like the token. Keep `.env` out of git.
+- On the platform, nobody can read a value back, not even the workflow's own token.
+
+## Order
+
+Items with an input from a dataset in `first` are handled before the others, in the order of the list. Everything else goes oldest first.
+
+```python
+@workflow(
+    inputs=["pollyxt/scc", "pollyxt/scc-calibration-candidates"],
+    outputs="pollyxt/scc-calibration",
+    keys=per_datum(),
+    first=["scc-calibration-candidates"],
+)
+def scc_sync(item: Item, ctx: Context) -> list[Output]:
+    ...
+```
+
+Here every calibration is uploaded, checked, and deleted from SCC if it fails, before any data file that would use it. A run takes items one at a time, so it finishes everything ready in a `first` dataset before it starts on the rest. It is an order, not a dependency: an item waiting to be retried doesn't hold back the others. The platform works out each item's place from all its inputs, also those planned in earlier runs, and changing `first` reorders items that are still waiting. `pangea-flows try` doesn't use the ledger, so it goes by key.
+
+## Options
+
+| `@workflow(...)` | |
+| --- | --- |
+| `inputs`, `outputs` | Datasets, see above |
+| `keys` | See above |
+| `schedule`, `timezone` | A cron expression, every 15 minutes by default; `None` to only run when asked |
+| `max_items` | Items per run, 10 by default |
+| `max_attempts` | Tries per item, 3 by default |
+| `lease` | How long a claim lasts without a heartbeat, 1 hour by default. Heartbeats are sent while the function runs. |
+| `start` | On the first run, also take data uploaded since this time |
+| `first` | Input datasets whose items go first, see [Order](#order) |
+| `secrets` | Environment variables with credentials, see [Secrets](#secrets) |
+| `name`, `token_env` | Default to the function's name and `PANGEA_TOKEN_<NAME>` |
+| Anything else | Shown with the workflow on the platform |
+
+## Development
+
+```bash
+uv sync
+uv run pytest                     # Unit tests
+uv run pytest -m integration      # Against a running API, see tests/README.md
+uv run pytest -m e2e              # pangea-flows serve, with podman
+```

pangea_flows-0.3.0/pyproject.toml

@@ -0,0 +1,31 @@
+[project]
+name = "pangea-flows"
+version = "0.3.0"
+description = "Write PANGEA processing workflows in Python"
+readme = "README.md"
+requires-python = ">=3.10"
+authors = [{ name = "Thanasis Georgiou", email = "ageorgiou@noa.gr" }]
+dependencies = [
+  "httpx>=0.27",
+  "pydantic>=2.6",
+]
+
+[project.optional-dependencies]
+serve = ["croniter>=2", "pynacl>=1.5"]
+
+[project.scripts]
+pangea-flows = "pangea_flows.cli:main"
+
+[dependency-groups]
+dev = ["pytest>=8"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+  "integration: needs a running PANGEA API, see tests/README.md",
+  "e2e: runs `pangea-flows serve` with podman against a running API, see tests/README.md",
+]