pyflowred 0.2.0__tar.gz → 0.3.0__tar.gz

{pyflowred-0.2.0 → pyflowred-0.3.0}/CHANGELOG.md

@@ -1,5 +1,63 @@
 # Changelog
 
+## 0.3.0 - 2026-09-27
+
+- **An end-to-end integration test** over the node set and its settings, run
+  against the service rather than in-process: `tools/e2eflows.py` deploys
+  eighteen tabs with 157 assertions, `tools/coverage.py` measures what they
+  reached. **66 of 66 countable node types** carry a message during a run
+  (97.1% counting the two set aside), and **183 of 191 reachable dialog
+  choices** are covered by a passing assertion (95.8%). Everything set aside
+  is named with its reason and both figures are printed, so an exclusion
+  cannot flatter the result. `tools/e2e/compose.yml` brings up the broker and
+  Kafka it wants.
+- Defects it found:
+  - **Window closing wedged for good** on a timestamp containing the byte
+    used as the index separator - about one millisecond in 250. The key was
+    split on that separator, the unpack raised, and because the index is
+    walked oldest-first the poisoned entry blocked every window behind it
+    from ever closing again. It is sliced by offset now.
+  - **The Assemble node shadowed `Node._complete`**, the runtime's own method
+    for finishing a message, so every message it emitted raised.
+  - `kafka out` took the topic and key from the message in preference to the
+    dialog, the opposite of the MQTT node and of what its own placeholder
+    says.
+  - The stream operators showed raw catalogue keys as their node status.
+  - A Global Config node carrying coordinates is treated as a placed node
+    and, belonging to no tab, dropped - so its environment variables were
+    never read.
+
+- **Stream processing.** Seven operators - `aggregate`, `derive`,
+  `threshold`, `transition`, `assemble`, `compress`, `pattern` - on a
+  `stream-state` configuration node backed by RocksDB. Windows tumble, hop,
+  slide, count, session, and close on a control message, which is what a
+  shift, a batch, a production order and a cleaning cycle all are. Event time
+  with watermarks and a second output for late messages; quality-aware
+  aggregation; a key limit that refuses rather than evicts; and
+  `GET <httpAdminRoot>stream-state` to see what is held.
+  RocksDB is the state store rather than a fallback: 100,000 keys each
+  holding a 300-sample window cost 1,193 MB as Python objects and 39 MB
+  resident through it, and its slowest hot path is eighteen times the rate
+  the runtime can feed it.
+
+- Google Cloud Storage nodes: a `gcs-config` bucket configuration plus
+  `gcs in` and `gcs out`, on obstore - the layer the flow store already uses.
+  Every operational value is a typed property, so bucket, path, format,
+  character set and content type can each be a literal, a message property, a
+  flow or global value, an environment variable or a JSONata expression; left
+  blank, each falls back to a documented message property. A flow that works
+  out where it is writing needs one node, not one per destination.
+  Credentials stay on the configuration node, where `${secret:NAME}` reaches
+  Secret Manager, and a path cannot climb out of the bucket's prefix.
+- `gcs list` and `gcs delete` alongside them. Listing is shallow (objects and
+  child folders at one level) or deep, with a limit and a `truncated` flag.
+  Deleting takes one object or a whole prefix, and is idempotent.
+- Folders work the same with and without a bucket's hierarchical namespace,
+  as far as the object API allows: shallow listing reports a folder as a
+  folder in either case, while obstore cannot address a name ending in `/`,
+  so a prefix delete leaves folder markers behind and reports them in
+  `msg.gcs.remainingFolders` rather than claiming a clean sweep.
+
 ## 0.2.0
 
 **Everything is now called `pyflowred`.** 0.1.0 shipped under that name on

{pyflowred-0.2.0 → pyflowred-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyflowred
-Version: 0.2.0
+Version: 0.3.0
 Summary: A Python port of Node-RED: the same editor, the same flow semantics, a Python runtime.
 Project-URL: Homepage, https://github.com/DataVoyage/pyflowred
 Project-URL: Documentation, https://github.com/DataVoyage/pyflowred/blob/main/docs/operations.md
@@ -37,6 +37,7 @@ Requires-Dist: prometheus-client>=0.21
 Requires-Dist: python-dateutil>=2.9
 Requires-Dist: python-multipart>=0.0.20
 Requires-Dist: pyyaml>=6.0
+Requires-Dist: rocksdict>=0.3.29
 Requires-Dist: starlette>=0.41
 Requires-Dist: uvicorn[standard]>=0.32
 Requires-Dist: watchfiles>=1.0
@@ -67,6 +68,8 @@ API — with a Python runtime and nodes written in Python.
 | Flow runtime, node registry, context, credentials, storage | Re-implemented in Python |
 | Projects (git-backed flows), multiplayer, palette catalogue | Re-implemented in Python |
 | Prometheus metrics endpoint and exporter nodes | New; the nodes ported from `node-red-contrib-prometheus-exporter` |
+| Kafka, Google Cloud Storage and secrets nodes | New; no Node-RED counterpart |
+| Stream processing: windows, aggregation, thresholds, state, patterns | New; RocksDB-backed keyed state |
 | Core nodes | Re-implemented in Python — all 50 types |
 
 The editor is not a reimplementation. It is Node-RED's own editor, talking to
@@ -120,9 +123,9 @@ Two images, same application. `Dockerfile` installs the released package with
 three stages, Node included.
 
 ```bash
-docker build -t pyflowred:0.2.0 .                        # released
+docker build -t pyflowred:0.3.0 .                        # released
 docker build -f Dockerfile.source -t pyflowred:source .  # this checkout
-docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.2.0
+docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.3.0
 ```
 
 Your own CA is a build argument or a mount, and reaches every library in the
@@ -188,6 +191,7 @@ src/pyflowred/
   registry/        node discovery, loading, palette installation
   api/             HTTP admin API, editor routes, comms WebSocket (Starlette)
   metrics/         the Prometheus registry, runtime instrumentation, endpoint
+  streaming/       keyed state on RocksDB, windows, aggregate functions
   nodes/core/      the core node set: Python implementations + Node-RED dialogs
   editor_client/   the Node-RED editor, vendored and built
 tools/
@@ -196,6 +200,14 @@ tools/
   uicheck.py       load the editor in a real browser and report on it
   uiflow.py        deploy and run a flow through the editor, in a real browser
   uiprojects.py    commit a deploy through the project UI, in a real browser
+  uigcs.py         check the Cloud Storage dialogs are typed inputs, in a browser
+  uistream.py      check the stream processing dialogs, in a browser
+  e2eflows.py      the end-to-end integration test: 18 tabs, 157 assertions
+  coverage.py      what a run reached: node types and dialog choices
+  e2e/             the MQTT broker and Kafka the test talks to
+  e2eflows.py      the end-to-end integration test: 12 tabs, 45 assertions
+  coverage.py      how much of the node set a run actually reached
+  e2e/             the MQTT broker and Kafka that test talks to
   benchflows.py    seven benchmark flows, 82 nodes across 33 node types
   loadtest.py      drive them and sample the runtime's CPU, RSS and fds
   rawbaseline.py   a bare Starlette endpoint, to measure the runtime against
@@ -213,6 +225,10 @@ tests/             the test suite, including a browser test of the editor
 .venv/bin/python -m pytest tests
 ```
 
+[docs/test-coverage.md](docs/test-coverage.md) is the standing record of what
+is tested, how, and what is not - including the 24% of end-to-end assertions
+that so far only check that something arrived rather than what it was.
+
 The suite covers every node group against real sockets (including an
 in-process MQTT broker), the runtime (subflows, groups, context, credentials,
 deploy types), the admin API, projects against a real `git` binary, and — when

{pyflowred-0.2.0 → pyflowred-0.3.0}/README.md

@@ -17,6 +17,8 @@ API — with a Python runtime and nodes written in Python.
 | Flow runtime, node registry, context, credentials, storage | Re-implemented in Python |
 | Projects (git-backed flows), multiplayer, palette catalogue | Re-implemented in Python |
 | Prometheus metrics endpoint and exporter nodes | New; the nodes ported from `node-red-contrib-prometheus-exporter` |
+| Kafka, Google Cloud Storage and secrets nodes | New; no Node-RED counterpart |
+| Stream processing: windows, aggregation, thresholds, state, patterns | New; RocksDB-backed keyed state |
 | Core nodes | Re-implemented in Python — all 50 types |
 
 The editor is not a reimplementation. It is Node-RED's own editor, talking to
@@ -70,9 +72,9 @@ Two images, same application. `Dockerfile` installs the released package with
 three stages, Node included.
 
 ```bash
-docker build -t pyflowred:0.2.0 .                        # released
+docker build -t pyflowred:0.3.0 .                        # released
 docker build -f Dockerfile.source -t pyflowred:source .  # this checkout
-docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.2.0
+docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.3.0
 ```
 
 Your own CA is a build argument or a mount, and reaches every library in the
@@ -138,6 +140,7 @@ src/pyflowred/
   registry/        node discovery, loading, palette installation
   api/             HTTP admin API, editor routes, comms WebSocket (Starlette)
   metrics/         the Prometheus registry, runtime instrumentation, endpoint
+  streaming/       keyed state on RocksDB, windows, aggregate functions
   nodes/core/      the core node set: Python implementations + Node-RED dialogs
   editor_client/   the Node-RED editor, vendored and built
 tools/
@@ -146,6 +149,14 @@ tools/
   uicheck.py       load the editor in a real browser and report on it
   uiflow.py        deploy and run a flow through the editor, in a real browser
   uiprojects.py    commit a deploy through the project UI, in a real browser
+  uigcs.py         check the Cloud Storage dialogs are typed inputs, in a browser
+  uistream.py      check the stream processing dialogs, in a browser
+  e2eflows.py      the end-to-end integration test: 18 tabs, 157 assertions
+  coverage.py      what a run reached: node types and dialog choices
+  e2e/             the MQTT broker and Kafka the test talks to
+  e2eflows.py      the end-to-end integration test: 12 tabs, 45 assertions
+  coverage.py      how much of the node set a run actually reached
+  e2e/             the MQTT broker and Kafka that test talks to
   benchflows.py    seven benchmark flows, 82 nodes across 33 node types
   loadtest.py      drive them and sample the runtime's CPU, RSS and fds
   rawbaseline.py   a bare Starlette endpoint, to measure the runtime against
@@ -163,6 +174,10 @@ tests/             the test suite, including a browser test of the editor
 .venv/bin/python -m pytest tests
 ```
 
+[docs/test-coverage.md](docs/test-coverage.md) is the standing record of what
+is tested, how, and what is not - including the 24% of end-to-end assertions
+that so far only check that something arrived rather than what it was.
+
 The suite covers every node group against real sockets (including an
 in-process MQTT broker), the runtime (subflows, groups, context, credentials,
 deploy types), the admin API, projects against a real `git` binary, and — when

{pyflowred-0.2.0 → pyflowred-0.3.0}/docker/README.md

@@ -11,7 +11,7 @@ Two images, same application.
 
 ```bash
 # released
-docker build -t pyflowred:0.2.0 .
+docker build -t pyflowred:0.3.0 .
 docker build -t pyflowred:0.3.0 --build-arg PYFLOWRED_VERSION=0.3.0 .
 
 # from this checkout
@@ -19,7 +19,7 @@ docker build -f Dockerfile.source -t pyflowred:source .
 ```
 
 ```bash
-docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.2.0
+docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.3.0
 ```
 
 The editor is then at <http://127.0.0.1:1880/>. Flows and settings live in
@@ -38,7 +38,7 @@ Settings can come from the environment, so nothing has to be baked in:
 docker run -d -p 1880:1880 \
     -e PYFLOWRED_UI_PORT=1880 \
     -e PYFLOWRED_FLOW_STORE=gs://my-bucket/pyflowred/production \
-    -v pyflowred-data:/data pyflowred:0.2.0
+    -v pyflowred-data:/data pyflowred:0.3.0
 ```
 
 ## Your own CA
@@ -51,13 +51,13 @@ proxy, where `uv` and `npm` cannot fetch anything without it:
 
 ```bash
 cp corp-root-ca.crt docker/ca/
-docker build -t pyflowred:0.2.0 .
+docker build -t pyflowred:0.3.0 .
 ```
 
 **Mounted at run time** - so a certificate can be rotated without rebuilding:
 
 ```bash
-docker run -d -p 1880:1880 -v ./ca:/certs:ro -v pyflowred-data:/data pyflowred:0.2.0
+docker run -d -p 1880:1880 -v ./ca:/certs:ro -v pyflowred-data:/data pyflowred:0.3.0
 ```
 
 Every `.crt` and `.pem` in `/certs` is picked up at start-up. Starting twice
@@ -97,7 +97,7 @@ at a time:
 cp corp-root-ca.crt docker/ca/
 
 # released: one index, which proxies PyPI
-docker build -t pyflowred:0.2.0 \
+docker build -t pyflowred:0.3.0 \
     --build-arg HTTPS_PROXY=http://proxy.corp:3128 \
     --build-arg HTTP_PROXY=http://proxy.corp:3128 \
     --build-arg NO_PROXY=.corp,localhost,127.0.0.1 \
@@ -123,7 +123,7 @@ well:
 docker run -d -p 1880:1880 \
     -e HTTPS_PROXY=http://proxy.corp:3128 \
     -e UV_INDEX_URL=https://nexus.corp/repository/pypi/simple \
-    -v pyflowred-data:/data pyflowred:0.2.0
+    -v pyflowred-data:/data pyflowred:0.3.0
 ```
 
 ## Compose

{pyflowred-0.2.0 → pyflowred-0.3.0}/docs/operations.md

@@ -3,8 +3,8 @@
 ## In a container
 
 ```bash
-docker build -t pyflowred:0.2.0 .
-docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.2.0
+docker build -t pyflowred:0.3.0 .
+docker run -d -p 1880:1880 -v pyflowred-data:/data pyflowred:0.3.0
 ```
 
 Two images: `Dockerfile` installs the released package with `uv` and carries
@@ -255,6 +255,317 @@ The Kafka tests use it and skip themselves when it is not running:
 .venv/bin/python -m pytest tests/test_nodes_kafka.py -m slow
 ```
 
+## Google Cloud Storage
+
+A `gcs-config` node holds the bucket and the identity, and any number of
+`gcs in` (read), `gcs out` (write), `gcs list` and `gcs delete` nodes point at
+it - the same shape as Kafka and MQTT. Built on `obstore`, the same layer the
+flow store uses.
+
+### Everything can come from the message
+
+This is the part that matters in a flow. Bucket, path, format, character set
+and content type are all typed properties: each can be a literal, a message
+property, a flow or global value, an environment variable or a JSONata
+expression. Leave a field blank and it falls back to a documented property of
+the message.
+
+| Field | Falls back to |
+|---|---|
+| Path | `msg.filename` |
+| Bucket | `msg.bucket` |
+| Output as (`gcs in`) | `msg.format` |
+| Content type (`gcs out`) | `msg.contentType` |
+| Encoding | `msg.encoding` |
+| Listing (`gcs list`) | `msg.depth` |
+| Delete (`gcs delete`) | `msg.mode` |
+| Limit | `msg.limit` |
+
+The order is always the field in the dialog, then the message, then the
+default. So a writer whose path is the JSONata expression
+
+```
+"orders/" & payload.tenant & "/" & payload.id & ".json"
+```
+
+serves every tenant and every order from one node.
+
+After either node, `msg.gcs` carries `bucket`, `path`, `key`, `size`,
+`contentType`, `eTag`, `lastModified` and, on a versioned bucket, `version`.
+`path` is what the node was asked for and `key` is where that landed once the
+bucket's prefix was applied - keeping both is what lets `msg.gcs.path` be fed
+straight into another node without the prefix being applied twice.
+
+### Credentials, and why they are not on the message
+
+Credentials are the one thing that stays on the configuration node. They
+belong to the connection rather than to the operation, and a message able to
+choose the key it is written with would be a way to write somewhere it should
+not.
+
+* **Application default** is the default and needs nothing configured: it
+  works from `GOOGLE_APPLICATION_CREDENTIALS` on a workstation, from the
+  metadata server on Compute Engine, and from Workload Identity in GKE, where
+  there is no key file at all.
+* **Service account key** takes the JSON itself, stored encrypted in the
+  credentials file. `${secret:NAME}` works here, so the key can live in Secret
+  Manager and be rotated without touching the flow.
+* **Service account file** takes a path, for a key mounted as a Kubernetes
+  secret.
+* **Anonymous** signs nothing, for a public bucket.
+
+The **Options** table takes any obstore GCS setting by name and is applied
+last, so it overrides the fields above - `base_url` against an emulator, for
+example.
+
+### Listing and deleting
+
+`gcs list` has two shapes. **Shallow** asks what is at one level: objects in
+`msg.payload`, child folders in `msg.folders`. **Deep** walks the whole
+subtree and returns no folders. Both stop at **Limit** (1000 by default) and
+set `msg.gcs.truncated` when there was more - a bucket can hold millions of
+objects, and a node that tried to put all of them on one message would take
+the runtime with it.
+
+Each entry carries `path` and `key`. `path` is relative to the bucket's
+prefix, so it goes straight into a `gcs in` node; a `split` node turns the
+array into one message per object.
+
+`gcs delete` removes one object, or everything under a prefix. It is
+idempotent: something already gone is not an error, it simply does not appear
+in `msg.gcs.deleted`.
+
+### Folders, with and without a hierarchical namespace
+
+A bucket has folders in one of two ways, and a flow should not have to know
+which. With a [hierarchical
+namespace](https://cloud.google.com/storage/docs/hns-overview) they are real
+resources. Without one - still the default, and it cannot be switched on
+afterwards - a folder is either implied by an object name containing `/`, or
+is a zero-byte object whose name ends in `/`, which is what the console and
+gsutil create.
+
+Measured against a real flat bucket:
+
+* **Shallow listing reports a folder as a folder** - as a child folder, not
+  as an object - even when nothing sits under it. That is why it is the
+  default here.
+* **Deep listing does not.** A zero-byte marker comes back as an ordinary
+  object, with its trailing slash removed, because obstore normalises paths.
+* **obstore cannot address a name ending in `/` at all.** It strips the slash
+  before the request, so it can neither create a marker nor delete one: the
+  delete returns "not found" while the object stays in the bucket.
+
+So a prefix delete removes the objects and leaves the markers, and says so:
+`msg.gcs.remainingFolders` lists what is still there.
+
+With a hierarchical namespace the first two hold, but two things differ -
+taken from Google's documentation rather than measured here:
+[empty folders are excluded from a delimited
+listing](https://cloud.google.com/storage/docs/hns-overview), so
+`remainingFolders` will not report one, and a folder is removed by its own
+operation, which the object API obstore speaks does not reach.
+
+Either way nothing breaks and no wrong object is touched. What can remain
+after a prefix delete is an empty folder.
+
+### The prefix is a boundary
+
+A bucket configuration may carry a prefix, and it is not only a convenience:
+a path is refused if it tries to climb out of the prefix with `..`. Paths
+normally come from messages, so without that a payload could read and write
+anywhere in the bucket.
+
+Cloud Storage has no append. Writing to a path that exists replaces it.
+
+### Testing against a real bucket
+
+The node tests run against an in-memory store, so they need no network. The
+round trip against a real bucket is marked slow and skips itself unless the
+environment names one:
+
+```bash
+GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json \
+PYFLOWRED_TEST_BUCKET=gs://a-bucket/a-prefix \
+    .venv/bin/python -m pytest tests/test_nodes_gcs.py -m slow
+```
+
+The edit dialogs are checked in a real browser, because whether a field is
+really a typed input is not visible from Python:
+`.venv/bin/python tools/uigcs.py http://127.0.0.1:1890/`.
+
+## Stream processing
+
+Seven operators that turn raw tag traffic into measurements and events, and
+one configuration node holding what they remember.
+
+| Node | What it does |
+|---|---|
+| `aggregate` | group by a key, collect into a window, compute |
+| `derive` | arithmetic against the previous value: delta, rate, integral, counter with rollover, running total |
+| `threshold` | level events with hysteresis and a dwell time |
+| `transition` | discrete state changes with duration and debounce |
+| `assemble` | last value per field into one record, with staleness |
+| `compress` | deadband and swinging-door reduction |
+| `pattern` | all / in order / absent, within a window |
+
+Every operational field is a typed property, so a flow can work out its own
+group key, value, limits and window.
+
+### Windows, including the one that matters most
+
+Tumbling, hopping and sliding by time; tumbling and sliding by count; session
+by a quiet gap - and **closed by a control message**. That last one is the
+generalisation: a shift, a batch, a production order and a cleaning cycle are
+all a period whose boundaries the plant announces. Send
+`msg.control = "close"` with the same group key and the window emits and
+resets.
+
+```
+inject (cron)  ──►  aggregate (external window)  ──►  …
+mqtt in  ──► transition ──────────┘
+```
+
+That is availability: `transition` gives run/idle/fault with durations, the
+cron `inject` closes the shift, `aggregate` sums the durations per state.
+
+### Time
+
+Processing time by default. With **event time** the timestamp comes from a
+message property and a window closes when the watermark - the highest
+timestamp seen, less the allowed lateness - passes its end. A message arriving
+after that leaves by the second output carrying `msg.late`, rather than being
+dropped where nobody can see it.
+
+### Quality
+
+Name a quality property and a sample it marks as bad is counted in
+`window.bad` and kept out of the arithmetic. A reading from a failed sensor
+averaged in with good ones is a number that is quietly wrong. `compress`
+always passes a bad sample through, because dropping it would hide the one
+thing somebody needs to see.
+
+### State
+
+`stream-state` is one RocksDB database, and there is usually more than one
+because retention is the reason to separate them.
+
+```python
+# a five-minute window and a thirty-day counter want different time-to-live
+```
+
+Why RocksDB rather than memory, measured here: 100,000 keys each holding a
+300-sample window cost **1,193 MB** as Python objects and **39 MB** resident
+through RocksDB. The slowest hot path through it - read, modify and write a
+300-sample buffer once per message - runs at 46,000 messages a second, which
+is eighteen times what the runtime can feed it. There is no memory tier in
+front of it on purpose.
+
+**Time to live reclaims space during compaction, not on read.** An entry whose
+time is up stays readable until a compaction happens to run. It is space
+reclamation and never semantics: every value carries its own timestamp and the
+operator decides what is stale.
+
+The database is **not closed on a deploy**. State is meant to outlive one, and
+RocksDB locks its directory, so closing on the way out would hand the
+replacement node a locked directory.
+
+### Not leaking, and being able to look
+
+Every operator takes a **key limit** and refuses new keys beyond it rather
+than evicting old ones - evicting would mean quietly not counting something
+the flow was asked to count. The limit bounds *open* state, which is the only
+thing that can run away.
+
+```bash
+curl <httpAdminRoot>stream-state
+curl '<httpAdminRoot>stream-state?node=<nodeId>'
+```
+
+reports every database and operator: keys tracked, refusals, late messages and
+how far the watermark has got. State you cannot look at is state you cannot
+trust, and an LSM tree is not readable in an editor - so this is the way in.
+
+## The end-to-end integration test
+
+A flow set that exercises the node types and their settings against the
+running service, with assertions, and a tool that measures how much of the
+node set a run really reached.
+
+```bash
+docker compose -f tools/e2e/compose.yml up -d          # MQTT and Kafka
+GOOGLE_APPLICATION_CREDENTIALS=... \
+    .venv/bin/python tools/e2eflows.py --deploy http://127.0.0.1:1880 --run
+.venv/bin/python tools/coverage.py http://127.0.0.1:1880 --run
+```
+
+Eighteen tabs, **157 assertions**. A simulated plant runs alongside so the
+MQTT, Kafka and stream paths are exercised continuously rather than once per
+button press.
+
+* **Run all checks** on *00 Overview* fires every subject at once.
+* `GET <httpNodeRoot>e2e/checks` returns the result as JSON,
+  `GET <httpNodeRoot>e2e/report` as a page.
+* A check whose dependency is not running reports **skip**; one that never
+  fires at all reports **never reported**, which is the failure mode that
+  otherwise looks like a check that was never written.
+
+### What it measures
+
+| | |
+|---|---|
+| **Node types** | 66 of 66 countable carry a message during a run — 97.1% counting the two set aside |
+| **Options within the nodes** | 183 of 191 reachable choices — 95.8% |
+
+`tools/coverage.py` reads the runtime's own message counters before and after
+a run, so a type counts only when a message actually went through it.
+Deployed is not tested.
+
+For options it reads every choice out of the node dialogs themselves, rather
+than from a list kept by hand that would go stale the first time somebody
+added one, and counts a choice when a node carrying it is upstream of an
+assertion that passed - or is named by one as something it proves, which is
+how the far side of a broker or a socket is accounted for.
+
+Read that precisely: it means *a node carrying this choice was proven to
+work*, not *this choice behaves differently from its neighbours*. A
+subscriber cannot tell a message published at quality of service 1 from one
+at 2. Distinguishing those is what the unit tests are for.
+
+Everything set aside is named and printed with its reason - a will message
+needs the client to vanish, SASL needs a broker configured for it, the
+Template node's syntax highlighting is never read at runtime - and both
+figures are shown, with and without the exclusions, so setting something
+aside cannot flatter the result.
+
+A detailed account of what is and is not tested - including where an
+assertion only checks that something arrived rather than what it was - is in
+[test-coverage.md](test-coverage.md).
+
+### Keeping it honest
+
+The assertions and the soak are separate. An assertion needs input it can
+predict, so each subject has a burst with known values. Counters inside the
+assertions are kept per run, because the stream state is durable on purpose
+and a second press of the button must not read as the first one having
+passed more.
+
+The generator refuses to emit a Function node whose body will not compile: a
+body built by string concatenation is one stray escape away from a syntax
+error, and the runtime reports that as a node that quietly never fires -
+which looks exactly like a feature that does not work.
+
+Deploying is idempotent and additive: the tool replaces every tab whose id
+starts with `t-` and leaves anything else alone. Credentials travel with the
+deploy rather than sitting in the flow file.
+
+The test needs a secrets provider, and the Cloud Storage tabs need
+credentials the runtime can find:
+
+```python
+secrets = {"provider": "file", "path": "~/.pyflowred/secrets", "ttl": 300}
+```
+
 ## Secrets
 
 A flow that calls a protected API or connects to a broker with authentication

{pyflowred-0.2.0 → pyflowred-0.3.0}/docs/porting.md

@@ -138,6 +138,20 @@ confluent-kafka, with the full librdkafka configuration surface reachable
 through a properties table. Everything on the wire is JSON, by design rather
 than by option. See [operations.md](operations.md#kafka).
 
+### Google Cloud Storage
+
+Also not a Node-RED node set, and the same shape again: a `gcs-config` bucket
+configuration plus `gcs in`, `gcs out`, `gcs list` and `gcs delete`. On
+obstore - the layer the flow store already uses, so there is one
+object-storage implementation in the process and not two.
+
+What is different from the `file` nodes it otherwise resembles: **every
+operational value is a typed property**, so bucket, path, format, character
+set and content type can each be a literal, a message property, a flow or
+global value, an environment variable or a JSONata expression. A flow that
+works out where it is writing does not need a node per destination. See
+[operations.md](operations.md#google-cloud-storage).
+
 ### Secrets
 
 Another PyFlowRED addition. `secrets` names a provider - Google Secret Manager or