@tikoci/centrs 0.1.6 → 0.1.8-next.0

package/commands/retrieve/README.md

@@ -1,8 +1,10 @@
 # retrieve
 
 Status: `CHR-passed` over `rest-api` and `native-api`, including multi-target
-fan-out (see **Target selection**). `snmp` is `not-started`. Matches
-`docs/MATRIX.md`.
+fan-out (see **Target selection**), `--query` row filters (see **Query**),
+`--follow` over `native-api` (see **Follow**) and `--sample` over both (see
+**Sample**) and bounded readiness/conditions (see **Wait**). `snmp` is
+`not-started`. Matches `docs/MATRIX.md`.
 
 Read RouterOS state. RouterOS menu reads model `<path>/<verb>` where the verb
 is `print`-style (`print`, `get`, and async POST-shaped reads as they're
@@ -50,6 +52,8 @@ duplicate that table. Behavior notes the generated reference cannot carry:
 - `--format json` (shortcut `--json`): set `CENTRS_FORMAT=json` to make the
   structured envelope the default. `--format yaml` renders the same envelope.
 - `--port`: native-api defaults to 8728; `--port 8729` selects TLS `api-ssl`.
+  A port typed into the target (`host:port`) is honored too, below the port
+  settings ladder (`--port`, `CENTRS_PORT`, a CDB `port=` comment, config).
 - `--query`/`--filter` are RouterOS **row** filters and always return an
   array (see Synopsis); `--where` filters *which devices* fan out, not rows.
 - `--all-attributes` conflicts with `--attribute(s)` (see Validation).
@@ -63,7 +67,7 @@ Spec-tier flags with no implementation yet — today they fail as unknown flags:
 
 | Flag                                | Designed behavior                                                                          |
 | ----------------------------------- | ----------------------------------------------------------------------------------------- |
-| `--once`                            | Bounded single read of a monitor-style menu (RouterOS `once`): returns **one** envelope and never follows. Open-ended follow is `api … --stream`. See constitution: protocol selection. |
+| `--once`                            | Bounded single read of a monitor-style menu (RouterOS `once`): returns **one** envelope and never follows. Open-ended follow is `--follow` (below), or `api … <menu>/listen` for the literal wire. See constitution: protocol selection. |
 | `--max-bytes <n>`                   | Byte budget for the rendered payload. If the response would exceed it, centrs truncates to fit, keeps `ok: true`, and adds a warning + `meta.truncated`. Not an error. (Will replace the implemented `--max-results`, which fails instead of truncating.) |
 | `--max-rows <n>`                    | Maximum row count for list reads. Excess rows are clipped; `ok: true` with a warning + `meta.truncated`.                                    |
 | `--yaml`                            | Shortcut for `--format yaml`.                                                             |
@@ -77,6 +81,15 @@ Default validator: `/console/inspect request=syntax path=<comma-joined>,print`
 unknown attributes with `validation/*` codes that include the inspect-suggested
 alternatives in the error envelope.
 
+`meta.validation.result` follows
+[the constitution](../../docs/CONSTITUTION.md#validation-is-the-product). It is
+`passed` once inspect accepts the read, even if the read fails afterward. It is
+`failed` when the gate refuses: `validation/*`, or `--query` on a singleton.
+It is absent when a `transport/*`, `auth/*`, `routeros/*` or `internal/*` error
+stopped inspect from answering; the device stage then says `not reached`
+(examples 12 and 16b; 16c is the `passed` case). With `--validate=false` it
+is `{enabled: false, source: "disabled"}`.
+
 If `--all-attributes` is combined with `--attribute(s)`, fail with
 `usage/conflicting-flags` *before* hitting the network.
 
@@ -140,6 +153,334 @@ warning entry — it is not an error. (Earlier text said a byte overflow returns
 *error* with the needed size; superseded — truncation is an `ok: true` footnote so
 partial data stays usable.)
 
+A failure **after** target resolution keeps what resolution found. The error
+envelope carries the resolved `meta.target` (host, port, base URL, record,
+sources), `meta.via`, the setting sources in `meta.settings`, and
+`meta.operation.auth` (`username`, `passwordProvided`; provenance only, never
+the secret). This holds for one-shot reads, `--follow`, `--sample`, `--wait`,
+each fan-out member and the MCP `centrs_retrieve` tool. A failure before
+resolution (bad shape, unknown target) keeps `meta.target.input` and a valid
+pinned `via`, with empty `meta.settings`. The
+library exposes this as `retrieveEnvelope()`, which never throws (examples 13,
+16b).
+
+## Query (`--query`)
+
+`retrieve <router> <menu> --query '<expr>'` returns only the rows that match,
+filtered on the router. `<expr>` is what you would write after `print where`:
+
+```bash
+centrs retrieve $R /interface --query 'type=ether and !disabled'
+centrs retrieve $R /ip/route --query '(dst-address="0.0.0.0/0" or distance>=10) and active'
+```
+
+`--filter` is the same flag. Repeat either, and the expressions are AND-ed. A
+list read with `--query` is always an array, `[]` when nothing matches. The
+result works with `--attribute(s)`, `--sample`, `--follow` and fan-out.
+A singleton (`/system/identity`) has no rows to filter and
+fails `usage/conflicting-flags`.
+
+**What it compiles to.** centrs parses the expression before connecting and
+sends it as the API query (native `?` words, REST `.query`; one language,
+`commands/api/AGENTS.md`). The router evaluates it, so comparisons use
+RouterOS's own per-property typing:
+
+| You write | RouterOS query words |
+| --------- | -------------------- |
+| `a=v`, `a!=v` | `a=v`; `a=v` `#!` |
+| `a<v`, `a>v` | `<a=v` `-a` `#\|`; `>a=v` |
+| `a<=v`, `a>=v` | `>a=v` `#!`; `<a=v` `-a` `#\|` `#!` |
+| bare `a` | `a=yes` if `a` is a boolean, otherwise `a` (has a value) |
+| `!x`, `x and y`, `x or y` | x `#!`; x y `#&`; x y `#\|` |
+
+As in `print where`, a row without the property counts as smaller than any
+value: `timeout<1d` includes static address-list entries (no `timeout`), and
+`timeout>1h` does not. API `<`/`>` skip such rows on their own, hence the `-a`
+("has no `a`") word. `and` (`&&`) binds tighter than `or` (`||`), as in RouterOS. Values run to the
+next space or parenthesis, and start right after the operator (RouterOS
+rejects `comment= x`). Quote a value with spaces or an `=` (`comment="uplink a"`,
+`comment="x=yes"`), using RouterOS escapes: `\"`, `\\`, `\$`, `\?`, `\_`, `\a`
+`\b` `\f` `\n` `\r` `\t` `\v`, and two uppercase hex digits (`\41`; `\ff` is `\f`
+then `f`). A backslash before whitespace drops it and the whole whitespace run
+after it, so `"x\<newline>  y"` is `xy`. A hex
+byte above `\7F` is refused: centrs sends values as UTF-8, so it could not match
+the raw byte RouterOS stores.
+
+**A bare name.** `where disabled` means `disabled=yes`, but `where comment`
+means "comment is set" (RouterOS stores an empty comment as absent). Only the
+router knows which kind a property is, so centrs asks it: `/console/inspect`
+completion after `where <name>=` offers exactly `yes`/`no` for a boolean. With
+`--validate=false` there is no one to ask, and a bare name fails
+`input/invalid-query`; write `disabled=yes`.
+
+**Validation.** Every property in the expression is checked against the
+menu's attributes (as for `--attributes`, plus `.id`). RouterOS answers a query
+on a misspelled property with no rows and no error, so a typo fails
+`validation/unknown-attribute` instead.
+
+**Same rows as `print where`, with these differences.** On CHR 7.23.7 and
+7.24.5, 58 expressions over three menus return exactly the rows
+`find where` selects (examples QY1, QY2). The exceptions come from the CLI
+typing an unquoted literal, which the query does not do:
+
+- `where dst-port=80` reads `80` as a number and matches nothing, and
+  `where protocol=tcp` likewise; the query matches the `"80"` and `"tcp"` that
+  `retrieve` shows. Quoted (`dst-port="80"`), the CLI agrees. This difference
+  is version-dependent: on CHR 7.25beta3, 7.25beta5, 7.25rc1 and 7.26beta1,
+  both unquoted CLI expressions also match. QY2 keeps exact expectations for
+  both behaviors; API query results stay the same over both transports.
+- Ordering (`<`, `>`, `<=`, `>=`) on a property RouterOS returns as text
+  (address-list `address`, `dst-port`) is text order: `192.0.2.10` sorts
+  before `192.0.2.9`. Numbers (`mtu`, `vlan-id`, `distance`) compare as
+  numbers.
+- The query accepts `disabled=true` as `yes`; the CLI rejects it.
+
+**Refused, never approximated.** `~` (regex) and `in` (prefix membership) have
+no query word, and a `$variable`, `[command]` or `{array}` value needs the
+script evaluator: these fail `input/unsupported-query` with a tip. Four
+spellings RouterOS reads differently from how they look fail
+`input/invalid-query`: `!a=v` (RouterOS reads `(!a)=v`; write `!(a=v)` or
+`a!=v`), `not` (not a RouterOS operator; use `!`), conditions separated
+only by a space (join them with `and` or `or`), and an unquoted `=` inside a
+value (RouterOS reads `comment=x=yes` as `(comment=x)=yes`, and `comment=x!=y`
+matches nothing; quote it: `comment="x=yes"`).
+
+## Follow (`--follow`)
+
+`retrieve <router> <menu> --follow` keeps a menu's rows current: first the
+rows that exist now, then a line for every change, as NDJSON. It is the
+friendly wrapper over native-API `listen`; `api <router> <menu>/listen` is the
+literal wire (`commands/api/README.md`). Decisions and CHR evidence: #396.
+
+**What starts it.** `--follow` on a list menu whose `print` takes
+`follow-only` in `/console/inspect` (checked live). A menu without it, a
+singleton included, fails `validation/not-followable` before anything is
+followed. Native-api only: an unset `via` infers native-api, and a `via` of
+`rest-api` from any source (flag, env, CDB, settings) fails
+`transport/capability-unsupported`; there is no polling fallback. Single
+target only: a fan-out selector fails `usage/fanout-not-supported`. Follow is
+filtered with the same `--query`/`--filter` grammar as a one-shot read (#397).
+
+**What each line means.** Every line is one envelope; read
+`meta.operation.stream`:
+
+| `kind` | When | `data` | Fields |
+| ------ | ---- | ------ | ------ |
+| `frame` | One row change | the row (`upsert`), `null` (`removed`) | `index`, `phase` (`snapshot`/`live`), `change` (`upsert`/`removed`), `id`, `source` (`print`/`listen`/`sweep`/`membership`) |
+| `synced` | Once, after the snapshot | `null` | `rows`: rows held at that point |
+| `notice` | First, only when there is advice | `null` | — (`tips` carries it) |
+| `summary` | Last, exactly once | the counts | `stopReason`, `frames`, `snapshot`, `changes`, `sweeps`, `synced`, `durationMs` |
+
+- **Bootstrap (A1).** centrs starts the `listen`, then the `print`, on one
+  connection. It emits the `print` rows, then every `listen` change received
+  since the `listen` started, all as `phase: "snapshot"`, then `synced`.
+  Applying the lines in order gives the menu's state at `synced`. That order
+  had 0 errors over about 14.7k writes on 7.23.7 and 7.24.5; merging by arrival
+  order resurrects deleted rows (#396 round 3). `synced` means the bootstrap
+  was processed. It is not an atomic snapshot, a full event history, or
+  routing convergence.
+- **`upsert`** carries the latest full row; replace what you hold for `id`.
+  **`removed`** carries `data: null`; drop `id`. The row identity is always
+  `meta.operation.stream.id`, even when `--attributes` leaves `.id` out of
+  `data`.
+- **Changes are coalesced state, not an event log.** RouterOS sends a change
+  about 200 ms after a write and folds writes in between. A row created and
+  deleted inside that window may never appear, and a flap may arrive as one
+  line or none. A `.dead` for an `id` centrs never reported is dropped, so a
+  removal the sweep saw first is reported once, as `source: "sweep"`.
+- **Removals RouterOS does not send.** View menus (`/interface/<type>`,
+  `/ip/route`, `/ipv6/route`) never send `.dead`. An `.id`-only sweep
+  (`print .proplist=.id`) runs every `--sweep` (default `10s`) on every followed
+  menu and reports an `id` that is gone as `removed` with `source: "sweep"`.
+  An `id` that changed since that sweep was sent is never removed by it. A
+  failed sweep ends the follow with an error; it never infers a removal. A
+  sweep only checks membership: it does not repair a missed field update or
+  report rows `listen` never announced. `--sweep` needs a unit (`10s`,
+  `500ms`); a bare number fails `settings/invalid-timeout`, as for `--sample`.
+  `--sweep 0` turns it off, with a notice tip saying view-menu removals will
+  not be reported.
+- **What `listen` never reports.** Hot counters (`rx-byte`, firewall
+  `bytes`/`packets`) and some protocol state (BFD sessions accept `listen` but
+  stay silent). The follow is correct but quiet there; `--sample` (below)
+  is the answer for those.
+- **Filtered membership.** The snapshot and sweep carry the predicate; the
+  listen stays unfiltered, retaining `.id,.dead`. For changed ids centrs reads
+  the current matching rows from RouterOS, in batches of at most 256 ids with
+  one membership read in flight. No client type coercion is involved, and
+  predicate fields need not appear in the caller's projection. These frames
+  have `source: "membership"`: a matching row is upserted, and a held row that
+  no longer matches (or no longer exists) is removed. This removal is a change
+  in the selected set, not a claim that RouterOS sent `.dead`. A real `.dead`
+  still has `source: "listen"` and only removes a held id. Delayed membership
+  results are discarded for ids with newer wire changes; sweeps and membership
+  reads are serialized. Bootstrap fixes its changed-id/version cutoff at the
+  snapshot's wire completion and reconciles those ids before `synced`. A
+  reply superseded by a newer change is discarded; that newer version stays
+  queued for live reconciliation. Continuous writes cannot extend bootstrap
+  indefinitely or postpone the first sweep and live `--count`. This may
+  coalesce more changes than unfiltered A1 replay.
+  The filtered sweep also detects silent exits from a predicate (e.g. a
+  decreasing address-list `timeout`). It does not discover silent entries or
+  repair silent field updates: use `--sample` for such state. With `--sweep 0`,
+  silent exits remain held until a listen notification arrives.
+- **Projection.** `--attribute(s)` projects `data`; centrs still asks RouterOS
+  for `.id,.dead` and removes them from `data` unless you named them.
+  `--all-attributes` sends `detail` to both `listen` and `print`.
+  `--list-attributes` and `--max-results` conflict with `--follow`
+  (`usage/conflicting-flags`).
+
+**What ends it.** `--count N` stops after N `live` frames; snapshot frames and
+`synced` never count, so `--count 1` is "the next observed change" (coalesced,
+so not a guaranteed transition). `--duration <d>` is a wall-clock bound that
+includes the bootstrap. Ctrl-C (`SIGINT`) stops it. Each ends with a
+successful summary whose `stopReason` is `count-reached`, `duration-elapsed`,
+or `interrupted`. A RouterOS error, a lost connection, a failed sweep or a full
+buffer ends it with a failed summary (`ok: false`, `stopReason`
+`routeros-error` or `transport-error`), whose counts are partial.
+`routeros-error` means RouterOS answered with an error; `transport-error`
+covers every other failure, and `error.code` names the cause. Buffers are
+bounded: if more than 50,000 changes are waiting (a consumer that stopped
+reading, or a burst during a long snapshot), the follow ends with
+`transport/stream-overflow`. The state you hold is stale; follow again to get a
+new snapshot. `--sweep` without `--follow`, and `--count` or `--duration`
+without `--follow` or `--sample`, fail `usage/conflicting-flags`.
+
+**Format.** Under `--follow`, `json`, `yaml` and `ndjson` all print one compact
+envelope per line (as `api --stream` does); `text` prints one short row per
+line. Every envelope, errors included, goes to stdout, and the exit code is
+nonzero for any failure, even after rows. Without `--follow`,
+`--format ndjson` prints the ordinary envelope as one compact line; it never
+starts a follow.
+
+Recipes (bounded, so an agent can run them unattended):
+
+```bash
+# Watch: hold /ip/address for 30s, then stop.
+centrs retrieve $R /ip/address --follow --duration 30s
+# Wait for the next change, at most 60s.
+centrs retrieve $R /ip/address --follow --count 1 --duration 60s
+```
+
+## Sample (`--sample`)
+
+`retrieve <router> <menu> --sample <interval>` reads the menu again every
+interval: one line per complete read, as NDJSON. It sees what `--follow`
+cannot (hot counters such as `rx-byte`, and state `listen` never announces),
+at the cost of missing whatever changes and changes back between two reads.
+
+**What starts it.** `--sample` with an interval that has a unit (`5s`,
+`500ms`, `1m`). A bare number fails `settings/invalid-timeout`: RouterOS reads
+`interval=5` as seconds and centrs reads `5` as milliseconds, so centrs does
+not guess. Any readable menu works, list or singleton. Each sample is the
+ordinary one-shot read (`print` or `get`) with the same validation (once, before
+the first read), projection and transport as `retrieve` without `--sample`:
+`rest-api` by default, `native-api` when pinned. Single target only; a fan-out
+selector fails `usage/fanout-not-supported`. `--follow` with `--sample` fails
+`usage/conflicting-flags`.
+
+Why reads and not RouterOS's own `print interval=`: on CHR 7.24.5 that feed has
+no end-of-tick marker, so a tick is only known complete when the next tick (or
+`!empty`) arrives, one whole interval later. A tick heavier than its interval
+also stalls the connection. A one-shot read ends with its own `!done` or HTTP
+response, so every sample is known complete when it arrives. The raw tick feed
+stays available as `api <router> <menu>/print -X POST -f interval=1s --stream`
+(`commands/api/README.md`).
+
+**What each line means.** Every line is one envelope; read
+`meta.operation.stream`:
+
+| `kind` | When | `data` | Fields |
+| ------ | ---- | ------ | ------ |
+| `sample` | One complete read | the whole read: a row array (list), the record (singleton), or the bare value (one `--attribute` on a singleton) | `index` (1-based), `at` (when the read was sent, host clock, ISO 8601), `readMs` |
+| `summary` | Last, exactly once | the counts | `stopReason`, `samples`, `durationMs` |
+
+- **A sample is the whole state at that read.** A row missing from a sample
+  was gone when it was read; there is no separate removal line. A list read
+  with no rows is `data: []`, still a complete sample.
+- **Cadence.** Reads start one interval apart, measured start to start. A
+  read slower than the interval delays the next read rather than overlapping
+  it, and missed slots are not made up in a burst. `at` gives the real spacing,
+  which is what a counter rate needs.
+- **Not an event log.** A change shorter than the interval can fall between
+  two reads. Use `--follow` for object changes RouterOS announces.
+
+**What ends it.** `--count N` stops after N samples. `--duration <d>` is a
+wall-clock bound counted from the end of validation. Ctrl-C (`SIGINT`) stops
+it. Each ends with a successful summary whose `stopReason` is `count-reached`,
+`duration-elapsed` or `interrupted`. A read still in flight when the run stops
+is not reported, because it is not complete. A failed read (a RouterOS error,
+a lost connection, a timeout) ends the run with a failed summary (`ok: false`,
+`stopReason` `routeros-error` or `transport-error`, as for `--follow`)
+carrying the partial `samples` count; there is no retry. `--list-attributes`, `--max-results` and
+`--sweep` conflict with `--sample` (`usage/conflicting-flags`).
+
+**Format.** As for `--follow`: `json`, `yaml` and `ndjson` print one compact
+envelope per line, and `text` prints `index`, `at` and the data on one line.
+Every envelope, errors included, goes to stdout; the exit code is nonzero for
+any failure, even after samples.
+
+Recipes (bounded, so an agent can run them unattended):
+
+```bash
+# Interface counters every 5s, three samples.
+centrs retrieve $R /interface --attributes name,rx-byte,tx-byte --sample 5s --count 3
+# CPU load once a second for a minute.
+centrs retrieve $R /system/resource --attribute cpu-load --sample 1s --duration 1m
+```
+
+## Wait (`--wait`, `--until`, `--until-empty`)
+
+`retrieve <router> <menu> --wait <deadline>` waits for a successful read.
+It retries refused, closed, reset and timed-out connections, and a host that is
+down or unreachable, so it can start while a router or CHR is rebooting or its
+services are coming up. Authentication,
+TLS, DNS, RouterOS and validation errors fail immediately. Identity/CDB/quickchr
+resolution errors also fail immediately; `--quickchr` must name a running VM.
+The caller's pinned protocol is preserved.
+
+```bash
+centrs retrieve --quickchr sun /system/resource --wait 5m --json
+centrs retrieve $R /routing/ospf/neighbor --wait 2m --until state=Full --json
+centrs retrieve $R /routing/ospf/neighbor --wait 2m --until-empty --json
+```
+
+- **Deadline and cadence.** `--wait` is the overall deadline, including
+  validation and retries, and needs a unit (`30s`, `5m`): a bare number is
+  refused, as for `--sample`. `--timeout` bounds each request. Reads begin one
+  second apart by default; `--sample 500ms` selects another cadence, measured
+  start to start without overlapping reads. With `--wait`, `--sample` yields
+  one final result rather than a stream. Slow reads delay the next one and
+  unfinished reads at the deadline are abandoned.
+- **Any matching row.** `--until '<expr>'` uses the same grammar, property
+  validation and device typing as `--query`. Repeat it to AND conditions.
+  `--query` selects the rows to consider; `--until` terminates the wait when
+  any of those rows match. The final `data` contains those matching rows,
+  projected as requested; predicate properties need not appear in `data`.
+  Already true succeeds on the initial read. Zero rows never satisfy it.
+  Predicates apply to list menus; singleton readiness uses `--wait` alone.
+- **Zero rows.** `--until-empty` succeeds only on a completed read with zero
+  rows selected by `--query`. Silence, transport failure, unfinished reads and
+  cancellation do not prove emptiness. It needs a list menu and conflicts
+  with `--until`.
+- **One result.** After request resolution, every terminal envelope carries `meta.operation.wait` with
+  `stopReason`, `elapsedMs`, `attempts` and `observations`. Attempts include
+  failed validation/connection attempts; observations count completed data
+  reads. `ready` and `condition-met` are successful. On deadline expiry,
+  `deadline-elapsed` is failed: it preserves the last transient transport error,
+  or reports `wait/deadline-exceeded` when the last completed observation did
+  not satisfy the condition. Ctrl-C or an aborted library signal fails with
+  `wait/interrupted`, `stopReason: "interrupted"`. Other errors have
+  `stopReason: "failed"`. CLI exit is 0 on success and 1 on any failure;
+  the one final envelope, including failures, is written to stdout.
+
+Single target only. `--until` and `--until-empty` require `--wait`.
+`--follow`, `--count`, `--duration` and `--list-attributes` conflict with it.
+This waiter uses completed reads over REST or native-api, including counters
+and protocol state that listen never reports. For open-ended change feeds use
+`--follow`; `--duration` finishing a watch does not establish a wait condition.
+The library entry is `retrieveWait(request, env?, { signal? })`, returning a
+success or failure envelope; `retrieve()` remains the one-shot entry.
+
 ## Target selection
 
 retrieve fans out over the **shared target-selection grammar** — multiple
@@ -155,7 +496,10 @@ stays the single-target envelope; a selector flag or a second positional target
 switches to fan-out. `--quickchr <name>` (the named-live-provider) behaves like a
 lone positional: one flag stays single-target, repeating it fans out, and mixing
 it with positionals or CDB selectors is `usage/conflicting-flags` (constitution:
-resolution providers).
+resolution providers). The `--host` **flag** with no `<router>` positional
+(`retrieve --host http://192.0.2.10:8080 /system/resource`) is a single target
+too: no CDB lookup, so auth comes from flags, env or config. `CENTRS_HOST` never
+stands in for `<router>`.
 
 retrieve-specific behavior on top of the shared engine: