bmweb-cli 0.1.4 → 0.1.7

package/README.md

@@ -1,11 +1,12 @@
 # bmweb-cli
 
 [BMWeb](https://bmweb.danner.ink/)'s tools as a command line. Read what an
-INPA `.IPO` script does, compile an `.IPS` / `.SRC` source, search every
-module the site ships for the key that does a thing, decode or compare the
-report links the app's Garage shares, and, with a K+DCAN cable, run jobs,
-read every fault memory of a car, and drive a module's INPA screens from
-the terminal.
+INPA `.IPO` script does, compile an `.IPS` / `.SRC` source into a real
+`.IPO`, search every module the site ships for the key that does a thing,
+decode or compare the report links the app's Garage shares, list what a
+module's SGBD declares (its jobs and their arguments, results and lookup
+tables), and, with a K+DCAN cable, run jobs, read every fault memory of a
+car, and drive a module's INPA screens from the terminal.
 
 The commands run the app's own code: the `.IPO` reader and the source
 compiler, the job search, the Garage report codec, the transport (framing,
@@ -28,9 +29,10 @@ Only the project's own code. Nothing BMW-derived ships with it: no scripts,
 no SGBDs, no fault database, no parts data. Everything the tool needs that
 it does not carry is fetched from the site (or `--api <url>`) and kept
 under `$XDG_CACHE_HOME/bmweb-cli/` (default `~/.cache/bmweb-cli/`) for a
-day: the job search index (about 2.5 MB), and, for the commands on the
-cable, the chassis archives the app itself loads (`api/chassis/E46.chassis`,
-about 20 MB for an E46), the group bytecode and the shared tables.
+day: the job search index (about 2.5 MB), and, for the `sgbd` commands,
+`job --info` and the commands on the cable, the chassis archives the app
+itself loads (`api/chassis/E46.chassis`, about 20 MB for an E46), the
+group bytecode and the shared tables.
 `--refresh` fetches again; when the site cannot be reached the cached copy
 is used and a warning says so.
 
@@ -97,7 +99,7 @@ m_main  F10        Back                         DIAGNOSE_ENDE, (exit)
 m_main  Shift+F10  Print                        (printscreen)
 ```
 
-### `bmweb ipo compile <file.IPS> [-I dir]... [-o out]`
+### `bmweb ipo compile <file.IPS> [-I dir]... [-o out] [--exec]`
 
 Compile an INPA source. Includes are looked up beside the script and in
 each `-I` directory, by file name, case-insensitively, the way INPA's own
@@ -106,18 +108,22 @@ tooling finds them; a missing one is named rather than half-compiled.
 ```
 $ bmweb ipo compile MY_SCRIPT.IPS -I ~/INPA/SGDAT
 MY_SCRIPT.IPS: compiled 8 procedures (2 menus, 3 screens, 3 functions, 0 state machines), includes INPA.H
-wrote /home/me/MY_SCRIPT.ipoexec.json (the app's exec form; not INPA's binary .IPO)
+wrote /home/me/MY_SCRIPT.IPO (44902 bytes, INPA's .IPO container)
 
 $ bmweb ipo compile MY_SCRIPT.IPS
 bmweb: MY_SCRIPT.IPS: missing include INPA.H (searched /home/me; pass -I <dir> with the INPA headers)
 ```
 
-What it writes is the form the app runs: the script's procedures as the
-token stream the runtime executes (`{procs, byid}`, the same shape the
-Script runner builds from a dropped file). The app's compiler emits that,
-not INPA's binary container, so the output is not a `.IPO` you could hand
-to INPA itself. The `.IPO` byte writer lives in the repository's Python
-tooling and is not part of this package.
+What it writes is a real `.IPO`: INPA's own binary container, the same
+block layout the shipped scripts use. The writer is the app's own, and it
+is held to reproduction -- every script in a full INPA install decodes and
+re-encodes to its own bytes exactly, so the bytes it writes for a source
+are the bytes that format calls for.
+
+`--exec` writes the app's exec form instead (`{procs, byid}` as JSON, the
+token stream the runtime executes and the same shape the Script runner
+builds from a dropped file), which is what this command wrote before it
+could write the container.
 
 ### `bmweb search <query...> [--chassis E46] [--limit N] [--json]`
 
@@ -197,6 +203,88 @@ Answering changed:
 + new   - cleared   = still present   ~ ident field changed
 ```
 
+### `bmweb sgbd jobs <sgbd> [--api url] [--refresh] [--json]`
+
+Every job an SGBD declares, with the arguments it takes, the results it
+returns, a comment, and whether the app's write gate would ask about it
+before sending it. Nothing is sent and no port is opened: this is the
+module's shipped declaration, the same one the app's Tool32 lists a job
+from. A diagnostic group (`d_motor`, `d_0012`) is read from its own group
+file rather than resolved on the wire, so it lists the job names and the
+tables the group carries.
+
+```
+$ bmweb sgbd jobs lws5
+NAME                       ARGS                                       RESULTS                                                                 COMMENT                              WRITE
+-------------------------  -----------------------------------------  ----------------------------------------------------------------------  -----------------------------------  -----
+ABGLEICH_LESEN                                                        JOB_STATUS, ABGL_LRW_OFFSET, ABGL_LWS_ID, ABGL_FGSTNR, _TEL_ANTWORT     OKAY, wenn fehlerfrei
+ABGLEICH_SCHREIBEN                                                    JOB_STATUS, _TEL_ANTWORT                                                 OKAY, wenn fehlerfrei                yes
+ABGLEICH_VORGEBEN          ABGL_LRW_OFFSET, ABGL_LWS_ID, ABGL_FGSTNR  JOB_STATUS, _TEL_AN_SG, _TEL_ANTWORT                                     OKAY, wenn fehlerfrei                yes
+CODIERUNG_LESEN            BLOCK                                      JOB_STATUS, COD_DATEN, _TEL_AN_SG, _TEL_ANTWORT                          OKAY, wenn fehlerfrei
+FS_LESEN                                                              JOB_STATUS, F_HEX_CODE, F_ORT_NR, F_ORT_TEXT, F_HFK, F_ART_ANZ, +4 more  OKAY, wenn fehlerfrei
+FS_LOESCHEN                                                           JOB_STATUS, _TEL_ANTWORT                                                 OKAY, wenn fehlerfrei                yes
+...
+
+24 jobs, 9 the app would ask about before sending
+```
+
+The WRITE column is the app's classifier (`isWriteJob`), the same verdict
+`bmweb job` gates on: a read token in the name wins, a write token makes a
+write, and an unknown name is a write. A cell that lists only the first few
+names says how many more there are; `--json` and `bmweb job <sgbd> <JOB>
+--info` carry every one.
+
+The site's export writes a job's results as `NAME : comment` and keeps no
+comment above them, so COMMENT is the `JOB_COMMENT` result where the SGBD's
+author wrote one and the first declared result's comment otherwise.
+
+### `bmweb sgbd tables <sgbd> [--api url] [--refresh] [--json]`
+
+The lookup tables an SGBD carries for its own bytecode -- the ones its jobs
+read with `tabset` to turn a byte into a word -- with the rows and columns
+of each.
+
+```
+$ bmweb sgbd tables ms450ds0
+NAME                      ROWS  COLS  COLUMNS
+------------------------  ----  ----  ---------------------------------------------------
+AUTHENTISIERUNG           4     2     AUTH_NR, AUTH_TEXT
+BAUDRATE                  7     3     NR, BAUD, BAUD_TEXT
+BETRIEBSSTUNDENSTATUS     4     2     WERT, UWTEXT
+BITS                      72    4     NAME, BYTE, MASK, VALUE
+CBSKENNUNG                16    3     NR, CBS_K, CBS_K_TEXT
+...
+
+53 tables (bmweb sgbd table ms450ds0 <NAME> prints one)
+```
+
+These are the module's own tables. The four shared table files (`t_pcod`,
+`t_scod`, `t_ausb`, `t_grtb`, which an SGBD reads with `tabsetex`) belong
+to no one module -- every job VM is handed all of them -- so they are not
+listed here, and a module is never said to carry them.
+
+### `bmweb sgbd table <sgbd> <NAME> [--api url] [--refresh] [--json]`
+
+One table's rows.
+
+```
+$ bmweb sgbd table ms450ds0 BAUDRATE
+NR    BAUD      BAUD_TEXT
+----  --------  --------------------
+0x01  PC9600    Baudrate 9.6 kBaud
+0x02  PC19200   Baudrate 19.2 kBaud
+0x03  PC38400   Baudrate 38.4 kBaud
+0x04  PC57600   Baudrate 57.6 kBaud
+0x05  PC115200  Baudrate 115.2 kBaud
+0x06  SB        Specific Baudrate
+0xXY  --        unbekannte Baudrate
+
+7 rows in BAUDRATE
+```
+
+The name is matched without regard to case, and a near miss is named:
+`bmweb sgbd table lws5 BITS` answers `lws5 carries no table BITS`.
+
 ## Commands on the cable
 
 These need a K+DCAN cable (an FTDI cable, or a clone) on the car's OBD
@@ -218,11 +306,18 @@ own bytecode branches on. An FTDI cable wants a 1 ms latency timer (set for
 you on Linux; on macOS and Windows a driver setting), and an echo failure
 (IFH-0003) says so.
 
-**Tested against the app's fake car, pending a real K+DCAN run.** The
-transport is exercised end to end against a fake cable (the DS2 and
-BMW-FAST framing, the reopen, the DTR sequence, the echo), and `job`,
-`scan` and `tui` against the fake car the app's own runtime tests use. No
-real cable has been on this code yet.
+**Run on a real car.** Since 0.1.4, `job`, `scan` and `tui` have been run
+on an E46 over an FTDI K+DCAN cable on macOS. What that run fixed is in
+`src/serial.ts`: on macOS a Node serial port never wakes on bytes an FTDI
+cable sends back, so the reader polls the port's modem lines while it is
+reading and picks the bytes up itself. The offline commands need no cable
+and never open one.
+
+Everything is still tested offline as well: the transport end to end
+against a fake cable (the DS2 and BMW-FAST framing, the reopen, the DTR
+sequence, the echo), and `job`, `scan` and `tui` against the fake car the
+app's own runtime tests use, both directly and through a gateway on a real
+socket. The gateway itself has not been on a car yet.
 
 ### `bmweb ports [--json]`
 
@@ -235,7 +330,116 @@ $ bmweb ports
 /dev/cu.usbserial-AB0JQ9XY  FTDI  0403:6001  sn AB0JQ9XY
 ```
 
-### `bmweb job <sgbd> <JOB> [arg] [--port p] [--yes] [--json]`
+### Gateway
+
+One machine owns the cable; another drives it. `bmweb gateway` opens the
+local port exactly as `job` does and serves it over a WebSocket, and
+`--gateway` on `job`, `scan` and `tui` (and the app in a browser) hands
+that socket to the same transport, which cannot tell it from a cable in
+its own USB port.
+
+Nothing about the wire moves. The framing, the checksums, the reopen for a
+concept change, DTR as the K-line transmit enable, the echo dropped by
+count, the timeouts measured to the first byte and every write
+confirmation all run on the machine you typed the command on, because
+that is where the bus is. The machine with the cable moves bytes and
+modem-line states, and nothing else.
+
+**It is a pipe, not a gate.** The gateway never sees a job name, only
+bytes, so it cannot tell a read from a write and does not try. Anyone who
+can reach the listening port can drive the car. It binds `127.0.0.1` by
+default, which is this machine only; `--listen 0.0.0.0:6801` opens it to
+whoever can route to the machine, and says so when it starts.
+
+On the machine with the cable:
+
+```
+$ bmweb gateway --listen 0.0.0.0:6801
+gateway: /dev/cu.usbserial-AB0JQ9XY served at ws://0.0.0.0:6801
+  this is a byte pipe with no gate of its own: anyone who can reach this port can drive the car.
+  listening beyond this machine; the port is open to whoever can route to it
+  one client at a time; the cable is closed when the client goes.
+  client 192.168.1.24:52233 connected
+  client 192.168.1.24:52233 disconnected, cable closed
+^C
+gateway: stopping, the cable is closed
+```
+
+On the machine driving it:
+
+```
+$ bmweb job ms450ds0 STATUS_LESEN --gateway 192.168.1.9:6801
+ms450ds0 MS450DS0 STATUS_LESEN: 1 set
+
+set 1
+  STAT_MOTORDREHZAHL_WERT  812.5
+  STAT_MOTORDREHZAHL_EINH  1/min
+  JOB_STATUS               OKAY
+
+$ bmweb job ms450ds0 FS_LOESCHEN --gateway 192.168.1.9:6801
+FS_LOESCHEN on ms450ds0 is a write (it changes the module or drives something). Send it? [y/N] n
+FS_LOESCHEN on ms450ds0: not sent (a write needs --yes or a y answer)
+```
+
+The question is asked here, on the driving machine, and a `n` means the
+bytes never leave it. `--gateway` takes `host:port`, a bare host (port
+6801 is assumed), or a full `ws://` / `wss://` URL. One client at a time:
+a second connection is refused with a message, and the cable is closed
+when the client goes, so the host machine can use its own port in
+between.
+
+#### From a browser
+
+The app drives a gateway too. Set the address in Settings, under "Cable on
+another machine", or open the page with `?gateway=ws://192.168.1.9:6801`,
+which stores it. The topbar's cable chip then reads
+`cable: gateway 192.168.1.9:6801 (/dev/cu.usbserial-AB0JQ9XY)`, so the car
+never looks local when it is not.
+
+Where this works is a browser rule, not ours: **a page served over http
+(localhost, an offline export opened from a file, an offline copy served
+on your own network) may open a plain `ws://` gateway. The hosted https
+site may not** -- browsers block `ws://` from an https page as mixed
+content, with no override. From https, only a `wss://` gateway works,
+which means putting a TLS front end with a certificate the browser trusts
+in front of it.
+
+#### The protocol
+
+JSON text frames for control, binary frames for the bytes the wire
+carries, so a read crosses as one frame with no base64 and no copy.
+
+| direction      | frame  | meaning                                                                                     |
+| -------------- | ------ | ------------------------------------------------------------------------------------------- |
+| client to host | text   | `{"id":1,"op":"open","config":{"baudRate":9600,"dataBits":8,"stopBits":1,"parity":"even"}}` |
+| client to host | text   | `{"id":2,"op":"close"}`                                                                     |
+| client to host | text   | `{"id":3,"op":"setSignals","signals":{"dataTerminalReady":true}}`                           |
+| client to host | text   | `{"id":4,"op":"getSignals"}`                                                                |
+| client to host | binary | the bytes to write, exactly as given                                                        |
+| host to client | text   | `{"id":1,"ok":true}`, or `{"id":4,"ok":true,"signals":{...}}`                               |
+| host to client | text   | `{"id":2,"ok":false,"error":"..."}`                                                         |
+| host to client | text   | `{"event":"hello","port":"/dev/cu.usbserial-AB0JQ9XY","gateway":"bmweb"}`                   |
+| host to client | text   | `{"event":"writeFailed","error":"..."}`                                                     |
+| host to client | binary | bytes as they arrive off the wire, streamed                                                 |
+
+A write needs no reply: the wire has no acknowledgement to give, and the
+transport never waited for one, so a round trip here would only lengthen
+the DTR hold and lose the answer. A write that FAILS does travel, as an
+unsolicited `writeFailed`, because a client that thinks a request went out
+when it never left the cable reports a phantom fault against a healthy
+module. Reads are streamed as the bytes arrive rather than gathered into
+an answer, so a timeout still means time-to-first-byte on the driving
+machine. `setSignals` keeps the lines it does not name, and keeps them
+across the reopen a concept change makes, exactly as a local port does.
+Every failure travels back as the same message text a local port would
+have raised, so the app throws it verbatim and cannot tell the difference.
+
+The WebSocket is RFC 6455 with no compression and no extensions, written
+in the package itself (`src/ws.ts`) rather than pulled in, because the
+package promises zero runtime dependencies and Node ships a WebSocket
+client only from version 22 and no server at all.
+
+### `bmweb job <sgbd> <JOB> [arg] [--results a,b] [--info] [--port p] [--gateway h:p] [--yes] [--json]`
 
 One raw job on one module, like the app's Tool32: the SGBD's own bytecode
 runs in the job VM over the cable, inside its EDIABAS session
@@ -263,7 +467,51 @@ name wins, a write token makes a write, and an unknown name is a write.
 A write goes out only with `--yes` or a `y` on the terminal; without a
 terminal the answer is no. A write is never sent silently.
 
-### `bmweb scan <chassis> [--port p] [--share] [--json]`
+`--results` names the results to print, comma-separated and without regard
+to case. The job still runs whole -- the flag narrows what is shown, not
+what is asked of the module -- and `--json` is filtered the same way, so
+the table and the JSON never disagree about what was read. A name that
+matched nothing is a warning on stderr and the exit code stays 0.
+
+```
+$ bmweb job ms450ds0 STATUS_LESEN --results stat_motordrehzahl_wert,JOB_STATUS
+ms450ds0 MS450DS0 STATUS_LESEN: 1 set
+
+set 1
+  STAT_MOTORDREHZAHL_WERT  812.5
+  JOB_STATUS               OKAY
+```
+
+`--info` prints what the SGBD declares about the job -- its arguments, its
+results and their comments -- instead of running it. It opens no port and
+sends nothing, so it works with the cable unplugged and with the car
+elsewhere; a job argument beside it is refused rather than quietly ignored.
+
+```
+$ bmweb job ms450ds0 AIF_LESEN --info
+ms450ds0 AIF_LESEN
+
+arguments (1)
+  AIF_NUMMER  int  ==0 : aktuelles AIF > 0 : Nummer des zu lesenden AIF default = 0 : aktuelles AIF
+
+results (21)
+  AIF_ADRESSE_HIGH  AIF Adresse des AIF, High-Word
+  AIF_ADRESSE_LOW   AIF Adresse des AIF, Low-Word
+  AIF_FG_NR         Fahrgestellnummer 7-stellig
+  AIF_FG_NR_LANG    Fahrgestellnummer 17-stellig falls vorhanden, sonst 7-stellig
+  AIF_DATUM         Datum der SG-Programmierung in der Form TT.MM.JJJJ
+  ...
+  JOB_STATUS        OKAY, wenn fehlerfrei
+```
+
+A job that writes is headed `[WRITE]` here too, and `bmweb sgbd jobs`
+below lists every job of a module at once.
+
+`--gateway <host:port>` runs the same job on a cable another machine is
+serving (see [Gateway](#gateway) below). The write gate does not move: it
+is asked here, on the machine you typed the command on.
+
+### `bmweb scan <chassis> [--port p] [--gateway h:p] [--share] [--json]`
 
 INPA's own whole-vehicle script (E46 E53 E65 E83 E85 E87 E89 E90 R50 R56):
 the script is opened, its fault-memory menu's read key is pressed, and what
@@ -295,7 +543,9 @@ Share: https://bmweb.danner.ink/#report/...
 The script's progress window (which module it is asking) goes to stderr;
 a key that would write is declined, a prompt is cancelled: the scan reads.
 
-### `bmweb tui [<chassis> <sgbd>] [--port p] [--menu m_x]`
+`--gateway <host:port>` scans a car on a cable another machine is serving.
+
+### `bmweb tui [<chassis> <sgbd>] [--port p] [--gateway h:p] [--menu m_x]`
 
 INPA's screens in the terminal. With a chassis and a module (SGBD or INPA
 code, or the chassis itself for its whole-vehicle script), that module's
@@ -315,7 +565,10 @@ cancels. The home starts with or without a cable.
 
 The TUI runs on the terminal's alternate screen (the buffer vim and htop
 use), so the shell's scrollback is never touched and quitting restores it;
-a screen redraws in place, only the lines that changed.
+a screen redraws in place, only the lines that changed. A viewer longer
+than the terminal (a fault protocol, a report) scrolls: Up/Down a line,
+PgUp/PgDn a page, Home/End to either end, with a line under it saying
+which rows are shown.
 
 Every dialog INPA opens is a prompt: a message waits for Enter, an input
 asks for the number (or hex, or text) within the declared range, the
@@ -332,6 +585,9 @@ performs.
 The module data the script needs (its `.IPO`, the SGBD bytecode, the
 tables) comes from the site's chassis archive, cached as described above.
 
+`--gateway <host:port>` drives a cable another machine is serving. Every
+confirmation still happens here, on your keyboard.
+
 ### Every command
 
 `--json` prints machine-readable output instead of a table. Errors are one
@@ -347,8 +603,15 @@ to `dist/bmweb.js` with esbuild and type-checks with `tsc`; `npm test` runs
 the `node:test` suites the build produced, every one of them offline: the
 serial tests drive the app's bus over a fake cable, the job, scan and tui
 tests drive the app's runtime against a fake car and a scripted terminal,
-with a module script written for the tests in INPA's language. The
-repository's `tools/check.sh` runs all of it.
+with a module script written for the tests in INPA's language, and the
+gateway tests run `job`, `scan` and `tui` a second time through a real
+gateway on a port the OS picks. The repository's `tools/check.sh` runs all
+of it.
+
+Set `BMWEB_VERBOSE=1` to see what the app's runtime logs: the wire trace
+the bus dumps after an error, each variant probe's verdict, the cable
+events. It goes to stderr; without it the runtime is silent and a command's
+output is only its own.
 
 ## License