@casadi/casadi-reader 0.0.0-poc → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (6) hide show
  1. package/CHANGELOG.md +14 -0
  2. package/README.md +121 -234
  3. package/bin/casadi-reader.js +12 -6
  4. package/dist/index.js +10037 -624
  5. package/package.json +4 -5
  6. package/dist/operations.js +0 -352
package/CHANGELOG.md ADDED
@@ -0,0 +1,14 @@
1
+ # 0.1.0
2
+
3
+ - Replace MX graph interpretation with a schema-driven structural reader.
4
+ - Read typed fields, inline structures, maps, shared references and byte ranges.
5
+ - Validate ordinary and debug serialization with the same generated layouts.
6
+ - Add SX, nested SX/MX calls, ONNX-backed function and additional MX fixtures.
7
+ - Defer byte payload decoding and File/Blob payload I/O.
8
+ - Move operation labels, mapping reconstruction and MX graph normalization to casadi-viz.
9
+
10
+ Breaking changes from 0.0.0-poc: `decodeCasadi` now returns
11
+ `casadi_serialization`, not `casadi_json`. Prefer `decode` and `open`.
12
+ The `/operations` export is removed. Raw Resource streams use
13
+ `decode(text, {type: 'Resource'})` or `open(file, {type: 'Resource'})`.
14
+ Other language prototypes remain deferred.
package/README.md CHANGED
@@ -1,273 +1,160 @@
1
- # casadi-reader
2
-
3
- Inspect `.casadi` files without installing or loading CasADi. One repository
4
- provides JavaScript/npm, Python/PyPI, C++, C, MATLAB and Julia interfaces.
5
- The current development focus is the **npm package**. The other language
6
- interfaces are experimental and deferred. This is not yet a published npm release.
7
- Files are inspected, never evaluated; no archives are extracted.
8
-
9
- | Interface | Implementation | Runtime dependencies |
10
- | --- | --- | --- |
11
- | `@casadi/casadi-reader` (npm) | Standalone JavaScript | None; Node 22+ or browser ESM |
12
- | `casadi-reader` (PyPI) | Standalone Python | None; Python 3.9+ |
13
- | C++ | Native reader, RAII `Document` | C++ standard library |
14
- | C | C ABI over the native reader | Same native library |
15
- | MATLAB | MATLAB package and MEX | Native reader compiled into the MEX |
16
- | Julia `CasadiReader` | Julia package calling the C ABI | Standalone native library, JSON.jl |
17
-
18
- All interfaces use the same object-table JSON contract and fixtures. The three
19
- reader implementations use source assets generated from the vendored scheme.
20
- MATLAB and Julia do not launch Python, Node or a subprocess to decode files.
21
-
22
- ## Python
23
-
24
- Install locally with `pip install .` (the distribution declares no runtime
25
- requirements):
26
-
27
- ```python
28
- from casadi_reader import read_casadi, read_resource, to_json
29
-
30
- graph = read_casadi('model.casadi')
31
- print(to_json(graph, indent=2))
32
-
33
- resource = read_resource('resource.casadi', lazy=True)
34
- blob = resource['resource']['blob']
35
- try:
36
- first_bytes = blob.read(0, 64)
37
- finally:
38
- blob.close()
39
- ```
1
+ # @casadi/casadi-reader
2
+
3
+ Read CasADi serialization into typed fields, values, containers and shared
4
+ references, without installing CasADi or loading its plugins.
40
5
 
41
- File input uses a read-only memory map. Lazy payloads keep that mapping alive;
42
- metadata and requested slices are decoded, without allocating the whole archive.
43
- `loads(text)` and `loads_resource(text, lazy=True)` also accept encoded strings.
44
- The CLI is `casadi-reader [--resource] [--lazy] input.casadi [output.json]`, or
45
- `python -m casadi_reader ...`.
6
+ The reader does not interpret SX/MX instructions, name mathematical operations,
7
+ reconstruct entry mappings, or build visualization graphs. Those tasks belong
8
+ in [casadi-viz](https://github.com/casadi/casadi-viz). An ONNX-backed function is
9
+ read as serialized configuration and model bytes; the reader never runs ONNX.
46
10
 
47
- ## JavaScript
11
+ The current development and release focus is npm. Other language prototypes in
12
+ this repository are deferred and still use the earlier MX-specific API.
48
13
 
49
- Install locally with `npm install /path/to/casadi-reader`:
14
+ ## API
50
15
 
51
16
  ```js
52
- import {readCasadi, decodeResource, openResource} from '@casadi/casadi-reader';
53
- const graph = readCasadi(await file.text());
54
- const {resource} = decodeResource(resourceStreamText, {lazy: true});
55
- const firstBytes = resource.blob.read(0, 64); // Uint8Array
56
-
57
- // File/Blob input also defers reading the encoded payload:
58
- const opened = await openResource(resourceFile, {lazy: true});
59
- const bytes = await opened.resource.blob.read(0, 64);
17
+ import {decode, open} from '@casadi/casadi-reader';
18
+
19
+ const document = decode(serializedText);
20
+ const fromFile = await open(file, {lazy: true}); // browser File/Blob
21
+ const root = document.objects[document.root];
22
+ console.log(root.type, root.fields);
60
23
  ```
61
24
 
62
- The npm tarball contains bundled ESM in `dist/`, with serialization metadata
63
- compiled into the JavaScript. It does not fetch or import the scheme JSON.
64
- After publication, a browser can import the versioned
65
- `https://unpkg.com/@casadi/casadi-reader@VERSION/dist/index.js` directly from a
66
- `<script type="module">`; no import map or bundler is required. `VERSION` is a
67
- placeholder: this prototype has not been published.
25
+ `decode()` accepts encoded `.casadi` text. `open()` accepts a File/Blob.
26
+ `decodeCasadi` remains an alias for `decode`; both return structural documents.
68
27
 
69
- The Node CLI is `node bin/casadi-reader.js [--resource] [--lazy] input.casadi`.
70
- Text mode retains the encoded string. File/Blob mode reads a bounded metadata
71
- prefix, then requested slices. File/Blob Resource input must have no surrounding
72
- whitespace. An application installing both CLIs should use the explicit module
73
- or Node entry point to avoid the common `casadi-reader` executable name.
28
+ The output is ordinary JavaScript data:
74
29
 
75
- ## C++ and C
30
+ ```js
31
+ {
32
+ format: 'casadi_serialization',
33
+ version: 1,
34
+ serializationProtocol: 3,
35
+ root: 8,
36
+ roots: [{$ref: 8}],
37
+ objects: [
38
+ // ... shared objects ...
39
+ {
40
+ type: 'Function',
41
+ fields: [
42
+ {name: 'Function::null', type: 'bool', value: false, offset: 19, byteLength: 1},
43
+ // Serialized field order and duplicate field names are preserved.
44
+ ],
45
+ layouts: ['MXFunction::serialize_body', /* base layouts ... */],
46
+ offset: 18,
47
+ byteLength: 1234
48
+ }
49
+ ]
50
+ }
51
+ ```
76
52
 
77
- Build with CMake and a C++11 compiler; no third-party library is needed:
53
+ The example's indices and byte counts are illustrative. `$ref` values are
54
+ zero-based indices into `objects`. Shared definitions appear once, even when
55
+ referenced by several functions. Inline structures have their own `type` and
56
+ `fields`. Vectors are arrays, pairs are two-element arrays, and maps are
57
+ `{$map: [[key, value], ...]}` so arbitrary key types and field order survive.
58
+ Repeated serializer fields remain repeated entries, not overwritten properties.
59
+ 64-bit integers outside JavaScript's exact range use `{$integer: "..."}`;
60
+ nonfinite floating-point values use `{$float: "..."}`. Byte offsets count decoded
61
+ wire bytes from the stream start, not characters in its a–p encoding.
78
62
 
79
- ```sh
80
- cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
81
- cmake --build build
82
- ctest --test-dir build --output-on-failure
83
- cmake --install build --prefix /your/prefix
84
- ```
63
+ A file can contain several roots; `roots` retains them in order. `root` is a
64
+ convenience index for a single shared-object root, otherwise null.
85
65
 
86
- Installed CMake consumers can use `find_package(casadi_reader CONFIG REQUIRED)`
87
- and link `casadi_reader::casadi_reader`. Headers are
88
- `<casadi_reader/reader.hpp>` and `<casadi_reader/reader.h>`.
66
+ ## Lazy bytes
89
67
 
90
- ```cpp
91
- casadi_reader::Document graph("model.casadi");
92
- std::string json = graph.json();
93
- casadi_reader::Document resource("resource.casadi", true, true);
94
- auto bytes = resource.read_blob(0, 0, 64);
68
+ ```js
69
+ const document = await open(resourceFile, {type: 'Resource', lazy: true});
70
+ const root = document.objects[document.root];
71
+ const blob = root.fields.find(f => f.name === 'ZipMemResource::blob').value;
72
+ const firstBytes = await blob.read(0, 64);
95
73
  ```
96
74
 
97
- ```c
98
- char error[1024];
99
- cr_document *doc = cr_open("model.casadi", 0, 0, error, sizeof(error));
100
- if (doc) {
101
- puts(cr_json(doc)); /* pointer valid until cr_close */
102
- cr_close(doc);
103
- } else {
104
- fprintf(stderr, "%s\n", error);
105
- }
106
- ```
75
+ `type` selects a raw `SerializingStream` root instead of FileSerializer framing.
76
+ Resource streams use the same archive representation embedded inside FMUs;
77
+ complete saved FMU functions are not yet a validated layout family.
107
78
 
108
- The C ABI catches exceptions and reports errors through caller-owned buffers.
109
- Handles own their source streams; lazy blob slices seek directly to encoded
110
- file ranges. C/C++ blob indices and byte offsets are zero-based. Do not share
111
- one handle between concurrent operations. Native eager archive-to-JSON expansion
112
- is limited to 1,000,000 bytes; use lazy mode for larger archives.
79
+ Lazy byte handles expose `offset`, `byteLength` and `read(offset, length)`.
80
+ Text-input reads are synchronous; File/Blob reads are asynchronous. JSON output
81
+ contains a small descriptor and requires the original source to retrieve bytes.
113
82
 
114
- ## MATLAB
83
+ Opaque streams are deferred in lazy mode. Large serialized strings are also
84
+ deferred (default `lazyThreshold: 65536` bytes). Smaller strings are decoded as
85
+ UTF-8 where possible; other strings retain their bytes as `{$bytes: [...]}` in
86
+ eager mode or a lazy byte handle in lazy mode. No archive extraction occurs.
115
87
 
116
- ```sh
117
- cmake -S . -B build -DCASADI_READER_MATLAB=ON
118
- cmake --build build
119
- ```
88
+ Lazy File/Blob opening loads metadata pages and skips payload pages by their
89
+ declared lengths. It requires unpadded encoded files, as emitted by CasADi.
90
+ Text input accepts surrounding whitespace but necessarily retains the supplied
91
+ encoded string. Graph fields and containers are currently eager; lazy mode
92
+ specifically concerns opaque byte payloads.
120
93
 
121
- ```matlab
122
- addpath('matlab');
123
- addpath('build/matlab');
124
- graph = casadi_reader.read('model.casadi');
125
- document = casadi_reader.Document('resource.casadi', true, true);
126
- cleanup = onCleanup(@() delete(document));
127
- metadata = document.data();
128
- bytes = document.readBlob(1, 0, 64);
94
+ ## Browser and CLI
95
+
96
+ Published packages contain bundled browser ESM with the layout data embedded.
97
+ They do not fetch or parse the source scheme JSON at runtime. A browser can
98
+ import a pinned `https://unpkg.com/@casadi/casadi-reader@VERSION/dist/index.js`
99
+ from a module script without a bundler or an import map.
100
+
101
+ ```sh
102
+ npx @casadi/casadi-reader model.casadi model.json
103
+ # Or, for a raw Resource stream:
104
+ npx @casadi/casadi-reader --type Resource --lazy resource.casadi
129
105
  ```
130
106
 
131
- `Document.json()` returns JSON text; `data()` uses MATLAB's `jsondecode`.
132
- The package ZIP can include the built MEX. MEX binaries are platform-specific;
133
- this prototype has been tested with MATLAB R2024b on Linux.
107
+ There are no runtime dependencies. Building the npm package needs Node 22+,
108
+ Python 3.9+ and the development dependencies in package-lock.json.
134
109
 
135
- ## Julia
110
+ ## Scheme and coverage
136
111
 
137
- From this checkout, build the library and instantiate the Julia environment:
112
+ CasADi's `misc/generate_serialization_scheme.py` produces the vendored
113
+ `schemes/serialization_scheme.json`, including lowered reader layouts.
114
+ `npm run generate` compiles that data into reader assets. The JavaScript engine
115
+ executes field, base-layout, repetition, condition and discriminator instructions;
116
+ it contains no SX/MX-specific decoding methods.
138
117
 
139
- ```sh
140
- julia --project=julia -e 'using Pkg; Pkg.instantiate(); include("julia/deps/build.jl")'
141
- ```
118
+ The extractor includes inline serializers, inheritance and tensor metadata
119
+ helpers. It derives operation dispatch families from CasADi's native dispatcher
120
+ and plugin registrations from the source. No mathematical evaluation occurs.
142
121
 
143
- ```julia
144
- using CasadiReader
145
- graph = read_casadi("model.casadi")
146
- document = read_resource("resource.casadi"; lazy=true)
147
- try
148
- metadata = data(document)
149
- bytes = read_blob(document, 1, 0, 64)
150
- finally
151
- close(document)
152
- end
153
- ```
122
+ Coverage is still experimental. Native plain/debug fixture pairs validate MX,
123
+ SX, nested calls, mappings, constants, ONNX-backed functions and Resource streams.
124
+ The debug files independently check serialized field names and primitive tags;
125
+ ordinary undecorated files use the same layouts. Tests also exercise a new
126
+ function discriminator supplied entirely as layout data.
154
127
 
155
- The source archive is a self-contained Julia project, including the native
156
- sources in `deps/reader`. `Pkg.build("CasadiReader")` needs CMake and a C++
157
- compiler. Alternatively set `CASADI_READER_LIBRARY` to an existing standalone
158
- reader library before starting Julia. Tested with Julia 1.6.2 on Linux.
159
-
160
- MATLAB and Julia blob indices are one-based; byte offsets remain zero-based.
161
- **Object references inside the JSON contract remain zero-based in every language.**
162
- For example, a Julia consumer accesses the root as `graph["objects"][graph["root"]+1]`.
163
-
164
- ## JSON contract
165
-
166
- Graph documents have `format: "casadi_json"`, `version: 1`,
167
- `serializationProtocol: 3`, a `root` object index, and an `objects` array.
168
- References are indices; shared nodes are stored once; null references stay null.
169
-
170
- - `kind: "function"`: name, type, input/output sparsities and names, input nodes,
171
- and instructions containing node references and argument/result work slots.
172
- - `kind: "mx"`: operation ID/name, dependency and sparsity references, constants
173
- and node-specific `info`. Indexing nodes also have a normalized `mapping`.
174
- Constants are numeric strings, including nonfinite values; decimal spelling
175
- may differ between implementations.
176
- - `kind: "sparsity"`: shape and compressed-column `colind` / `row` arrays.
177
-
178
- `info` is reconstructed from serialized members and tested against native
179
- `MX.info()`. It does not prescribe viewer layout or styling. The representation
180
- is not lossless and cannot be written back to `.casadi` by this prototype.
181
-
182
- Lazy Resource payloads serialize as small descriptors with `offset`, `byteLength`
183
- and `encoding`. Retrieving bytes requires the live document/blob and its original
184
- source; descriptors alone are not portable archive copies. Lazy mode defers
185
- opaque payloads, not all numerical arrays or graph nodes. Eager mode is default.
186
- Payload encoding is checked on access; declared extents are checked on opening.
187
-
188
- ## Supported subset
189
-
190
- The fixtures come from CasADi 3.8.1 default `Function.save()` files: MX arithmetic,
191
- sparse matrices, gathers, slices and entry assignments. Supported layouts are
192
- ProtoFunction 2, FunctionInternal 8, XFunction 1, MXFunction 3, little-endian
193
- protocol 3. Unsupported operations/layouts fail explicitly. SX functions, nested
194
- function calls, plugins, JIT, nonempty option/cache dictionaries and debug
195
- serialization are outside this prototype. It is not a general reader for every
196
- CasADi file or version, nor a hardened untrusted-file service.
197
-
198
- Resource entry points accept a **raw SerializingStream containing one Resource**.
199
- This tests the embedded ZIP representation used by FMUs, but **does not yet read
200
- an entire saved FmuFunction**. No FMU is loaded or executed.
201
-
202
- ## Scheme, tests and packaging
203
-
204
- `schemes/serialization_scheme.json` vendors CasADi's checked-in
205
- `misc/serialization_scheme.json`. It is a source-derived serializer index and
206
- named-field contract, **not a complete executable deserialization grammar**.
207
- `scripts/generate-reader-assets.py` consumes this JSON and emits three checked-in
208
- source assets: `src/scheme.js`, `python/casadi_reader/_scheme.py`, and
209
- `native/src/scheme.hpp`. These contain runtime protocol constants, operation IDs
210
- and class versions, omitting the C++ source index. Supported positional layouts
211
- are still implemented explicitly in the readers.
212
-
213
- `npm run build` regenerates the assets and bundles the JavaScript. `npm pack`
214
- runs that build automatically. Python packages contain the generated Python
215
- module, so neither Python nor JavaScript parses the scheme JSON at runtime.
216
- Build tools are development dependencies only.
217
-
218
- CI checks `npm run check:generated` **before** rebuilding, rejecting stale
219
- committed assets. The initial CI builds and tests the npm package and produces its tarball. A browser test serves the extracted npm tarball
220
- over HTTP and checks that decoding needs one JavaScript request and no scheme
221
- JSON request. CI uploads build artifacts; it does not publish packages.
128
+ This is **not yet a guarantee that every CasADi class/version can be read**.
129
+ Unsupported lowering, missing field types, absent layouts and unknown
130
+ discriminators fail explicitly. Current fixtures target the CasADi 3.8.1 source
131
+ snapshot, little-endian protocol 3. Expanding coverage belongs in the scheme
132
+ extractor and fixtures, not mathematical interpretation in the reader. Advanced
133
+ consumers can supply a compiled schema through `decode(text, {scheme})`.
222
134
 
223
135
  ```sh
224
- node scripts/vendor-scheme.mjs ../serialization-scheme/misc/serialization_scheme.json
225
- python scripts/generate-reader-assets.py
226
136
  npm ci
227
137
  npm run check:generated
228
138
  npm test
229
139
  npm run test:browser
230
- PYTHONPATH=python python -m unittest discover -s python/tests -v
231
- CASADI_READER_LIBRARY="$PWD/build/libcasadi_reader.so" julia --project=julia julia/test/runtests.jl
232
- ```
233
-
234
- Set `CASADI_READER_NATIVE` to the native CLI to enable Python/native agreement
235
- tests. Python tests also compare JavaScript when Node is available. Native CasADi
236
- is used only to regenerate fixtures (`scripts/generate-fixtures.py` and
237
- `scripts/generate-resource-fixture.cpp`), never to run the readers. Tests check
238
- native graph/metadata oracles and deferred 16 MiB payload access.
239
-
240
- Create unpublished artifacts with:
241
-
242
- ```sh
243
- python -m build
244
140
  npm pack
245
- python scripts/package-bindings.py --output dist --mex build/matlab/casadi_reader_mex.mexa64
246
141
  ```
247
142
 
248
- This produces Python wheel/sdist, npm tarball, native and Julia source archives,
249
- and a MATLAB ZIP. It does not register packages or upload releases. The adjacent
250
- `casadi-viz` proof of concept consumes the npm package through a local dependency.
251
-
252
- The reader implementation is MIT licensed. The vendored scheme includes
253
- upstream CasADi source excerpts that retain their original license; see NOTICE.
254
-
255
- ## npm publishing
143
+ CI rejects stale generated assets before rebuilding. Browser tests serve the
144
+ extracted npm tarball over HTTP and verify one JavaScript request with no scheme
145
+ JSON or CasADi runtime request. Native CasADi is used only when regenerating the
146
+ fixtures, via `scripts/generate-fixtures.py` and the Resource fixture generator.
256
147
 
257
- `.github/workflows/publish.yml` publishes on a published GitHub release, using
258
- OIDC Trusted Publishing and provenance. Its release tag must equal `v` plus the
259
- version in `package.json`. Prerelease versions use npm's `next` tag; stable
260
- versions use `latest`. It runs the tests and packed-browser check before publishing.
148
+ ## Publishing
261
149
 
262
- Once the package exists on npm, open its Settings → Trusted publishing and add:
150
+ `publish.yml` publishes on a published GitHub release, after tests. The release
151
+ tag must equal `v` plus the package version. Prereleases use npm's `next` tag;
152
+ stable versions use `latest`.
263
153
 
264
- - Provider: GitHub Actions
265
- - Organization: `casadi`
266
- - Repository: `casadi-reader`
267
- - Workflow filename: `publish.yml`
268
- - Environment: leave blank (the workflow does not use a GitHub environment)
269
- - Allowed actions: enable direct `npm publish`
154
+ The npm Trusted Publisher configuration is GitHub organization `casadi`,
155
+ repository `casadi-reader`, workflow `publish.yml`, with no environment name and
156
+ with direct `npm publish` allowed. Publishing uses OIDC and provenance, with no
157
+ npm token secret. See [npm's documentation](https://docs.npmjs.com/trusted-publishers/).
270
158
 
271
- No npm token secret is needed. See the current
272
- [npm Trusted Publishing documentation](https://docs.npmjs.com/trusted-publishers/).
273
- The workflow is prepared; the package has not yet been published to npm.
159
+ The reader implementation is MIT licensed. Source excerpts in the vendored
160
+ scheme retain CasADi's original license; see NOTICE.
package/bin/casadi-reader.js CHANGED
@@ -1,11 +1,17 @@
1
1
  #!/usr/bin/env node
2
2
  import {readFile,writeFile} from 'node:fs/promises';
3
- import {decodeCasadi,decodeResource} from '../dist/index.js';
3
+ import {decode} from '../dist/index.js';
4
4
  try {
5
- const args=process.argv.slice(2),options={lazy:args.includes('--lazy')},resource=args.includes('--resource');
6
- const [input,output,...extra]=args.filter(a=>a!=='--lazy'&&a!=='--resource');
7
- if(!input||extra.length)throw Error('Usage: casadi-reader [--lazy] [--resource] INPUT [OUTPUT.json]');
8
- const read=resource?decodeResource:decodeCasadi;
9
- const json=JSON.stringify(read(await readFile(input,'utf8'),options),null,2)+'\n';
5
+ const args=process.argv.slice(2),options={};const positional=[];
6
+ for(let i=0;i<args.length;i++){
7
+ if(args[i]==='--lazy')options.lazy=true;
8
+ else if(args[i]==='--type') {if(!args[i+1])throw Error('Missing --type value');options.type=args[++i];}
9
+ else if(args[i]==='--resource')options.type='Resource';
10
+ else if(args[i].startsWith('--'))throw Error('Unknown option '+args[i]);
11
+ else positional.push(args[i]);
12
+ }
13
+ const [input,output,...extra]=positional;
14
+ if(!input||extra.length)throw Error('Usage: casadi-reader [--lazy] [--type TYPE] INPUT [OUTPUT.json]');
15
+ const json=JSON.stringify(decode(await readFile(input,'utf8'),options),null,2)+'\n';
10
16
  if(output)await writeFile(output,json);else process.stdout.write(json);
11
17
  }catch(error){console.error(error.message);process.exitCode=1;}