sdn-flow 0.2.0

package/docs/ARCHITECTURE.md

@@ -0,0 +1,200 @@
+# Architecture
+
+## Package Boundary
+
+`sdn-flow` is the portable layer between:
+
+- UI flow authoring
+- host-side runtime execution
+- signed artifact dependency resolution
+- generated C++ bundle compilation
+- deployment authorization
+- deployment transport
+
+It is intentionally host-agnostic so different applications and runtimes can
+consume the same contracts.
+
+## Core Rule
+
+A deployable flow is a compiled single WASM runtime artifact.
+
+The source graph exists for authoring, validation, tracing, and compilation.
+It is not the deployable unit.
+There is no separate interpreted deployment engine.
+There is no JS source generator on the deploy path; source synthesis happens in
+a native C++ generator compiled to WASM.
+
+## Single Plugin Rule
+
+A single plugin invocation is modeled as a one-node flow with optional triggers
+and zero or more sinks. There is no separate deployment path for "just a
+plugin."
+
+## Package Layers
+
+### Runtime Contracts
+
+The runtime layer defines normalized manifest/program objects used by authoring,
+validation, and host integration:
+
+- plugin manifests declare methods, ports, stream limits, and drain policies
+- flow programs declare nodes, edges, triggers, and trigger bindings
+- deployed execution lives in the generated C++ runtime, not in package JS
+
+### Designer
+
+The designer layer is UI-facing but DOM-free. It owns:
+
+- graph mutation helpers
+- external input/output and capability summaries for visual editors
+- single-plugin flow creation
+- compile orchestration through an injected compiler adapter
+- deployment orchestration through an injected deployment client
+
+### Host Planning
+
+The host-planning layer owns portable hosted-runtime description:
+
+- runtime kind (`flow`, `plugin`, `service`)
+- startup phase (`bootstrap`, `early`, `session`, `on-demand`)
+- local vs remote authority
+- runtime dependencies
+- transport bindings (`same-app`, `direct`, `webrtc`, `sdn-protocol`, `http`)
+
+This keeps startup-only services such as local licensing in the same runtime
+model as any other flow/plugin deployment.
+
+See also:
+
+- [Host Capability Model](./HOST_CAPABILITY_MODEL.md)
+
+### Compiler
+
+The compiler layer owns:
+
+- signed artifact catalog resolution
+- single-source C++ runtime generation
+- native C++ source-generator tool compilation and invocation
+- `emception`-compatible compile planning
+- artifact assembly back into the deploy contract
+
+### Authorization
+
+Deployment permissions are explicit signed envelopes, designed for SDN HD-wallet
+signatures.
+
+The package defines:
+
+- canonical payload encoding
+- deploy authorization payload shape
+- signer and verifier contracts
+- scope checks against target, artifact, and required capabilities
+
+### Transport
+
+Remote deployment payloads may be encrypted with PKI. The transport layer
+provides:
+
+- X25519 ephemeral key agreement
+- HKDF-derived AES-256-GCM content encryption
+- JSON payload helpers for deployment envelopes
+
+### Deploy
+
+The deploy layer only ships compiled artifacts. It does not accept raw flow
+graphs as the deployment boundary.
+
+## Artifact Contract
+
+A compiled flow artifact contains:
+
+- `artifactId`
+- `programId`
+- `format`
+- `wasm`
+- `manifestBuffer`
+- `manifestExports.bytesSymbol`
+- `manifestExports.sizeSymbol`
+- `entrypoint`
+- `graphHash`
+- `requiredCapabilities`
+- `pluginVersions`
+- `schemaBindings`
+- `abiVersion`
+
+The compile plan that produces that artifact may also include:
+
+- generator request bytes consumed by the native generator WASM tool
+- generated `main.cpp`
+- generated runtime topology descriptors and mutable node-state storage
+- signed plugin artifact dependency descriptors
+- signed plugin manifest bytes
+- the exact `em++` command issued to `emception`
+
+## Editor And Capability Model
+
+Programs may declare:
+
+- `externalInterfaces`
+- `artifactDependencies`
+- `editor` layout metadata
+
+Plugin manifests may declare:
+
+- `capabilities`
+- `externalInterfaces`
+
+The designer layer can merge those declarations with trigger bindings and
+resolved plugin manifests so the operator can see:
+
+- which network, timer, protocol, filesystem, and database bindings are needed
+- which capabilities must be approved and signed
+- which signed plugin artifacts must exist before compilation
+
+## Host Integration
+
+Hosts are expected to provide adapters for:
+
+- graph validation against host capabilities
+- manifest building into canonical FlatBuffer bytes
+- invoking the native C++ source generator tool
+- compilation via `emception` or equivalent
+- local deployment/install
+- remote deployment endpoint verification and installation
+- protocol/pubsub registration and network transport
+
+Hosts may also use the portable host-planning surface to describe early-start
+services, same-app loopback bindings, or WebRTC-connected local runtimes
+without changing the flow/plugin ABI.
+
+Capability gaps must be documented at the host-profile level. Do not solve a
+missing runtime capability by inventing a second deploy/runtime model.
+
+This package deliberately leaves those host integrations outside the portable
+core.
+
+## Manifest Rule
+
+Every plugin and every compiled flow artifact must embed a FlatBuffer manifest
+buffer and expose a callable export for it. The default flow export names are:
+
+- `flow_get_manifest_flatbuffer`
+- `flow_get_manifest_flatbuffer_size`
+
+The default plugin export names are:
+
+- `plugin_get_manifest_flatbuffer`
+- `plugin_get_manifest_flatbuffer_size`
+
+## Example
+
+The repo includes a concrete example in
+[examples/flows/iss-proximity-oem/flow.json](../examples/flows/iss-proximity-oem/flow.json)
+that:
+
+- streams OMMs from SDN pubsub
+- ingests them into a FlatSQL storage plugin
+- queries objects within `50 km` of object `25544`
+- propagates `90` samples for one orbit
+- generates OEMs
+- writes and republishes those OEMs through explicit sink nodes

package/docs/HOST_CAPABILITY_MODEL.md

@@ -0,0 +1,317 @@
+# Host Capability Model
+
+## Purpose
+
+This document defines how compiled `sdn-flow` runtimes consume the canonical
+module-level host capability model from `space-data-module-sdk` and project it
+onto flow-level runtime artifacts.
+
+The canonical module-facing capability vocabulary, module-facing hostcall ABI,
+and single-module conformance rules live in `space-data-module-sdk`.
+This document focuses on how `sdn-flow` composes those module contracts into
+compiled flow runtimes.
+
+The goal is to keep one runtime model across OrbPro, `sdn-js`, Go SDN, and
+WASI-style hosts without creating a different flow runtime model or a different
+deploy shape for each environment.
+
+## Hard Rules
+
+1. Every deployable unit is one compiled WASM runtime artifact.
+2. Every module is a flow. A "single plugin" is a degenerate one-node flow,
+   but its standalone artifact contract still comes from `space-data-module-sdk`.
+3. Hosts provide capabilities and placement, not business logic.
+4. Capability requirements must be explicit in manifests, flow programs,
+   hosted-runtime plans, or deployment envelopes. Do not hide them in ad hoc
+   host config or implicit imports.
+5. Use the minimum host-adapter set needed to cover environment families.
+   Do not add a new harness just because one runtime exposes a feature
+   differently.
+6. Generic "WASI" is not a sufficient capability claim. If a flow needs more
+   than the portable baseline, the required host profile and imports must be
+   documented explicitly.
+7. If there is no compiled-host path for a required feature in a target
+   environment, treat that as a release-blocking alert. Do not paper over it
+   with loose JS wrappers, host cron jobs, or sidecar-only glue.
+
+## Capability Declaration Surfaces
+
+Use the surfaces below together. They solve different problems.
+
+| Surface                                         | Purpose                                    | Example                                           |
+| ----------------------------------------------- | ------------------------------------------ | ------------------------------------------------- |
+| `manifest.capabilities`                         | coarse approval/signing scope              | `["timers", "http", "storage_write"]`             |
+| `manifest.externalInterfaces`                   | concrete plugin-level bindings             | outbound HTTP, FlatSQL adapter, filesystem path   |
+| `program.externalInterfaces`                    | flow-level bindings beyond a single plugin | shared HTTPS listener, shared pubsub topic        |
+| `program.triggers`                              | trigger ownership                          | timer, protocol request, HTTP request             |
+| hosted-runtime plan `requiredCapabilities`      | runtime placement/startup requirements     | early local service needs `protocol_handle`       |
+| deployment authorization `requiredCapabilities` | signed approval at deploy time             | remote install allowed to use `pubsub` and `ipfs` |
+
+Keep capability IDs coarse and stable. Put transport-specific details in the
+interface object, not in a proliferating list of near-duplicate capability
+strings.
+
+## Canonical Capability Guidance
+
+Recommended coarse capability IDs:
+
+- `clock`
+- `random`
+- `timers`
+- `http`
+- `network`
+- `filesystem`
+- `pipe`
+- `pubsub`
+- `protocol_handle`
+- `protocol_dial`
+- `database`
+- `storage_adapter`
+- `storage_query`
+- `storage_write`
+- `wallet_sign`
+- `ipfs`
+- `scene_access`
+- `render_hooks`
+
+Guidance:
+
+- Use `network` for raw sockets, TCP, UDP, QUIC, or similar transport access.
+- Use `http` for HTTP client/server semantics instead of encoding HTTP as a
+  special case of raw sockets.
+- Use `filesystem` for path-backed storage and `database` for logical database
+  interfaces such as FlatSQL.
+- Use `pipe` for stdin/stdout/stderr, named pipes, or stream-style host I/O.
+- Keep `ipfs` coarse. Pinning, block get/put, and publish behavior belong in
+  interface metadata.
+- Compute-only infrastructure plugins such as DA FlatBuffers usually need no
+  host capability at all. `hd-wallet-wasm` style plugins should only declare
+  host capabilities such as `random` or `wallet_sign` when they genuinely need
+  host-provided entropy or resident key material.
+
+## External Interface Patterns
+
+External interfaces carry the precise runtime binding details. Keep the
+interface kinds small and use `properties` for specialization.
+
+### Timer / Clock / Randomness
+
+- Timers normally enter through `program.triggers`.
+- If a plugin needs direct time or entropy outside trigger scheduling, declare
+  `clock` and `random` capabilities explicitly.
+- Do not let native code silently reach into browser globals or host globals.
+  Entropy and time should enter through explicit host capabilities.
+
+### HTTP
+
+Use `kind: "http"` for both outbound requests and inbound handlers.
+
+Example:
+
+```json
+{
+  "interfaceId": "celestrak-fetch",
+  "kind": "http",
+  "direction": "output",
+  "capability": "http",
+  "resource": "https://celestrak.org",
+  "required": true,
+  "properties": {
+    "methods": ["GET"],
+    "purpose": "Fetch OMM updates"
+  }
+}
+```
+
+### Raw Network / TCP / UDP
+
+Use `kind: "network"` and put socket details in `properties`.
+
+Example:
+
+```json
+{
+  "interfaceId": "udp-observation-ingest",
+  "kind": "network",
+  "direction": "input",
+  "capability": "network",
+  "resource": "udp://0.0.0.0:40123",
+  "required": true,
+  "properties": {
+    "transport": "udp",
+    "mode": "listen"
+  }
+}
+```
+
+This avoids separate ABIs for TCP, UDP, raw sockets, and runtime-specific
+socket APIs.
+
+### Filesystem / Pipes
+
+Use `kind: "filesystem"` for path-backed files and `kind: "pipe"` for stream
+or pipe semantics.
+
+Example:
+
+```json
+{
+  "interfaceId": "oem-export-dir",
+  "kind": "filesystem",
+  "direction": "output",
+  "capability": "filesystem",
+  "resource": "file:///var/lib/sdn/oem",
+  "required": true,
+  "properties": {
+    "access": "read-write"
+  }
+}
+```
+
+```json
+{
+  "interfaceId": "stderr-log",
+  "kind": "pipe",
+  "direction": "output",
+  "capability": "pipe",
+  "resource": "stderr",
+  "required": false
+}
+```
+
+### Database / FlatSQL
+
+Storage engines remain plugins or explicit host services, not hidden host
+subsystems.
+
+- Use `kind: "database"` for the logical database surface seen by the flow.
+- Use `kind: "host-service"` when the flow needs a host-provided storage
+  adapter contract.
+
+### IPFS / SDS Protocol / Other Services
+
+If a function is logically a networked platform service rather than a raw
+socket, prefer `kind: "host-service"`, `kind: "protocol"`, or `kind: "pubsub"`
+plus explicit `resource`, `protocolId`, `topic`, or `properties`.
+
+Examples:
+
+- IPFS pinning service
+- SDS PNM notification channel
+- local orbit-determination service runtime
+- same-app licensing runtime
+
+## Environment Profiles
+
+The canonical model is shared, but capability availability differs by host
+profile. Document the profile. Do not claim "runs anywhere" unless it is true
+for the declared interfaces.
+
+### OrbPro Browser Host
+
+Portable expectations:
+
+- `clock`, `random`, `timers`
+- outbound HTTP
+- same-app bindings
+- browser-managed storage surfaces
+- scene/render integration
+- `sdn-js` protocol/pubsub integration when configured
+
+Non-portable or unavailable by default:
+
+- raw TCP/UDP sockets
+- arbitrary listen sockets
+- unrestricted filesystem access
+- OS pipes
+
+If a flow needs those features, place that runtime in a different host profile
+instead of pretending the browser host can satisfy them.
+
+### Node / Deno / Bun Through The `sdn-js` Host Family
+
+Expected direction:
+
+- keep one `sdn-js` host family for browser-capable and JS-capable runtimes
+- expose files, pipes, outbound HTTP, protocol/pubsub, IPFS, and local
+  listeners through the same capability model
+- document unsupported features per runtime instead of creating a new plugin
+  ABI for each JS engine
+
+If one engine cannot provide a capability in compiled-host form, mark that
+capability unsupported for that profile.
+
+### Go SDN Host
+
+This is the preferred profile for long-running service flows such as:
+
+- timer-triggered OMM sync
+- FlatSQL-backed local storage
+- HTTPS serving
+- IPFS publish and pin
+- SDS protocol notifications
+- local orbit determination and association jobs
+
+The flow still deploys as one compiled WASM artifact. The Go host supplies the
+bindings.
+
+### Generic WASI Runtime
+
+Treat generic WASI as a narrow portability baseline only.
+
+Portable assumptions:
+
+- startup entrypoint
+- stdio-like behavior
+- linear memory/exported symbol access
+- whatever explicit imported host functions the host provides
+
+Do not assume from "WASI" alone:
+
+- outbound HTTP
+- sockets
+- persistent filesystem semantics
+- timers beyond what the host explicitly exposes
+- pubsub, protocol, IPFS, or database services
+
+### WasmEdge / Wasmtime / Wasmer / Wazero
+
+Vendor or runtime extensions may be used, but they do not change the canonical
+flow ABI.
+
+Rules:
+
+- declare the same capability IDs and interface shapes
+- document the exact host profile or extension required
+- do not describe the result as generic-WASI portable unless the same compiled
+  contract is actually supported elsewhere
+
+## Alert Conditions
+
+Treat these as architecture failures:
+
+- a host-side cron job driving loose plugin calls instead of a timer trigger
+- JSON moving between compiled plugins on flow edges
+- separate deployment of "scheduler" and "plugins" as different runtime models
+- a browser-only JS implementation and a different Go/WASI implementation for
+  the same canonical feature
+- a claim that a module runs on "any WASI runtime" when it actually needs
+  custom imports or a vendor extension
+- sidecar metadata or sidecar policy as the only source of runtime truth
+
+## Example Deployment Shape
+
+The following workload is valid in the canonical model:
+
+1. A timer-triggered flow wakes on a declared interval.
+2. It fetches OMMs from CelesTrak over an explicit HTTP interface.
+3. It writes aligned records through a FlatSQL storage plugin/runtime.
+4. It exposes a declared HTTPS interface for download.
+5. It publishes content to IPFS and requests pinning through declared host
+   services.
+6. It notifies peer SDN nodes over a declared SDS protocol binding.
+7. It kicks off downstream association and orbit-analysis nodes.
+
+This still compiles to one signed deployable WASM artifact. Local and remote
+deployment may wrap that artifact in the same signed and optionally encrypted
+deployment envelope.

package/docs/PLUGIN_ARCHITECTURE.md

@@ -0,0 +1,145 @@
+# Plugin Architecture
+
+## Scope
+
+This document defines how `sdn-flow` consumes the standalone plugin
+architecture sourced from `space-data-module-sdk`.
+
+It covers:
+
+- plugin identity and manifest rules
+- streaming method contracts
+- external interface declarations
+- callable manifest exports
+- the relationship between plugins and flows
+- local and remote deployment expectations
+
+## Core Rule
+
+The runtime does not distinguish between "a plugin" and "a flow" at deployment
+time. A single plugin invocation is a one-node flow.
+
+That means:
+
+- single-plugin execution uses the same deploy model as multi-node graphs
+- compiled deployment is always one WASM runtime artifact
+- trigger, queueing, and drain semantics stay uniform
+
+## Plugin Contract
+
+Every plugin participates in the flow system through a manifest-defined method
+surface.
+
+A method declares:
+
+- stable `methodId`
+- one or more input ports
+- zero or more output ports
+- accepted schema sets per port
+- stream-count limits
+- `maxBatch`
+- `drainPolicy`
+
+Methods are invoked over schema-tagged frame streams, not ad hoc JSON payloads.
+
+Plugins may also declare `externalInterfaces` so visual editors and deployment
+tooling can show the real network, protocol, filesystem, database, or host
+service bindings required to make the graph run.
+
+Keep capability IDs coarse and portable. Use interface metadata to capture
+runtime-specific details such as:
+
+- HTTP method sets
+- TCP/UDP/raw-socket transport mode
+- filesystem sandbox or mount location
+- pipe or stream direction
+- IPFS or SDS service operation
+
+Storage engines follow the same rule. A FlatSQL database should be represented
+as a storage plugin/runtime that:
+
+- ingests aligned FlatBuffer records
+- exposes query methods
+- declares whether it is backed by transient memory or a host-provided storage
+  adapter
+
+The host should not hide a second engine-owned SQL source of truth behind that
+plugin surface.
+
+Infrastructure libraries follow the same rule. If a deployable flow depends on
+`hd-wallet-wasm`, DA FlatBuffers, or similar runtime libraries for signing,
+envelope encryption, field-level encryption, schema transforms, or repacking,
+those surfaces should appear as normal manifest-defined plugins. Do not hide
+them behind host-only helper code.
+
+For the canonical host capability rules and environment profiles, see:
+
+- [Host Capability Model](./HOST_CAPABILITY_MODEL.md)
+
+## Manifest Rule
+
+Every plugin must embed a FlatBuffer manifest buffer and expose callable exports
+for it.
+
+Default export names:
+
+- `plugin_get_manifest_flatbuffer`
+- `plugin_get_manifest_flatbuffer_size`
+
+Compiled flow artifacts follow the same rule with flow-scoped export names:
+
+- `flow_get_manifest_flatbuffer`
+- `flow_get_manifest_flatbuffer_size`
+
+The manifest is part of the runtime contract. Sidecar metadata files are not
+sufficient for deployable artifacts.
+
+## Streaming Model
+
+Each input port accepts one or more streams of schema-tagged frames.
+
+Each frame carries runtime type identity:
+
+- `schemaName`
+- `fileIdentifier`
+- optional `schemaHash`
+- `streamId`
+- `sequence`
+- `traceId`
+
+If a method only processes one frame per invocation, the runtime keeps invoking
+it until the backlog is drained or the scheduler yields.
+
+## Plugin Families
+
+`sdn-flow` does not hardcode host-specific plugin classes. Common families
+include:
+
+- propagators
+- sensors
+- analyzers
+- publishers
+- responders
+- infrastructure plugins
+
+Examples of infrastructure plugins include FlatSQL storage, `hd-wallet-wasm`
+sign/encrypt nodes, and DA FlatBuffers schema/field transform nodes.
+
+All of them enter the runtime through the same manifest + method contract.
+
+## Host Boundary
+
+Hosts are responsible for:
+
+- manifest decoding
+- plugin loading and registration
+- capability enforcement
+- editor integration for external interface inspection and approval
+- compilation of flow graphs into deployable WASM artifacts
+- installation and execution of those artifacts
+
+`sdn-flow` owns the portable runtime and deployment model, not host-specific
+loading code.
+
+Host-specific runtime differences must stay in capability bindings and hosted
+runtime plans. They must not fork the plugin ABI.