@techspokes/typescript-wsdl-client 0.16.1 → 0.17.0

package/docs/decisions/002-streamable-responses.md

@@ -0,0 +1,308 @@
+# ADR 002: Streamable SOAP Responses and JSON Gateway Output
+
+Proposal for opt-in streamable SOAP response support, with Escapia EVRN Content Service as the concrete driver.
+
+See the root [README](../../README.md) for the authoritative documentation index and [Architecture](../architecture.md) for the current generator pipeline.
+
+## Status
+
+Accepted (2026-04-20). Implementation shipped in 0.17.0 across five phases
+(Phase 0 research → Phase 5 integration). See
+`scratches/plans/v017/streamable-responses.plan.yaml` for phase-by-phase
+notes and verification gates.
+
+## Context
+
+The current generator handles WSDL operations as buffered request and response calls. A generated client method awaits the SOAP runtime callback, receives a fully materialized result object, and returns `{ response, headers, responseRaw, requestRaw }`. The generated OpenAPI document describes each operation as `application/json`. The generated Fastify route calls the client method, optionally unwraps array wrapper types, and returns the standard response envelope.
+
+Escapia exposes two related SOAP services:
+
+- `EVRNService` contains the normal operation and type model.
+- `EVRNContentService` contains content-oriented operations that are intended for large responses.
+
+The Escapia content WSDL includes stream-like operations such as `UnitDescriptiveInfoStream` and `UnitCalendarAvailBatch`. Their output wrapper types contain schema and wildcard payload sections instead of clean concrete response members. The practical record shapes come from the main EVRN service WSDL, for example `UnitDescriptiveContentType` and `UnitCalendarAvailType`.
+
+This means the current package has two gaps:
+
+- It does not stream SOAP responses through generated clients or gateways.
+- It does not let users map opaque WSDL response wrappers to concrete JSON record shapes from the same or a companion WSDL.
+
+## Current Architecture Review
+
+The compile stage produces `catalog.json`, which is the only artifact that understands WSDL types, operation input and output elements, attribute metadata, child type metadata, and primitive mapping options.
+
+The client stage generates a `node-soap` wrapper. Every operation is generated as a promise-returning method. The public operations interface mirrors that buffered shape, which is useful for mocks but cannot currently express an `AsyncIterable` or stream response.
+
+The OpenAPI stage generates paths from catalog operations. It assumes every operation has a JSON request body and a JSON response body. Since version 0.7.1, successful responses are always wrapped in the standard envelope schema.
+
+The gateway stage reads the OpenAPI document first and enriches operation metadata from `catalog.json` when the catalog is available. This lets route generation stay OpenAPI-driven, but stream support needs metadata that OpenAPI alone cannot represent.
+
+The app and test generators assume ordinary JSON route responses. Generated tests use mock clients that implement the buffered operations interface.
+
+## Requirements
+
+- Existing buffered generation must remain unchanged unless a user opts in.
+- Stream support must be configurable per operation.
+- Users must be able to map an opaque output wrapper to a concrete record type.
+- Record types must be resolvable from the current catalog or from a companion catalog.
+- Gateway output must be able to emit records before the full SOAP response has arrived.
+- Generated routes must respect Node and Fastify backpressure.
+- OpenAPI output must describe the wire media type without pretending that NDJSON is a single JSON object.
+- Error handling must distinguish errors before the first byte from errors after streaming has started.
+- The feature must be testable with a local chunked SOAP server, not only static XML fixtures.
+
+## Proposed Configuration Model
+
+Add a stream configuration file passed with `--stream-config`. The same option should be available on `compile`, `client`, `openapi`, `gateway`, and `pipeline` so each stage can be run independently.
+
+```json
+{
+  "shapeCatalogs": {
+    "evrn": {
+      "wsdlSource": "https://api.escapia.com/EVRNService.svc?singleWsdl"
+    }
+  },
+  "operations": {
+    "UnitDescriptiveInfoStream": {
+      "mode": "stream",
+      "format": "ndjson",
+      "mediaType": "application/x-ndjson",
+      "recordType": "UnitDescriptiveContentType",
+      "recordPath": [
+        "UnitDescriptiveInfoStream",
+        "EVRN_UnitDescriptiveInfoRS",
+        "UnitDescriptiveContents",
+        "UnitDescriptiveContent"
+      ],
+      "shapeCatalog": "evrn"
+    },
+    "UnitCalendarAvailBatch": {
+      "mode": "stream",
+      "format": "ndjson",
+      "mediaType": "application/x-ndjson",
+      "recordType": "UnitCalendarAvailType",
+      "recordPath": [
+        "EVRN_UnitCalendarAvailBatchRS",
+        "EVRN_UnitCalendarAvailBatchRS",
+        "UnitCalendarAvails",
+        "UnitCalendarAvail"
+      ],
+      "shapeCatalog": "evrn"
+    }
+  }
+}
+```
+
+`shapeCatalogs` names additional WSDL or catalog inputs used only to resolve stream record shapes.
+
+`operations` marks specific WSDL operations as streamed. Operations not listed keep the existing buffered behavior.
+
+`recordPath` is an ordered XML element path from the SOAP body payload to the repeated record element. Duplicate local names must be allowed because Escapia uses nested elements with the same name in at least one response.
+
+`recordType` is the TypeScript and schema model to emit for each streamed record.
+
+`format` should support `ndjson` first. `json-array` can be added later for clients that require a single JSON array response.
+
+## CLI and Programmatic API
+
+The CLI should add `--stream-config <file>` to all generation commands. The pipeline command should pass the parsed and normalized configuration to every stage.
+
+The programmatic API should add optional stream configuration fields to:
+
+```typescript
+export interface PipelineOptions {
+  streamConfigFile?: string;
+  streamConfig?: StreamConfig;
+}
+
+export interface GenerateOpenAPIOptions {
+  streamConfigFile?: string;
+  streamConfig?: StreamConfig;
+}
+
+export interface GenerateGatewayOptions {
+  streamConfigFile?: string;
+  streamConfig?: StreamConfig;
+}
+```
+
+The compiler options should accept normalized stream metadata only when compilation needs to persist it into the catalog.
+
+## Catalog Model
+
+The compiled catalog should carry normalized stream metadata beside operation metadata:
+
+```typescript
+export interface Operation {
+  name: string;
+  inputTypeName?: string;
+  outputTypeName?: string;
+  stream?: OperationStreamMetadata;
+}
+
+export interface OperationStreamMetadata {
+  mode: "stream";
+  format: "ndjson" | "json-array";
+  mediaType: string;
+  recordPath: string[];
+  recordTypeName: string;
+  shapeCatalogName?: string;
+  sourceOutputTypeName?: string;
+}
+```
+
+The compiler should also represent wildcard schema particles explicitly. Today Escapia stream wrappers compile into misleading `xs:schema` properties and drop the wildcard payload. A better model is to retain both the schema marker and a wildcard marker so diagnostics and stream candidate detection are honest.
+
+## Shape Resolution
+
+The first implementation should support companion catalogs in one of two forms:
+
+- A `catalogFile` path for a catalog already produced by `wsdl-tsc compile`.
+- A `wsdlSource` path or URL that the pipeline compiles before normalizing stream metadata.
+
+For generated types, the MVP should copy the reachable record type graph from the companion catalog into the current generated output when there is no name collision. If a collision occurs, generation should fail with a clear diagnostic instead of silently renaming public API types.
+
+A future version can support external type imports from another generated client directory.
+
+## Client Runtime
+
+Generated clients should keep the existing buffered method signatures for ordinary operations.
+
+Stream operations should return a separate response type:
+
+```typescript
+export type StreamOperationResponse<RecordType, HeadersType = Record<string, unknown>> = {
+  records: AsyncIterable<RecordType>;
+  headers: HeadersType;
+  requestRaw?: string;
+};
+```
+
+The operations interface should use the same stream response type so generated gateway tests and user mocks can implement streaming behavior without importing the concrete SOAP client.
+
+The transport layer should be proven by a chunked integration test. If `node-soap` with stream or SAX options can deliver records incrementally, it can be used for the MVP. If it buffers before yielding, the feature must use a dedicated SOAP HTTP transport for stream operations before release.
+
+> **Phase 0 research outcome (2026-04-20):** `node-soap` **buffers** the full response before invoking the operation callback. Measured with `test/research/node-soap-streaming.test.ts` against a chunked HTTP fixture: first chunk flushed at ~48 ms, server closed at ~701 ms, node-soap callback fired at ~659 ms (~40 ms before close — that is parse time, not streaming). Generated stream operations therefore use a dedicated streaming HTTP transport emitted into the client runtime; `node-soap` remains the transport for buffered operations only. SAX streaming was validated with `saxes` 6.0.0; `test/research/sax-record-path.test.ts` chunk-fuzzes the Escapia-shaped XML (duplicate `EVRN_UnitDescriptiveInfoRS` wrappers) across every single-byte split and every byte-pair split and produces identical records each time. See `scratches/plans/v017/streamable-responses.plan.yaml` for the full findings.
+
+## XML to JSON Streaming Conversion
+
+The runtime needs a streaming XML parser. `fast-xml-parser` is suitable for buffered documents but not for this use case. A SAX-style parser should track the configured `recordPath`, build a record object as the target element closes, and yield each record immediately.
+
+The converter must reuse catalog metadata for:
+
+- XML attributes and the configured attributes key
+- Child element type lookup
+- Repeated elements
+- Nillable elements
+- Text content and `$value`
+- Primitive mapping
+
+The converter should collect SOAP faults, response errors, and warnings when they appear before the first record. After streaming starts, terminal errors cannot be represented as a normal JSON error envelope for NDJSON. The runtime should either emit a final error record with a documented extension shape or abort the stream and log the classified error.
+
+## OpenAPI Output
+
+Stream operations should not use the standard success envelope for `200` responses. The response content should use the configured stream media type.
+
+```json
+{
+  "description": "Successful streamed SOAP operation response",
+  "content": {
+    "application/x-ndjson": {
+      "schema": { "type": "string" },
+      "x-wsdl-tsc-stream": {
+        "format": "ndjson",
+        "itemSchema": {
+          "$ref": "#/components/schemas/UnitDescriptiveContentType"
+        }
+      }
+    }
+  }
+}
+```
+
+OpenAPI 3.1 cannot fully describe an NDJSON sequence as a standard JSON Schema document. The `x-wsdl-tsc-stream` extension makes the item schema explicit for generated gateways, documentation tools, and future SDK generators.
+
+## Gateway Output
+
+Generated stream routes should call the stream operation and send a Node stream through Fastify.
+
+```typescript
+import { Readable } from "node:stream";
+
+handler: async (request, reply) => {
+  const client = fastify.escapiaContentClient;
+  const result = await client.UnitDescriptiveInfoStream(request.body);
+
+  reply.type("application/x-ndjson");
+  return reply.send(Readable.from(toNdjson(result.records)));
+}
+```
+
+The generated operation schema should keep request validation but omit the Fastify response serialization schema for streamed `200` responses. Fastify cannot serialize an unbounded stream with a normal JSON response schema.
+
+The gateway runtime should add `toNdjson()` and, later, `toJsonArrayStream()` helpers. These helpers should be deterministic, backpressure-aware, and safe for large payloads.
+
+## Test Strategy
+
+Add unit tests for stream config parsing, operation matching, catalog normalization, wildcard compiler metadata, and record path matching.
+
+Add converter tests that split XML chunks across element boundaries to prove the parser does not rely on convenient chunking.
+
+Add a local Escapia-like WSDL fixture with `xs:any` output wrappers and a companion WSDL fixture with the concrete record types.
+
+Add an integration test with a fake SOAP HTTP server that writes one record, waits, writes another record, and then closes the envelope. The test should assert that the gateway emits the first NDJSON line before the upstream response completes.
+
+Add snapshot tests for generated `client.ts`, `operations.ts`, OpenAPI output, gateway route files, and gateway runtime helpers.
+
+Run `npm run smoke:pipeline` after implementation to verify ordinary buffered generation is unchanged.
+
+## Implementation Plan
+
+1. Define `StreamConfig`, parser, validation errors, and normalized operation metadata.
+2. Add `--stream-config` to the CLI and thread it through the programmatic API.
+3. Update the compiler to preserve wildcard particles and emit stream candidate diagnostics.
+4. Implement companion catalog loading and record type graph copying.
+5. Generate stream method signatures in `client.ts` and `operations.ts`.
+6. Add a streaming XML converter with catalog-aware JSON shape conversion.
+7. Generate OpenAPI stream responses with `x-wsdl-tsc-stream` metadata.
+8. Generate Fastify streaming handlers and runtime helpers.
+9. Add unit, snapshot, and chunked integration tests.
+10. Update README, CLI reference, configuration docs, gateway guide, testing guide, and supported patterns.
+
+## Acceptance Criteria
+
+- Existing generated output is unchanged when no stream config is provided.
+- A stream-configured Escapia content WSDL generates typed stream client methods.
+- The generated gateway emits `application/x-ndjson` records incrementally.
+- The generated OpenAPI document identifies stream operations and record schemas.
+- The converter maps XML attributes, arrays, text values, and nillable values consistently with buffered responses.
+- The chunked integration test proves the first record is sent before the full SOAP response is available.
+- Stream route errors before the first byte use the normal gateway error envelope.
+- Stream route errors after the first byte follow a documented terminal error policy.
+
+## Consequences
+
+The generator gains a second response execution model. This increases complexity in the client, OpenAPI, gateway, and test generators, but keeps the complexity opt-in and operation-scoped.
+
+The catalog becomes more important as the shared source of truth because OpenAPI alone cannot carry enough stream conversion metadata.
+
+NDJSON becomes the recommended stream format because it is simple, broadly consumable, and does not require buffering a complete JSON array before sending data.
+
+Companion catalogs are required for vendors that split stream wrappers and concrete record shapes across separate WSDLs.
+
+## Implementation Notes
+
+Captured after the 0.17.0 ship for future maintainers:
+
+- `--stream-config` is wired onto the `compile`, `client`, and `pipeline` CLI commands. It is intentionally not accepted on `openapi`, `gateway`, or `app` because those commands consume a pre-compiled `catalog.json` that already carries the normalized `OperationStreamMetadata`. The original proposal text that listed the flag on every command reflected the design intent; the shipped surface is narrower for that reason.
+- `GenerateOpenAPIOptions` and `GenerateGatewayOptions` in the programmatic API do not carry stream-config fields for the same reason. `compileWsdlToProject` and `runGenerationPipeline` (PipelineOptions) do.
+- The client stream transport is emitted from two templates (`clientStreamMethods.tpl.txt`, `operationsStreamHelper.tpl.txt`). They embed the `StreamOperationResponse<T>` type and a `callStream()` method that POSTs a hand-built SOAP envelope via global `fetch`, bypassing `node-soap` as required by the phase-0 finding.
+- `saxes ^6.0.0` was promoted from devDependency to runtime dependency on the package, and added as a pinned dependency in the generated app scaffold so stream-enabled consumers install it automatically.
+- The `json-array` format is reserved in the config parser but not yet implemented by the emitters. Entries using `format: "json-array"` parse successfully and can be used to forward-declare intent; they do not currently generate routes or client methods.
+
+## References
+
+- Escapia EVRN API documentation: <https://eweb.escapia.com/distribution/api/evrn-api-documentation>
+- Escapia EVRN service WSDL: <https://api.escapia.com/EVRNService.svc?WSDL>
+- Escapia EVRN content service WSDL: <https://api.escapia.com/EVRNContentService.svc?WSDL>
+- Escapia batch API support article: <https://support.escapia.com/articles/en_US/Article/HASW-Batch-API-Methods-EVRN?category=Website_Services>

package/docs/gateway-guide.md

@@ -81,6 +81,43 @@ export async function registerRoute_v1_weather_getcityforecastbyzip(fastify: Fas
 }
 ```
 
+### Streaming Handlers
+
+Operations opted in via `--stream-config` emit an NDJSON response pipe
+instead of the standard envelope. The generated handler streams records as
+they arrive, and the Fastify response is flushed line-by-line with
+backpressure (ADR-002).
+
+```typescript
+import type { FastifyInstance } from "fastify";
+import type { GetWeatherInformation } from "../../client/types.js";
+import schema from "../schemas/operations/getweatherinformation.json" with { type: "json" };
+import { toNdjson } from "../runtime.js";
+
+// Response schema omitted: stream operations emit NDJSON, not a single JSON object
+const { response: _response, ...routeSchema } = schema as Record<string, unknown>;
+
+export async function registerRoute_v1_weather_getweatherinformation(fastify: FastifyInstance) {
+  fastify.route<{ Body: GetWeatherInformation }>({
+    method: "POST",
+    url: "/get-weather-information",
+    schema: routeSchema,
+    handler: async (request, reply) => {
+      const client = fastify.weatherClient;
+      const result = await client.GetWeatherInformation(request.body as GetWeatherInformation);
+      reply.type("application/x-ndjson");
+      return reply.send(toNdjson(result.records));
+    },
+  });
+}
+```
+
+The client method returns `StreamOperationResponse<RecordType>` with a
+`records: AsyncIterable<RecordType>`. Errors raised before the first record
+use the normal error envelope; errors raised mid-stream trip the chunked
+response's truncation — consumers detect these via the absence of a clean
+terminating zero-chunk.
+
 ## Error Handling
 
 The centralized error handler (runtime.ts) automatically classifies errors:

package/docs/generated-code.md

@@ -36,6 +36,23 @@ const info = await client.GetWeatherInformation({});
 console.log(info.GetWeatherInformationResult.WeatherDescriptions);
 ```
 
+## Calling Stream Operations
+
+Operations opted into streaming with `--stream-config` return `StreamOperationResponse<RecordType>` instead of the buffered shape. `records` is a single-pass `AsyncIterable<RecordType>` that pulls bytes from the upstream SOAP response on demand, so memory stays bounded regardless of payload size.
+
+```typescript
+const result = await client.UnitDescriptiveInfoStream({});
+for await (const record of result.records) {
+  console.log(record);
+}
+```
+
+Because `records` is single-pass, each response can be iterated once. Headers captured before the first record are available on `result.headers`, and the serialized SOAP envelope that was sent upstream is on `result.requestRaw` when populated.
+
+Streaming requires the `saxes` runtime dependency. The generated app scaffold pins `saxes ^6.0.0` automatically; consumers that integrate the generated client into their own project must install it explicitly.
+
+See [Stream Configuration](configuration.md#stream-configuration) for how to opt specific operations in, and [ADR-002](decisions/002-streamable-responses.md) for the rationale and terminal-error policy.
+
 ## Attributes and Text Content
 
 When an element has both attributes and text content, use the $value convention:
@@ -134,6 +151,8 @@ Key features of the generated handlers:
 - Envelope wrapping: `buildSuccessEnvelope()` wraps the raw SOAP response in the standard `{ status, message, data, error }` envelope
 - Route file header comments include propagated operation summary and description when present
 
+Stream-configured operations generate a different handler shape: the response serialization schema is omitted, `reply.type("application/x-ndjson")` is set, and `reply.send(toNdjson(result.records))` streams records with backpressure. See the [Gateway Guide Streaming Handlers section](gateway-guide.md#streaming-handlers) for the full example and terminal-error policy.
+
 See [Gateway Guide](gateway-guide.md) for the full architecture and [CLI Reference](cli-reference.md) for generation flags.
 
 ## Operations Interface
@@ -158,4 +177,6 @@ The gateway plugin accepts any `WeatherOperations` implementation, so you can pa
 app.register(weatherGateway, { client: mock });
 ```
 
+For stream operations, mock implementations return an async-iterable `records` field. See the [Testing Guide](testing.md) for the full pattern with async generators.
+
 See [Testing Guide](testing.md) for full integration test patterns.

package/docs/migration-playbook.md

@@ -58,6 +58,39 @@ Open `openapi.json` and review the paths, request/response schemas, and descript
 
 Validate the spec with any OpenAPI tool. Built-in validation runs by default; disable with `--openapi-validate false` if you need to inspect an intermediate state.
 
+## Step 2b (Optional): Opt Into Streaming for Large Responses
+
+Skip this step unless a specific SOAP operation returns payloads too large or too slow to buffer in memory. Good candidates are batch-style operations with repeated record elements, responses that use `xs:any` wildcards to point at concrete records in a companion WSDL, and any operation where first-byte latency matters more than total duration.
+
+Author a stream config naming those operations:
+
+```json
+{
+  "operations": {
+    "MyBatchOp": {
+      "recordType": "MyRecordType",
+      "recordPath": ["MyBatchOpResponse", "Records", "Record"]
+    }
+  }
+}
+```
+
+Regenerate with `--stream-config`:
+
+```bash
+npx wsdl-tsc pipeline \
+  --wsdl-source your-service.wsdl \
+  --client-dir ./generated/client \
+  --openapi-file ./generated/openapi.json \
+  --gateway-dir ./generated/gateway \
+  --gateway-service-name your-service \
+  --gateway-version-prefix v1 \
+  --stream-config ./stream.config.json \
+  --init-app
+```
+
+The opted-in operations now return `StreamOperationResponse<RecordType>` on the client, serve `application/x-ndjson` on the gateway, and advertise the record schema in OpenAPI via `x-wsdl-tsc-stream`. Operations not listed in the config are unchanged. See [Stream Configuration](configuration.md#stream-configuration) for the full file reference and [ADR-002](decisions/002-streamable-responses.md) for the terminal-error policy.
+
 ## Step 3: Generate the REST Gateway
 
 Generate Fastify route handlers that translate JSON HTTP requests into SOAP calls.

package/docs/migration.md

@@ -16,12 +16,37 @@ These steps apply to every version upgrade:
 
 ## Version Compatibility
 
-| wsdl-tsc | Node.js | TypeScript | soap | Fastify |
-|----------|---------|------------|------|---------|
-| 0.10.x | >= 20.0 | >= 5.6 | >= 1.3 | >= 5.2 |
-| 0.9.x | >= 20.0 | >= 5.6 | >= 1.3 | >= 5.2 |
-| 0.8.x | >= 20.0 | >= 5.6 | >= 1.3 | >= 5.2 |
-| 0.7.x | >= 20.0 | >= 5.6 | >= 1.3 | N/A |
+| wsdl-tsc | Node.js | TypeScript | soap | Fastify | saxes |
+|----------|---------|------------|------|---------|-------|
+| 0.17.x | >= 20.0 | >= 6.0 | >= 1.9 | >= 5.8 | >= 6.0 |
+| 0.16.x | >= 20.0 | >= 6.0 | >= 1.9 | >= 5.8 | N/A |
+| 0.15.x | >= 20.0 | >= 6.0 | >= 1.8 | >= 5.4 | N/A |
+| 0.11.x to 0.14.x | >= 20.0 | >= 5.6 | >= 1.3 | >= 5.2 | N/A |
+| 0.10.x | >= 20.0 | >= 5.6 | >= 1.3 | >= 5.2 | N/A |
+| 0.9.x | >= 20.0 | >= 5.6 | >= 1.3 | >= 5.2 | N/A |
+| 0.8.x | >= 20.0 | >= 5.6 | >= 1.3 | >= 5.2 | N/A |
+| 0.7.x | >= 20.0 | >= 5.6 | >= 1.3 | N/A | N/A |
+
+Versions 0.11 through 0.16 are additive and non-breaking. See `CHANGELOG.md` for per-version detail rather than dedicated upgrade sections here.
+
+## Upgrading to 0.17.x from 0.16.x
+
+This upgrade adds opt-in streamable SOAP responses. No breaking changes; generated output is byte-for-byte unchanged when `--stream-config` is not provided.
+
+### What Changed in 0.17.x
+
+The CLI gains a `--stream-config <file>` flag on `compile`, `client`, and `pipeline`. Operations listed in that file emit a new client method signature returning `StreamOperationResponse<RecordType>` with `records: AsyncIterable<RecordType>`, an OpenAPI 200 response typed as `application/x-ndjson` with an `x-wsdl-tsc-stream` extension, and a Fastify route that streams NDJSON with backpressure. The compiler now retains `xs:any` wildcard particles on compiled types (previously dropped silently), enabling honest stream-candidate detection and companion-catalog shape resolution. `saxes ^6.0.0` is now a runtime dependency of the package and is pinned automatically into the generated app scaffold.
+
+### Steps to Upgrade to 0.17.x
+
+1. Update the package and regenerate; no flag or code changes are required for buffered operations
+2. If consumers integrate the generated client directly (not via the app scaffold), install `saxes ^6.0.0` as a runtime dependency before using any stream operation
+3. To opt into streaming for specific operations, author a stream-config file (see [Stream Configuration](configuration.md#stream-configuration)) and pass it via `--stream-config`
+4. Review the new generated client method signatures for any opted-in operation; consumers must use `for await (const record of result.records)` instead of awaiting the full response
+
+### Is 0.17.x Breaking?
+
+No. Without `--stream-config`, generated output is byte-for-byte unchanged. Consumers only see new surfaces when they opt in.
 
 ## Upgrading to 0.10.x from 0.9.x
 

package/docs/output-anatomy.md

@@ -43,6 +43,24 @@ Exports a TypeScript interface with the same method signatures as the client cla
 
 Contains runtime metadata and helper functions. Includes `unwrapArrayWrappers()` for bridging between SOAP array wrapper objects and flattened OpenAPI array schemas.
 
+### Stream Operations
+
+When `--stream-config` opts an operation into streaming (see [ADR-002](decisions/002-streamable-responses.md)), the client emits additional surfaces alongside the buffered ones. Buffered operations in the same client are unaffected.
+
+`operations.ts` gains a `StreamOperationResponse<RecordType>` type that stream methods return instead of the buffered `{ response, headers, responseRaw, requestRaw }` shape:
+
+```typescript
+export type StreamOperationResponse<RecordType, HeadersType = Record<string, unknown>> = {
+  records: AsyncIterable<RecordType>;
+  headers: HeadersType;
+  requestRaw?: string;
+};
+```
+
+`client.ts` gains a protected `callStream()` transport. Stream methods bypass `node-soap` (which buffers the full response before invoking its callback, as confirmed in phase-0 research) and instead POST a hand-built SOAP envelope via global `fetch`, piping the response body through a SAX-driven record parser. `node-soap` remains the transport for buffered operations.
+
+Generated imports required at runtime: `saxes` for SAX parsing. The generated app scaffold pins `saxes ^6.0.0` automatically; manual consumers must install it explicitly.
+
 ## OpenAPI Output
 
 Generated at the path specified by `--openapi-file`.
@@ -55,6 +73,29 @@ The spec includes one POST path per WSDL operation, request and response schemas
 
 OpenAPI validation runs by default using `@apidevtools/swagger-parser`. Disable with `--openapi-validate false`.
 
+### Stream Schema Extension
+
+Stream operations do not use the standard success envelope for `200` responses. The response content declares the configured stream media type (default `application/x-ndjson`) with `schema: { "type": "string" }` and an `x-wsdl-tsc-stream` extension that carries the record schema reference:
+
+```json
+{
+  "200": {
+    "description": "Successful streamed SOAP operation response",
+    "content": {
+      "application/x-ndjson": {
+        "schema": { "type": "string" },
+        "x-wsdl-tsc-stream": {
+          "format": "ndjson",
+          "itemSchema": { "$ref": "#/components/schemas/UnitDescriptiveContentType" }
+        }
+      }
+    }
+  }
+}
+```
+
+OpenAPI 3.1 cannot fully describe an NDJSON sequence as a standard JSON Schema document, so the extension makes the item schema explicit for generated gateways, documentation tools, and future SDK generators. Error responses (400, 502, and the rest) still use the normal envelope.
+
 ## Gateway Output
 
 Generated into the directory specified by `--gateway-dir`.
@@ -74,6 +115,14 @@ Generated into the directory specified by `--gateway-dir`.
 
 Each route file in `routes/` follows the same pattern: validate the JSON request body against the operation schema, call the corresponding SOAP operation via the typed client, transform the SOAP response to JSON, and return it with the appropriate response schema.
 
+### Stream Routes
+
+Stream-configured operations generate a different route shape. The Fastify response serialization schema is omitted because Fastify cannot serialize an unbounded stream with a normal JSON response schema. The handler sets `reply.type("application/x-ndjson")` and returns `reply.send(toNdjson(result.records))`. `runtime.ts` gains a `toNdjson<T>(records: AsyncIterable<T>): Readable` helper that wraps the async iterable in a backpressure-aware Node `Readable`. See the [Gateway Guide](gateway-guide.md#streaming-handlers) for the full handler example and terminal-error policy.
+
+### Generated Test Surface
+
+When `--test-dir` is combined with `--stream-config`, the generated happy-path tests for stream operations assert on the `application/x-ndjson` content-type and parse each line as a separate JSON record. Mock clients use async-generator overrides that yield records to drive those tests; see the [Testing Guide](testing.md) for the pattern.
+
 ### Plugin registration
 
 The generated plugin exports a Fastify plugin function. Register it with your Fastify app and provide a client instance (real or mock) and a route prefix.

package/docs/production.md

@@ -61,6 +61,30 @@ npx wsdl-tsc openapi --catalog-file ./build/catalog.json --openapi-file ./docs/a
 }
 ```
 
+## Streaming Operations
+
+Operations opted into streaming with `--stream-config` change the production profile of the gateway. Plan for these characteristics before rolling out.
+
+### Backpressure
+
+Records flow through `Readable.from` in the generated `runtime.ts`. The iterator's `next()` is not called until the internal buffer has room, so a slow HTTP client propagates backpressure all the way to the upstream SOAP server. A client that opens a connection and stops reading will eventually stall the upstream SOAP socket; set idle timeouts on both the gateway and the SOAP endpoint accordingly.
+
+### Connection Timeouts
+
+Stream responses can stay open for the full duration of the upstream SOAP call. The Fastify `keepAliveTimeout`, `requestTimeout`, and any reverse proxy (nginx, ALB, CloudFront) must allow the maximum expected upstream duration. Default Fastify timeouts are shorter than typical long-running SOAP batch responses.
+
+### Memory Profile
+
+Memory stays bounded regardless of payload size because records are parsed and emitted one at a time. The SAX parser buffers only within the current record element. This is the primary reason to opt into streaming: buffered operations load the full response into memory before yielding.
+
+### Observability
+
+Log the time to first record, not just the time to response completion. First-record time is the most useful SLO signal because it measures the combined latency of upstream connect, first-byte, and parser spin-up. Track it separately from total stream duration.
+
+### Terminal-Error Policy
+
+Errors raised before the first record use the normal gateway error envelope (client sees a standard JSON error). Errors raised mid-stream truncate the chunked response without a terminating zero-chunk. Consumers detect this as an incomplete HTTP response. Document this behavior for downstream API consumers so they distinguish truncation from a legitimate empty stream.
+
 ## Known Limitations
 
 ### Choice Elements
@@ -78,3 +102,11 @@ Security hints extracted from policies. Custom policies may require manual secur
 ### Array Wrapper Flattening
 
 Single-child sequences with maxOccurs>1 become array schemas. Sequences with multiple children preserve wrapper.
+
+### Stream Format Coverage
+
+Only `ndjson` is emitted today. `json-array` is reserved in the config schema but not yet implemented; opting in with `format: "json-array"` parses successfully but generates no routes for that operation.
+
+### Stream Transport Bypasses node-soap
+
+Stream operations bypass `node-soap` entirely and POST a hand-built SOAP envelope via `fetch`. Any `node-soap` middleware, interceptors, or custom security handlers that your buffered operations rely on will not apply to stream operations. Authentication headers, proxies, and TLS options must be configured on the stream transport path separately.

package/docs/start-here.md

@@ -64,6 +64,38 @@ The gateway transforms JSON HTTP requests into SOAP calls and returns JSON respo
 
 Next: [Gateway Guide](gateway-guide.md) for integration details, then [Migration Playbook](migration-playbook.md) for the full modernization workflow
 
+### I need to stream large SOAP responses
+
+Some SOAP services return payloads that are too large or too slow to buffer in memory. Opt selected operations into streaming with a small JSON config and the `--stream-config` flag. Operations not listed keep their buffered behavior, so existing output stays byte-for-byte unchanged.
+
+```bash
+npx wsdl-tsc pipeline \
+  --wsdl-source your-service.wsdl \
+  --client-dir ./generated/client \
+  --openapi-file ./generated/openapi.json \
+  --gateway-dir ./generated/gateway \
+  --gateway-service-name my-service \
+  --gateway-version-prefix v1 \
+  --stream-config ./stream.config.json
+```
+
+Minimal `stream.config.json`:
+
+```json
+{
+  "operations": {
+    "MyStreamOp": {
+      "recordType": "MyRecordType",
+      "recordPath": ["MyStreamOpResponse", "Records", "Record"]
+    }
+  }
+}
+```
+
+Stream operations return `StreamOperationResponse<RecordType>` on the client (`records: AsyncIterable<RecordType>`), emit `application/x-ndjson` on the gateway, and advertise the record schema in OpenAPI via the `x-wsdl-tsc-stream` extension.
+
+Next: [ADR-002: Streamable Responses](decisions/002-streamable-responses.md) for rationale and terminal-error policy, then [Stream Configuration](configuration.md#stream-configuration) for the full file reference.
+
 ## What NOT to Expect
 
 This package does not replace a full API management platform. It does not provide rate limiting, policy enforcement, or multi-language SDK generation. It generates code; it does not run a proxy or manage deployments.
@@ -80,3 +112,4 @@ For more on scope boundaries, see the "When NOT to Use This" section of the [REA
 | Plan a full SOAP-to-REST migration | [Migration Playbook](migration-playbook.md) |
 | Set up testing for generated code | [Testing Guide](testing.md) |
 | Review all CLI flags | [CLI Reference](cli-reference.md) |
+| Opt specific operations into NDJSON streaming | [ADR-002](decisions/002-streamable-responses.md) and [Stream Configuration](configuration.md#stream-configuration) |

package/docs/supported-patterns.md

@@ -19,6 +19,8 @@ These patterns are handled end-to-end: WSDL parsing, TypeScript type generation,
 - Circular type references detected and broken with minimal stub types
 - Multiple WSDL ports and bindings; the first SOAP binding is selected, all ports are documented in service metadata
 - SOAP 1.1 and SOAP 1.2 binding detection
+- Streamable SOAP responses, opt-in per operation via `--stream-config` (ADR-002): client exposes `AsyncIterable<RecordType>`, gateway emits NDJSON with backpressure, OpenAPI advertises the record schema via `x-wsdl-tsc-stream`
+- `xs:any` wildcard particles retained on compiled types (they used to be dropped silently) — enables honest stream-candidate detection and companion-catalog shape resolution
 
 ### Named simple types and same-name elements