@t4h.framework/transforms 0.2.11 → 0.2.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/.ai/skills/framework-transforms/SKILL.md +53 -104
  2. package/CHANGELOG.md +6 -0
  3. package/README.md +17 -30
  4. package/package.json +3 -3
package/.ai/skills/framework-transforms/SKILL.md CHANGED
@@ -1,122 +1,71 @@
1
1
  ---
2
2
  name: framework-transforms
3
3
  description: >-
4
- Guides correct use of @t4h.framework/transforms in workflows: converting a
5
- JSON document (an array of records, possibly nested inside an envelope) into
6
- NDJSON so it can be fanned out with `split` from @t4h.framework/fs. Use
7
- whenever a workflow receives a JSON batch file that is NOT NDJSON — an API
8
- response or uploaded file shaped like `{ "a": { "b": [ ...records ] } }` —
9
- and each record must become its own workflow run, or when the user mentions
10
- converting JSON to NDJSON, splitting a JSON array, a records envelope, or
11
- the symbols `json`, `Json`, `toNDJson`, `JsonToNDJsonActivity`. Prefer this
12
- skill over parsing JSON in workflow code or hand-rolling delimiters over
4
+ Guides the use of @t4h.framework/transforms in T4H Framework workflows:
5
+ json(binary, { path }).toNDJson() changes a JSON document (an array of
6
+ records, possibly inside an envelope such as { "a": { "b": [ ... ] } }) into
7
+ NDJSON, so that split from @t4h.framework/fs starts one workflow for each
8
+ record. Use when a workflow receives a JSON batch file that is not NDJSON
9
+ (an API response or an uploaded file), when each record must become its own
10
+ workflow run, or when the user mentions JSON to NDJSON, splitting a JSON
11
+ array, a records envelope, json, Json, toNDJson or JsonToNDJsonActivity.
12
+ Use it instead of JSON.parse in workflow code or a hand-made delimiter over
13
13
  JSON text.
14
14
  ---
15
15
 
16
16
  # @t4h.framework/transforms
17
17
 
18
- Format conversions for workflow files. Package path:
19
- `framework/packages/transforms`. Peer dependencies: `@t4h.framework/core`,
20
- `@t4h.framework/fs`.
18
+ `json(binary, options)` only keeps the source and the options. The work runs
19
+ when you call `.toNDJson()`, which is an activity and returns the `Binary` of
20
+ a new NDJSON file.
21
21
 
22
- ## Import surface
22
+ ## Fan out a JSON batch file
23
23
 
24
- ```typescript
25
- import { json, Json, JsonToNDJsonActivity } from '@t4h.framework/transforms'
26
- ```
27
-
28
- `json(binary, options)` is the entry point: it captures the source document
29
- and returns a `Json` value whose `.toX()` methods run the actual transform —
30
- today `toNDJson()`, each future target (`toCsv`, ...) being one more method.
31
- Only the `.toX()` call does work; `json(...)` itself is synchronous and free.
32
-
33
- ## Workflow usage
34
-
35
- The canonical pipeline — download a JSON batch file, normalize it to NDJSON,
36
- fan it out one workflow per record:
24
+ 1. Get the source as a `Binary` (for example `response.body.binary` from
25
+ `@t4h.framework/http`).
26
+ 2. Call `json(binary, { path }).toNDJson()`. `path` is a dot path to the array.
27
+ Omit it when the root is the array.
28
+ 3. Give the result to `split` with `Split.line()`.
29
+ 4. In the child workflow, the input is one JSON line: use `Type.String()` as
30
+ the schema and `JSON.parse` it.
37
31
 
38
32
  ```typescript
39
33
  import { split, Split } from '@t4h.framework/fs'
40
- import { request } from '@t4h.framework/http'
41
34
  import { json } from '@t4h.framework/transforms'
42
35
 
43
- const main = new Workflow({ id: 'main', schema: inputSchema }, async ({ url }) => {
44
- const response = await request({ url })
45
-
46
- // { "registros": { "registro": [ {...}, {...} ] } }
47
- const ndjson = await json(response.body.binary, {
48
- path: 'registros.registro',
49
- }).toNDJson()
36
+ const ndjson = await json(response.body.binary, {
37
+ path: 'registros.registro',
38
+ }).toNDJson()
50
39
 
51
- return await split(ndjson, processRecord, { delimiter: Split.line() })
52
- })
53
- ```
54
-
55
- Each child workflow receives one record as a raw JSON line (`Type.String()`
56
- input, `JSON.parse` it) — see the **framework-fs-split** skill for the
57
- `split`/`join` side.
58
-
59
- ## `json(binary, options).toNDJson()`
60
-
61
- ```typescript
62
- function json(binary: Binary, options?: JsonOptions): Json
63
- Json.prototype.toNDJson(): Promise<Binary>
40
+ return await split(ndjson, processRecord, { delimiter: Split.line() })
64
41
  ```
65
42
 
66
- `toNDJson()` rewrites the document as NDJSON — one line per element of the
67
- target array — and resolves with the `Binary` of the new file. Only that
68
- reference reaches workflow history; the content never does.
69
-
70
- | Option | Default | Behavior |
71
- | ---------- | ------- | ---------------------------------------------------------------------------------------------- |
72
- | `path` | — | Dot path to the array of records (`'registros.registro'`). Omit when the document root is the array. |
73
- | `encoding` | `utf8` | Encoding used to decode the source binary before parsing. |
74
-
75
- Behaviors you can rely on:
76
-
77
- - **Formatting-agnostic.** Minified, pretty-printed, arbitrary whitespace and
78
- BOM-prefixed documents all produce identical output. Values containing
79
- `"},{"`, escaped quotes, or nested structures are preserved exactly —
80
- records are re-serialized, never text-sliced.
81
- - **Any element type.** Array elements may be objects, arrays or scalars;
82
- each becomes one `JSON.stringify`'d line. An empty array yields an empty
83
- (zero-line) file, which `split` resolves as an empty batch.
84
- - **Fail-fast.** `toNDJson()` rejects on a bad `path` or invalid JSON, with
85
- no partial output.
86
-
87
- ## Limits
88
-
89
- The source document is parsed **in memory** inside the activity — fits files
90
- up to tens of megabytes comfortably. For gigabyte-scale JSON a streaming
91
- scanner would be needed; this package does not provide one today.
92
-
93
- ## Runtime requirements
94
-
95
- - Claim provider: `FileSystemClaim` (from `@t4h.framework/fs`).
96
- - Production whitelist: register `JsonToNDJsonActivity` via
97
- `runtime.addActivity`.
98
-
99
- ## Anti-patterns
100
-
101
- - **Parsing the batch file in workflow code.** `response.body.json()` is for
102
- small API payloads: on a batch file it re-reads and re-parses the whole
103
- document on every replay and holds it in workflow memory. Normalize with
104
- `toNDJson()` and fan out with `split` instead.
105
- - **Hand-rolling delimiters over JSON text.** `Split.char('},{')` over a JSON
106
- document breaks on formatting changes and on values containing structural
107
- characters. JSON is parsed, never text-split.
108
- - **Expecting records in the workflow.** The transform resolves with a
109
- `Binary` reference; records only materialize inside each child workflow run
110
- started by `split`.
111
- - **Iterating records and calling `startWorkflow` per item.** That pushes N
112
- entries into parent history; `split` keeps it O(1). See
113
- **framework-fs-split**.
114
-
115
- ## Related skills
116
-
117
- - `framework-fs-split` — `split`/`join`/`read`: fanning the NDJSON out into a
118
- batch of workflows and folding results back.
119
- - `framework-http` — downloading the source file; `response.body.binary` is
120
- the natural input to `json(...)`.
121
- - `framework-workflow-testing` — the harness for testing workflows that use
122
- this pipeline.
43
+ ## Traps
44
+
45
+ - Do not `JSON.parse` a batch file or call `response.body.json()` on it in
46
+ workflow code. The workflow keeps the full document in memory and parses it
47
+ again on each replay.
48
+ - Do not cut JSON text with `Split.char('},{')`. It breaks when the format
49
+ changes or when a value contains `},{`.
50
+ - The workflow gets a `Binary` reference, not records. Records exist only in
51
+ the child workflows that `split` starts.
52
+ - Do not loop over records and start one workflow for each. Use `split`.
53
+ - `toNDJson()` rejects with `TypeError` when `path` does not resolve or does
54
+ not point to an array, and with `SyntaxError` when the JSON is not valid.
55
+ - An empty array gives an empty file. `split` gives an empty batch.
56
+ - The runtime must give a `FileSystemClaim` (from `@t4h.framework/fs`),
57
+ because the activity writes the output file through it.
58
+
59
+ ## Test
60
+
61
+ Test the workflow with `TestWorkflowEnvironment` from
62
+ `@t4h.framework/core/testing`. Give a `FileSystemClaim` with `mockClaim`.
63
+ `toNDJson()` reads the source with `binary.stream()` and writes the output with
64
+ `fs.write`.
65
+
66
+ ## Reference
67
+
68
+ - Guide, options and guarantees:
69
+ https://tech4humans-brasil.github.io/framework/transforms/
70
+ - `split`: https://tech4humans-brasil.github.io/framework/fs/split
71
+ - Types: `node_modules/@t4h.framework/transforms/dist/json.d.ts`
package/CHANGELOG.md CHANGED
@@ -1,5 +1,11 @@
1
1
  # @t4h.framework/transforms
2
2
 
3
+ ## 0.2.12
4
+
5
+ ### Patch Changes
6
+
7
+ - [#257](https://github.com/tech4humans-brasil/framework/pull/257) [`586bfb4`](https://github.com/tech4humans-brasil/framework/commit/586bfb4663f4a478527ee0c8db360603f034d615) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - The transforms README is short and points to the docs site. The README and the skill no longer say that the source document is parsed in memory, because the transform reads it as a stream.
8
+
3
9
  ## 0.2.11
4
10
 
5
11
  ### Patch Changes
package/README.md CHANGED
@@ -1,43 +1,30 @@
1
1
  # @t4h.framework/transforms
2
2
 
3
- Data transform activities for workflows.
3
+ Durable data transforms for workflows. Today it has one transform:
4
+ `json(...).toNDJson()` changes a JSON document into NDJSON (one record on each
5
+ line), so that `split` from `@t4h.framework/fs` can start one workflow for
6
+ each record.
4
7
 
5
- ## JSON to NDJSON
8
+ ## Installation
6
9
 
7
- Rewrites a JSON document into NDJSON — one record per line — so it can be
8
- consumed by line-oriented activities such as `split` from
9
- `@t4h.framework/fs`. The transform is formatting-agnostic: minified,
10
- pretty-printed and BOM-prefixed documents all produce the same output.
10
+ ```bash
11
+ npm install @t4h.framework/transforms
12
+ ```
13
+
14
+ Peer dependencies: `@t4h.framework/core`, `@t4h.framework/fs`.
15
+
16
+ ## Usage
11
17
 
12
18
  ```typescript
13
19
  import { split, Split } from '@t4h.framework/fs'
14
20
  import { json } from '@t4h.framework/transforms'
15
21
 
16
- // { "registros": { "registro": [{ ... }, { ... }] } }
17
- const ndjson = await json(response.body.binary, {
18
- path: 'registros.registro',
19
- }).toNDJson()
22
+ // file: { "records": { "record": [{ ... }, { ... }] } }
23
+ const ndjson = await json(file, { path: 'records.record' }).toNDJson()
20
24
 
21
- const batch = await split(ndjson, registroWorkflow, {
22
- delimiter: Split.line(),
23
- })
25
+ await split(ndjson, processRecord, { delimiter: Split.line() })
24
26
  ```
25
27
 
26
- Omit `path` when the document root is the array itself. `json` captures the
27
- source document and its options; each `.toX()` method runs a dedicated
28
- transform activity, leaving room for other targets (`toCsv`, ...) later.
29
-
30
- ### Options
28
+ ## Documentation
31
29
 
32
- | Option | Default | Description |
33
- | ---------- | ------- | -------------------------------------------------- |
34
- | `path` | — | Dot path to the array of records (`a.b.c`) |
35
- | `encoding` | `utf8` | Encoding used to decode the source binary |
36
-
37
- ### Limits
38
-
39
- The source document is parsed in memory, so this activity fits files up to
40
- tens of megabytes. Only the resulting `Binary` reference crosses the
41
- activity boundary — the content itself never enters the workflow history.
42
-
43
- Peer dependencies: `@t4h.framework/core`, `@t4h.framework/fs`.
30
+ https://tech4humans-brasil.github.io/framework/transforms/
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@t4h.framework/transforms",
3
- "version": "0.2.11",
3
+ "version": "0.2.12",
4
4
  "description": "Data transform activities for the T4H Framework",
5
5
  "homepage": "https://github.com/tech4humans-brasil/framework/tree/main/packages/transforms",
6
6
  "bugs": "https://github.com/tech4humans-brasil/framework/issues",
@@ -33,8 +33,8 @@
33
33
  "test:watch": "vitest"
34
34
  },
35
35
  "devDependencies": {
36
- "@t4h.framework/core": "^0.20.0",
37
- "@t4h.framework/fs": "^0.13.1",
36
+ "@t4h.framework/core": "^0.21.0",
37
+ "@t4h.framework/fs": "^0.14.0",
38
38
  "typescript": "^5.9.3",
39
39
  "vitest": "^4.0.18"
40
40
  },