map-transform 1.6.0-rc.7 → 1.6.0-rc.9

package/NEXT.md

@@ -0,0 +1,123 @@
+# Proposals for MapTransform v2 (`next`)
+
+A list of suggested changes to the `next` mutation language, to be taken on one
+at a time. Each section is self-contained: what the problem is, what the change
+is, and which verified findings it addresses. Findings marked _(verified)_ were
+reproduced by running `mapTransform` from `src/mapTransform.ts` on small inputs.
+
+## Background: how the engine models context and reverse today
+
+**Context is structural.** Only drilling down adds a level, operations are
+opaque, set steps don't move, `^^` is depth zero of the current tree and `^^^`
+the original input. In reverse, a target context mirrors the context, so parent
+and root paths set on the parent or root target
+(`src/run/resolveParentSets.ts`). See "Parent and root paths" in the README.
+
+**Reverse is a syntactic reversal.** `adjustPipelineToDirection`
+(`src/run/index.ts`) reverses the prepped steps. Pipelines whose first step is
+not a path are silently prefixed with `.` or given a reverse plug
+(`src/prep/mutation.ts`).
+
+**`flip` is runtime state.** `isRev = xor(rev, flip)`. `$apply` clears it, `$if`
+conditions force forward, `$array` forces a direction per branch, `$direction`
+ignores it, and `lookup` special-cases it.
+
+## Reverse by rule, not by heuristic
+
+### Problem
+
+- `$flip: false` is dropped at prep time, so a nested object cannot un-flip.
+- `$direction` ignores `flip`, but paths honor it.
+- Pipelines that do not start with a path are silently plugged or prefixed with
+  `.` in reverse (`addStepWhenNoGetStep` in `src/prep/mutation.ts`).
+
+### Change
+
+- Add an explicit two-sided form for the hard cases, JSON-friendly:
+  `{ "$fwd": [...], "$rev": [...] }`. This replaces legacy `divide`, `fwd`, and
+  `rev`, and most uses of `$direction` on two adjacent steps.
+- `$alt` in reverse: set with the first pipeline; if the value is a nonvalue,
+  use the last `$default`. Losing pipelines are never run as setters.
+- `$flip` becomes a prep-time rewrite: swap key and value, reverse each
+  pipeline, toggle `>`. `flip` leaves `State`. Transformers that need to know,
+  such as `lookup`, get a flag on the transform step at prep time.
+
+## One modifier for "the other direction"
+
+### Problem
+
+Direction pairs are spelled five ways: `$concat`/`$concatRev`,
+`merge`/`mergeRev`, `$lookup`/`$lookdown`, `explode`/`implode`, and `$array`
+with `$flip`.
+
+### Change
+
+Allow `$flip: true` on any operation object, and drop `$concatRev`, `mergeRev`,
+`$lookdown`, `implode`, and the special `$flip` handling on `$array`. Keep
+`explode`/`implode` names only if they read better than `explode` + `$flip`.
+
+## Constants vs defaults
+
+### Problem
+
+Two concepts have five spellings: `$value`, `$fixed`, `$value` with
+`fixed: true`, and the `value` and `fixed` transformers. `$value` being
+suppressed by `noDefaults` is surprising when it is used as a constant, which is
+why `fixed` exists.
+
+### Change
+
+- `$value: x` is a constant and is never suppressed.
+- `$default: x` is a fallback and is suppressed by `noDefaults`.
+- Drop `$fixed`, `fixed: true`, and the `fixed` transformer.
+
+`$alt: ['fullname', { $default: 'Anon' }]` then reads as intended.
+
+## Reverse-only props without slashed keys
+
+### Problem
+
+Reverse-only props use the lexical trick `name/1`, with `\/` escaping for
+literal slashes.
+
+### Change
+
+A `$rev: { name: [...] }` block on a mutation object holds props that only apply
+in reverse, merged over the main props. `$fwd: { ... }` for symmetry. `name/1`
+and its escaping go away.
+
+## Fail loudly at prep time
+
+### Problem
+
+Bad input is accepted silently:
+
+- Unknown `$direction` values, including typos, mean "both" _(verified)_.
+- `[1:3]` and `[abc]` go through `parseInt` and become `[1]` and `[NaN]`
+  _(verified)_.
+- Unknown `$`-keys on a mutation object, boolean values, `$alt: []`, non-array
+  `$nonvalues`, and unknown `compare` operators are dropped or ignored without a
+  message.
+- Setting to parent or root degrades silently instead of failing.
+
+### Change
+
+Throw at prep time on all of the above. Prep runs once, so this costs nothing at
+run time.
+
+## Trim compatibility leftovers
+
+### Problem
+
+Kept for compatibility but pure noise in v2: the `<` get prefix, the
+`$undefined` alias of `$nonvalues`, `fwdAlias`/`revAlias`, `$value` accepting a
+function, and `$if` using unprefixed `then`/`else` while everything else is
+`$`-prefixed.
+
+`$alwaysApply` is not inherited, so it must be repeated on every nested mutation
+object.
+
+### Change
+
+Drop the leftovers listed above. Make `$alwaysApply` inherited by nested
+mutation objects.

package/README.md

@@ -176,26 +176,16 @@ await mapTransform(def)(source, { target }) // We're reusing `def` from the prev
 ```
 
 Finally, if you're using the same `options` across several `mapTransform` calls,
-you may gain some optimization by preparing the options up front. Do this with
-the exported `prepareOptions` function like so:
+pass the same options object each time. Calls with the same options object share
+prepared pipelines automatically, so a pipeline is only prepared once. Changing
+pipeline definitions on the options object after it has been used is not
+supported.
 
-```javascript
-import mapTransform, { prepareOptions, transform } from 'map-transform'
-
-// ...
-
-const preppedOptions = prepareOptions(options)
-await mapTransform(def1, preppedOptions)(source)
-
-// ...
-
-await mapTransform(def2, preppedOptions)(source)
-```
-
-Most of the preparations are still done just-in-time, but when it's done once,
-it won't happen again on the next call. `prepareOptions` is idempotent, so
-passing already prepared options to `mapTransform` -- or calling
-`prepareOptions` on them again -- costs nothing.
+`map-transform/next` works the same way. Like in the legacy API, the default
+export is async, and it is also exported as `mapTransformAsync()`, with
+`mapTransformSync()` as its sync counterpart. Note that `mapTransformSync()` and
+the async `mapTransformAsync()` cache their prepared pipelines separately, so
+mixing both on the same options object will prepare each pipeline twice.
 
 > [!NOTE] We are preparing for an upcoming 2.0 version, which will include
 > breaking changes.
@@ -221,6 +211,27 @@ passing already prepared options to `mapTransform` -- or calling
 > new `mapTransformSync` and `mapTransformAsync`, but there might still be gaps
 > in the documentation here.
 
+#### Deviations from v1 behaviour in `next`
+
+The following behaviours of `mapTransformSync` and `mapTransformAsync` differ
+from the default export, on purpose:
+
+- The root form without a dot after a single caret, `^path`, is removed. Write
+  `^^.path`.
+- `$alt` no longer changes the levels for parent paths after it. Before, a
+  `$alt` that found no value added a level, so the next step needed `^.^` to
+  reach the parent. After, the next step sees the same levels as before the
+  `$alt`, so `^` is enough.
+- Parent and root paths in reverse set on the parent or root level of the target
+  instead of on the current level. Before, `section: '^.^.meta.section'` inside
+  an iteration reversed to `items[].section`, and a root path in reverse set
+  nothing. After, the same definition sets `meta.section` on the object holding
+  `items`. When the parent level does not exist, the value is dropped.
+- Mutation objects iterate arrays by default, just like get paths map over
+  arrays. Before, you had to set `$iterate: true`, and without it the entire
+  array was passed to the mutation object, giving one object with an array of
+  values on every property. Set `$iterate: false` to get that behaviour back.
+
 ### The mutation object
 
 Think of a mutation object as a description of the object structure you want.
@@ -267,6 +278,10 @@ mutation object. If you don't, the entire array will be passed to the mutation
 object. This will in most cases have the (propably unwanted) effect of returning
 one object with an array of values on every property.
 
+> [!NOTE] In the next version, mutation objects iterate arrays by default, so
+> `$iterate: true` is not needed. Set `$iterate: false` to pass the entire array
+> to the mutation object instead.
+
 ```javascript
 const def = {
   $iterate: true,
@@ -307,8 +322,9 @@ as its value, this mutation object will be iterated by default (no need to set
 the `$iterate` property). This does not happen to pipelines, paths, or
 operations.
 
-> [!NOTE] This automatic iteration will disappear in v2.0 and is also removed in
-> the new `mapTransformSync` and `mapTransformAsync` exports.
+> [!NOTE] This changes slightly in the next version, as mutation objects iterate
+> by default. A key with the `[]` suffix will still be iterated, but not because
+> of that suffix.
 
 #### Values on the mutation object
 
@@ -395,6 +411,25 @@ This will essentially make MapTransform treat `null` the same way as
 You could in principle include any primitive value in `nonvalues` and it will be
 treated as `undefined`, e.g. an empty string or the number `0`.
 
+You may also set `$nonvalues` on an operation or mutation object, to use another
+set of non-values for that operation and any pipelines within it. This overrides
+the `nonvalues` option for that part of the definition. `$undefined` is an alias
+of `$nonvalues`, and when both are set, `$nonvalues` is used. As a definition
+may be written in JSON, where `undefined` cannot be expressed, the value
+`'**undefined**'` is treated as `undefined` in these lists:
+
+```javascript
+const def = {
+  // Treat an empty string as `undefined` when picking a default here
+  name: [
+    {
+      $alt: ['data.name', { $value: 'Anonymous' }],
+      $nonvalues: ['**undefined**', ''],
+    },
+  ],
+}
+```
+
 A mutation object will be skipped in its entirety when it encounters a non-value
 in the pipeline. This is not always wanted, e.g. when the mutation object has
 defaults that we want to set in the mutation object. In these cases, you may set
@@ -459,8 +494,11 @@ const def = {
 
 The `$modify: true` notation is actually an alias for `$modify: '.'`.
 
-This is the way to set it for reverse direction (this is the exact same as the
-example above, just reversed):
+`$modify` works like a path segment that merges instead of replacing, and it
+takes effect on the side of the prop that is being set. When it is in a key, it
+modifies going forward. When it is in a value, it modifies in reverse. This is
+the way to set it for reverse direction (this is the exact same as the example
+above, just reversed):
 
 ```javascript
 const def = {
@@ -479,6 +517,18 @@ const def = {
 }
 ```
 
+> [!NOTE] In the next version, you may put `$modify` on both sides of a prop to
+> modify in both directions. `$modify: '$modify'` modifies the pipeline value
+> both going forward and in reverse, and `$modify: 'response.$modify'` does the
+> same with the object at `response`:
+>
+> ```javascript
+> const def = {
+>   $modify: 'response.$modify',
+>   data: 'response.data.deeply.placed.items',
+> }
+> ```
+
 ### Transform pipelines
 
 The idea of the transform pipeline, or "pipeline" for short, is that you
@@ -544,6 +594,10 @@ specified. In the example above, we have set `$iterate: true` on the mutation
 object, to signal that we want the mutation to be applied to each item of any
 array. See also [the `iterate` operation](#iteratepipeline-operation) for more.
 
+> [!NOTE] In the next version, mutation objects will iterate by default, so to
+> get the behavior described as default for mutation objects above, you will
+> have to set `$iterate: false`.
+
 > Editors note: We should think through how we use the word "pipeline", as it is
 > sometimes ment to refer to an array of operations that a value may be
 > transformed through, other times any operation that could have been a part of
@@ -654,41 +708,68 @@ In the example with the `content` path, you may access the `id` with the path
 much the same way as `../` in file paths on any computer. You may go up several
 levels with e.g. `^.^.^.prop` (not applicable to our example).
 
-In an iteration you need to remember that the array counts as one level, so if
-iterating the `tags[]` array from our example, you would have to use the path
-`^.^.id` to get to the id. You could also use `^.[0]` to get the first item in
-the array you're iterating.
-
-> Editors note: We should probably have more examples here, especially on what
-> happens in a pipeline where we go up and down.
-
-The root notation follows the same logic, but will always go to the base level
-of the data structure, regardless of how many levels down you have moved. Roots
-are specified with double carets, so the path `^^.id` will get the id from our
-data from anywhere in the data structure, be it in `content` or when iterating
-through `tags[]`.
-
-There is a gotcha here, for both parent and root paths, relating to what data
-you're moving up in, as you are mutating the data as you move "down". The short
-answer is that you're moving in the mutated data – if you are mutating it. Say
-you have a pipeline where you first mutate the data on the level you're at with
-a mutation object. The following steps in the pipeline will relate to the
-mutated object, and if you move into the mutated data with a path, and then go
-up again with a parent path, you are moving up in the mutated data. But if you
-instead just move into the data without mutating it, moving up with a parent
-path will give you the original data – there's nothing else, as you have not
-mutated it.
-
-This is probably as expected, but the confusing part comes into play when there
-is some "distance" between mutating the data and referencing it with a root or
-parent path, especially the root. Say you first mutate the root level with a
-mutation object, then move on with other operations, and then at some point,
-further down in the data, you want to reference something at the root level in
-_the original data_. This is where the **original root** notation comes in.
-
-Use triple carets (`^^^`) to always reference the original, untransformed source
-data, regardless of any prior transformations in the pipeline. This is useful in
-multi-step pipelines where earlier steps may have transformed the root:
+The number of carets you need follows from the structure of the definition
+alone, by these rules:
+
+1. **Only drilling down adds a level.** A key segment like `content` adds one
+   level, and so does an index like `[0]`. Entering an item in an array adds one
+   level too, whether you get there by iterating a path with `[]`, by a mutation
+   object iterating an array, or by an `$iterate` operation. A bare `[]` that
+   only makes sure you have an array, adds nothing. So when iterating the
+   `tags[]` array from our example above, you would use `^.^.id` to get to the
+   id: one caret to the array and one more to the object holding it. You could
+   also use `^.[0]` to get the first item in the array you're iterating.
+2. **Operations are opaque.** After a `$transform`, a mutation object, `$alt`,
+   `$if`, `$filter`, `$apply`, or `$array`, the levels are what they were before
+   the operation, no matter what the operation did on the inside. Inside an
+   operation, you inherit the levels from where the operation is used.
+3. **Set paths don't move.** A `>prop` step does not add a level, so `^` after
+   it is the same as `^` before it.
+4. **`^^` is the top of the tree you're in, `^^^` is the original data.** The
+   root notation `^^.id` goes to the base level of the data structure,
+   regardless of how many levels down you have moved.
+5. **In reverse, parent and root paths set on the parent or root level.** See
+   [Setting on parent and root](#setting-on-parent-and-root) below.
+
+The parents you move up in, are the data you moved down in. When you first
+mutate the data with a mutation object and then move into the result with a
+path, going up again with a parent path takes you up in the mutated data:
+
+```javascript
+const def = [
+  { id: 'key', item: { title: 'content.heading' } },
+  { title: 'item.title', id: ['item.title', '^.^.id'] },
+]
+// From { key: 'ent1', content: { heading: 'The heading' } }
+// we get { title: 'The heading', id: 'ent1' }
+```
+
+Here, `item.title` in the second mutation object moves two levels down into the
+object from the first, and `^.^.id` goes back up to the mutated object and gets
+the `id` we set there. Note that within a mutation object, all pipelines start
+from the same object, so the pipeline for `id` is not affected by the pipeline
+for `title`.
+
+Rule 2 means you don't have to know what an operation did to get the levels
+right after it. In this example, `$alt` may find the article on different
+depths, but the mutation object after it sees the same levels either way:
+
+```javascript
+const def = [
+  'items[]',
+  {
+    $iterate: true,
+    article: [
+      { $alt: ['content', 'original.content'] },
+      { title: 'title', section: '^.^.meta.section' },
+    ],
+  },
+]
+// From { meta: { section: 'news' }, items: [{ content: { title: 'A' } }] }
+// we get [{ article: { title: 'A', section: 'news' } }]
+```
+
+Rule 4 in a multi-step pipeline, where the first step mutates the root:
 
 ```javascript
 const def = [
@@ -703,22 +784,43 @@ const def = [
 ]
 ```
 
-This is especially useful when you write a named pipeline and reference it with
-an `$apply` operation, as you have no knowledge in the named pipeline about any
-mutation of the root outside it. Using `^^^` guarantees access to the original
-source data in all cases.
+The same holds inside a named pipeline applied with `$apply`: `^^` is the top of
+the tree the pipeline is applied in, mutated or not, and `^^^` is the original
+source data.
 
-Note also, that if you are within a mutation object and move down into the data,
-the data above where you are isn't transformed yet. As long as you are within a
-mutation object, you're referencing the data as it was before entering that
-mutation object. But as soon as you move to the next step in the pipeline –
-after the mutation object, you are left with the results of that mutation
-object.
+##### Setting on parent and root
 
-> Editor's note: We need examples.
+When you go in reverse, a parent or root path becomes a set path, and it sets on
+the parent or root level of the target, with the same number of carets as going
+forward. In this example, `section` is read from two levels up going forward,
+and in reverse it is set two levels up: on the object holding `items`.
 
-> [!NOTE] Setting on parent and root paths is currently not supported, but may
-> be in the future.
+```javascript
+const def = [
+  'items[]',
+  { $iterate: true, id: 'key', section: '^.^.meta.section' },
+]
+// Forward, from { meta: { section: 'news' }, items: [{ key: 'ent1' }] }
+// we get [{ id: 'ent1', section: 'news' }]
+// In reverse, from [{ id: 'ent1', section: 'news' }]
+// we get { meta: { section: 'news' }, items: [{ key: 'ent1' }] }
+```
+
+A root path sets on the root level of the target in the same way, so
+`'^^.meta.section'` in the example above would give the same result in reverse.
+When you set on a parent level, nothing is set on the current level, so `id` is
+the only prop on the items here.
+
+Set paths may also use parent and root notation going forward, with a `>` prefix
+or as a key on a mutation object. `{ '^.^.meta.section': 'section' }` sets
+`meta.section` two levels up in the target.
+
+When there is no parent level to set on, e.g. when a mutation object with
+`section: '^.section'` is run in reverse at the top level, the value is dropped.
+
+A parent path _within_ a path is different from these cases. The path
+`item.^.count` moves down into `item` and up again, and is the same as `count`.
+In reverse, it sets `count`, not `item.count`.
 
 #### Setting on a path
 
@@ -927,6 +1029,12 @@ would be a truly pure function.
 > call the outer function yourself with any relevant props. This option will be
 > removed in v2.0, but is still available to the regular MapTransform function.
 
+Transformers may be asynchronous, but only when you run the transformation with
+`mapTransformAsync`. When you run it with `mapTransformSync` and a transformer
+returns a promise, MapTransform will throw, telling you the id of the offending
+transformer. This also applies to transformers used by other operations, like
+`filter`.
+
 #### `filter` operation
 
 The `filter` operation will use a transformer function or a pipeline to decide
@@ -1087,6 +1195,9 @@ const def = [
 For mutation objects, you may set `$iterate: true` to apply the mutation to each
 item in an array.
 
+> [!NOTE] In the next version, mutation objects do this by default, and
+> `$iterate: false` turns it off.
+
 When you need to iterate a pipeline or an operation that is not a mutation
 object or an operation that supports the `$iterate` flag direction, you may wrap
 it in an `$iterate` operation object like this:
@@ -1238,6 +1349,10 @@ don't think too much about it. See
 [the `get` and `set` operations](#getpath-and-setpath-operation) for more on how
 `get` works in reverse.
 
+When the value is `undefined` in reverse, the alternative pipelines are used to
+look for a default value. Only pipelines without set paths (e.g. a `$value`
+pipeline) are used for this, so put the default in its own pipeline.
+
 To apply the `alt` operation to each item in an array, set `$iterate: true` on
 the operation object. You may also set `$direction: 'fwd'` or
 `$direction: 'rev'` to limit it to one direction only.
@@ -1829,6 +1944,10 @@ except that the value set with `fixed` will be included even when
 `state.noDefaults` is `true`. Use `value` for default values, and `fixed` for
 values that should be set no matter what.
 
+Like the `value` transformer, `fixed` has its own short-hand operation object
+notation, so `{ $transform: 'fixed', value: 'customer' }` may be written
+`{ $fixed: 'customer' }`.
+
 #### `flatten` transformer
 
 Will flatten an array in the pipeline. The default is to flatten one layer deep,