@pdtf/schemas 3.6.0-dev.19 → 3.6.0-dev.21

package/docs/v4-mapping-and-recomposition.md

@@ -0,0 +1,416 @@
+# V4 mapping and recomposition
+
+PDTF 1.x verified claims are JSON Pointers against the V3 transaction schema.
+PDTF 2.0 credentials are per-entity documents. The V4 generator
+(`src/utils/generateV4Schemas.js`) already encodes every rule needed to move
+between the two — but until now it emitted only schemas, so any consumer
+wanting to map a claim onto a credential (or back) had to re-implement those
+rules by hand and then keep them in step.
+
+This adds three things, all produced by the same generation run:
+
+| Artefact | What it is |
+| --- | --- |
+| `src/schemas/v4/mapping.json` | Generated manifest: V3 pointer → entity + pointer, and the inverse |
+| `src/utils/v4.js` | `decompose` / `recompose` / `resolveV3Pointer`, driven entirely by the manifest |
+| `src/tests/v4/roundTrip.test.js` | V3 → entities → V3 equality, with documented exceptions |
+
+Regenerate everything with:
+
+```bash
+npm run generate:v4
+```
+
+The generator remains read-only against V3.
+
+## Using it
+
+```js
+const {
+  decomposeToV4,
+  recomposeFromV4,
+  resolveV3Pointer,
+  v4Mapping,
+} = require("@pdtf/schemas");
+
+const entities = decomposeToV4(v3Transaction);
+// { Property, Title: [...], Transaction, Person: [...] }
+
+const v3Again = recomposeFromV4(entities);
+
+// Map a verified-claim pointer onto the credential that now carries it
+resolveV3Pointer("/propertyPack/ownership/ownershipsToBeTransferred/1/ownershipType");
+// { entity: "Title", entityPointer: "/ownershipType", rule: "title.ownershipsToBeTransferred" }
+```
+
+Pass `idFactory` to `decomposeToV4` to mint real URNs/DIDs; the default is
+deterministic and derived from `transactionId`, so repeated runs are stable and
+diffable. A supplied factory may be **partial** — anything it does not implement
+falls back to the default, so adding an entity type here cannot break a caller's
+factory on upgrade.
+
+## The manifest
+
+Rules are **prefix rewrites**, not a leaf enumeration. A V3 pointer is resolved
+by longest-prefix match against `rule.v3Pointer` (with `{index}` matching an
+array index), which means pointers far deeper than anything listed — the kind
+claims actually carry — resolve correctly, and the file does not churn every
+time a field is added to V3.
+
+`mapping.source` records exactly which V3 input produced the output:
+
+```json
+{
+  "packageVersion": "3.6.0-31",
+  "v3SchemaId": "https://trust.propdata.org.uk/schemas/v3/pdtf-transaction.json",
+  "combinedSha256": "…"
+}
+```
+
+The same block is stamped on every entity schema as `x-pdtf-source`, so a
+consumer holding a single schema can still pin.
+
+The content hash is deliberately the only identifier. A commit sha would tie
+generated output to git history: it could never be correct in the same commit
+that changes `combined.json`, and it would churn on every rebase or cherry-pick
+without the schema content differing at all. To find the commit for a hash,
+search the history for it.
+
+### It cannot drift
+
+`verifyCoverage()` runs at generation time and **fails the build** if any V3 key
+is not claimed by a rule, or if a rule claims a key that is absent from its
+target entity schema. Adding a field to V3 that has no home in V4 is a hard
+error rather than a field that silently disappears from the round trip.
+
+## The four ambiguous cases
+
+### 1. `Title` merges two V3 arrays
+
+`propertyPack.titlesToBeSold` and
+`propertyPack.ownership.ownershipsToBeTransferred` both contribute to `Title`.
+
+- **Key collision:** `titleNumber`, and only `titleNumber`. It is the same fact
+  on both sides, and it is declared in `mapping.collisions`.
+- **How the indices correlate: they don't.** The arrays are correlated by
+  `titleNumber` value. A V3 instance may order them differently or list a title
+  on only one side, and the tests cover both.
+- **Recomposition** splits a `Title` back by the `ownedKeys` each rule declares.
+  A title carrying only shared keys goes to `titlesToBeSold` alone
+  (`instance.primarySource`).
+
+### 2. `propertyPack.ownership` splits two ways
+
+`ownershipsToBeTransferred` → `Title`; everything else →
+`Transaction.saleContext`.
+
+`propertyPack.legalOwners` also lands in `saleContext`. It is excluded from
+`Property` and, being an unidentified name list with no link to `participants`,
+cannot be correlated to `Person`/`Organisation` entities. It is a sale-level
+ownership fact of the same family as `numberOfSellers` and
+`isLimitedCompanySale`, so it sits alongside them — which keeps the round trip
+lossless. Previously it was dropped entirely.
+
+### 3. Relationship fields on `participants`
+
+`role`, `sellersCapacity`, `organisation`, `organisationReference`,
+`participantId` and `actingFor` describe a party's relationship to *this
+transaction*, not the party, so they stay off `Person`. They are carried on
+`Transaction.participants[]`, each entry pairing a `participant` DID with those
+fields.
+
+This means the round trip needs only four entities — `Property`, `Title`,
+`Transaction`, `Person` (`mapping.roundTrip.entities`). `Organisation`,
+`Representation`, `SellerCapacity` and `Offer` are credential-facing
+projections of data those four already carry; they are not required to
+reconstruct V3.
+
+### 4. Array order
+
+V3 arrays become id-keyed entities, so order has to live somewhere. It lives on
+`Transaction` — `titlesToBeSold` and `participants` are ordered reference lists,
+and they are authoritative for recomposition. No index or provenance field is
+added to the entity documents themselves, so a credential never carries a
+position that could diverge from another producer's.
+
+## Documented round-trip exceptions
+
+`mapping.roundTrip.knownExceptions` carries these at runtime:
+
+1. **Overlay metadata** (`*Ref`, `*Required` keys) is stripped from V4. It is
+   schema-level only and never appears in instance data.
+2. A `Title` carrying only keys shared by both source arrays recomposes into
+   `titlesToBeSold` only, not into `ownershipsToBeTransferred`.
+3. **`ownershipsToBeTransferred` order is canonicalised** to `titlesToBeSold`
+   order. The array is correlated by `titleNumber` and its order carries no
+   meaning in V3, so this is a deliberate normalisation that removes the
+   index-divergence class of bug rather than reproducing it. Entry *content* is
+   preserved exactly.
+
+4. **An explicitly empty container recomposes as absent** — `propertyPack {}`,
+   `propertyPack.ownership {}` and `propertyPack.titlesToBeSold []` are all valid
+   V3 but carry no data, and the entity model has no place to record "present but
+   empty". (`participants []` is invalid V3 — `minItems: 1` — so it does not
+   arise.)
+5. **`recompose` always emits `$schema`**, set to `source.v3SchemaId`. An
+   instance that omitted one gains it.
+
+Everything else round-trips to deep equality, including the full example
+transaction in `src/examples/v3/exampleTransaction.json`.
+
+## What the round trip is actually tested against
+
+Worth being precise about, because "it round-trips" is easy to overclaim:
+
+| | |
+| --- | --- |
+| `exampleTransaction.json` | One real instance — roughly **12% of the V3 leaf-path surface** (456 of ~3,300 paths) |
+| Synthetic awkward instance | The structural transforms: two-array merge, index divergence, one-sided titles |
+| `roundTripEdges.test.js` | The six top-level keys the example never touches, plus the boundaries below |
+
+The untested majority is `propertyPack` subtree, which passes through wholesale
+under a single prefix rule — low risk, but not zero. The transforms that could
+actually lose data (title merge/split, participant split, ownership split) are
+covered directly.
+
+**Refused rather than guessed:** a `titleNumber` repeated within one source
+array throws. It is valid V3 — neither array declares uniqueness — but a title
+number identifies a title, so merging the two entries would silently drop one.
+
+**Not yet done:** generating fixtures from the schema to exercise the remaining
+~88%, and round-tripping real production transactions. Either would turn "no
+known failures" into something stronger.
+
+## Defects this surfaced in the generated schemas
+
+Two fields were being lost because the generator read only `node.properties` and
+ignored V3's `discriminator` + `oneOf` branches:
+
+- **`Title` was missing every tenure detail** — `leaseholdInformation`,
+  `managedFreeholdOrCommonholdInformation`, `estateRentcharges`,
+  `wholeFreeholdForSale`, `otherOwnershipDetails` all live on
+  `ownershipsToBeTransferred` branches.
+- **`Person`/`SellerCapacity` were missing** `sellersCapacity`,
+  `dateBecameOwnerOrAuthority` and `offerId`, which live on `participants`
+  branches. `SellerCapacity` was hand-authored with a four-value `capacityType`
+  enum against V3's nine-value `capacity`.
+
+`mergedProperties()` now folds branch properties in, and `SellerCapacity` is
+derived from the V3 seller branch rather than hand-authored.
+
+
+## Representation
+
+V3 previously referred to people **by role only** (`originatorRole`,
+`destinationRole` on enquiry messages) and nothing anywhere referenced a
+specific participant. That made a representation relationship underivable — you
+could see that a transaction had a "Seller's Conveyancer", but not which seller
+they acted for — and it made two sellers with separate conveyancers
+unrepresentable, because both participants would carry the same role with
+nothing to tell them apart.
+
+Two additive, optional fields close that gap:
+
+| Field | Purpose |
+| --- | --- |
+| `participants[].participantId` | Stable identity within the transaction |
+| `participants[].actingFor` | The `participantId`s this participant is instructed by |
+
+Both follow the existing key idiom (`^[A-Za-z0-9][A-Za-z0-9._:+-]*$`, the same
+pattern `offers` and `enquiries` keys use). Neither carries a form `*Ref`, so no
+overlay changes, and both are optional, so existing instances stay valid.
+
+### Cardinality
+
+One `Representation` per **(representative, represented party)** pair.
+
+- A conveyancer instructed jointly by two sellers → `actingFor: ["s1", "s2"]` →
+  **two** Representations. Each is a separate statement that can be presented,
+  or revoked, on its own.
+- A separated couple instructing their own conveyancers → two representatives,
+  each `actingFor` one seller → **one each**. This is the case the old
+  role-only model could not express at all.
+
+### A conveyancer is instructed by a person, not by an offer
+
+`actingFor` references participants. The retainer is with the client, so a
+buyer's conveyancer acts for the buyer; `participants[].offerId` remains the
+separate link from that buyer to their offer. This also means a **prospective
+buyer**, who has no `offerId` until they make an offer, can still be
+represented — which a buyer-via-offer model could not do.
+
+### The relationship credentials
+
+`Representation`, `SellerCapacity`, `Offer`, `Gift` and `TransactionRole` are
+`kind: "credential"`. They are the linking entities that **embody a party's
+role**, and role is stored nowhere else.
+
+| Credential | Asserts | Role |
+| --- | --- | --- |
+| `SellerCapacity` | this person sells, in this capacity | **implied**: Seller |
+| `Offer` | this person made this offer | **implied**: Buyer |
+| `Gift` | this person gifts funds towards a purchase | **implied**: Gift Donor |
+| `Representation` | this party is instructed by that party | explicit — which kind |
+| `TransactionRole` | this party takes this role | explicit — which role |
+
+For the first three, role is implied 1:1 by the credential's existence: V3 puts
+`sellersCapacity` only on the Seller branch and `giftDetails` only on Gift Donor.
+`offerId` is on the Buyer **and** Gift Donor branches, so a donor's offer link
+lives on `Gift` rather than `Offer`, which keeps `Offer ⇒ Buyer` exact.
+
+`Representation` and `TransactionRole` each span several roles, so they carry
+role as their own discriminator. That is not a duplicated participant attribute:
+"what kind of representation is this" is a property of the representation, and
+asserting a role is `TransactionRole`'s entire purpose. Representation's role
+cannot be derived from the edge — acting for a seller tells you the side, not
+whether the party is the conveyancer, the agent or the surveyor.
+
+### Identity: `participants[].did`
+
+V3 participants carry a `did` — the party's own identifier, minted when the
+party is created, so in practice always present. V4 uses it verbatim as
+`Person.id`; the `idFactory` mints one only for instances that carry none.
+
+That matters because identity must be stable *across* transactions. A person
+selling one property and buying another appears in two transactions, and should
+be the same subject in both — one wallet, accumulating credentials, not one
+identity per participation. A transaction-scoped id (`did:web:…:transactions:
+<txId>:participants:<pid>`) cannot do that; `did:key` minted per person can.
+
+`participantId` remains as the transaction-local fallback for implementations
+that do not mint DIDs. `actingFor` names a party by `did` where one exists and
+by `participantId` otherwise.
+
+**One party, one entry.** A `did` repeated within a transaction is the same
+party listed twice, which `decompose` refuses: a transaction is a single sale,
+so nobody is both its buyer and its seller. The same person across a sale and an
+onward purchase is two transactions, each with its own credentials, and needs
+nothing special.
+
+### Why role is not on the participant
+
+Because a second copy would not be revoked.
+
+Firing a conveyancer means revoking their `Representation`. If the transaction
+also recorded `role: "Seller's Conveyancer"` against that participant, the
+revocation would remove the relationship and leave the role assertion standing —
+two sources of truth, disagreeing, with the stale one still readable.
+
+So `Transaction.participants[]` is a **roster**, carrying only what remains true
+of a party regardless of any relationship:
+
+```jsonc
+{ "participant": "did:pdtf:person-tx-c1",
+  "participantId": "c1",
+  "organisation": "Suemme and Profitt",
+  "organisationReference": "R1" }
+```
+
+`organisation` stays here rather than on the credential for the same
+single-home reason: where someone works does not stop being true when a
+representation ends. A credential names its parties by DID, so resolving the
+representative gives the firm.
+
+### They are part of the round trip
+
+Because role and every relationship live only on the credentials, `recompose`
+needs them — they are not redundant copies. `decompose` therefore returns them
+alongside the other entities:
+
+```js
+const { decomposeToV4, recomposeFromV4 } = require("@pdtf/schemas");
+
+const entities = decomposeToV4(v3Transaction);
+// { Property, Title, Transaction, Person,
+//   Representation, SellerCapacity, Offer, Gift, TransactionRole }
+
+recomposeFromV4(entities); // === v3Transaction
+```
+
+Dropping a credential and recomposing gives back a V3 instance in which that
+party has no role and no relationship — which is exactly what revocation should
+mean, and is asserted by a test.
+
+Every participant carrying a role receives exactly one role-bearing credential:
+the specific one where the relationship exists, `TransactionRole` otherwise. A
+participant with no role receives none.
+
+`SellerCapacity` is scoped to the transaction rather than to a title, since the
+transaction already carries the titles to be sold. It is emitted for **every**
+seller, whether or not a capacity has been declared, so the credential embodying
+the Seller role is never missing while a form is still being filled in.
+
+### If you want an identifier everyone can mint
+
+Three patterns, of which V3 already uses two:
+
+- **First-writer-mints** — whoever adds the item sets the canonical id, and
+  everyone else adopts it because it is in the shared payload. What the seven new
+  identifiers assume. A source can simply use its own id as the value, promoting
+  it out of its private namespace. Its failure mode is duplication, not ambiguity:
+  two parties adding the same document independently mint two ids.
+- **Authority-issued** — `uprn`, `titleNumber`, `lmkKey` and `hmlrReference`
+  already work this way. It needs a registry and one authoritative issuer, which
+  exists for properties, titles and EPCs but not for documents. Transactions are
+  identified by `transactionId` and need no second identifier alongside it.
+- **Content-derived** — a hash of the file bytes. Interoperable with no issuer,
+  and it deduplicates for free, which first-writer-mints does not. It changes if a
+  file is re-rendered, and you need the bytes to compute it. Not unprecedented
+  here: `contracts[].signatures` already requires a `contractHash`.
+
+
+## Identifiers and `did:web`
+
+Entity ids come from the `idFactory` you pass to `decompose`. The built-in
+default is deterministic and derived from `transactionId`, which keeps repeated
+runs stable and diffable, but it is a placeholder — supply your own to mint real
+identifiers.
+
+`did:web` is a natural fit, because the DID document is hosted rather than
+anchored to a ledger and can carry service endpoints, so a party holding a
+credential can resolve where to fetch what it refers to:
+
+```js
+const entities = decomposeToV4(v3Transaction, {
+  idFactory: {
+    transaction: () => `did:web:moverly.com:transactions:${v3Transaction.transactionId}`,
+    person: (participant, index) =>
+      `did:web:moverly.com:transactions:${v3Transaction.transactionId}` +
+      `:participants:${participant?.participantId ?? index}`,
+    property: () => `urn:pdtf:property:${v3Transaction.transactionId}`,
+    title: (title, index) => `urn:pdtf:title:…`,
+    representation: (representative, representedParty) => `urn:pdtf:representation:…`,
+    sellerCapacity: (seller) => `urn:pdtf:sellercapacity:…`,
+  },
+});
+```
+
+`did:web:moverly.com:transactions:<id>` resolves to
+`https://moverly.com/transactions/<id>/did.json`.
+
+### The DID pattern accepts colons, as the spec requires
+
+A `did:web` encodes its path as colon-separated segments, and a port is
+percent-encoded (`did:web:example.com%3A3000:…`). The W3C ABNF allows both —
+`method-specific-id = *( *idchar ":" ) 1*idchar` — so the schema pattern follows
+it exactly rather than approximating:
+
+```
+^did:[a-z0-9]+:(?:(?:[a-zA-Z0-9._-]|%[0-9A-Fa-f]{2})*:)*(?:[a-zA-Z0-9._-]|%[0-9A-Fa-f]{2})+$
+```
+
+An earlier pattern here allowed no colons after the method name, which rejected
+every `did:web` carrying a path — in nine places across five entities.
+
+### It does not disturb the round trip
+
+`Transaction.id` is synthetic: `decompose` mints it and `recompose` drops it,
+because V3 has no field for it. `transactionId` is the real V3 data and survives
+untouched. So a `did:web` transaction id **embeds** the V3 identifier rather than
+replacing it, both are present on the entity, and which DID method you choose
+makes no difference to the V3 that comes back. There is a test asserting exactly
+that.
+
+`Property` and `Title` keep `urn:` ids: they are things described, not parties
+that resolve to a DID document. Nothing stops you minting `did:web` for them —
+the URN pattern already permits colons — but nothing requires it either.

package/index.js

@@ -821,6 +821,16 @@ const setCacheMaxSize = (maxSize) => {
   return keysToDelete.length;
 };
 
+const {
+  mapping: v4Mapping,
+  decompose: decomposeToV4,
+  recompose: recomposeFromV4,
+  arrayKeyFor: v4ArrayKeyFor,
+  identifyV3Pointer,
+  resolveV3Pointer,
+  resolveV3PointerAll,
+} = require("./src/utils/v4");
+
 module.exports = {
   ajv,
   getTransactionSchema,
@@ -843,4 +853,12 @@ module.exports = {
   warmupCache,
   pruneCacheByAge,
   setCacheMaxSize,
+  // V3 <-> V4 entity decomposition (see src/schemas/v4/mapping.json)
+  v4Mapping,
+  decomposeToV4,
+  recomposeFromV4,
+  v4ArrayKeyFor,
+  identifyV3Pointer,
+  resolveV3Pointer,
+  resolveV3PointerAll,
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pdtf/schemas",
-  "version": "3.6.0-dev.19",
+  "version": "3.6.0-dev.21",
   "description": "Property Data Trust Framework Schemas and Utilities",
   "main": "index.js",
   "files": [
@@ -19,6 +19,7 @@
     "test:watch": "jest --watch",
     "extract-overlays": "cd src/utils && node extractOverlay.js",
     "extract-extension-overlays": "cd src/utils && node extractExtensionOverlays.js",
+    "generate:v4": "node src/utils/generateV4Schemas.js",
     "publish:dev": "npm publish --tag dev",
     "publish:next": "npm publish --tag next",
     "publish:latest": "npm publish --tag latest"