@pdtf/schemas 3.6.0-dev.3 → 3.6.0-dev.30

package/docs/ta6-certificate-permit-contract.md

@@ -0,0 +1,51 @@
+# TA6 certificate and permit document contract
+
+Candidate: `@pdtf/schemas@3.6.0-dev.30`, targeting `dev`. Published baseline checked on 27 October 2026: `3.6.0-dev.29`. Publication requires release approval; this document does not assert publication.
+
+## Paths and refs
+
+Both `ta6ed6` and `ta6ed6v2` use the same stored paths and values.
+
+| Path | Values | TA6 ref in both overlays |
+| --- | --- | --- |
+| `/propertyPack/electricalWorks/electricalWorkSince2005/suppliedCertificate/yesNo` | `Yes`, `No` | `11.2.1` |
+| `/propertyPack/electricalWorks/electricalWorkSince2005/suppliedCertificate/attachments` | `Attached`, `To follow` | `11.2.2` (formerly `11.2`) |
+| `/propertyPack/waterAndDrainage/drainage/mainsFoulDrainage/offMainsDrainageSystem/permitDocuments` | `Attached`, `To follow`, `Not applicable` | `11.7j` (unchanged) |
+
+The certificate object retains ref `11.2` and the existing `certificateType` and `attachments` paths. `certificateType` remains defined in the base and BASPI/TA6 fourth-edition overlays; it is not newly displayed in sixth-edition forms.
+
+## Compatibility and requiredness
+
+The new answer is a flat optional string. It has no default, discriminator, `oneOf`, or additional requiredness. The sixth-edition attachment requirement is removed. Explicit No needs neither attachments nor certificate type; Yes supports both attachment choices and can be saved without a status. Attachment-only records remain valid. An empty certificate object remains unanswered, never inferred as No. Existing outer electrical-work conditions and requirements for details and the certificate object are unchanged.
+
+Permit documents stay optional in all four applicable branches: Septic tank, Cesspit, Sewerage treatment plant and Other. Absence stays absent.
+
+No other v3 overlay or extension overlay artifact changes. The shared base acquires the optional certificate field and wider permit enum; consumers validating these paths with the base therefore see the same additions. BASPI and TA6 fourth-edition certificate requirements are unchanged, including when combined with a sixth-edition overlay. Heating/mains-gas schema is unchanged.
+
+Differential verification against the registry tarball uses `getSubschemaValidator` and `getValidator` with complete baseline-valid transaction fixtures for both editions:
+
+```bash
+node src/utils/checkTa6OmissionCompatibility.js /path/to/unpacked/dev.29/package
+```
+
+352 comparisons: legacy shapes have 64 valid-to-valid, 24 invalid-to-valid, 64 invalid-to-invalid and **zero valid-to-invalid** results. New-answer cases have 32 valid-to-valid, 16 invalid-to-valid, 104 invalid-to-invalid and 48 valid-to-invalid results. The 48 tightenings are exclusively `Maybe`, empty string or null already stored at the previously unconstrained new `yesNo` path, combined with a valid legacy attachment status (including both validators and editions). This is not a claim of universal compatibility with arbitrary additional-property data; no production corpus scan was performed.
+
+## Buyer Ready dependency
+
+Buyer Ready main `6d2e4cbfe` pins dev.29 in its lockfile. Sellers need a frontend dependency and lockfile bump to the published candidate version, followed by a frontend release, to see these choices. A backend-only bump does not update the seller UI.
+
+22 mounted `FormElement.vue` tests against the candidate package confirm both overlays: the certificate answer and attachment status are optional; attachment choices remain visible for Yes, No and unanswered; no update/default is emitted on mount; the existing outer work-Yes condition is preserved; and Not applicable is available in each drainage branch. The current UI does not hide attachment controls on certificate No. The backend must suppress stale attachment ticks, without deleting seller data. No frontend product code change is needed for basic visibility or requiredness. These are component-level checks, not a deployed browser acceptance claim.
+
+## Backend mapping contract
+
+For §11.2, explicit Yes/No takes precedence. No ticks only `11.2 no` and suppresses stale attachment ticks. Yes ticks `11.2 yes` and uses the attachment status. With no answer, Attached/To follow retain the legacy Yes inference. With neither, leave all four boxes blank.
+
+For §11.7(j), use the actual `11.7j na` widget. The three statuses are mutually exclusive. Prefer `permitDocuments`; fall back to `plantRegistered/attachments` only when the newer value is absent. Explicit new values override conflicting legacy statuses.
+
+The backend MR is separate from the six mapping fixes in MR !4579 (open at initial verification, head `7f650a32f`). It must not merge or deploy without approval. Do not regenerate signed PDFs or modify production claims.
+
+## Release checks
+
+Edit only `src/schemas/v3/combined.json`; regenerate with `extract-overlays`, `extract-extension-overlays`, and `generate:v4` after the package version bump. All v4 headers must carry the candidate version and combined-source SHA-256.
+
+After approval: merge the schema PR into dev, publish with `npm run publish:dev`, verify the exact npm version and dev dist-tag from the registry, then bump backend/frontend package metadata and lockfiles to that verified version. Recheck concurrent versions before publishing. Never publish this candidate to latest.

package/docs/v4-mapping-and-recomposition.md

@@ -0,0 +1,419 @@
+# V4 mapping and recomposition
+
+PDTF 1.x verified claims are JSON Pointers against the V3 transaction schema.
+PDTF 2.0 credentials are per-entity documents. The V4 generator
+(`src/utils/generateV4Schemas.js`) already encodes every rule needed to move
+between the two — but until now it emitted only schemas, so any consumer
+wanting to map a claim onto a credential (or back) had to re-implement those
+rules by hand and then keep them in step.
+
+This adds three things, all produced by the same generation run:
+
+| Artefact | What it is |
+| --- | --- |
+| `src/schemas/v4/mapping.json` | Generated manifest: V3 pointer → entity + pointer, and the inverse |
+| `src/utils/v4.js` | `decompose` / `recompose` / `resolveV3Pointer`, driven entirely by the manifest |
+| `src/tests/v4/roundTrip.test.js` | V3 → entities → V3 equality, with documented exceptions |
+
+Regenerate everything with:
+
+```bash
+npm run generate:v4
+```
+
+The generator remains read-only against V3.
+
+## Using it
+
+```js
+const {
+  decomposeToV4,
+  recomposeFromV4,
+  resolveV3Pointer,
+  v4Mapping,
+} = require("@pdtf/schemas");
+
+const entities = decomposeToV4(v3Transaction);
+// { Property, Title: [...], Transaction, Person: [...] }
+
+const v3Again = recomposeFromV4(entities);
+
+// Map a verified-claim pointer onto the credential that now carries it
+resolveV3Pointer("/propertyPack/ownership/ownershipsToBeTransferred/1/ownershipType");
+// { entity: "Title", entityPointer: "/ownershipType", rule: "title.ownershipsToBeTransferred" }
+```
+
+Pass `idFactory` to `decomposeToV4` to mint real URNs/DIDs; the default is
+deterministic and derived from `transactionId`, so repeated runs are stable and
+diffable. A supplied factory may be **partial** — anything it does not implement
+falls back to the default, so adding an entity type here cannot break a caller's
+factory on upgrade.
+
+## The manifest
+
+Rules are **prefix rewrites**, not a leaf enumeration. A V3 pointer is resolved
+by longest-prefix match against `rule.v3Pointer` (with `{index}` matching an
+array index), which means pointers far deeper than anything listed — the kind
+claims actually carry — resolve correctly, and the file does not churn every
+time a field is added to V3.
+
+`mapping.source` records exactly which V3 input produced the output:
+
+```json
+{
+  "packageVersion": "3.6.0-31",
+  "v3SchemaId": "https://trust.propdata.org.uk/schemas/v3/pdtf-transaction.json",
+  "combinedSha256": "…"
+}
+```
+
+The same block is stamped on every entity schema as `x-pdtf-source`, so a
+consumer holding a single schema can still pin.
+
+The content hash is deliberately the only identifier. A commit sha would tie
+generated output to git history: it could never be correct in the same commit
+that changes `combined.json`, and it would churn on every rebase or cherry-pick
+without the schema content differing at all. To find the commit for a hash,
+search the history for it.
+
+### It cannot drift
+
+`verifyCoverage()` runs at generation time and **fails the build** if any V3 key
+is not claimed by a rule, or if a rule claims a key that is absent from its
+target entity schema. Adding a field to V3 that has no home in V4 is a hard
+error rather than a field that silently disappears from the round trip.
+
+## The four ambiguous cases
+
+### 1. `Title` merges two V3 arrays
+
+`propertyPack.titlesToBeSold` and
+`propertyPack.ownership.ownershipsToBeTransferred` both contribute to `Title`.
+
+- **Key collision:** `titleNumber`, and only `titleNumber`. It is the same fact
+  on both sides, and it is declared in `mapping.collisions`.
+- **How the indices correlate: they don't.** The arrays are correlated by
+  `titleNumber` value. A V3 instance may order them differently or list a title
+  on only one side, and the tests cover both.
+- **Recomposition** splits a `Title` back by the `ownedKeys` each rule declares.
+  A title carrying only shared keys goes to `titlesToBeSold` alone
+  (`instance.primarySource`).
+
+### 2. `propertyPack.ownership` splits two ways
+
+`ownershipsToBeTransferred` → `Title`; everything else →
+`Transaction.saleContext`.
+
+`propertyPack.legalOwners` also lands in `saleContext`. It is excluded from
+`Property` and, being an unidentified name list with no link to `participants`,
+cannot be correlated to `Person`/`Organisation` entities. It is a sale-level
+ownership fact of the same family as `numberOfSellers` and
+`isLimitedCompanySale`, so it sits alongside them — which keeps the round trip
+lossless. Previously it was dropped entirely.
+
+### 3. Relationship fields on `participants`
+
+`role`, `sellersCapacity`, `organisation`, `organisationReference`,
+`participantId` and `actingFor` describe a party's relationship to *this
+transaction*, not the party, so they stay off `Person`. They are carried on
+`Transaction.participants[]`, each entry pairing a `participant` DID with those
+fields.
+
+This means the round trip needs only four entities — `Property`, `Title`,
+`Transaction`, `Person` (`mapping.roundTrip.entities`). `Organisation`,
+`Representation`, `SellerCapacity` and `Offer` are credential-facing
+projections of data those four already carry; they are not required to
+reconstruct V3.
+
+### 4. Array order
+
+V3 arrays become id-keyed entities, so order has to live somewhere. It lives on
+`Transaction` — `titlesToBeSold` and `participants` are ordered reference lists,
+and they are authoritative for recomposition. No index or provenance field is
+added to the entity documents themselves, so a credential never carries a
+position that could diverge from another producer's.
+
+## Documented round-trip exceptions
+
+`mapping.roundTrip.knownExceptions` carries these at runtime:
+
+1. **Overlay metadata** (`*Ref`, `*Required` keys) is stripped from V4. It is
+   schema-level only and never appears in instance data.
+2. `Transaction.titlesToBeSold` lists the titles V3 listed under
+   `propertyPack.titlesToBeSold`. A title known only from
+   `ownershipsToBeTransferred` is a `Title` entity but is not listed there, which
+   is how recomposition tells the two apart — and why a title whose only listed
+   key is `titleNumber` still gets its entry back.
+3. **`ownershipsToBeTransferred` order is canonicalised** to `titlesToBeSold`
+   order. The array is correlated by `titleNumber` and its order carries no
+   meaning in V3, so this is a deliberate normalisation that removes the
+   index-divergence class of bug rather than reproducing it. Entry *content* is
+   preserved exactly.
+
+4. **An explicitly empty container recomposes as absent** — `propertyPack {}`,
+   `propertyPack.ownership {}` and `propertyPack.titlesToBeSold []` are all valid
+   V3 but carry no data, and the entity model has no place to record "present but
+   empty". (`participants []` is invalid V3 — `minItems: 1` — so it does not
+   arise.)
+5. **`recompose` always emits `$schema`**, set to `source.v3SchemaId`. An
+   instance that omitted one gains it.
+
+Everything else round-trips to deep equality, including the full example
+transaction in `src/examples/v3/exampleTransaction.json`.
+
+## What the round trip is actually tested against
+
+Worth being precise about, because "it round-trips" is easy to overclaim:
+
+| | |
+| --- | --- |
+| `exampleTransaction.json` | One real instance — roughly **12% of the V3 leaf-path surface** (456 of ~3,300 paths) |
+| Synthetic awkward instance | The structural transforms: two-array merge, index divergence, one-sided titles |
+| `roundTripEdges.test.js` | The six top-level keys the example never touches, plus the boundaries below |
+
+The untested majority is `propertyPack` subtree, which passes through wholesale
+under a single prefix rule — low risk, but not zero. The transforms that could
+actually lose data (title merge/split, participant split, ownership split) are
+covered directly.
+
+**Refused rather than guessed:** a `titleNumber` repeated within one source
+array throws. It is valid V3 — neither array declares uniqueness — but a title
+number identifies a title, so merging the two entries would silently drop one.
+
+**Not yet done:** generating fixtures from the schema to exercise the remaining
+~88%, and round-tripping real production transactions. Either would turn "no
+known failures" into something stronger.
+
+## Defects this surfaced in the generated schemas
+
+Two fields were being lost because the generator read only `node.properties` and
+ignored V3's `discriminator` + `oneOf` branches:
+
+- **`Title` was missing every tenure detail** — `leaseholdInformation`,
+  `managedFreeholdOrCommonholdInformation`, `estateRentcharges`,
+  `wholeFreeholdForSale`, `otherOwnershipDetails` all live on
+  `ownershipsToBeTransferred` branches.
+- **`Person`/`SellerCapacity` were missing** `sellersCapacity`,
+  `dateBecameOwnerOrAuthority` and `offerId`, which live on `participants`
+  branches. `SellerCapacity` was hand-authored with a four-value `capacityType`
+  enum against V3's nine-value `capacity`.
+
+`mergedProperties()` now folds branch properties in, and `SellerCapacity` is
+derived from the V3 seller branch rather than hand-authored.
+
+
+## Representation
+
+V3 previously referred to people **by role only** (`originatorRole`,
+`destinationRole` on enquiry messages) and nothing anywhere referenced a
+specific participant. That made a representation relationship underivable — you
+could see that a transaction had a "Seller's Conveyancer", but not which seller
+they acted for — and it made two sellers with separate conveyancers
+unrepresentable, because both participants would carry the same role with
+nothing to tell them apart.
+
+Two additive, optional fields close that gap:
+
+| Field | Purpose |
+| --- | --- |
+| `participants[].participantId` | Stable identity within the transaction |
+| `participants[].actingFor` | The `participantId`s this participant is instructed by |
+
+Both follow the existing key idiom (`^[A-Za-z0-9][A-Za-z0-9._:+-]*$`, the same
+pattern `offers` and `enquiries` keys use). Neither carries a form `*Ref`, so no
+overlay changes, and both are optional, so existing instances stay valid.
+
+### Cardinality
+
+One `Representation` per **(representative, represented party)** pair.
+
+- A conveyancer instructed jointly by two sellers → `actingFor: ["s1", "s2"]` →
+  **two** Representations. Each is a separate statement that can be presented,
+  or revoked, on its own.
+- A separated couple instructing their own conveyancers → two representatives,
+  each `actingFor` one seller → **one each**. This is the case the old
+  role-only model could not express at all.
+
+### A conveyancer is instructed by a person, not by an offer
+
+`actingFor` references participants. The retainer is with the client, so a
+buyer's conveyancer acts for the buyer; `participants[].offerId` remains the
+separate link from that buyer to their offer. This also means a **prospective
+buyer**, who has no `offerId` until they make an offer, can still be
+represented — which a buyer-via-offer model could not do.
+
+### The relationship credentials
+
+`Representation`, `SellerCapacity`, `Offer`, `Gift` and `TransactionRole` are
+`kind: "credential"`. They are the linking entities that **embody a party's
+role**, and role is stored nowhere else.
+
+| Credential | Asserts | Role |
+| --- | --- | --- |
+| `SellerCapacity` | this person sells, in this capacity | **implied**: Seller |
+| `Offer` | this person made this offer | **implied**: Buyer |
+| `Gift` | this person gifts funds towards a purchase | **implied**: Gift Donor |
+| `Representation` | this party is instructed by that party | explicit — which kind |
+| `TransactionRole` | this party takes this role | explicit — which role |
+
+For the first three, role is implied 1:1 by the credential's existence: V3 puts
+`sellersCapacity` only on the Seller branch and `giftDetails` only on Gift Donor.
+`offerId` is on the Buyer **and** Gift Donor branches, so a donor's offer link
+lives on `Gift` rather than `Offer`, which keeps `Offer ⇒ Buyer` exact.
+
+`Representation` and `TransactionRole` each span several roles, so they carry
+role as their own discriminator. That is not a duplicated participant attribute:
+"what kind of representation is this" is a property of the representation, and
+asserting a role is `TransactionRole`'s entire purpose. Representation's role
+cannot be derived from the edge — acting for a seller tells you the side, not
+whether the party is the conveyancer, the agent or the surveyor.
+
+### Identity: `participants[].did`
+
+V3 participants carry a `did` — the party's own identifier, minted when the
+party is created, so in practice always present. V4 uses it verbatim as
+`Person.id`; the `idFactory` mints one only for instances that carry none.
+
+That matters because identity must be stable *across* transactions. A person
+selling one property and buying another appears in two transactions, and should
+be the same subject in both — one wallet, accumulating credentials, not one
+identity per participation. A transaction-scoped id (`did:web:…:transactions:
+<txId>:participants:<pid>`) cannot do that; `did:key` minted per person can.
+
+`participantId` remains as the transaction-local fallback for implementations
+that do not mint DIDs. `actingFor` names a party by `did` where one exists and
+by `participantId` otherwise.
+
+**One party, one entry.** A `did` repeated within a transaction is the same
+party listed twice, which `decompose` refuses: a transaction is a single sale,
+so nobody is both its buyer and its seller. The same person across a sale and an
+onward purchase is two transactions, each with its own credentials, and needs
+nothing special.
+
+### Why role is not on the participant
+
+Because a second copy would not be revoked.
+
+Firing a conveyancer means revoking their `Representation`. If the transaction
+also recorded `role: "Seller's Conveyancer"` against that participant, the
+revocation would remove the relationship and leave the role assertion standing —
+two sources of truth, disagreeing, with the stale one still readable.
+
+So `Transaction.participants[]` is a **roster**, carrying only what remains true
+of a party regardless of any relationship:
+
+```jsonc
+{ "participant": "did:pdtf:person-tx-c1",
+  "participantId": "c1",
+  "organisation": "Suemme and Profitt",
+  "organisationReference": "R1" }
+```
+
+`organisation` stays here rather than on the credential for the same
+single-home reason: where someone works does not stop being true when a
+representation ends. A credential names its parties by DID, so resolving the
+representative gives the firm.
+
+### They are part of the round trip
+
+Because role and every relationship live only on the credentials, `recompose`
+needs them — they are not redundant copies. `decompose` therefore returns them
+alongside the other entities:
+
+```js
+const { decomposeToV4, recomposeFromV4 } = require("@pdtf/schemas");
+
+const entities = decomposeToV4(v3Transaction);
+// { Property, Title, Transaction, Person,
+//   Representation, SellerCapacity, Offer, Gift, TransactionRole }
+
+recomposeFromV4(entities); // === v3Transaction
+```
+
+Dropping a credential and recomposing gives back a V3 instance in which that
+party has no role and no relationship — which is exactly what revocation should
+mean, and is asserted by a test.
+
+Every participant carrying a role receives exactly one role-bearing credential:
+the specific one where the relationship exists, `TransactionRole` otherwise. A
+participant with no role receives none.
+
+`SellerCapacity` is scoped to the transaction rather than to a title, since the
+transaction already carries the titles to be sold. It is emitted for **every**
+seller, whether or not a capacity has been declared, so the credential embodying
+the Seller role is never missing while a form is still being filled in.
+
+### If you want an identifier everyone can mint
+
+Three patterns, of which V3 already uses two:
+
+- **First-writer-mints** — whoever adds the item sets the canonical id, and
+  everyone else adopts it because it is in the shared payload. What the seven new
+  identifiers assume. A source can simply use its own id as the value, promoting
+  it out of its private namespace. Its failure mode is duplication, not ambiguity:
+  two parties adding the same document independently mint two ids.
+- **Authority-issued** — `uprn`, `titleNumber`, `lmkKey` and `hmlrReference`
+  already work this way. It needs a registry and one authoritative issuer, which
+  exists for properties, titles and EPCs but not for documents. Transactions are
+  identified by `transactionId` and need no second identifier alongside it.
+- **Content-derived** — a hash of the file bytes. Interoperable with no issuer,
+  and it deduplicates for free, which first-writer-mints does not. It changes if a
+  file is re-rendered, and you need the bytes to compute it. Not unprecedented
+  here: `contracts[].signatures` already requires a `contractHash`.
+
+
+## Identifiers and `did:web`
+
+Entity ids come from the `idFactory` you pass to `decompose`. The built-in
+default is deterministic and derived from `transactionId`, which keeps repeated
+runs stable and diffable, but it is a placeholder — supply your own to mint real
+identifiers.
+
+`did:web` is a natural fit, because the DID document is hosted rather than
+anchored to a ledger and can carry service endpoints, so a party holding a
+credential can resolve where to fetch what it refers to:
+
+```js
+const entities = decomposeToV4(v3Transaction, {
+  idFactory: {
+    transaction: () => `did:web:moverly.com:transactions:${v3Transaction.transactionId}`,
+    person: (participant, index) =>
+      `did:web:moverly.com:transactions:${v3Transaction.transactionId}` +
+      `:participants:${participant?.participantId ?? index}`,
+    property: () => `urn:pdtf:property:${v3Transaction.transactionId}`,
+    title: (title, index) => `urn:pdtf:title:…`,
+    representation: (representative, representedParty) => `urn:pdtf:representation:…`,
+    sellerCapacity: (seller) => `urn:pdtf:sellercapacity:…`,
+  },
+});
+```
+
+`did:web:moverly.com:transactions:<id>` resolves to
+`https://moverly.com/transactions/<id>/did.json`.
+
+### The DID pattern accepts colons, as the spec requires
+
+A `did:web` encodes its path as colon-separated segments, and a port is
+percent-encoded (`did:web:example.com%3A3000:…`). The W3C ABNF allows both —
+`method-specific-id = *( *idchar ":" ) 1*idchar` — so the schema pattern follows
+it exactly rather than approximating:
+
+```
+^did:[a-z0-9]+:(?:(?:[a-zA-Z0-9._-]|%[0-9A-Fa-f]{2})*:)*(?:[a-zA-Z0-9._-]|%[0-9A-Fa-f]{2})+$
+```
+
+An earlier pattern here allowed no colons after the method name, which rejected
+every `did:web` carrying a path — in nine places across five entities.
+
+### It does not disturb the round trip
+
+`Transaction.id` is synthetic: `decompose` mints it and `recompose` drops it,
+because V3 has no field for it. `transactionId` is the real V3 data and survives
+untouched. So a `did:web` transaction id **embeds** the V3 identifier rather than
+replacing it, both are present on the entity, and which DID method you choose
+makes no difference to the V3 that comes back. There is a test asserting exactly
+that.
+
+`Property` and `Title` keep `urn:` ids: they are things described, not parties
+that resolve to a DID document. Nothing stops you minting `did:web` for them —
+the URN pattern already permits colons — but nothing requires it either.