@pdtf/schemas 3.6.0-8 → 3.6.0-dev.1

package/CLAUDE.md

@@ -0,0 +1,101 @@
+# CLAUDE.md
+
+This file provides guidance for AI coding agents when working with code in this repository.
+
+## Overview
+
+This repository contains the Property Data Trust Framework (PDTF) schemas - JSON Schema-based definitions for digital residential property data exchange in England and Wales. The framework enables frictionless exchange of property data between software products and services.
+
+## Commands
+
+### Testing
+```bash
+npm test          # Run all tests
+npm run test:watch # Run tests in watch mode
+```
+
+### Development
+```bash
+npm install                       # Install dependencies
+npm run extract-overlays          # Regenerate v3 schemas and overlays from combined.json
+npm run extract-extension-overlays # Regenerate v3 extension overlays from combined.json
+npm run publish:dev               # Publish the dev branch package to npm tag dev
+npm run publish:next              # Publish the next branch package to npm tag next
+npm run publish:latest            # Publish the main branch package to npm tag latest
+```
+
+**Important**: `src/schemas/v3/combined.json` is the source of truth for v3 schema changes. Never edit generated schema artifacts directly; there are no exceptions. Utility/source files may be edited normally. After changing `combined.json`, always run both overlay extraction commands so derived files stay in sync.
+
+## Architecture
+
+### Schema System
+The PDTF uses a flexible overlay system where a base transaction schema can be extended with form-specific overlays:
+
+- **Base Schema**: Core transaction schema at `/src/schemas/v3/pdtf-transaction.json`
+- **Overlays**: Form-specific extensions in `/src/schemas/v3/overlays/` (e.g., baspi5.json, ta6.json)
+- **Merging**: Overlays are merged with the base schema using deepmerge with custom merge strategies
+- **Generated Files**: `pdtf-transaction.json`, `skeleton.json`, `compactSkeleton.txt`, `src/schemas/v3/overlays/*.json`, and `src/schemas/v3/overlays/extensions/*.json` are generated from `combined.json`
+
+### Key Components
+
+1. **index.js**: Main module exposing schema utilities
+   - `getTransactionSchema(schemaId, overlays)`: Merges base schema with overlays
+   - `getValidator(schemaId, overlays)`: Returns AJV validator for schema
+   - `validateVerifiedClaims()`: Validates verified claims structure
+
+2. **Schema Versions**: Currently on v3, with v2 still supported
+   - Use `https://trust.propdata.org.uk/schemas/v3/pdtf-transaction.json` for v3
+
+3. **Overlay System**: Supports multiple UK property forms
+   - BASPI (v4 and v5)
+   - Law Society TA forms (TA6, TA7, TA10)
+   - NTS forms (2023 and 2025 versions)
+   - CON29R, PIQ, RDS, and others
+
+### Testing Approach
+Tests in `/src/tests/` validate:
+- Schema structure with different overlay combinations
+- Path validation for nested properties
+- Custom overlay merging
+- Verified claims validation
+
+Use existing test patterns when adding new tests.
+
+## Branch And Release Workflow
+
+This repository publishes `@pdtf/schemas` through three branches and npm
+dist-tags:
+
+- `main` -> npm `latest`: stable PDTF releases for broad industry use.
+- `next` -> npm `next`: tested capabilities proposed for the next managed PDTF
+  point release.
+- `dev` -> npm `dev`: the unstable coal face for faster-moving integration work
+  and early adopters.
+
+PRs may be merged into `dev` to prove out work before it is ready for the managed
+release path. Once a `dev` PR has been tested and is suitable for standards
+review, promote the same change into `next` via a focused PR. Do not treat `dev`
+as the standard, and do not publish `dev` or `next` builds to npm `latest`.
+
+Promotion is automatic by default: after a PR is merged into `dev`,
+`.github/workflows/promote-dev-to-next.yml` creates a matching PR into `next` by
+cherry-picking the merged dev PR. Add the `no-promote-next` label before merging
+a dev PR only when it should stay out of the `next` review queue. Reviewers
+should still inspect the promotion PR, resolve conflicts, and adjust
+release/version metadata if needed before merging to `next`.
+
+Each npm publish needs a unique semver version. Use `-dev.N` versions on `dev`,
+prerelease versions on `next`, and normal release versions on `main`.
+
+Consumer-specific compatibility profiles belong in the consuming application or
+service unless the capability becomes generic PDTF package functionality.
+Consumers should pin/hash supplied schemas and use their own target-profile
+configuration for API calls and webhooks.
+
+See `docs/branch-release-workflow.md` for the full workflow.
+
+## Important Notes
+
+- All schemas use JSON Schema Draft 07
+- The repository publishes to npm as `@pdtf/schemas`
+- Overlays may contain fields from licensed forms (BASPI, PIQ, Law Society) - ensure compliance when rendering data

package/README.md

@@ -7,7 +7,12 @@ The Property Data Trust Framework (PDTF) Schemas provide standardized JSON Schem
 
 ## Project Goals & Status
 
-**Current Version:** 3.5.0 (3.5.1-1 available as a pre-release on branch/package-tag `next`)
+**Current Version:** 3.5.0 on `main` / npm `latest`.
+
+Additional release channels are available for earlier adoption:
+
+- `next` branch / npm `next`: tested capabilities proposed for the next PDTF release.
+- `dev` branch / npm `dev`: fast-moving integration work for Moverly and other early adopters. This channel may include changes that are not yet accepted as PDTF standard capability.
 
 This schema framework aims to support the [Home Buying and Selling Group](https://homebuyingandsellinggroup.co.uk) 'Property Pack' initiative, encompassing all requirements starting with the Buyers and Sellers Property Information set ([BASPI v4.0](https://homebuyingandsellinggroup.co.uk/baspi/)).
 
@@ -62,7 +67,7 @@ if (!isValid) {
 
 - **🏗️ Modular Schema System** - Base schemas with flexible overlay composition
 - **📋 Multiple Form Support** - BASPI, NTS, Law Society TA forms, and more
-- **🧩 Extension Overlays** - Granular NTS2 features as individual modules
+- **🧩 Extension Overlays** - Granular NTS2 and SEF25 features as individual modules
 - **✅ JSON Schema Validation** - Full JSON Schema Draft 07 support with AJV
 - **🔗 Verified Claims** - Support for verified data provenance tracking
 - **📚 Comprehensive Documentation** - Detailed usage guides and examples
@@ -82,7 +87,7 @@ if (!isValid) {
 │   │   │   │   └── extensions/        # Modular NTS2 extensions
 │   │   │   │       ├── jk.json        # Japanese Knotweed
 │   │   │   │       ├── tf.json        # Transfer Fees
-│   │   │   │       └── ...            # 16 total extensions
+│   │   │   │       └── ...            # NTS2 + SEF25 extensions
 │   │   │   └── combined.json          # Master schema for generation
 │   │   ├── v2/                        # Legacy schema version
 │   │   ├── verifiedClaims/            # Verified claims schemas
@@ -163,31 +168,56 @@ const errors = validateVerifiedClaims(verifiedClaims, schemaId, ["nts2023"]);
 
 #### Extension Overlays
 
-Modular NTS2 features for selective adoption:
+Modular features for selective adoption. Extensions are merged on top of an NTS base overlay.
 
 ```javascript
-// Individual extensions
-const schema = getTransactionSchema(schemaId, ["nts2023", "jk", "tf"]);
-
-// Multiple specialist issues
-const schema = getTransactionSchema(schemaId, [
-  "nts2023",
-  "as",
-  "dr",
-  "jk",
-  "sb",
-]);
+// NTS2 extensions
+const schema = getTransactionSchema(schemaId, ["nts2023", "jk", "tf", "ma"]);
+
+// SEF25 extensions (Seller Enquiry Form)
+const sef25 = ["sc", "pc", "ph", "dk", "rw", "sd", "lc", "wg", "ic", "nd", "mi", "tr"];
+const schema = getTransactionSchema(schemaId, ["nts2023", ...sef25]);
 ```
 
-| Extension         | Code | Description                      |
-| ----------------- | ---- | -------------------------------- |
-| Japanese Knotweed | `jk` | Knotweed presence and management |
-| Transfer Fees     | `tf` | Additional leasehold fees        |
-| Managing Agent    | `ma` | Leasehold managing agent details |
-| Solar Panels      | `sl` | Solar panel ownership details    |
-| Asbestos          | `as` | Asbestos presence and management |
+**NTS2 Extensions:**
 
-[View all extensions →](src/schemas/v3/overlays/README.md#extension-overlays)
+| Extension | Code | Description |
+| --- | --- | --- |
+| Japanese Knotweed | `jk` | Knotweed presence and management |
+| Asbestos | `as` | Asbestos presence and management |
+| Dry Rot | `dr` | Dry rot treatment |
+| Subsidence | `sb` | Subsidence or structural fault |
+| Health & Safety | `hs` | Ongoing health or safety issues |
+| Outside Areas | `oa` | Outside areas details |
+| Main Construction | `mc` | Construction type if standard form |
+| Loft Access | `la` | Loft access and details |
+| Spray Foam | `sf` | Spray foam insulation |
+| Solar Panels | `sl` | Solar panel ownership details |
+| Heating Installed | `hi` | Central heating installation date |
+| Flood Defences | `fd` | Flood defence information |
+| Estate Rentcharges | `er` | Estate rentcharges for freehold |
+| Managing Agent | `ma` | Leasehold managing agent details |
+| Transfer Fees | `tf` | Additional leasehold fees |
+| Onward Chain | `oc` | Other property in chain |
+
+**SEF25 Extensions (Seller Enquiry Form):**
+
+| Extension | Code | Description |
+| --- | --- | --- |
+| Supply Costs | `sc` | Private water/sewerage costs |
+| Parking Permit Cost | `pc` | Parking permit frequency |
+| Property Hazards | `ph` | 4 hazard Yes/No questions |
+| Dropped Kerb | `dk` | Dropped kerb access to parking |
+| Private Right of Way | `rw` | Private right of way |
+| Storm/Fire/Flood Damage | `sd` | Storm, fire or flood damage |
+| Solar Lease Costs | `lc` | Solar panel lease costs |
+| Warranties & Guarantees | `wg` | 7 warranty categories upfront |
+| Insurance Claims | `ic` | Insurance claims upfront |
+| Neighbour Development | `nd` | Neighbour development |
+| Material Issue | `mi` | Other material issue upfront |
+| Title Restrictions | `tr` | Title restrictions for freehold |
+
+[View full SEF25 UI spec →](docs/sef25-extensions-ui-spec.md) | [View all extensions →](src/schemas/v3/overlays/README.md#extension-overlays)
 
 ## Usage Examples
 
@@ -204,7 +234,7 @@ const legalSchema = getTransactionSchema(schemaId, ["ta6ed4", "ta7ed3"]);
 const ntsSchema = getTransactionSchema(schemaId, ["nts2023"]);
 ```
 
-### Modular NTS2 Features
+### Modular Extension Features
 
 ```javascript
 // Selective NTS2 adoption
@@ -215,15 +245,9 @@ const partialNts2 = getTransactionSchema(schemaId, [
   "ma", // Managing Agent
 ]);
 
-// Full specialist issues
-const specialistIssues = getTransactionSchema(schemaId, [
-  "nts2023",
-  "as",
-  "dr",
-  "jk",
-  "sb",
-  "hs",
-]);
+// SEF25 Seller Enquiry Form extensions
+const sef25 = ["sc", "pc", "ph", "dk", "rw", "sd", "lc", "wg", "ic", "nd", "mi", "tr"];
+const sellerEnquiry = getTransactionSchema(schemaId, ["nts2023", ...sef25]);
 ```
 
 ### Data Validation

package/docs/backend-lms-profile-filtering-spec.md

@@ -0,0 +1,180 @@
+# Backend LMS Profile Filtering Spec
+
+## Context
+
+Moverly will consume `@pdtf/schemas@dev` for early access to schema changes, but
+LMS should only receive data shaped to the `pdtf-transaction.json` profile they
+currently support.
+
+This compatibility layer belongs in Moverly backend services because it is an
+integration-boundary concern. The PDTF schema package should not carry
+LMS-specific profile code unless the capability later becomes generic and useful
+to PDTF consumers.
+
+## Target Profile
+
+Initial LMS profile:
+
+- Source: LMS-supplied `pdtf-transaction.json`
+- Profile id: `lms-2026-05-08`
+- Schema id: `https://trust.propdata.org.uk/schemas/v3/pdtf-transaction.json`
+- Raw SHA-256:
+  `5135ffd6158bbe48955ae630cf9cae42ba18f6cd522daed771cd3e44136283d7`
+- Canonical SHA-256:
+  `98aeebe4231557c1aa7bc5182cbb5318d28af32cdcbe3b293feb61dca1e8d6f5`
+
+The LMS profile is not an official PDTF release. It is a pinned recipient
+profile supplied by LMS. When LMS can support an official PDTF release, move the
+target profile to that release artifact instead.
+
+## Configuration
+
+Add a backend environment variable:
+
+```bash
+LMS_PDTF_TARGET_PROFILE=lms-2026-05-08
+```
+
+Behaviour:
+
+- Non-production may default to `lms-2026-05-08`.
+- Production must fail startup or alert loudly if the variable is missing or
+  unknown.
+- Logs and telemetry must include the selected profile id.
+
+## Scope
+
+Apply LMS profile filtering to:
+
+- backend endpoints that expose PDTF transaction data to LMS;
+- webhooks sent to LMS;
+- verified claim payloads included in LMS-facing API responses or webhooks.
+
+Do not apply LMS filtering to:
+
+- stored/source Moverly verified claims;
+- internal Moverly workflows;
+- non-LMS partners;
+- frontend schema rendering, except where frontend displays backend-provided
+  compatibility metadata.
+
+## Filtering Rules
+
+Treat the target profile as closed for outbound LMS payloads, even though PDTF
+schemas generally do not declare `additionalProperties: false`.
+
+For transaction payloads:
+
+1. Build the normal internal Moverly/PDTF payload.
+2. Filter unknown top-level and nested fields against the selected LMS target
+   schema.
+3. Validate the filtered payload against the selected LMS target schema.
+4. Send only the filtered payload.
+
+For verified claims:
+
+1. Iterate each verified claim envelope.
+2. Inspect each `claims` path/value pair independently.
+3. Keep the line item only if:
+   - the path exists in the selected LMS target schema; and
+   - the value validates against that path's subschema.
+4. Drop unsupported or invalid line items.
+5. Preserve the verified claim envelope if at least one line item remains.
+6. Drop the envelope only if all line items are suppressed.
+7. Preserve envelope metadata such as `id`, `transactionId`, `schemaVersion`,
+   `verification`, and `terms_of_use`.
+8. Do not mutate the source verified claim.
+
+Do not map paths or transform values. If a field needs structural mapping, that
+is a separate business/product decision and should not happen implicitly in this
+filtering layer.
+
+## Selective Disclosure Semantics
+
+Suppression means "not disclosed to LMS under this profile".
+
+It does not mean "the seller/source did not assert this". Any backend feature
+answering questions such as "did the seller make an assertion about flooding?"
+must query the source verified claims or explicit suppression metadata, not the
+LMS-filtered payload.
+
+## Suggested Backend Module
+
+Create a small backend-owned module, for example:
+
+```text
+src/integrations/lms/pdtfProfile/
+  lms-2026-05-08/pdtf-transaction.json
+  lms-2026-05-08/metadata.json
+  filterTransactionForProfile.ts
+  filterVerifiedClaimsForProfile.ts
+  profileRegistry.ts
+```
+
+The implementation should:
+
+- load the selected profile by id;
+- verify the pinned file hash in tests;
+- use JSON Schema parsing rather than hand-maintained path allowlists;
+- compile validators with the same AJV draft/settings used for PDTF validation;
+- cache subschema validators by `profileId:path`;
+- return suppression metadata without logging claim values.
+
+## Observability
+
+Record:
+
+- partner: `LMS`;
+- target profile id;
+- profile raw/canonical hash;
+- endpoint or webhook name;
+- transaction id;
+- claim line items before filtering;
+- claim line items after filtering;
+- verified claim envelopes before filtering;
+- verified claim envelopes after filtering;
+- envelopes dropped;
+- unsupported path count;
+- invalid value count;
+- suppressed paths, where safe;
+- validation errors, without sensitive values.
+
+## Tests
+
+Backend tests should cover:
+
+- selected profile id is read from env;
+- unknown production profile id fails loudly;
+- profile metadata hashes match the bundled LMS schema file;
+- transaction filtering removes unknown top-level fields;
+- transaction filtering removes unknown nested fields;
+- filtered transaction validates against the LMS target profile;
+- a field valid in `@pdtf/schemas@dev` but absent from LMS is suppressed;
+- verified claim line items are filtered independently;
+- verified claim envelope metadata is preserved;
+- envelopes are dropped only when all line items are suppressed;
+- source verified claims are not mutated;
+- suppression metadata is returned and contains no claim values.
+
+## Rollout
+
+1. Publish and install `@pdtf/schemas@dev` in Moverly backend.
+2. Add the LMS target profile module and tests.
+3. Set `LMS_PDTF_TARGET_PROFILE=lms-2026-05-08` in non-production.
+4. Run LMS endpoint and webhook regression tests.
+5. Deploy on Monday morning behind normal backend release controls.
+6. Monitor suppression and validation telemetry.
+7. Set the production env var and complete go-live.
+8. Keep the previous behaviour available for rollback until LMS confirms the new
+   payloads are accepted.
+
+## Future Profiles
+
+For a new LMS-supported PDTF release:
+
+1. Add the official released `pdtf-transaction.json` artifact or reference.
+2. Give it a clear profile id, for example `pdtf-3.6.0`.
+3. Add metadata and hash tests.
+4. Update `LMS_PDTF_TARGET_PROFILE`.
+5. Deploy backend and monitor telemetry.
+6. Retain the previous profile until rollback is no longer needed.

package/docs/branch-release-workflow.md

@@ -0,0 +1,138 @@
+# Branch And Release Workflow
+
+This repository uses three long-lived release channels for `@pdtf/schemas`.
+
+## Branches And npm Tags
+
+| Branch | npm dist-tag | Purpose |
+| --- | --- | --- |
+| `main` | `latest` | Stable PDTF releases for broad industry use. |
+| `next` | `next` | Tested capabilities intended for the next managed PDTF point release. |
+| `dev` | `dev` | Unstable coal face for faster-moving integration work and early adopters. |
+
+`dev` is not the standard. It is a proving ground for changes that may later be
+promoted to `next` through focused PRs. Consumers who need the stable standard
+should use `@pdtf/schemas` from `latest`/`main`.
+
+## Versioning
+
+Every npm publish must have a unique semver version.
+
+- `main` publishes normal release versions, for example `3.5.0`, on `latest`.
+- `next` publishes prerelease versions on the `next` tag.
+- `dev` publishes prerelease versions on the `dev` tag, using a `-dev.N`
+  suffix, for example `3.6.0-dev.0`.
+
+Do not publish `dev` or `next` builds to `latest`.
+
+## Dev Fast-Path Workflow
+
+Use this workflow when work needs to move faster than the managed PDTF release
+cycle, or when a change needs practical integration testing before it is proposed
+for the standards-track path.
+
+1. Open a PR against `dev`.
+2. Treat `dev` as the unstable coal face. Keep the change reviewable and, where
+   possible, structured so that accepted parts can later be promoted to `next`.
+3. Change `src/schemas/v3/combined.json` for schema changes, then run both
+   extraction scripts:
+
+   ```bash
+   npm run extract-overlays
+   npm run extract-extension-overlays
+   ```
+
+4. Add or update tests for schema, overlay, path-validation, and utility
+   behaviour.
+5. Bump the package version to the next unique `-dev.N` version if the PR will
+   be published from `dev`.
+6. If the change should not enter the `next` review queue after it has landed in
+   `dev`, add the `no-promote-next` label before merging.
+7. Merge to `dev`.
+8. If publishing the dev channel, publish from `dev`:
+
+   ```bash
+   npm run publish:dev
+   ```
+
+9. Consumers can use `@pdtf/schemas@dev`, or pin the exact dev version in
+   lockfiles when deterministic rollout is required.
+
+## Automated Promotion To next
+
+The `.github/workflows/promote-dev-to-next.yml` workflow can open the matching
+`next` PR automatically.
+
+Trigger:
+
+- A PR is merged into `dev`.
+- The merged PR does not have the `no-promote-next` label.
+
+Behaviour:
+
+1. Check out `next`.
+2. Create a branch named `promote/dev-pr-<number>-to-next`.
+3. Cherry-pick the merged dev PR's effective diff onto that branch.
+4. Open a PR into `next` with a title and body referencing the original dev PR.
+
+If the cherry-pick conflicts, the workflow creates an issue instead of a PR so a
+maintainer can promote the change manually.
+
+The promotion PR is intentionally still reviewed. Before merging to `next`,
+maintainers should:
+
+- confirm the change is appropriate for the managed release path;
+- remove any dev-only scaffolding;
+- adjust package version metadata from `-dev.N` to the correct `next`
+  prerelease version where needed;
+- run tests and schema extraction checks.
+
+## Manual Promotion To next
+
+If automation is not used, or if the automated cherry-pick conflicts:
+
+1. Branch from `next`.
+2. Cherry-pick the merged dev PR commit or recreate the focused diff.
+3. Remove dev-only changes.
+4. Bump to the next unique `next` prerelease version if the PR will be published
+   from `next`.
+5. Open a PR into `next` that links back to the original dev PR.
+6. Merge to `next` after review.
+7. Publish from `next`:
+
+   ```bash
+   npm run publish:next
+   ```
+
+The `next` channel should remain tested and coherent. It can be ahead of
+`main`, but it should not contain unrelated experimental work merely because it
+exists on `dev`.
+
+## Promotion To main
+
+When the PDTF point-release cycle is ready:
+
+1. Select accepted `next` changes for the point release.
+2. Merge or cherry-pick the accepted changes into `main`.
+3. Bump to the release version, for example `3.6.0`.
+4. Publish from `main`:
+
+   ```bash
+   npm run publish:latest
+   ```
+
+Long-running `dev` work should not be merged wholesale into `main` unless the
+full contents have been reviewed and accepted for the PDTF release.
+
+## Consumer Compatibility Profiles
+
+Consumer-specific compatibility profiles are owned by the consuming application
+or service unless and until they become a reusable PDTF capability.
+
+Consumers may pin exact supplied schemas, hash them, and filter outbound API
+calls or webhooks accordingly. That code should not be added to this schema
+package unless the capability is generic and appropriate for PDTF consumers.
+
+If a consumer later supports an official PDTF release, its target profile should
+move from the consumer-specific schema to the relevant published
+`@pdtf/schemas` release or prerelease.