json-rest-api 1.0.22 → 1.0.24

package/CLAUDE.md

@@ -71,7 +71,7 @@ This separation ensures backend developers have full control and visibility into
    - `url-helpers.js` - Centralized URL generation functions
    - `knex-json-api-transformers-querying.js` - JSON:API transformations
    - `relationship-processor.js` - Handles relationship updates
-   - `payload-validators.js` - Request validation
+   - `request-contracts.js` - Compiled request validation contracts
    - `knex-query-helpers.js` - Database query building
    - `simplified-helpers.js` - Simplified API mode transformations
 
@@ -158,4 +158,3 @@ const book = await api.resources.books.post({
 3. **ID Type Confusion**: SQLite returns numeric IDs, but JSON:API expects strings
 4. **Context Placement**: Context is a separate parameter, not inside the first parameter
 5. **Cleanup**: Always destroy knex connection in `after()` hook
-

package/JSKIT_BACKEND_MIGRATION_TRACKER.txt

@@ -0,0 +1,180 @@
+JSKIT Backend Migration Tracker
+
+Scope
+- Move DB, schema, casting, and normalization ownership out of jskit-ai.
+- Enrich json-rest-schema and json-rest-api so they can natively cover the required capability set.
+- Do not add compatibility runtimes.
+- Do not bend json-rest-api or json-rest-schema around jskit-ai shapes unless the capability is a clean native addition.
+
+Status Legend
+- done
+- in_progress
+- pending
+- blocked
+
+Execution Rule
+- Complete one numbered point at a time.
+- After each point, write a full report:
+  - what changed
+  - how it worked before
+  - how it works now
+  - relevant code snippets
+- Stop and wait for the next user prompt before starting the next numbered point.
+
+Work List
+
+0. [done] Create and maintain this tracker as the canonical work ledger.
+   Notes:
+   - This file is the progress source of truth for the migration work.
+   - Status changes must be recorded here before moving to the next point.
+
+1. [done] Tighten json-rest-schema scalar semantics.
+   Deliverables:
+   - make `id` represent the real external/resource ID contract
+   - tighten boolean parsing
+   - tighten number/integer parsing where schema should reject invalid inputs instead of silently coercing
+
+2. [done] Define a native json-rest-schema create/replace/patch validation contract.
+   Deliverables:
+   - explicit operation mode semantics
+   - presence-aware field handling
+   - patch behavior that preserves omission and only normalizes touched fields
+   - switch json-rest-api call sites to the explicit schema contract
+   Notes:
+   - verification blocker removed by deleting stale DB-helper tests from `json-rest-schema`
+   - DB-helper coverage now lives in `json-rest-api`, which owns `createKnexTable()` and `generateKnexMigration()`
+   - verified end-to-end against a local symlinked `json-rest-schema@1.0.13`; `json-rest-api` test suite passes with the new schema contract
+
+3. [done] Add transport-schema export from json-rest-schema.
+   Deliverables:
+   - export usable by JSON Schema / Ajv style consumers
+   - enough structure for pre-handler validation in connectors such as Fastify
+   Notes:
+   - added `schema.toJsonSchema({ mode, additionalProperties })` in `json-rest-schema`
+   - export is draft-07, mode-aware, and transport-facing rather than a clone of every in-process coercion path
+   - custom transport export hooks are supported through `handler.toJsonSchema`
+   - unknown external metadata keys are ignored; custom registered validators without a transport hook fail loudly
+
+4. [done] Extend json-rest-schema field metadata only where it remains schema-level meaning.
+   Deliverables:
+   - preserve useful passive metadata such as nullability, defaults, precision, scale, unsigned, enum values, and temporal precision
+   Notes:
+   - made `enum` a first-class built-in rule in `json-rest-schema`
+   - added canonical passive metadata support for `temporalPrecision`
+   - transport export now preserves passive schema metadata under `x-json-rest-schema.metadata`
+   - passive metadata remains schema-owned metadata; no DB-specific runtime logic was added to `json-rest-schema`
+
+5. [done] Add repository/storage mapping in json-rest-api.
+   Deliverables:
+   - API field to DB column mapping
+   - virtual/non-column fields
+   - write serializers where persistence formatting is required
+   Notes:
+   - added `field.storage = { column, serialize }` as the native repository/storage surface in `json-rest-api`
+   - storage metadata is compiled once into `schemaInfo.storageInfo`
+   - Knex writes now translate logical attributes to storage columns through the compiled mapping
+   - table creation and migration generation now respect storage column overrides
+   - JSON:API transformers translate storage rows back to logical field names before response shaping
+   - storage adapters now own row/id/select translation used by Knex callers, reducing repeated mapping helpers
+   - runtime row access now uses the same logical field path for `id` as for other fields: `getFieldValue(record, 'id')`
+   - include handling and relationship links now use logical record IDs consistently, including custom-id resources
+   - direct DB helper surfaces now support logical `id` fields mapped to custom storage columns without duplicate-column drift
+   - verified with dedicated storage-mapping tests and the full `json-rest-api` suite
+
+6. [done] Add DB introspection snapshot support in json-rest-api.
+   Deliverables:
+   - column type and shape introspection
+   - enum/set values
+   - indexes
+   - foreign keys
+   - check constraints
+   - precision/scale/temporal precision
+   Notes:
+   - added `introspectKnexTableSnapshot(knex, { tableName, idColumn })` as the native normalized snapshot helper
+   - snapshot support is dialect-aware for `better-sqlite3`/SQLite and `mysql2`
+   - SQLite captures named check constraints and foreign keys from table SQL, while indexes/columns/foreign-key actions come from PRAGMA metadata
+   - `RestApiKnexPlugin` now exposes `api.resources.{resource}.introspectKnexTableSnapshot()` for table-backed resources
+   - verified with real SQLite snapshot tests, MySQL raw-double snapshot tests, and a public scope-method test
+
+7. [done] Add deterministic migration generation and diffing in json-rest-api.
+   Deliverables:
+   - create and alter support
+   - named/composite indexes
+   - FK rules
+   - enum/set handling
+   - defaults and safety warnings for destructive changes
+   Notes:
+   - `dbTablesOperations` now normalizes table-level metadata (`indexes`, `foreignKeys`, `checkConstraints`) and uses one contract for live table creation, create-migration generation, and snapshot diffing
+   - field-level `unique` / `index` / `references` are compiled into deterministic named indexes and foreign keys instead of relying on ad hoc column builder side effects
+   - added `generateKnexMigrationDiff(tableName, currentSnapshot, schema, options)` returning `{ migration, warnings, plan }`
+   - additive diff generation supports columns, indexes, and foreign keys; destructive or unsupported changes stay explicit as warnings
+   - `RestApiKnexPlugin` now exposes `generateKnexMigration()` and `generateKnexMigrationDiff()` on table-backed scopes and threads table-level metadata through `schemaInfo`
+
+8. [done] Preserve and clarify the pass-through searchSchema contract in json-rest-api.
+   Deliverables:
+   - searchSchema validates allowed filter keys and normalizes values only
+   - filter meaning remains backend-owned and server-defined
+   - arbitrary public filter keys are supported when the backend maps them intentionally
+   - regression coverage for shared merge/validation behavior and invalid filter keys
+   Notes:
+   - did not introduce a generic filter DSL
+   - documented that searchSchema keys are public filter names, not a portable operator language
+   - verified that alias filters (`oneOf`) and custom backend filters (`applyFilter`) work as intended in the Knex path, where their execution semantics are owned by the backend
+
+9. [done] Add declarative autofiltering/scoping in json-rest-api.
+   Deliverables:
+   - public
+   - workspace
+   - user
+   - workspace_user
+   Notes:
+   - replaced the purpose-specific multihome surface with a generic `AutoFilterPlugin`
+   - resources now declare `autofilter` presets or inline filters, while the plugin resolves scope values from configured resolver functions
+   - `workspace`, `user`, `workspace_user`, and `public` are example presets, not built-in auth meanings
+   - the plugin scopes collection queries and single-record lookups, stamps scoped fields on `POST`/`PUT`, and rejects inconsistent scoped updates
+   - scoped `belongsTo` relationships are supported by injecting or validating JSON:API relationship identifiers before write validation
+   - auth/admission semantics remain out of scope for `json-rest-api`; core keeps `checkPermissions` hooks as a neutral extension point only
+   - shipped `AccessPlugin` was removed before this point so scoping does not inherit built-in auth assumptions
+
+10. [done] Add SQL-backed query projections and stable list behavior in json-rest-api.
+    Deliverables:
+    - SQL-backed output-only query fields via an optional plugin
+    - sparse-field and include support for declared query fields
+    - deterministic ordered-list / cursor behavior with stable tie-breakers
+    Notes:
+    - added an optional `QueryProjectionsPlugin` that compiles resource-level `queryFields`, separate from normal schema fields and response-layer computed fields
+    - query fields are selected by default, can be requested in sparse fieldsets, and are stripped from write input
+    - query fields can drive sorting and cursor pagination; `id` is appended as a deterministic tie-breaker
+    - query-field selection is supported on the Knex path and the AnyAPI Knex path, including included resources
+    - documented as a separate advanced guide so it does not blur with response-layer computed fields
+
+11. [done] Add a Fastify-oriented connector layer in json-rest-api.
+    Deliverables:
+    - consume exported schema before request handling
+    - no jskit-ai compatibility shim
+    Notes:
+    - added `FastifyPlugin` as a transport connector that registers REST routes on a supplied Fastify instance
+    - resource `POST`/`PUT`/`PATCH` routes now attach transport-facing JSON Schemas built from `schema.toJsonSchema({ mode })`, wrapped in the JSON:API document envelope
+    - connector-level transport schemas strip output-only/computed fields and direct `belongsTo` foreign keys from `data.attributes`, matching the existing transport write contract
+    - connector registers an `application/vnd.api+json` parser and rejects unsupported write content types before route handlers run
+    - no jskit-ai compatibility shim or duplicate hand-written field validation layer was added
+
+12. [pending] Remove the corresponding DB/schema/normalization ownership from jskit-ai and verify parity.
+    Deliverables:
+    - no jskit-ai-owned DB/schema normalization path remains for the migrated surface
+    - verification notes for anything still intentionally left outside scope
+
+ADDENDUM ITEMS TO POINT 12:
+
+* This is the reason we did points 1 to 11 (!).
+* Needs to be done with a lot of care, since it's harder than it looks 
+* You MUST read AGENTS.md in jskit-ai VERY carefully, follow it VERY carefully. One of the previous attempts (4 failed in total) was unsuccessful because this step was missed. That file has important instructions on how to avoid repeated helpers and a pointer to important function maps
+* There must be NOTHING left of the in house DB stuff. jskit is only a user of what json-rest-api offers.
+* NO SHIMS. NO COMPATIBILITY LAYERS. NO LAYERS AT ALL. NO RUNTIMES.
+* In an earlier attempt, you made a runtime that mimiked the repository's functions. NO. Repository functions will use json-rest-api's functions to do what they need to do
+* A repository should be simple. It can have helper functions, but if it has a LOT of helper functions, there is a problem -- interrupt and let's talk. This happened in the workspace repository
+* Multi homing is to be done with the generic filtering option of json-rest-api
+* Take a LOT of care with the definiton of the resource file. Do NOT keep the old Type + Normalisation flow. That is GONE. We now have json-rest-schema, and that's what you need to use. One of the 4 failed attempts failed because you were stubborn about reusing that paradigm
+* The resource file (which is what is used by the server, AND by the client) needs to be clean, and contain the right information (schemas, relationships, etc). It's a DATA ONLY file.
+* The goal is to devoid jskit of any db logic/layering/
+

package/README.md

@@ -72,4 +72,25 @@ app.listen(3000, () => {
 - [Quick Start](QUICKSTART.md) - Quick start
 - [Complete guide](GUIDE/) - Comprehensive guide
 - [API Reference](API.md) - API reference
-- [Why json-rest-api?](docs/COMPARISON.md) - Step-by-step tutorial
+- [Why json-rest-api?](docs/COMPARISON.md) - Step-by-step tutorial
+
+## Development
+
+```bash
+# Run the normalizeId contract suites under the default backend and AnyAPI,
+# then rebuild the checked-in docs site
+npm run verify
+
+# Run lint separately
+npm run lint
+
+# Run the focused normalizeId suites directly
+npm run test:id-contracts
+npm run test:id-contracts:anyapi
+
+# Rebuild the checked-in docs site
+npm run docs
+
+# Serve the docs locally with live rebuild
+npm run docs:dev
+```

package/SINGLE_VALIDATION_CONTRACT_TODO.txt

@@ -0,0 +1,172 @@
+Single Validation Contract TODO
+
+Goal
+- Make `json-rest-schema` the single authored validation contract in `json-rest-api`.
+- Keep plain JSON Schema only as a derived transport artifact for connectors.
+- Remove the split between:
+  - resource/search validation via `json-rest-schema`
+  - hand-written JSON:API/query validators in `plugins/core/lib/querying-writing/payload-validators.js`
+
+Current audit status
+- [done] Audit the live validation paths and identify the actual indirection.
+
+Root cause
+- Validation is split across three places:
+  1. `plugins/core/lib/querying-writing/compile-schemas.js`
+     - compiles resource and search schemas with `createSchema(...)`
+  2. `plugins/core/lib/querying-writing/payload-validators.js`
+     - manually validates JSON:API request envelopes and query parameter objects
+  3. `plugins/core/connectors/lib/transport-route-schemas.js`
+     - separately builds connector-facing JSON Schemas from `schemaInfo`
+- This duplicates contract meaning across runtime and transport.
+- Route methods still call imperative payload validators before calling
+  `schemaInstance.create()`, `schemaInstance.replace()`, or `schemaInstance.patch()`.
+- `setupCommonRequest()` in `plugins/core/rest-api-plugin-methods/common.js`
+  also repeats some checks that are already transport-contract concerns.
+
+What the neat end state looks like
+- Authored contract:
+  - `json-rest-schema` only
+- Derived transport schema:
+  - plain JSON Schema, generated from the same compiled contract
+- Runtime write validation:
+  - uses the same compiled contract as Fastify pre-validation
+- Remaining imperative checks:
+  - only stateful/business checks that are not schema concerns
+  - examples: permission checks, record existence, relationship access, complete-replace semantics
+
+What should stay imperative
+- `validateRelationshipAccess()` in `plugins/core/rest-api-plugin-methods/common.js`
+- `validateCompleteReplacePayload()` in `plugins/core/rest-api-plugin-methods/common.js`
+- path/body cross-source checks such as URL id vs body id consistency
+- permission and data existence checks
+
+What should stop existing
+- `validateGetPayload()`
+- `validateQueryPayload()`
+- `validatePostPayload()`
+- `validatePutPayload()`
+- `validatePatchPayload()`
+- duplicate route/body/query syntax checks spread across route methods and `setupCommonRequest()`
+
+Work list
+
+1. [done] Define one compiled request-contract surface for resource `get/query/post/put/patch`.
+   Target files:
+   - `plugins/core/lib/querying-writing/compile-schemas.js`
+   - `plugins/core/connectors/lib/transport-route-schemas.js`
+   Deliverables:
+   - compile request contracts once per scope
+   - store them on `scope.vars.schemaInfo` or an adjacent compiled-contract object
+   - separate:
+     - write document contracts (`post`, `put`, `patch`)
+     - query/get contracts (`get`, `query`)
+     - relationship route contracts
+   Status:
+   - resource route contracts now compile once in `request-contracts.js`
+   - relationship route contracts are still separate and remain pending work
+
+2. [done] Replace manual write-document validation with compiled schema contracts.
+   Target files:
+   - `plugins/core/lib/querying-writing/payload-validators.js`
+   - `plugins/core/rest-api-plugin-methods/post.js`
+   - `plugins/core/rest-api-plugin-methods/put.js`
+   - `plugins/core/rest-api-plugin-methods/patch.js`
+   Deliverables:
+   - one compiled JSON:API document contract per write mode
+   - `data.type` constrained to the scope name inside the contract
+   - `data.id` required/optional according to mode
+   - `attributes` contract derived from `schemaInstance.toJsonSchema({ mode })`
+   - relationship identifier contracts derived from `schemaRelationships` and `belongsTo` metadata
+   - `included` rules expressed once in the contract rather than repeated manually
+
+3. [done] Replace manual GET/query parameter validation with compiled schema contracts.
+   Target files:
+   - `plugins/core/lib/querying-writing/payload-validators.js`
+   - `plugins/core/rest-api-plugin-methods/get.js`
+   - `plugins/core/rest-api-plugin-methods/query.js`
+   Deliverables:
+   - a compiled `get` contract for:
+     - `id`
+     - `queryParams.include`
+     - `queryParams.fields`
+   - a compiled `query` contract for:
+     - `include`
+     - `fields`
+     - `filters`
+     - `sort`
+     - `page`
+   - include-depth validation and sortable-field restrictions represented in the compiled contract, not duplicated in ad hoc code
+   Notes:
+   - `searchSchemaInstance` should remain the value contract for `filters`
+   - the query contract should validate the envelope/shape, not duplicate filter semantics
+
+4. [done] Collapse Fastify pre-validation and in-process validation onto the same source for resource write routes.
+   Target files:
+   - `plugins/core/connectors/lib/transport-route-schemas.js`
+   - `plugins/core/connectors/fastify-plugin.js`
+   - route methods in `plugins/core/rest-api-plugin-methods/`
+   Deliverables:
+   - Fastify consumes transport JSON Schema exported from the compiled request contract
+   - route methods consume the same compiled contract directly for runtime validation
+   - no second hand-written envelope contract path
+
+5. [done] Shrink `setupCommonRequest()` to non-schema responsibilities.
+   Target file:
+   - `plugins/core/rest-api-plugin-methods/common.js`
+   Deliverables:
+   - keep simplified-mode conversion, transaction setup, context wiring
+   - keep stateful/business checks only
+   - remove syntax/envelope checks that belong in the compiled contract
+   - keep only cross-input checks that a single schema cannot express cleanly
+
+6. [done] Delete `payload-validators.js`.
+   Target file:
+   - `plugins/core/lib/querying-writing/payload-validators.js`
+   Deliverables:
+   - no route method imports it anymore
+   - helper logic worth keeping is either:
+     - absorbed into compiled contracts
+     - or moved into narrowly named stateful helpers
+
+7. [done] Tighten tests around the single-contract flow.
+   Target areas:
+   - `tests/fastify-plugin.test.js`
+   - route method coverage exercised by the existing REST suites
+   - add one focused contract test suite if needed
+   Deliverables:
+   - prove Fastify pre-validation and runtime validation reject the same bad inputs
+   - cover:
+     - wrong `data.type`
+     - missing/invalid `data.id`
+     - invalid relationships
+     - invalid query/include/page/sort shapes
+   - verify there is no drift between transport schema and runtime schema behavior
+
+8. [done] Proof cleanup.
+   Deliverables:
+   - `rg` proof that route methods no longer import `payload-validators.js`
+   - `rg` proof that `validateGetPayload`, `validateQueryPayload`, `validatePostPayload`,
+     `validatePutPayload`, and `validatePatchPayload` are gone
+   - `npm test`
+
+Current state
+- Resource `get/query/post/put/patch` now use one compiled contract source.
+- Fastify transport schemas for those resource routes are derived from the same compiled contract.
+- `payload-validators.js` is deleted.
+- `npm test` passes.
+- The main remaining simplification is the separate relationship-route contract path.
+
+File map for the next implementation pass
+- Remaining relationship-route cleanup:
+  - `plugins/core/rest-api-plugin-methods/post-relationship.js`
+  - `plugins/core/rest-api-plugin-methods/patch-relationship.js`
+  - `plugins/core/rest-api-plugin-methods/delete-relationship.js`
+- Transport schema generator to reuse, not duplicate:
+  - `plugins/core/connectors/lib/transport-route-schemas.js`
+
+Explicit non-goals
+- Do not move permission checks into schemas.
+- Do not move DB existence checks into schemas.
+- Do not turn `searchSchema` into a generic filter DSL.
+- Do not add a compatibility layer that preserves both manual validators and compiled contracts.