@dialecte/core 0.7.0 → 0.8.0

package/skills/dialecte/references/core-api.md

@@ -0,0 +1,204 @@
+# @dialecte/core API map
+
+A map to find the right existing function, not a signature reference. Confirm the exact signature
+in the installed `.d.ts` or the repo before calling. Paths are relative to the core package `src/`.
+
+## Contents
+
+- Entry points
+- Project
+- Document
+- Query
+- Transaction
+- Records, refs, operations
+- Hooks (signatures and rules)
+- Extensions mechanism
+- Helpers (`@dialecte/core/helpers`)
+- Utils (`@dialecte/core/utils`)
+- Errors, diagnostics, perf
+- Stores
+- Easy-to-duplicate list
+
+## Entry points
+
+| Import                   | What is there                                                                                                                                    |
+| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `@dialecte/core`         | Project, Document, Query, Transaction, `defineDialecte`, `createProjectFactory`, stores, XML parse/build, errors, diagnostics, perf, all types   |
+| `@dialecte/core/helpers` | record converters, guards, `readAttribute(s)`, `standardizeRecord`, `mergeExtensions`, `stripAttributes`, write-path guards, namespace constants |
+| `@dialecte/core/utils`   | attribute rules, definition resolution, xsi:type, namespaces, ordering, `invariant`, `formatXml` / `formatEmbeddedXml`, `saveToDisk`             |
+| `@dialecte/core/test`    | `createTestHelpers`, `createTestRunner`, `createTestProject`, XML/XPath assertions, test dialecte, `normalizeUuids`                              |
+| `@dialecte/core/sqlite`  | opt-in SQLite storage: importing it registers `{ type: 'sqlite' }`; `connectHub`, `adoptHubPort`, `listProjects`                                 |
+
+## Project (`project/project.ts`)
+
+- `open(name)` - must come first; opens the store, hydrates `state.documents`. `close()` is async
+  and waits for running imports. `destroy()` deletes the database; it is refused `PROJECT_OPEN_ELSEWHERE`,
+  nothing removed and the project closed, while another connection of the name is open (another project
+  or store, in this page or another tab): close that one, then call `destroy()` again. A test closes the stores
+  `createTestContext` opened by name with `closeTestContexts()` before destroying their project.
+- `importFile(file)` → `{ documentId, fileName, recordCount, diagnostics }` or throws; `import(files)`
+  → per-file `{ status: 'imported' | 'failed' }`, never rejects. The import runs inside the store; imports
+  are serialised per project; a document being imported is invisible until ready.
+- `export({ documentId, indent?, withDatabaseIds?, xmlDocument?, download? })` → `{ file: Blob, filename }`
+  (+ `xmlDocument` when asked, page only). The store writes the file without a DOM, so export works in a
+  worker and in Node.
+- `initEmptyDocument()`, `removeDocument(id)`, `getDocuments()`, `getDocument(id)`,
+  `openDocument(id)`, `getDocumentStatus(id)`, `getDocumentConfig(id)`.
+- `undo(id)` / `redo(id)` - store-level change log.
+- Blobs: `addBlob`, `getBlob`, `exportBlob`, `attachBlob`, `detachBlob`, `removeBlob`,
+  `getBlobsByDocument`, `getBlobsByRecord`, `getStandaloneBlobs`. `getDatabaseInstance()` for the raw DB.
+- Cross-document: `queryFirst(fn)`, `queryAll(fn)`.
+- `state` (documents map), `perf`, `createChannel()` (a BroadcastChannel on the project's channel: what changed,
+  this tab's changes and other tabs'). On a store that announces its changes (`Store.subscribe`: SQLite,
+  in-memory) the channel is the Project's own and reposts what the store tells; on `local` (IndexedDB) it is
+  shared by project name.
+- `listProjects({ dialecteId? })` from `@dialecte/core/sqlite` (SQLite projects of the origin, read from the private
+  file system) and `factory.destroyProject({ name })` (delete one without opening it).
+
+## Document (`document/document.ts`)
+
+- `query` - getter, a new Query each access (extensions bound).
+- `transaction(fn, { label? })` - callback → `beforeCommit` → atomic commit → broadcast → history.
+- `prepare(fn)` → `{ operations, summary, query, commit(), discard() }` - preview before commit.
+- `state` (`loading, error, progress, history, lastUpdate`), `subscribe(cb)`, `documentId`.
+
+## Method index (generated from the source)
+
+<!-- dialecte-skills:generated core-methods -->
+
+- Query (`doc.query`, also on `tx`): `findAncestors`, `findByAttributes`, `findDescendants`, `getAttribute`, `getAttributes`, `getChild`, `getChildren`, `getDefinition`, `getDocumentInfo`, `getRecord`, `getRecords`, `getRecordsByTagName`, `getRoot`, `getSnapshot`, `getTree`
+- Transaction only (`tx`): `addChild`, `clearRecordCache`, `clearStagedOperations`, `commit`, `deepClone`, `delete`, `ensureChild`, `finishStaging`, `getStagedOperations`, `update`
+- Document: `close`, `destroy`, `prepare`, `subscribe`, `transaction`
+- Project: `addBlob`, `attachBlob`, `close`, `createChannel`, `destroy`, `detachBlob`, `export`, `exportBlob`, `getBlob`, `getBlobsByDocument`, `getBlobsByRecord`, `getDatabaseInstance`, `getDocument`, `getDocumentConfig`, `getDocumentStatus`, `getDocuments`, `getStandaloneBlobs`, `import`, `importFile`, `initEmptyDocument`, `open`, `openDocument`, `queryAll`, `queryFirst`, `redo`, `removeBlob`, `removeDocument`, `undo`
+<!-- /dialecte-skills:generated -->
+
+## Query (`document/query/main.ts`)
+
+| Method                                                                            | Use it for                                                                                                                            |
+| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| `getRoot()`                                                                       | the root record                                                                                                                       |
+| `getRecord(ref)` / `getRecords(refs)`                                             | one / many records, `undefined` when missing                                                                                          |
+| `getChild(ref, tag)` / `getChildren(ref, tag)`                                    | direct children by tag                                                                                                                |
+| `getRecordsByTagName(tag)`                                                        | every record of a tag in the document                                                                                                 |
+| `findDescendants(ref, { collect, omit })`                                         | descendants grouped by tag (NOT document order)                                                                                       |
+| `findAncestors(ref, { depth, stopAtTagName, order })`                             | the ancestor chain (bottom-up by default)                                                                                             |
+| `findByAttributes({ tagName, attributes })`                                       | lookup by attribute values                                                                                                            |
+| `getTree(ref, { select: { where, ... }, omit, unwrap, depth })`                   | a subtree in document order; unwraps transparent elements by default; whole-document load unless a committed read with `id` + `depth` |
+| `getSnapshot({ ref, ancestors, siblings, depth, as: 'tree' \| 'xml' \| 'both' })` | a view with staged ops overlaid                                                                                                       |
+| `getAttribute(ref, { name, namespace?, fullObject?, defaults? })`                 | one attribute (`''` when absent)                                                                                                      |
+| `getAttributes(ref, { namespace?, fullObject?, defaults? })`                      | value object of attributes                                                                                                            |
+| `getDefinition(ref)`                                                              | resolved definition (parent- and xsi:type-aware)                                                                                      |
+| `getDocumentInfo()`                                                               | the document's registry record                                                                                                        |
+| `any`                                                                             | untyped escape hatch - avoid in domain code                                                                                           |
+
+## Transaction (`document/transaction/main.ts`, extends Query)
+
+| Method                                                               | Use it for                                                                             |
+| -------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
+| `addChild(parent, { tagName, attributes, namespace?, value?, id? })` | create; standardised, hooks fire                                                       |
+| `ensureChild(parent, params)`                                        | get-or-create among direct children (match by attributes, then id, then tag)           |
+| `update(ref, { attributes?, value? })`                               | change; an `undefined`/`null` attribute value removes it; no-op updates stage nothing  |
+| `delete(ref)`                                                        | delete with cascade; root is protected                                                 |
+| `deepClone(parent, treeRecord)`                                      | copy a `getTree` result under a parent; always new ids; returns source→target mappings |
+| `progress`                                                           | `plan`, `nextStep`, `endPlan` for long operations                                      |
+
+`finishStaging`, `commit`, `getStagedOperations`, `clear*` are internal - Document calls them.
+
+## Records, refs, operations (`types/records.ts`, `document/types.ref.ts`, `types/operations.ts`)
+
+- `RawRecord` `{ id, tagName, namespace, attributes[], value, parent: {id, tagName} | null, children[] }`
+  (stored shape; a root's `parent` is `null`, not `undefined`).
+- `TrackedRecord` = raw + `status` (created/updated/deleted/unchanged) - what queries return.
+- `TreeRecord` = tracked + `tree[]` - what `getTree` returns.
+- `Ref` `{ tagName, id }` (id optional only for singletons). Methods accept any `RefOrRecord`.
+- `Operation`: created(new) | updated(old, new) | deleted(old).
+- Attribute forms: value object uses stored names (`'ext:cA'`, `'xsi:type'`); full-object form
+  takes a local `name` plus `namespace` - a prefixed name there is rejected.
+
+## Hooks (`types/dialecte-config.ts`, `types/io.ts`)
+
+Provided by the dialecte package (on the Project, built once per project), not by app code in
+general. Signatures in the installed `.d.ts`; behaviour:
+
+| Hook                                                                                                              | Receives                               | Returns                                                                                             | Notes                                                                                                                         |
+| ----------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| `beforeClone`                                                                                                     | `{ record }` (tree)                    | `{ shouldBeCloned, transformedRecord }`                                                             | sync, per node; false drops the subtree                                                                                       |
+| `afterStandardizedRecord`                                                                                         | `{ record }`                           | record                                                                                              | sync, idempotent; runs on create, update, import, init                                                                        |
+| `afterCreated`                                                                                                    | `{ childRecord, parentRecord, query }` | `Operation[]`                                                                                       | after child + parent staged                                                                                                   |
+| `afterUpdated`                                                                                                    | `{ oldRecord, newRecord, query }`      | `Operation[]`                                                                                       | skipped for no-op updates                                                                                                     |
+| `beforeDelete`                                                                                                    | `{ record, query }`                    | `Operation[]`                                                                                       | once on the deletion root, subtree still readable                                                                             |
+| `afterDelete`                                                                                                     | `{ record, parentRecord, query }`      | `Operation[]`                                                                                       | subtree gone                                                                                                                  |
+| `beforeCommit`                                                                                                    | `{ operations, query }`                | `Operation[]`                                                                                       | once per transaction; its ops fire no hooks                                                                                   |
+| `beforeImport`                                                                                                    | xml string                             | xml string                                                                                          | whole-file normalisation                                                                                                      |
+| `createImportSession` → `beforeImportRecord` (one session per file; `recordTagNames` narrows the records it sees) | `{ record, ancestry }`                 | void                                                                                                | sync, per record at its closing tag (children before parent); `ancestry` = still-open parents, not standardised; collect only |
+| `afterImport`                                                                                                     | -                                      | `{ creates, updates, deletes, diagnostics }` (the diagnostics reach the app with the import result) | resolve from collected state; bulk write, no undo                                                                             |
+
+Hook-returned operations skip standardisation, guards and further hooks. A returned `deleted`
+needs the parent's `updated` (children list) too.
+
+## Extensions mechanism (`document/types.extensions.ts`, `helpers/merge-extensions.ts`, `document/bind-extensions.ts`)
+
+- A module is `{ query?, transaction? }`, groups nest to any depth.
+- `mergeExtensions({ base, custom })` merges per method at Project construction; duplicates throw
+  `EXTENSION_METHOD_COLLISION`.
+- `bindExtensions` drops the first argument: `fn(query, params)` becomes `doc.query.<ns>.fn(params)`.
+- Inside a transaction both query and transaction groups are bound to `tx`.
+- Naming an augmented document: `ExtendedDocument<Config, Modules>`; do not hand-write intersections.
+
+## Helpers (`@dialecte/core/helpers`)
+
+- `toRef(recordOrRef)`, `toRawRecord`, `toTrackedRecord`, `toTreeRecord`, `widen`, `toFullAttributeArray`.
+- Guards: `isRawRecord`, `isTrackedRecord`, `isTreeRecord`, `isRecordOf`, `isChildOf`, `isParentOf`,
+  `isElementOf`, `isAttributeOf`, `isFullAttributeArray`.
+- `readAttribute` / `readAttributes` - sync, same options and results as the query methods, on a
+  record as held (no staged writes); take `dialecteConfig`. A dialecte package may export a bound
+  version - prefer it.
+- `standardizeRecord`, `stripAttributes`, `mergeExtensions`.
+- Write guards: `assertNoFixedViolation`, `assertXsiTypeAdmitted`, `assertXsiTypeChangeLeavesNothing`.
+- Constants: `CUSTOM_RECORD_ID_ATTRIBUTE` (`dev:db-id`), `DIALECTE_DEV_NAMESPACE`, `XSI_NAMESPACE`, ...
+
+## Utils (`@dialecte/core/utils`)
+
+- `invariant(condition, { detail, key?, ref?, meta? })`.
+- Attribute rules: `getAttributeRules`, `resolveSchemaAttributeValue`, `isSchemaDefaultValue`, `extractLocalName`.
+- Definitions: `resolveDefinition(record)` (needs a record; for a ref use `query.getDefinition`),
+  `admittedXsiTypes`, `resolveXsiType`, `canonicalXsiType`.
+- Datatypes: `enumeratedValues`, `valuePatterns`.
+- Namespaces: `resolveNamespaceByPrefix`, `resolveNamespaceByScope`, `resolvePrefixByNamespaceScope`,
+  `compareQualifiedAttributes`, `orderAttributesBySequence`, `isNamespaceAllowedByWildcard`.
+- Ordering: `orderByConfigSequence`.
+- XML text: `formatXml` (template-tolerant, for fixtures), `formatEmbeddedXml` - only from
+  `@dialecte/core/utils`; the main entry has no `formatXml` (`project.export` formats its file).
+
+## Errors, diagnostics, perf
+
+- `throwDialecteError(key, { detail, ref?, meta?, cause? })`, `createDialecteError`, `toDialecteError`,
+  `isKnownDialecteError(err, { catalogues })`, `defineErrors(catalogue)` for a dialecte's own codes.
+- Core codes by range: D0 general, D1 store, D2 records, D3 constraint violations, D4 transaction,
+  D5 IO, D6 config (extension collision), D7 project.
+- Diagnostics (import warnings, validation): `createDiagnostic(catalogue, code, { ref, meta })`,
+  `isKnownDiagnostic`. LSP-shaped `{ code, severity, source, message, ref }`.
+- Perf: `dev: { perf: true }` on project creation (when the package factory accepts `dev`; check its signature); `perf.start/stop/count/time/report()`; marks are
+  `dialecte::<layer>::<name>` on the standard performance timeline.
+
+## Stores (`store/`)
+
+- `storage` param: `{ type: 'local' }` (IndexedDB, the default), `{ type: 'sqlite' }` (the SQLite
+  engine in a worker, one per project shared by every tab through the hub; opt-in: the app imports
+  `@dialecte/core/sqlite` once, else `PROJECT_SQLITE_NOT_IMPORTED`; opened from a dialecte's project
+  factory, which names its engine worker),
+  `{ type: 'inMemory', writable? }`,
+  `{ type: 'custom', store }`. The store is a switch: results must be identical on every store,
+  only speed changes. A difference is a store bug.
+- `Store` interface is public surface for custom stores. The patch merge every built-in store uses
+  is internal to the stores (not exported): a custom store merges a `RecordPatch` itself.
+
+## Easy-to-duplicate list - grep these before writing
+
+ref conversion `toRef` · record shapes `toRawRecord`/`toTreeRecord`/`widen` · attribute reads
+`getAttributes`/`readAttributes` · lookup by attributes `findByAttributes` · canonical order
+`orderAttributesBySequence`/`orderByConfigSequence` · definition `getDefinition`/`resolveDefinition`
+· xsi:type `canonicalXsiType` · namespaces `utils/namespaces.ts` · errors `invariant`/`throwDialecteError`
+· diagnostics `createDiagnostic` · binding `bindExtensions` ·
+subtree copy `getTree` + `deepClone` (but check the dialecte for a faithful domain copy first) ·
+XML text `formatXml`.

package/skills/dialecte/references/testing.md

@@ -0,0 +1,116 @@
+# Testing with Dialecte
+
+Every dialecte package ships table-driven runners built on core's `createTestRunner`. Use them;
+do not hand-roll project setup, import, export and cleanup in each test.
+
+| Package                        | Import                     | Runner                                                                                                                |
+| ------------------------------ | -------------------------- | --------------------------------------------------------------------------------------------------------------------- |
+| core (its dummy test dialecte) | `@dialecte/core/test`      | `createTestHelpers({ dialecte })` → `runTestCases`, `createTestProject`, ...; the dummy dialecte's own `runTestCases` |
+| a dialecte package             | `<package>/<version>/test` | `run<Name>TestCases`, `create<Name>TestProject`, ... (see that dialecte's skill)                                      |
+
+## The three modes
+
+- `.withExport({ testCases, act })` - the outcome is XML. `act` performs the write and may return
+  `{ assertOn: 'source' | 'target', withDatabaseIds? }` (default: source, ids kept) or
+  `{ assertOn: 'custom', xmlString }`; each case lists `expectedQueries` / `unexpectedQueries`.
+  XPath prefixes are the config's namespace KEYS, not the XML prefixes: `default:` for the default
+  namespace, the config key for every other one.
+- `.withoutExport({ testCases, act })` - the outcome is a value or a call. `act` asserts with
+  `expect` itself and returns nothing.
+- `.generic(testCases, act)` - pure synchronous functions, no XML, no database.
+
+## Shape
+
+```ts
+// core's own tests, on its dummy schema (Root > A > AA_1 > AAA_1, required attributes aA, aAA_1, ...)
+import { CUSTOM_RECORD_ID_ATTRIBUTE } from '@dialecte/core/helpers'
+import { XMLNS_DEFAULT_NAMESPACE, XMLNS_DEV_NAMESPACE, runTestCases } from '@dialecte/core/test'
+
+import type { ActParams, ActResult, BaseXmlTestCase, TestDialecteConfig } from '@dialecte/core/test'
+
+type TestCase = BaseXmlTestCase & { params: { value: string } }
+
+const testCases: Record<string, TestCase> = {
+	'A without AA_1 → AA_1 added with the value': {
+		sourceXml: /* xml */ `
+			<Root ${XMLNS_DEFAULT_NAMESPACE} ${XMLNS_DEV_NAMESPACE} ${CUSTOM_RECORD_ID_ATTRIBUTE}="root">
+				<A ${CUSTOM_RECORD_ID_ATTRIBUTE}="a-without-child" aA="one" />
+				<!-- no AA_1 yet -->
+			</Root>`,
+		params: { value: 'added' },
+		expectedQueries: ['//default:A[@aA="one"]/default:AA_1[@aAA_1="added"]'],
+	},
+}
+
+async function act({
+	testCase,
+	source,
+}: ActParams<TestDialecteConfig, TestCase>): Promise<ActResult> {
+	await source.transaction(async (tx) => addFirstChild(tx, testCase.params))
+	return { assertOn: 'source' }
+}
+
+runTestCases.withExport({ testCases, act })
+```
+
+A dialecte package's runner has the same shape with its own types (`<Name>Test.TestCases`,
+`<Name>Test.ActParams`) and its root element.
+
+## Rules
+
+- **Red first, one case at a time.** Add one case with `only: true`, run it, see it fail for the
+  right reason (an assertion, not a missing module - stub the function with the right type first),
+  write the minimum code, then the next case. Remove every `only` at the end. A table written after
+  the implementation only pins what the code happens to do.
+- **Case keys** read `<initial state> → <outcome>`.
+- **Fixtures are inline XML** in the test file. `dev:db-id` (`CUSTOM_RECORD_ID_ATTRIBUTE`) gives
+  records stable ids; choose ids that say intent (`orphaned-ref`, `a-without-child`). Comment the
+  elements that are deliberately absent. No `<?xml ?>` declaration in template literals.
+- **Expectations are structural XPath with meaningful predicates.** `//default:AA_1` alone proves
+  nothing; a case can pass because ANOTHER element matched. Do not assert on `dev:db-id` in XPath.
+  Pair `unexpectedQueries` with the commented-out source when asserting a deletion.
+- **Fixtures are validated against the schema before a case runs** by the runners `createTestHelpers`
+  builds - every dialecte package's `run<Name>TestCases` / `create<Name>TestProject` (a dialecte may
+  add its own checks, see its skill). Opt out of those extra checks only with a written reason
+  (`skipExtraChecks: '<why>'` on `createTestProject`). Core's own `runTestCases` on the dummy
+  dialecte (`createTestRunner`) validates nothing.
+- **Test the function directly** (`fn(source.query, params)` / `fn(tx, params)`), not through the
+  bound extension method. Use public interfaces; no mocks of internal collaborators.
+- **No test names its store: the runner chooses it.** A node vitest project whose setup file calls
+  `useTestStorage({ type: 'inMemory' })` (core 0.8+) runs your tests in Node on the in-memory store;
+  without one, the runner's default store. A second project whose setup imports
+  `@dialecte/core/sqlite` and calls `useTestStorage({ type: 'sqlite' })` (core 0.8+) runs the same
+  suites on the SQLite engine in the test's own process, no test changed. A test file never calls `useTestStorage` and passes no
+  `storage`; only a test about one store itself (an IndexedDB-only behaviour) names it. The
+  interface is the truth: a result that differs per store is a store bug, never a reason to pin one
+  store.
+- **XML assertions take XML text.** `expectedQueries`, `assertOn: 'custom'` (`xmlString`), and the
+  `assertExpectedElementQueries({ xml, queries })` / `assertUnexpectedElementQueries({ xml, queries })`
+  that `createXmlAssertions({ namespaces })` returns, read a string, the same way in Node and in the
+  browser. Holding a DOM? Serialize it first. Never build one for a test (`xmlDocument: true`,
+  `DOMParser`, `document.evaluate`): they exist only in the page.
+- **XPath for small exact output, record expectations for big files.** On a fixture of megabytes, a
+  `//` query over the whole export takes seconds and its failure prints too much. State what must
+  hold as record expectations - a tag, attribute values, the parent, a count - and check them on
+  the records in `.withoutExport`, with the query API:
+
+  ```ts
+  // .withoutExport: the expectations read the records, no export, no XPath
+  async function act({ testCase, source }: ActParams<Config, TestCase>): Promise<void> {
+  	await source.transaction(async (tx) => renameParts(tx, testCase.params))
+  	const root = await source.query.getRoot()
+  	const { Part = [], Assembly = [] } = await source.query.findDescendants(root, {
+  		collect: ['Part', 'Assembly'],
+  	})
+  	expect(Part).toHaveLength(3) // count
+  	for (const assembly of Assembly) {
+  		const parent = await source.query.getRecord(assembly.parent) // the parent
+  		expect(parent?.tagName).toBe('Line')
+  	}
+  }
+  ```
+
+- **Green by accident** is the classic trap: last-wins indexes, id order, an XPath matching a
+  sibling. When a case passes on the first run, break the code on purpose and watch it fail.
+- Re-indent `/* xml */` fixtures with core's `dialecte-xmlfmt "<glob>"`; the code formatter does not
+  touch them.