@peter.prib/renderengine 0.2.7 → 0.2.11

package/README.md

@@ -1,11 +1,12 @@
 # RenderEngine
 
-**An ArchiMate modeller, a diagram editor and a chart library that run in any browser tab, with no install and no
-runtime dependencies.**
+**An ArchiMate modeller, a data modeller, a diagram editor and a chart library that run in any browser tab, with
+no install and no runtime dependencies.**
 
 Open an Archi model, edit it the way you would in Archi, and save it back to the same `.archimate` file, without
-installing anything. Or put the same editor in your own app: it's a JavaScript library and a set of workbench parts,
-not a desktop application.
+installing anything. Model data at three levels, governed and checked the way DAMA-DMBOK2 has it, with banking at
+BIAN level built in. Or put the same editors in your own app: it's a JavaScript library and a set of workbench
+parts, not a desktop application.
 
 ![The ArchiMate modeller: a layered view of an online shop, the model tree by layer with each type's picture, and the palette](https://gitlab.com/peter.prib/renderengine/-/raw/master/docs/images/archimate-modeller.png)
 
@@ -18,7 +19,7 @@ git clone https://gitlab.com/peter.prib/renderengine.git
 cd renderengine
 npm start          # the ArchiMate modeller
 npm run diagram    # the diagram editor (flowcharts, UML and more)
-npm run datamodel  # a data model at each level, in each notation
+npm run datamodel  # the data modeller: an enterprise ontology and a retail bank at BIAN level
 ```
 
 No server is needed, and nothing needs installing first, because the built bundles are in `dist/`. You can also
@@ -26,7 +27,7 @@ open the pages directly:
 
 - `examples/archimate.html`: the ArchiMate modeller
 - `examples/diagram-editor.html`: the diagram editor
-- `examples/data-model.html`: a data model (conceptual, logical, physical; Chen, Crow's Foot, UML)
+- `examples/data-model.html`: the data modeller
 
 All work in current Chrome, Edge, Firefox and Safari. Save writes back to the file you opened in Chrome and Edge;
 other browsers download the file instead.
@@ -111,6 +112,61 @@ It runs in the browser and embeds in other apps. It also has:
 
 ---
 
+## The data modeller
+
+One model at the conceptual, logical and physical levels, read by each stakeholder in their own terms, governed and
+checked the way DAMA-DMBOK2 has a model reviewed, and ready for banking at BIAN level. It shares the modeller's
+workbench: several models open at once, undo for everything, files saved back where they came from, models kept in
+the browser and backed up.
+
+![The data modeller: a retail bank's logical model through its Sensitive data lens, each entity coloured by how sensitive its data is](https://gitlab.com/peter.prib/renderengine/-/raw/master/docs/images/data-model.png)
+
+### One model, three levels
+
+- **Concepts, logical entities and tables, each a schema of its own**, mapped any number either way: a concept
+  realised by several entities, several entities kept in one table. The **mapping view** draws the three side by
+  side, and **lineage** shows, for anything, what it realises and what realises it, all the way up and down.
+- **Perspectives and contexts**: each stakeholder's names for the concepts, and what they mean in a context, with
+  its rules and where and when it applies, drawn as notes.
+- **Chen, Crow's Foot and UML**, switched with one choice; **subtypes** (complete or not, disjoint or overlapping);
+  **data types and enumerations** defined once, code sets with their source and version, each linked to its
+  **ISO 20022** type.
+- **Edit it on the diagram**: double-click an entity for its attribute table (keys, required, what a foreign key
+  references, classification), right-click for new and delete. Every edit is one undo step.
+
+### Banking at BIAN level
+
+- **Service domains** with their control records, behaviour qualifiers and the service operations BIAN names for
+  them, and the **service landscape**: every service domain in its business area and domain.
+- **Read BIAN's own published material instead of retyping it.** A BIAN semantic API (all 258 of release 14 read,
+  plain or with ISO 20022) becomes a control record, its qualifiers, their data types and ISO 20022 types in one
+  step. BIAN's service landscape in ArchiMate becomes service domains in their areas and domains: run over BIAN's
+  v14 model, every one it places lands in the right place.
+- BIAN's vocabulary, its functional patterns, artifacts, action terms and qualifier types, is as BIAN's v14 model
+  has it.
+
+### Governed and checked (DAMA-DMBOK2)
+
+- **Stewardship**: a business owner, data steward and technical custodian on each entity.
+- **Classification**: sensitivity levels and categories (PII, PCI...) on entities and attributes, **checked down
+  the mappings**, so a table is never classified lower than the data it holds; a **lens** colours the model by it.
+- **Check model** reviews it as DMBOK2 would: missing keys, foreign keys without relationships, repeating groups,
+  unresolved many-to-many, one name with two types, concepts without definitions, names against the **naming
+  standard** you set, **data quality rules** by dimension (completeness, validity, uniqueness...), and BIAN's one
+  owner per business object.
+- **The Data Model Scorecard**: Steve Hoberman's ten categories, six measured from the model and four for a
+  reviewer, weighted as you choose, in one total.
+
+### To the database and the warehouse
+
+- **SQL DDL**: the physical level as `CREATE TABLE`s with keys, foreign keys and comments.
+- **A dimensional scheme**: facts at their grain with additive, semi-additive or non-additive measures, dimensions
+  with their history type and conformed across facts, a **star schema** view, and checks as Kimball would make
+  them.
+- **Bases** to start a model from, and **compare and merge** between models.
+
+---
+
 ## The diagram editor
 
 The ArchiMate modeller is built on a general diagram editor, and every editing feature above is part of that
@@ -181,9 +237,10 @@ page:
 </script>
 ```
 
-Use `editorType: 'diagram-editor'` with `DiagramEditorPart` for the diagram editor. The model is a plain JavaScript
-API (`ArchimateModel`, `DiagramModel`), so you can also build, check, search, report on and export models with no
-UI at all.
+Use `editorType: 'diagram-editor'` with `DiagramEditorPart` for the diagram editor, and `editorType: 'data-model'`
+with `DataModelPart` for the data modeller. The models are plain JavaScript APIs (`ArchimateModel`, `DiagramModel`,
+`DataModel`), so you can also build, check, score, search, report on and export models with no UI at all, under
+Node too: `new DataModel(spec).checker().run()`, `new SqlDdl().write(model)`.
 
 ---
 

package/REFERENCE.md

@@ -811,6 +811,109 @@ types — see `ARCHITECTURE.md`):
   one, or a qualifier, that is no logical entity, or with a pattern BIAN lacks, and a mapping joining an entity that
   is not there. Model ▸ **Check model…** lists the findings in the `ModelCheckDialog` the diagram editors' check uses;
   choosing one shows it (the entity's level, the service domain seen through, the mapping view).
+
+  **Checked as DMBOK2 reviews a model.** The check also finds, at the logical and physical levels: an entity with
+  attributes and no primary key (`no-primary-key`); a foreign key whose `references` (an entity, or `entity.attribute`,
+  by id or name) is not there (`reference-missing`) or has no relationship with its entity either way
+  (`foreign-key-without-relationship`), one saying nothing on an entity with no relationship (`foreign-key-unrelated`);
+  attributes differing only by a number at the end, a repeating group (`repeating-group`); one name with different
+  types at a level, in any case (`type-conflict`); and at the physical level a many-to-many relationship, which a table
+  between resolves (`many-to-many`). A concept without a definition (`definition-missing`) and anything defined only
+  by its name (`definition-restates-name`) are reported too. A model may keep a **naming standard**
+  (`model.namingStandard`, a `NamingStandard`; none by default): per level, for its entities and attributes, a
+  `style` (a `NameStyle`: Title Case, lower case words, snake_case, UPPER_SNAKE_CASE, camelCase, PascalCase; each
+  says whether a name is written so and suggests it), `classWords` an attribute name must end with, a `maxLength`,
+  and `abbreviations` a name must use (`{ number: 'num' }`); names that break it are reported (`naming`), with how
+  they would be written. Model ▸ **Naming standard** ▸ sets it (`NamingStandard.fields`), starts from
+  `NamingStandard.CONVENTIONAL` or removes it, one undo step each (`model.namingStandardCommand(spec|null)`).
+
+  ```javascript
+  new NamingStandard(NamingStandard.CONVENTIONAL).check('physical', 'attributes', 'account_number');
+  // ['has "number" where the standard writes "num"']
+  ```
+
+  **Stewardship, classification, lineage.** Any entity may say who answers for it (`stewardship: { owner, steward,
+  custodian }`: business owner, data steward, technical custodian) and how it is classified (`sensitivity`, a level of
+  the model's scheme, and `categories`); so may an attribute (its row's Sensitivity and Categories). The entity dialog
+  has both, its attribute table the attributes'. A model keeps a `ClassificationScheme` (`model.classificationScheme`:
+  sensitivity `levels` in order, lowest first, each with a colour, and `categories`; the default Public, Internal,
+  Confidential, Restricted and PII, PCI, PHI, Financial, saved only when changed; Model ▸ **Classification scheme…**).
+  An entity's effective classification (`scheme.effective(entity)`) is the highest of its own and its attributes', with
+  all their categories. The check reports what realises a classified entity, down the mappings, classified lower or
+  without its categories (`classification-lost`), and a sensitivity the scheme lacks (`classification-unknown`). A
+  **classification lens** (a perspective kind, `ClassificationLens`; Model ▸ New classification lens…) colours each
+  entity by its effective sensitivity, unclassified grey. `model.lineage(level, id)` gives what an entity realises
+  (`up`, to its concepts) and what realises it (`down`, to its tables), through the mappings; on hover an entity
+  shows its stewardship, classification and lineage (*realises*, *realised by*). Right-click an entity ▸ **Show
+  lineage** opens the mapping view with its lineage as it is and the rest greyed; there, right-click another for its
+  lineage, or the canvas ▸ Show all. The banking base classifies its party data and has a *Sensitive data* lens.
+
+  **Subtypes.** A relationship's `kind` says what it is (`RelationshipKind`, registered as perspective kinds are): an
+  association (`AssociationKind`, the default: a cardinality at each end, Chen's diamond) or a generalization
+  (`GeneralizationKind`, `kind: 'generalization'`: the source a subtype of the target, `complete` and `disjoint`
+  said on it). Each kind draws its own edge (`DataModelDiagram.buildEdge`): a generalization in every notation as a
+  line to its supertype ending in a hollow triangle, labelled `{complete, disjoint}` (in Chen, `ISA {…}`, a line,
+  not a diamond), with no cardinalities. The relationship dialog has the kind and a generalization's constraints.
+  The check reports generalizations that go round (`generalization-cycle`) and takes none for many-to-many. The
+  ontology's logical Party is the supertype of Person and Organisation.
+
+  **Data quality rules.** A context's rule may say which data quality dimension it measures (`dimension`, one of
+  `DataModelContext.QUALITY_DIMENSIONS`: Completeness, Validity, Uniqueness, Timeliness, Accuracy, Consistency) and
+  which attribute of what realises its concepts it is on (`attribute`); it then reads *Completeness of legal name:
+  always given* in its note and on hover (`DataModelContext.ruleText`). A context's dialog edits its rules as a table
+  (the rule, its concepts by name, its dimension, its attribute). The check reports a rule on an attribute nothing
+  realising its concepts has (`rule-attribute-missing`; names compared as words, so *tax number* is `tax_number`)
+  and a dimension DMBOK2 does not name (`quality-dimension-unknown`).
+
+  **Code sets.** An enumeration, as reference data, may say where it comes from (`source`), who keeps it (`owner`) and
+  which version it is (`version`): in its dialog, saved, and on hover in the data types view (`codeSet`).
+
+  **SQL DDL.** Export ▸ **SQL DDL (physical level)** writes the physical level as SQL (`SqlDdl`, `SqlDdlFileFormat`, an
+  export-only `ModelFileFormat`): a `CREATE TABLE` per table, its columns' types (`defaultType` where one has none)
+  and `NOT NULL` for required and key columns, its `PRIMARY KEY` and `UNIQUE` columns; a `FOREIGN KEY` per column
+  that references another table (`references`: a table, by id or name, meaning its primary key, or `table.column`),
+  a comment where one cannot be made; `COMMENT ON`s for definitions (`comments: false` leaves them out). Names SQL
+  would not take bare are quoted. ANSI SQL as PostgreSQL reads it.
+
+  A table's name may be schema-qualified (`sales.customer`, as a database lists its tables): the DDL names it in its
+  schema (`sales.customer`, each part quoted as it needs, `SqlDdl.tableName()`), and a reference to it is
+  `sales.customer` or `sales.customer.id`. The check and the DDL read a reference the one way,
+  `LevelSchema.resolveReference(ref)`: the whole text as a table (by id or name, any case), else what is before its
+  last dot as the table and the rest as its column — `{ entity, column }`, or null.
+
+  **A dimensional scheme.** At the logical and physical levels an entity may have a role (`EntityRole`, registered as
+  relationship kinds are): a **fact** (`FactRole`, `role: 'fact'`: what is measured, at a stated `grain`, its
+  attributes' `measure` additive, semi-additive or non-additive) or a **dimension** (`DimensionRole`: its `scdType`,
+  0 to 3, how it keeps history, `conformed` when facts share it). The entity dialog has the role and what that role
+  has; the attribute table a Measure column. Level ▸ **Star schemas** (offered once there is a fact; `DataModel.STAR`,
+  `model.starView()`) draws, at the physical level if it has a fact (else the logical), each fact in a row of stars
+  with the dimensions it is related to around it, a shared dimension once, by the first, in their roles' colours. The
+  check reports a fact without its grain (`fact-no-grain`), related to no dimension (`fact-no-dimension`) or with no
+  measure (`fact-no-measure`), a measure on what is no fact (`measure-outside-fact`), and two facts joined directly
+  (`fact-to-fact`). The banking base has a daily balance star (`fact_daily_balance`, `dim_account` of type 2,
+  a conformed `dim_date`).
+
+  **The Data Model Scorecard.** Model ▸ **Scorecard…** scores the model on the ten categories of Steve Hoberman's Data
+  Model Scorecard (`ModelScorecard`, `model.scored()`), the total in its title. Six are measured from the model, a
+  `MeasuredCategory` each, with notes on why: **Completeness** (the share, averaged, of entities defined, concepts
+  realised at the logical level, logical entities realised by a table, logical entities with an owner or steward),
+  **Structure** (logical entities and tables the check finds nothing structural on), **Standards** (names keeping the
+  naming standard; none without one), **Readability** (boxes clear of each other, associations named),
+  **Definitions** (definitions more than the name and at least four words), **Consistency** (logical entities and
+  tables the check finds nothing inconsistent on). **Correctness**, **Scheme**, **Abstraction** and **Data** are for a
+  reviewer (`ScorecardCategory`). A reviewer's score and note, kept in the model (`model.scorecard.reviews`), override
+  a measured score too. The total is the weighted average of the categories with a score, with how much of the weight
+  that covers; weights are the model's own (`model.scorecard.weights`), equal (10 each) by default: the book gives its
+  own weighting, and an organisation sets the one it reviews by. Saving is one undo step (`scorecardCommand`).
+
+  ```javascript
+  model.scored().total();   // { score: 91, coverage: 50 }
+  ```
+
+  ```javascript
+  new SqlDdl().write(model);   // 'CREATE TABLE party (
+    party_id BIGINT NOT NULL, …'
+  ```
   The ontology base keeps BIAN's **Agreement** (the contract) apart from its **Arrangement** (the terms a product is
   provided on), and has **Product Feature** and **Condition**; its concepts say which BIAN business object they are
   (`details.bian`, on hover).
@@ -849,6 +952,27 @@ types — see `ARCHITECTURE.md`):
   model.view({ level: DataModel.LANDSCAPE });   // { entities, folders, elementFolders, ... }
   ```
 
+  **Editing.** At a level of the schema, double-click an entity (or a relationship's line or Chen diamond) to edit it;
+  right-click one for Edit… and Delete, or the canvas for **New entity here…** and **New relationship…**; Model ▸ New
+  concept/entity/table… and New relationship… do the same. Each level says what its entities hold and gives its
+  dialog's fields (`DataModelNotation.level(name)`: a `ConceptualLevel` a name and a definition; a `LogicalLevel` and
+  a `PhysicalLevel`, both `AttributedLevel`s, the attributes or columns too, as a `rows` table: name, type (the model's
+  data types and plain types suggested at the logical level, SQL types at the physical), PK, FK, unique, required,
+  what a foreign key references, a definition), and reads it back over the entity (`entitySpecFrom`), so where it is
+  drawn and what the dialog does not show stay; `relationshipFields` and `relationshipSpecFrom` do the same for a
+  relationship (ends, cardinalities, name, identifying, definition). Every edit is one undo step, a
+  `DataModelEditCommand` (the model as it was before and after, `DataModel#restore` putting either back):
+  `saveEntityCommand(level, id|null, spec)`, `removeEntityCommand(level, id)` (its relationships and mapping pairs go,
+  and each perspective, service domain and context forgets it: `forgetEntity`), `saveRelationshipCommand(level,
+  index|null, spec)`, `removeRelationshipCommand(level, index)`; `relationshipIndex(level, drawn)` says which
+  relationship a line or a diamond is. These stand in for graphDiagram's own Rename and Delete, which change only
+  the drawing.
+
+  ```javascript
+  model.history.execute(model.saveEntityCommand('logical', null, { name: 'Phone', attributes: [{ name: 'number', required: true }] }));
+  model.history.undo();   // gone again
+  ```
+
   **Data types.** A model's `dataTypes` are the types its attributes have, each defined once: a `DataType` (a plain
   named type, `basedOn` another), an `EnumerationType` (`values: [{ name, description? }]`) or a `StructuredType`
   (`fields: [{ name, type?, description?, iso20022? }]`), its `kind` in a spec picking the class (`DataType.from`,
@@ -2430,6 +2554,11 @@ offers `text` as its first choice; any other field, a checkbox too, gets a check
 fields: every field on "Keep" showing the value they share, read-only fields left out.
 `onApply(values)` receives every field; `onClose(applied)` is told once when the dialog closes, whether it
 was applied (false for Cancel, Escape or a click outside), so a caller waiting on a choice can settle.
+A `rows` field (`RowsField`) edits a list as a table: `columns` `[{ key, label, type?: 'text'|'checkbox'|'select',
+choices?, suggestions?, placeholder?, width? }]`, `value` the items, `addLabel`; a row each, moved up and down or removed,
+**+ Add** for a new one. It reads as the items in their order, each the item it was made from with the columns laid
+over it (what no column shows is kept; an empty text or an unticked box leaves its key out), a new row left empty
+dropped. It takes the dialog's whole width, its label above it.
 `EdgeStyleDialog`, the grid-size and diagram-name prompts are all `FormDialog`s. A dialog fits the window
 (at most 90% of its height): its fields scroll between the title and the buttons, which stay in view, and
 dragging the title moves it. A `textarea` scrolls inside its own box (at most 40% of the window) and can be