domainforge 0.13.0

package/docs/reference/README.md

@@ -0,0 +1,199 @@
+# Reference — DomainForge
+
+Purpose: Canonical descriptions of APIs, data models, grammar, CLI commands, configuration and error codes.
+
+MECE checklist:
+
+- Stable API surface docs and usage examples
+- Grammar and AST specification
+- Error codes with examples and recommended handling
+- CLI reference command list with flags and examples
+- Configuration schema and environment variables
+
+## How to use this section
+
+- Start with `grammar-spec.md` to understand supported syntax.
+- Consult `primitives-api.md` to see the in-memory structures created by parsing.
+- Use `cli-commands.md` when working from the terminal or CI.
+- Binding-specific references (`python-api.md`, `typescript-api.md`, `wasm-api.md`) describe constructors, methods, and platform notes.
+- `error-codes.md` provides remediation steps for validation output.
+- `calm-mapping.md` documents integration with FINOS CALM.
+- `protobuf-api.md` documents the Protobuf projection engine API.
+- `configuration.md` centralizes environment variables and registry options.
+- `registry.md` documents workspace namespace mappings (`.sea-registry.toml`).
+
+## File map and highlights
+
+### grammar-spec.md
+
+- Pest grammar rules mirrored verbatim from `sea.pest`.
+- Examples for every declaration type (namespace, dimension, unit, entity, resource, flow, instance, role, relation, policy).
+- Edge cases (escaping, numeric formats, regex validation) documented for parser debugging.
+
+### primitives-api.md
+
+- Fields and constructors for Entity, Resource, Flow, ResourceInstance, Role, Relation, Policy.
+- Cross-language method parity tables.
+- JSON serialization examples and CALM mapping notes.
+
+### cli-commands.md
+
+- Subcommand-by-subcommand reference with flags and exit codes.
+- Scenario-driven examples (batch validation, registry usage, JSON output piping).
+- Pointers to CLI source files for maintainers.
+
+### python-api.md
+
+- Installation instructions (pip, maturin, platform wheels).
+- Usage examples for Graph parsing, export/import, policy evaluation.
+- Exception types and how validation errors surface in Python.
+
+### typescript-api.md
+
+- npm installation guidance (Node 18+, napi prebuilds).
+- Graph API coverage with examples using Vitest.
+- Notes on CommonJS vs ESM and bundler integration.
+
+### wasm-api.md
+
+- Browser and bundler setup (wasm-pack, npm scripts, MIME types).
+- API surface for parsing/exporting in WASM contexts.
+- Performance and memory considerations for large graphs.
+
+### error-codes.md
+
+- Exhaustive catalog of validation and parsing errors grouped by category.
+- Examples demonstrating each code and suggested fixes.
+- Binding notes on how errors are propagated.
+
+### calm-mapping.md
+
+- Bidirectional mapping tables between SEA primitives and CALM JSON fields.
+- Implementation pointers to export/import modules and round-trip tests.
+- Known limitations and schema alignment notes.
+
+### configuration.md
+
+- CLI/environment defaults, registry schema, binding configuration flags.
+- Platform-specific guidance and migration notes across versions.
+
+## Navigational checklist by task
+
+- **Model authoring**: read `grammar-spec.md` and `primitives-api.md` first.
+- **Running tools**: rely on `cli-commands.md` and `configuration.md`.
+- **Debugging**: use `error-codes.md` plus `cli-commands.md` (`explain`), optionally cross-check `calm-mapping.md` for projection issues.
+- **Integrations**: choose the binding reference that matches your runtime (Python/TS/WASM) and pair with `calm-mapping.md` or RDF guides in how-tos.
+
+## Maintenance guidance
+
+- When updating parser rules, edit `grammar-spec.md` and add examples mirroring new syntax.
+- Any new primitive fields must be reflected in `primitives-api.md` and binding references.
+- New CLI flags require updates to `cli-commands.md` and `configuration.md`.
+- Adding or changing validation errors demands edits to `error-codes.md` and backfill of examples.
+- Projection changes (CALM/RDF/SBVR) should be mirrored in `calm-mapping.md` with notes on compatibility.
+
+## Cross-references
+
+- Tutorials and how-tos in `docs/new_docs/tutorials/` and `docs/new_docs/how-tos/` provide guided flows; use this reference section to double-check APIs used there.
+- Explanations in `docs/new_docs/explanations/` describe the rationale behind design choices documented here.
+
+## Glossary
+
+- **Namespace**: logical partition for IDs; default is `"default"`.
+- **Dimension/Unit**: measurement system for quantities; see `create-custom-units` how-to for authoring guidance.
+- **Policy**: logical statement evaluated over graph elements; see `policy-evaluation-logic.md` explanation.
+- **CALM**: FINOS Architecture-as-Code JSON format for interoperability.
+
+## Sample workflow tying references together
+
+1. Review `grammar-spec.md` to model a payments domain with roles and relations.
+2. Use `cli-commands.md` to parse and validate the DSL: `domainforge validate payments.sea --format human`.
+3. If errors appear, look up codes in `error-codes.md` and adjust accordingly.
+4. Export to CALM using `cli-commands.md` guidance, then cross-check fields using `calm-mapping.md`.
+5. Embed the graph into your runtime using `python-api.md` or `typescript-api.md` examples.
+6. Configure CI with settings from `configuration.md` to keep validation automated.
+
+## Example table of common references
+
+| Topic               | Primary file      | Secondary references                                   |
+| ------------------- | ----------------- | ------------------------------------------------------ |
+| Syntax              | grammar-spec.md   | explanations/architecture-overview.md                  |
+| API structs         | primitives-api.md | python-api.md, typescript-api.md                       |
+| CLI usage           | cli-commands.md   | how-tos/install-cli.md                                 |
+| Errors              | error-codes.md    | how-tos/debugging-parser-failures.md                   |
+| CALM integration    | calm-mapping.md   | how-tos/export-to-calm.md, how-tos/import-from-calm.md |
+| Protobuf projection | protobuf-api.md   | how-tos/export-to-protobuf.md                          |
+| Configuration       | configuration.md  | how-tos/run-cross-language-tests.md                    |
+
+## See also
+
+- `docs/new_docs/how-tos/` for task-based guidance.
+- `docs/new_docs/tutorials/` for guided, end-to-end walkthroughs.
+- `docs/new_docs/playbooks/` for operational procedures (releases, migrations).
+
+## Quality bar for reference updates
+
+- **Accuracy**: validate examples against current code. Run `cargo test -p domainforge-core --features "cli,python,typescript"` when modifying APIs.
+- **Completeness**: ensure new primitives or flags are reflected across all relevant reference files.
+- **Clarity**: prefer short code snippets with expected outputs; avoid pseudocode.
+- **Traceability**: link to source files and tests that define behavior.
+
+## Review checklist
+
+1. Are grammar changes mirrored in `grammar-spec.md` with examples?
+2. Are new fields or methods documented in `primitives-api.md` and binding references?
+3. Do CLI flag additions include exit code expectations?
+4. Are new error codes documented with example inputs and fixes?
+5. Do CALM/RDF projection changes mention compatibility considerations?
+6. Are configuration defaults and environment variables updated?
+7. Are cross-references updated so tutorials/how-tos remain accurate?
+
+## Updating this directory
+
+When adding or editing reference files:
+
+- Follow the heading hierarchy (H1 title, H2 sections, H3 subsections).
+- Keep line-wrapped bullet lists for readability.
+- Include “See also” links at the bottom of each file to guide readers to related content.
+- Use Markdown tables where comparisons help (e.g., CLI flags, binding parity).
+
+## Frequently asked questions
+
+- **Where do the binding APIs come from?**
+  - Python: `domainforge-core/src/python/` via PyO3 classes.
+  - TypeScript: `domainforge-core/src/typescript/` via napi-rs.
+  - WASM: `domainforge-core/src/wasm/` via wasm-bindgen.
+- **How stable is the CLI?**
+  - Semantic versioning applies; minor releases add flags, major releases break.
+- **Can I generate docs automatically?**
+  - Use `domainforge --help` and bindings’ docstrings as sources; this reference is hand-curated to stay concise.
+
+## Example contribution workflow
+
+```bash
+# Update grammar-spec after adding a new declaration
+vim docs/new_docs/reference/grammar-spec.md
+just fmt
+just all-tests
+```
+
+- Submit PR with links to relevant tests verifying the change.
+- Keep commit messages descriptive (e.g., "Document new module import syntax").
+
+## Contact and ownership
+
+- Primary maintainers: see `CODEOWNERS` and `docs/plans/` for domain leads.
+- File issues with documentation bugs referencing the file path and version.
+
+## Release alignment
+
+- Align documentation updates with version bumps recorded in `CHANGELOG.md`.
+- When cutting releases, verify reference links against tagged source to avoid drift.
+- Archive previous versions in branch tags if API changes are significant.
+
+## Metrics for completeness
+
+- Each reference file should include: summary, API or schema details, examples with expected outputs, troubleshooting notes, and cross-references.
+- Run link checkers (e.g., `markdown-link-check`) before merging large doc updates.
+
+(End of reference overview)

package/docs/reference/ast-json-api.md

@@ -0,0 +1,427 @@
+# AST & JSON Output Reference
+
+This reference documents the Abstract Syntax Tree (AST) structure and JSON output capabilities in DomainForge. It covers the AST schema, how to generate JSON output from SEA DSL source code, and the relationship between the `Ast` and `Graph` structures.
+
+## Overview
+
+DomainForge provides multiple ways to work with parsed SEA DSL content as structured data:
+
+1. **AST (Abstract Syntax Tree)**: The direct representation of parsed SEA source code, preserving declaration order and source location information.
+2. **Graph**: The semantic domain model built from the AST, with entities, resources, flows, and policies organized for validation and querying.
+3. **JSON Output**: Serialized representations of both structures for interoperability with other tools.
+
+## AST Schema
+
+The JSON schema for AST output is defined in `schemas/ast-v3.schema.json`. This schema is **programmatically generated** from the Rust type definitions to ensure accuracy.
+
+> [!IMPORTANT]
+> The schema is auto-generated from Rust types using [`schemars`](https://docs.rs/schemars). Do not edit `ast-v3.schema.json` manually—regenerate it instead.
+
+### Schema Location
+
+| File                         | Description                                                    |
+| ---------------------------- | -------------------------------------------------------------- |
+| `schemas/ast-v3.schema.json` | Current programmatically-generated schema (recommended)        |
+| `schemas/ast-v2.schema.json` | Previous version (deprecated, lacks Resource/Flow annotations) |
+| `schemas/ast-v1.schema.json` | Legacy hand-written schema (deprecated)                        |
+
+### Top-Level Structure
+
+```json
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "Ast",
+  "description": "Abstract Syntax Tree for SEA DSL",
+  "type": "object",
+  "required": ["declarations", "metadata"],
+  "properties": {
+    "declarations": {
+      "type": "array",
+      "items": { "$ref": "#/definitions/SpannedAstNode" }
+    },
+    "metadata": { "$ref": "#/definitions/FileMetadata" }
+  }
+}
+```
+
+### Regenerating the Schema
+
+To regenerate the AST JSON schema from the Rust type definitions:
+
+```bash
+cd domainforge-core
+cargo test --lib generate_ast_schema -- --ignored --nocapture
+```
+
+This runs the schema generation test which:
+
+1. Uses `schemars` to derive a JSON Schema from the Rust `Ast` type
+2. Writes the output to `schemas/ast-v3.schema.json`
+
+> [!TIP]
+> Run this command after modifying any AST-related types in `domainforge-core/src/parser/ast.rs` or `domainforge-core/src/policy/expression.rs`.
+
+### Schema Generation Implementation
+
+The schema is generated from types defined in `domainforge-core/src/parser/ast_schema.rs`. This module mirrors the AST types with `JsonSchema` derives:
+
+```rust
+use schemars::{schema_for, JsonSchema};
+use serde::{Deserialize, Serialize};
+
+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
+pub struct Ast {
+    pub metadata: FileMetadata,
+    pub declarations: Vec<SpannedAstNode>,
+}
+
+// Test that generates the schema file
+#[test]
+#[ignore]
+fn generate_ast_schema() {
+    let schema = schema_for!(Ast);
+    let json = serde_json::to_string_pretty(&schema).unwrap();
+    std::fs::write("../schemas/ast-v3.schema.json", &json).unwrap();
+}
+```
+
+## AST Node Types
+
+The schema includes all 15 AST node types:
+
+| Node Type         | Description                            | Fields                                                                      |
+| ----------------- | -------------------------------------- | --------------------------------------------------------------------------- |
+| `Export`          | Export wrapper for public declarations | `declaration`                                                               |
+| `Entity`          | Entity declaration                     | `name`, `annotations` (optional)                                            |
+| `Resource`        | Resource declaration                   | `name`, `annotations` (optional)                                            |
+| `Flow`            | Resource transfer between entities     | `resource_name`, `from_entity`, `to_entity`, `annotations` (optional)       |
+| `Pattern`         | Named regex for string validation      | `name`, `regex`                                                             |
+| `Role`            | Participant category                   | `name`                                                                      |
+| `Relation`        | Predicate connecting roles             | `name`, `subject_role`, `predicate`, `object_role`                          |
+| `Dimension`       | Dimension declaration for units        | `name`                                                                      |
+| `UnitDeclaration` | Custom unit definition                 | `symbol`, `dimension`, `factor`, `base_unit`                                |
+| `Policy`          | Validation rule or constraint          | `name`, `metadata`, `expression`                                            |
+| `Instance`        | Entity instance with field values      | `name`, `entity_type`                                                       |
+| `ConceptChange`   | Version migration                      | `name`, `from_version`, `to_version`, `migration_policy`, `breaking_change` |
+| `Metric`          | Observable metric                      | `name`, `expression`, `metadata`                                            |
+| `MappingDecl`     | Format mapping rules                   | `name`, `target`, `rules`                                                   |
+| `ProjectionDecl`  | Projection configuration               | `name`, `target`, `overrides`                                               |
+
+### Expression Types
+
+Policy and metric expressions use a rich expression language:
+
+| Expression Type            | Description                                         |
+| -------------------------- | --------------------------------------------------- |
+| `Literal`                  | JSON literal value                                  |
+| `QuantityLiteral`          | Decimal value with unit                             |
+| `TimeLiteral`              | ISO 8601 timestamp                                  |
+| `IntervalLiteral`          | Time interval (start, end)                          |
+| `Variable`                 | Named variable reference                            |
+| `Binary`                   | Binary operation (AND, OR, comparisons, arithmetic) |
+| `Unary`                    | Unary operation (NOT, negate)                       |
+| `Cast`                     | Type cast                                           |
+| `Quantifier`               | ForAll, Exists, ExistsUnique                        |
+| `MemberAccess`             | Object member access                                |
+| `Aggregation`              | COUNT, SUM, MIN, MAX, AVG                           |
+| `AggregationComprehension` | Aggregation with comprehension syntax               |
+| `GroupBy`                  | Grouping operation                                  |
+
+## Generating JSON Output
+
+### Via Rust Library
+
+The `Graph` struct implements `Serialize` from serde, enabling direct JSON serialization:
+
+```rust
+use domainforge_core::parser::parse_to_graph;
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let source = r#"
+        Entity "Customer"
+        Entity "Vendor"
+        Resource "Payment" USD
+        Flow "Payment" from "Customer" to "Vendor"
+    "#;
+
+    let graph = parse_to_graph(source)?;
+
+    // Serialize the full graph to JSON
+    let json = serde_json::to_string_pretty(&graph)?;
+    println!("{}", json);
+
+    // OR: Serialize the AST directly (preserves source structure)
+    // using the new stable schema types
+    let ast_json = domainforge_core::parser::parse_to_ast_json(source)?;
+    println!("{}", ast_json);
+
+    Ok(())
+}
+```
+
+To convert a `Graph` back to an `Ast` structure:
+
+```rust
+use domainforge_core::Graph;
+
+let ast = graph.to_ast();
+// The Ast can then be used for code generation or formatting
+```
+
+### Via WASM (JavaScript/TypeScript)
+
+The WASM bindings expose a `toJSON()` method on the `Graph` class:
+
+```typescript
+import init, { Graph } from "domainforge_core";
+
+async function parseAndSerialize() {
+  await init();
+
+  const source = `
+        Entity "Customer"
+        Entity "Vendor"
+        Resource "Payment" USD
+        Flow "Payment" from "Customer" to "Vendor"
+    `;
+
+  const graph = Graph.parse(source);
+  const json = graph.toJSON();
+
+  console.log(JSON.stringify(json, null, 2));
+}
+```
+
+### Via CLI
+
+The SEA CLI provides multiple commands for JSON output:
+
+#### Validate with JSON Output
+
+```bash
+# Validate and output results as JSON
+domainforge validate --format json input.sea
+```
+
+Output structure:
+
+```json
+{
+  "error_count": 0,
+  "violations": []
+}
+```
+
+#### Project to CALM Format
+
+```bash
+# Export graph to CALM JSON format
+domainforge project --format calm input.sea output.json
+```
+
+#### Export AST as JSON
+
+To export the raw Abstract Syntax Tree (useful for tools that need source preservation):
+
+```bash
+domainforge parse --ast --format json input.sea > output.ast.json
+```
+
+This output conforms to `ast-v3.schema.json` and preserves all declarations, comments, and line/column numbers.
+
+#### Project to Knowledge Graph
+
+```bash
+# Export to RDF/Turtle or XML
+domainforge project --format kg input.sea output.ttl
+domainforge project --format kg input.sea output.xml
+```
+
+## Graph JSON Structure
+
+When serializing a `Graph` to JSON, the structure includes all domain primitives:
+
+```json
+{
+  "entities": {
+    "<uuid>": {
+      "id": "<uuid>",
+      "name": "Customer",
+      "namespace": "default",
+      "version": null,
+      "attributes": {}
+    }
+  },
+  "resources": {
+    "<uuid>": {
+      "id": "<uuid>",
+      "name": "Payment",
+      "namespace": "default",
+      "unit": "USD",
+      "has_units": true,
+      "attributes": {}
+    }
+  },
+  "flows": {
+    "<uuid>": {
+      "id": "<uuid>",
+      "resource_id": "<uuid>",
+      "from": "<uuid>",
+      "to": "<uuid>",
+      "quantity": null,
+      "namespace": "default",
+      "attributes": {}
+    }
+  },
+  "roles": {},
+  "relations": {},
+  "instances": {},
+  "entity_instances": {},
+  "policies": {},
+  "patterns": {},
+  "concept_changes": {},
+  "metrics": {},
+  "mappings": {},
+  "projections": {},
+  "entity_roles": {},
+  "config": {
+    "use_three_valued_logic": true
+  }
+}
+```
+
+### Field Descriptions
+
+| Field              | Type                            | Description                          |
+| ------------------ | ------------------------------- | ------------------------------------ |
+| `entities`         | `Map<UUID, Entity>`             | All declared entities in the graph   |
+| `resources`        | `Map<UUID, Resource>`           | All declared resources               |
+| `flows`            | `Map<UUID, Flow>`               | Resource transfers between entities  |
+| `roles`            | `Map<UUID, Role>`               | Participant categories               |
+| `relations`        | `Map<UUID, RelationType>`       | Predicates connecting roles          |
+| `instances`        | `Map<UUID, ResourceInstance>`   | Concrete resource instances          |
+| `entity_instances` | `Map<UUID, Instance>`           | Entity instances with field values   |
+| `policies`         | `Map<UUID, Policy>`             | Validation rules and constraints     |
+| `patterns`         | `Map<UUID, Pattern>`            | Regex patterns for string validation |
+| `concept_changes`  | `Map<UUID, ConceptChange>`      | Version migration declarations       |
+| `metrics`          | `Map<UUID, Metric>`             | Observable metrics                   |
+| `mappings`         | `Map<UUID, MappingContract>`    | Format mapping declarations          |
+| `projections`      | `Map<UUID, ProjectionContract>` | Projection declarations              |
+| `entity_roles`     | `Map<UUID, Vec<UUID>>`          | Role assignments to entities         |
+| `config`           | `GraphConfig`                   | Evaluation configuration             |
+
+## Spanned AST Structure
+
+The internal `Ast` structure contains `SpannedAstNode` elements, where each node includes source location:
+
+```json
+{
+  "node": { "type": "Entity", "name": "Customer", ... },
+  "line": 1,
+  "column": 1
+}
+```
+
+This enables precise error messages and IDE integration with accurate source positions.
+
+## Converting Between AST and Graph
+
+### AST → Graph (Parsing)
+
+The standard parsing pipeline converts source code to a `Graph`:
+
+```rust
+use domainforge_core::parser::{parse_to_graph, parse_to_graph_with_options, ParseOptions};
+
+// Simple parsing
+let graph = parse_to_graph(source)?;
+
+// With options (namespace, registry, etc.)
+let options = ParseOptions {
+    default_namespace: Some("my_namespace".to_string()),
+    ..Default::default()
+};
+let graph = parse_to_graph_with_options(source, &options)?;
+```
+
+### Graph → AST (Reconstruction)
+
+Convert a `Graph` back to an `Ast` for code generation or formatting:
+
+```rust
+let ast = graph.to_ast();
+// Note: Currently reconstructs Entity and Resource nodes only
+```
+
+> [!NOTE]
+> The `to_ast()` method is a partial implementation that currently reconstructs only Entity and Resource nodes. Full round-trip support is planned for future releases.
+
+## Projection Formats
+
+SEA supports projecting graphs to various external formats:
+
+| Format          | Command             | Output                  |
+| --------------- | ------------------- | ----------------------- |
+| CALM            | `--format calm`     | FINOS CALM JSON         |
+| Knowledge Graph | `--format kg`       | RDF Turtle or XML       |
+| Protobuf        | `--format protobuf` | Protocol Buffers schema |
+
+### CALM Export Example
+
+```bash
+domainforge project --format calm domains/finance.sea output/finance.calm.json
+```
+
+Produces FINOS CALM-compatible JSON following the schema in `schemas/calm-v1.schema.json`.
+
+## Validation and Error Handling
+
+JSON output from validation includes structured error information:
+
+```json
+{
+  "error_count": 2,
+  "violations": [
+    {
+      "severity": "error",
+      "policy_name": "reference_integrity",
+      "message": "Entity 'Unknown' not found",
+      "context": { "line": 5, "column": 12 }
+    },
+    {
+      "severity": "warning",
+      "policy_name": "naming_convention",
+      "message": "Entity name should be PascalCase",
+      "context": { "line": 3, "column": 8 }
+    }
+  ]
+}
+```
+
+### Severity Levels
+
+| Severity  | Description                                   |
+| --------- | --------------------------------------------- |
+| `error`   | Validation failure that blocks processing     |
+| `warning` | Potential issue that doesn't block processing |
+| `info`    | Informational message                         |
+
+## Version Compatibility
+
+| Schema               | Version | Status                                                  |
+| -------------------- | ------- | ------------------------------------------------------- |
+| `ast-v3.schema.json` | 3.0.0   | **Current** (auto-generated, Resource/Flow annotations) |
+| `ast-v2.schema.json` | 2.0.0   | Deprecated (lacks Resource/Flow annotations)            |
+| `ast-v1.schema.json` | 1.0.0   | Deprecated (hand-written)                               |
+
+- Minimum domainforge-core version: `0.6.0`
+- JSON output is stable across minor versions
+
+## See Also
+
+- [primitives-api.md](primitives-api.md) — Core data structure reference
+- [cli-commands.md](cli-commands.md) — CLI operations
+- [calm-mapping.md](calm-mapping.md) — CALM format mapping details
+- [wasm-api.md](wasm-api.md) — WASM bindings reference
+- [typescript-api.md](typescript-api.md) — TypeScript API reference
+- [python-api.md](python-api.md) — Python API reference