domainforge 0.13.0

package/docs/how-tos/export-to-calm.md

@@ -0,0 +1,110 @@
+# Export to CALM
+
+Goal: Export a SEA DSL model to FINOS CALM and verify the payload is valid.
+
+## Prerequisites
+
+- SEA CLI installed (`cargo install --path domainforge-core --features cli`).
+- A validated `.sea` model file with entities/resources/flows (and optional roles/relations).
+- Optional: Python/TypeScript bindings for programmatic export.
+- JSON tooling such as `jq` for inspection.
+
+## Steps (be concise)
+
+1. **Validate the model first (CLI)**
+
+   ```bash
+   domainforge validate --format human path/to/model.sea
+   ```
+
+   - Fails fast on syntax/semantic errors; fix these before exporting.
+
+2. **Export to CALM JSON (CLI)**
+
+   ```bash
+   domainforge project --format calm path/to/model.sea calm.json
+   ```
+
+   - On success, the command prints `Projected to CALM: calm.json`.
+   - Output includes `metadata.sea:version` set to the current SEA version (e.g., `0.1.0`).
+
+3. **Inspect the output**
+
+   ```bash
+   jq '.metadata, (.models[0].roles | length), (.models[0].relations | length)' calm.json
+   ```
+
+   - Confirm roles/relations/flows are present and IDs look like UUIDs.
+
+4. **Re-import to double-check**
+
+   ```bash
+   domainforge import --format kg calm.json
+   ```
+
+   - Uses the KG importer to ensure the CALM payload can be transformed back into a graph.
+
+5. **Export programmatically in Python**
+
+   ```python
+   from domainforge import Graph
+
+   with open("path/to/model.sea") as f:
+       graph = Graph.parse(f.read())
+
+   calm_json = graph.export_calm()
+
+   with open("calm.json", "w") as f:
+       f.write(calm_json)
+   ```
+
+6. **Export programmatically in TypeScript**
+
+   ```ts
+   import { Graph } from "@godspeedai/domainforge";
+   import { readFileSync, writeFileSync } from "fs";
+
+   const graph = Graph.parse(readFileSync("path/to/model.sea", "utf8"));
+   const calm = graph.exportCalm();
+   writeFileSync("calm.json", calm);
+   ```
+
+7. **Validate against the CALM schema (optional)**
+
+   ```bash
+   ajv validate -s calm.schema.json -d calm.json
+   ```
+
+   - Use the FINOS Architecture-as-Code schema for strict validation if a downstream system requires it.
+
+## Mapping Notes
+
+- Roles and relations emit as typed facts; verify `subjectRole`, `predicate`, and `objectRole` fields reference valid role IDs.
+- Instances and resource quantities retain their units; ensure custom units are defined before export to avoid validation failures.
+- Namespaces: unset namespaces default to `"default"` and may be omitted in the CALM output.
+- Metadata includes `sea:exported`, `sea:version`, and `sea:timestamp`; keep them for traceability.
+
+## Validation Tips
+
+- Use `jq` to spot-check key sections (roles, relations, flows) and ensure UUIDs are present.
+- If a relation references a flow ID, confirm the flow exists and carries a valid UUID; missing IDs cause export failures.
+- For large models, prefer `domainforge project --format calm input.sea | gzip > calm.json.gz` to stream-compress during export (portable across Bourne-like shells and Windows with compatible tools).
+
+## Troubleshooting
+
+- **Parse errors**: Run `domainforge validate` before exporting; export stops on parse failure.
+- **Missing roles/relations**: Ensure they are declared in the DSL; empty arrays in CALM indicate nothing was declared.
+- **Schema validation failures**: Align with the official CALM schema and update your DSL model to supply required fields (e.g., `id`, `name`).
+- **Version drift**: If consumers require a specific SEA version, check `metadata["sea:version"]` and consider pinning the toolchain.
+
+## Verification Checklist
+
+- [ ] `domainforge validate` passes on the source `.sea` file.
+- [ ] `domainforge project --format calm` succeeds and outputs metadata with `sea:version`.
+- [ ] CALM payload re-imports through `domainforge import --format kg` without losing entities/resources/roles/relations.
+- [ ] Optional schema validation (AJV) passes when required by downstream systems.
+
+## Links
+
+- Tutorials: [Getting Started](../tutorials/getting-started.md)
+- Reference: [CALM Mapping](../reference/calm-mapping.md), [CLI Commands](../reference/cli-commands.md), [Primitives API](../reference/primitives-api.md)

package/docs/how-tos/export-to-protobuf.md

@@ -0,0 +1,312 @@
+# Export to Protobuf
+
+This guide explains how to export SEA models to Protocol Buffers (`.proto`) files for use with gRPC services, binary serialization, and cross-language contracts.
+
+## Prerequisites
+
+- `sea` CLI installed ([install guide](./install-cli.md))
+- A valid `.sea` model file
+
+## Quick Start
+
+Export an entity model to a `.proto` file:
+
+```bash
+domainforge project --format protobuf model.sea output.proto
+```
+
+This generates a complete `.proto` file with:
+
+- All entities as Protobuf messages
+- All resources as Protobuf messages
+- Type mappings from SEA types to proto types
+- Metadata header with generation timestamp
+
+## Example
+
+Given a SEA model (`payment.sea`):
+
+```sea
+Namespace "com.example.payments"
+
+Entity "Customer" {
+    name: String
+    email: String
+    balance: Money
+}
+
+Entity "Payment" {
+    amount: Money
+    status: String
+    timestamp: DateTime
+}
+
+Resource "PaymentRequest" USD
+```
+
+Running:
+
+```bash
+domainforge project --format protobuf payment.sea payment.proto
+```
+
+Produces:
+
+```protobuf
+// Generated by SEA Projection Framework
+// Projection: protobuf
+// Source Namespace: com.example.payments
+// Generated At: 2025-12-14T20:00:00Z
+// DO NOT EDIT - This file is auto-generated
+
+syntax = "proto3";
+
+package com.example.payments;
+
+option java_package = "com.example.payments";
+option java_multiple_files = true;
+option go_package = "github.com/example/payments";
+
+import "google/protobuf/timestamp.proto";
+
+message Customer {
+  string balance = 1;
+  string email = 2;
+  string name = 3;
+}
+
+message Payment {
+  string amount = 1;
+  string status = 2;
+  google.protobuf.Timestamp timestamp = 3;
+}
+
+message PaymentRequest {
+  string id = 1;
+  double value = 2;
+  string unit = 3;
+}
+```
+
+## Options
+
+### Include gRPC Services
+
+Generate gRPC service definitions from Flow patterns:
+
+```bash
+domainforge project --format protobuf --include-services model.sea output.proto
+```
+
+This converts SEA Flows into gRPC RPC methods:
+
+| SEA Construct           | gRPC Output            |
+| ----------------------- | ---------------------- |
+| Flow destination entity | Service name           |
+| Flow name               | RPC method name        |
+| Flow resource           | Request/Response types |
+
+### Multi-File Output
+
+Split output by namespace for large models:
+
+```bash
+domainforge project --format protobuf --multi-file --output-dir ./proto model.sea
+```
+
+This creates a directory structure:
+
+```
+proto/
+├── common/
+│   └── common.proto
+├── payments/
+│   └── payments.proto
+└── customers/
+    └── customers.proto
+```
+
+### Specify Package Name
+
+Override the default package:
+
+```bash
+domainforge project --format protobuf --package "org.example.api" model.sea output.proto
+```
+
+### Language-Specific Options
+
+Set language-specific proto options:
+
+```bash
+domainforge project --format protobuf \
+  --option java_package="com.example.api" \
+  --option go_package="github.com/example/api" \
+  --option java_multiple_files=true \
+  model.sea output.proto
+```
+
+## Type Mapping
+
+SEA types are mapped to Protobuf types as follows:
+
+| SEA Type         | Protobuf Type               |
+| ---------------- | --------------------------- |
+| `String`         | `string`                    |
+| `Int`            | `int64`                     |
+| `Float`          | `double`                    |
+| `Bool`           | `bool`                      |
+| `Date`           | `google.protobuf.Timestamp` |
+| `DateTime`       | `google.protobuf.Timestamp` |
+| `Duration`       | `google.protobuf.Duration`  |
+| `UUID`           | `string`                    |
+| `Money`          | Custom `Money` message      |
+| `List<T>`        | `repeated T`                |
+| `Optional<T>`    | `optional T`                |
+| Entity reference | Message reference           |
+
+## Buf.build Integration
+
+If `buf` CLI is installed, use it for linting and validation:
+
+```bash
+# Lint generated protos
+domainforge project --format protobuf --buf-lint model.sea output.proto
+
+# Check breaking changes
+domainforge project --format protobuf --buf-breaking --against ./previous model.sea output.proto
+```
+
+If `buf` is not installed, these flags emit a warning and continue.
+
+## Programmatic Usage
+
+### Rust
+
+```rust
+use domainforge_core::parser::parse_source;
+use domainforge_core::parser::ast::ast_to_graph;
+use domainforge_core::projection::protobuf::ProtobufEngine;
+
+let source = std::fs::read_to_string("model.sea")?;
+let ast = parse_source(&source)?;
+let graph = ast_to_graph(&ast)?;
+
+let proto_file = ProtobufEngine::project(&graph, "my_namespace", "com.example");
+println!("{}", proto_file.to_proto_string());
+```
+
+### Python
+
+```python
+from domainforge_core import parse_source, ast_to_graph
+from domainforge_core.projection import protobuf_engine
+
+source = open("model.sea").read()
+ast = parse_source(source)
+graph = ast_to_graph(ast)
+
+proto = protobuf_engine.project(graph, "my_namespace", "com.example")
+print(proto.to_proto_string())
+```
+
+### TypeScript
+
+```typescript
+import { parseSource, astToGraph } from "domainforge-core";
+import { ProtobufEngine } from "domainforge-core/projection";
+
+const source = fs.readFileSync("model.sea", "utf-8");
+const ast = parseSource(source);
+const graph = astToGraph(ast);
+
+const protoFile = ProtobufEngine.project(graph, "my_namespace", "com.example");
+console.log(protoFile.toProtoString());
+```
+
+## Compatibility Checking
+
+The engine supports schema compatibility checking to prevent breaking changes:
+
+### Compatibility Modes
+
+| Mode       | Allowed Changes                           |
+| ---------- | ----------------------------------------- |
+| `Additive` | Only additions (new fields/messages)      |
+| `Backward` | Additions + safe removals (with reserved) |
+| `Breaking` | Any changes allowed                       |
+
+### CLI Usage
+
+```bash
+# Check compatibility against previous version
+domainforge project --format protobuf --compatibility additive \
+  --against ./previous/output.proto \
+  model.sea output.proto
+```
+
+## gRPC Streaming
+
+SEA Flows can specify streaming mode for gRPC:
+
+```sea
+// Unary (default)
+Flow from Client to Server of Request
+
+// Server streaming
+Flow @streaming from Client to Server of EventStream
+
+// Client streaming
+Flow @client_streaming from Client to Server of DataChunks
+
+// Bidirectional
+Flow @bidirectional from Client to Server of ChatMessage
+```
+
+Generated proto:
+
+```protobuf
+service ServerService {
+  rpc ProcessRequest(Request) returns (RequestResponse);
+  rpc StreamEvents(EventStreamRequest) returns (stream EventStream);
+  rpc UploadChunks(stream DataChunks) returns (DataChunksResponse);
+  rpc Chat(stream ChatMessage) returns (stream ChatMessage);
+}
+```
+
+## Troubleshooting
+
+### Missing Well-Known Types
+
+If you see errors about missing `google.protobuf.*` types:
+
+1. Ensure your protoc includes path has the Google WKT:
+
+   ```bash
+   protoc -I /usr/include -I . --go_out=. output.proto
+   ```
+
+2. Or install the WKT package for your language.
+
+### Field Number Stability
+
+Field numbers are assigned deterministically by alphabetical order of field names. To maintain backward compatibility:
+
+- Don't rename fields
+- Don't remove fields without using `reserved`
+- Add new fields at the end (they'll get higher numbers)
+
+### Import Errors
+
+For multi-file output, ensure your import paths match the generated structure:
+
+```bash
+protoc -I ./proto --go_out=. proto/**/*.proto
+```
+
+## See Also
+
+- [SDS-001: Protobuf Projection Engine](../specs/SDS-001-protobuf-projection-engine.md) - System design spec
+- [ADR-003: Protobuf as Projection Target](../specs/ADR-003-protobuf-projection-target.md) - Architecture decision
+- [Protobuf Advanced Features Plan](../plans/protobuf_advanced_features_plan.md) - Roadmap
+- [CLI Commands Reference](../reference/cli-commands.md) - All CLI options

package/docs/how-tos/extend-grammar.md

@@ -0,0 +1,133 @@
+# Extend Grammar
+
+Goal: Extend the SEA grammar in `domainforge-core/grammar/sea.pest`, update the parser/AST, and refresh tests across languages.
+
+## Prerequisites
+
+- Rust toolchain with `cargo` and `just` available.
+- Familiarity with Pest grammar syntax and the existing rules in `domainforge-core/grammar/sea.pest`.
+- Ability to run Rust/Python/TypeScript tests (`just all-tests`) to ensure binding parity.
+
+## Steps (be concise)
+
+1. **Plan the syntax change**
+
+   - Identify the new construct (e.g., `Constraint`, `Annotation`).
+   - Decide whether it is a statement, expression, or modifier of existing rules.
+   - Sketch the EBNF and how it should appear in AST structs.
+
+2. **Update the grammar**
+
+   ```bash
+   sed -n '1,160p' domainforge-core/grammar/sea.pest  # review nearby rules
+   ```
+
+   - Add or modify rules using Pest syntax. Prefer `silent` rules (`_`) for whitespace handling.
+   - Keep tokens unambiguous; if you add keywords, update `identifier` to prevent collisions.
+
+3. **Regenerate parser expectations**
+
+   - If the change affects parsing order, adjust combinators in `domainforge-core/src/parser.rs` or helpers in `domainforge-core/src/ast.rs`.
+   - Update AST types to include new fields (e.g., add `Annotation` struct) and ensure `FromStr` implementations parse them.
+
+4. **Propagate to projections**
+
+   - **CALM/KG/SBVR**: Update projectors (`domainforge-core/src/calm/mod.rs`, `domainforge-core/src/kg.rs`, `domainforge-core/src/sbvr.rs`) so the new construct is emitted.
+   - **Validation**: Add semantic checks in `domainforge-core/src/validator.rs` to keep error reporting meaningful.
+
+5. **Expose through bindings**
+
+   - Python: Add new fields/classes in `domainforge-core/src/python/primitives.rs` and wire them into `domainforge-core/src/python/lib.rs` or `graph.rs` if they are part of the `Graph` API.
+   - TypeScript: Mirror changes in `domainforge-core/src/typescript/primitives.rs` and adjust `domainforge-core/src/typescript/index.d.ts` typing.
+   - WASM: Update `domainforge-core/src/wasm/mod.rs` (or `domainforge-core/src/wasm/lib.rs`) exports to expose the new types if browser consumers need the feature.
+
+6. **Add parser tests (Rust)**
+
+   ```bash
+   cargo test -p domainforge-core parser_tests -- --nocapture
+   ```
+
+   - Create or edit fixtures under `domainforge-core/tests/` or `domainforge-core/examples/` to cover the new syntax and failure cases.
+   - Add golden tests if the construct affects export formats.
+
+7. **Add binding tests (Python + TypeScript)**
+
+   - Python: write a `tests/test_<feature>.py` that parses the DSL snippet and asserts field values.
+   - TypeScript: add a Vitest case under `typescript-tests/` that mirrors the Python assertion to maintain parity.
+
+8. **Run cross-language suites**
+
+   ```bash
+   just rust-test
+   just python-test
+   just ts-test
+   ```
+
+   - Use `just all-tests` to run everything sequentially before committing.
+
+9. **Document the new syntax**
+
+   - Update `docs/new_docs/reference/grammar-spec.md` with the new rule and an example.
+   - Add a short section to the relevant how-to or tutorial to show usage.
+
+## Cross-Bindings and Snapshots
+
+- Keep serialized forms stable; if the new construct appears in CALM, adjust mapping tables and fixtures.
+- Update `index.d.ts` and Python type hints to match the Rust structs so auto-completion remains accurate.
+- Regenerate any snapshot tests that capture textual output (e.g., KG/Turtle strings) after confirming the change is intentional.
+
+## Spec and Example Tests
+
+- Write minimal DSL examples that highlight the new rule with and without namespaces.
+- Add negative tests for malformed input to ensure errors are descriptive (use error codes from `validation_error.rs`).
+- If the rule impacts policies, add evaluation tests in `domainforge-core/src/policy` or binding-specific test suites.
+
+## Common Pitfalls / Troubleshooting
+
+- Forgetting to update one binding often leads to runtime errors (`AttributeError` in Python, `undefined` in TS). Keep a checklist per language.
+- Grammar precedence issues: if a new rule causes existing tests to fail, inspect operator precedence and adjust via Pest ordering or explicit `_` separators.
+- When changing keywords, ensure backwards compatibility or document the breaking change with a migration note.
+
+## Links
+
+- Tutorials: [Extend Grammar Walkthrough](../tutorials/getting-started.md)
+- Reference: [Grammar Spec](../reference/grammar-spec.md), [Primitives API](../reference/primitives-api.md), [Cross-Language Binding Strategy](../explanations/cross-language-binding-strategy.md)
+
+## Verification Checklist
+
+- [ ] Parser accepts the new syntax (Rust tests passing).
+- [ ] CALM/KG/SBVR exports reflect the construct without dropping fields.
+- [ ] Python and TypeScript bindings expose the new fields with the same naming.
+- [ ] Documentation updated (how-to + grammar spec) with a runnable example.
+- [ ] Version bumped or changelog entry added if the change is user-visible.
+
+## Example Workflow (adding a `Note` keyword)
+
+1. Add grammar rule:
+
+   ```pest
+   note = { "Note" ~ string }
+   statement = { role_decl | relation_decl | note | flow_decl | resource_decl | entity_decl }
+   ```
+
+2. Update AST:
+
+   ```rust
+   pub struct Note { pub text: String }
+   ```
+
+3. Wire projection:
+
+   - CALM: add `notes` array under model metadata.
+   - KG: emit as annotation triples.
+
+4. Add tests:
+
+   - Rust: `tests/note_tests.rs` covering parse success/failure.
+   - Python: assert `graph.notes()[0].text == "Payment delayed"`.
+   - TypeScript: similar Vitest assertion.
+
+5. Document:
+
+   - Add a snippet to `grammar-spec.md` and this how-to.
+   - Mention migration steps if the keyword conflicts with existing identifiers.

package/docs/how-tos/generate-rdf-turtle.md

@@ -0,0 +1,106 @@
+# Generate RDF/Turtle
+
+Goal: Output SEA models as RDF/Turtle (or RDF/XML) for knowledge graph pipelines and validate them before loading into a triple store.
+
+## Prerequisites
+
+- SEA CLI installed with `cli` feature.
+- Rust toolchain and optional SHACL feature if you need RDF/XML import validation (`cargo test -p domainforge-core --features "cli shacl"`).
+- Optional bindings: Python/TypeScript packages built locally for programmatic export.
+- A `.sea` file (e.g., `domainforge-core/examples/basic.sea`).
+
+## Steps (be concise)
+
+1. **Export to Turtle using the CLI**
+
+   ```bash
+   domainforge project --format kg domainforge-core/examples/basic.sea /tmp/basic.ttl
+   ```
+
+   - The `project` command parses the DSL, builds a graph, and writes Turtle by default.
+   - The CLI prints `Projected to KG: /tmp/basic.ttl` on success.
+
+2. **Export to RDF/XML instead of Turtle**
+
+   ```bash
+   domainforge project --format kg domainforge-core/examples/basic.sea /tmp/basic.rdf
+   ```
+
+   - The exporter inspects the output extension; `.rdf` or `.xml` triggers RDF/XML formatting.
+
+3. **Export programmatically in Rust**
+
+   ```rust
+   use domainforge_core::{KnowledgeGraph, parser::parse_to_graph};
+
+   let source = std::fs::read_to_string("domainforge-core/examples/basic.sea")?;
+   let graph = parse_to_graph(&source)?;
+   let kg = KnowledgeGraph::from_graph(&graph)?;
+   let turtle = kg.to_turtle();
+   std::fs::write("/tmp/basic.ttl", turtle)?;
+   ```
+
+   - Use `kg.to_rdf_xml()` if you need XML output for SHACL validators.
+
+4. **Export programmatically in Python**
+
+   ```python
+   from domainforge import Graph
+   import json
+
+   graph = Graph.parse(open("domainforge-core/examples/basic.sea").read())
+   calm = graph.export_calm()
+   print("CALM length", len(calm))
+   # Convert to Turtle via the CLI or call Rust from Python if you need KG directly
+   ```
+
+   - Python bindings expose CALM export; convert CALM to Turtle using the CLI (`domainforge import --format kg /tmp/basic.calm.json`) if you need a one-liner.
+
+5. **Export programmatically in TypeScript**
+
+   ```ts
+   import { Graph } from "@godspeedai/domainforge";
+   import { writeFileSync } from "fs";
+
+   const graph = Graph.parse(require("fs").readFileSync("domainforge-core/examples/basic.sea", "utf8"));
+   const calm = graph.exportCalm();
+   writeFileSync("/tmp/basic.calm.json", calm);
+   ```
+
+   - Feed the CALM output to the CLI or a Rust helper to generate Turtle. Direct KG export is currently provided by the Rust core.
+
+6. **Validate the Turtle with SHACL (optional)**
+
+   ```bash
+   domainforge validate-kg --file /tmp/basic.ttl
+   ```
+
+   - Requires building with `--features shacl`. Validation fails if the KG violates the SEA SHACL shapes.
+
+7. **Load into a triple store**
+
+   - Use `riot --validate /tmp/basic.ttl` (Apache Jena) to check syntax.
+   - Load into GraphDB or Blazegraph with the namespace you expect; Turtle output uses CURIE prefixes derived from namespaces in the source DSL.
+
+## Metadata and Namespaces
+
+- Default namespace resolves from `.sea-registry.toml` if present. If you do not set one, resources/entities use the `default` namespace prefix.
+- Custom prefixes appear in the Turtle header. Use consistent namespace configuration to avoid collisions when merging multiple models.
+- CALM metadata fields (`sea:version`, `sea:timestamp`) are not embedded directly in Turtle; store them alongside the graph in your catalog.
+
+## Post-processing and Validation Tips
+
+- **Minify or canonicalize**: Run `riot --out=N-TRIPLE /tmp/basic.ttl` to flatten prefixes for diff-friendly output.
+- **SHACL diagnostics**: When `domainforge validate-kg` fails, inspect the error for the offending triple and fix the originating DSL element (often a missing role or flow ID).
+- **Round-trip**: Convert DSL → Turtle → Import (`domainforge import --format kg`) → Export CALM to ensure no information loss before loading to production stores.
+
+## Common Pitfalls / Troubleshooting
+
+- Missing namespaces lead to verbose IRIs; define them in the registry to keep triples stable.
+- Ensure flows have UUIDs; the KG exporter relies on deterministic IDs for edges.
+- If you see `Failed to convert to Knowledge Graph`, re-run `domainforge validate` first—semantic errors in the DSL often surface during KG projection.
+
+## Links
+
+- Tutorials: [Getting Started](../tutorials/getting-started.md)
+- Reference: [CALM Mapping](../reference/calm-mapping.md), [CLI Commands](../reference/cli-commands.md), [Grammar Spec](../reference/grammar-spec.md)