@ldclabs/kip-lang 0.3.1 → 2.0.0

package/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to `@ldclabs/kip-lang` are documented here.
+
+## 2.0.0
+
+Targets KIP specification revision `2.0-draft`. This is a rewrite of the
+language surface, not an increment: KIP 2.0 command text is not backward
+compatible with the 1.0 grammar the 0.x line implemented.
+
+### Added
+
+- **Executable AST.** `lower` / `lowerAll` / `lowerStatement` turn a syntax
+  tree into the closed `Command` AST an engine runs, rejecting what the
+  grammar admits but the language does not with a KIP error code.
+- **KQL 2.0**: the five Core element patterns (`CONCEPT`, `PROPOSITION`,
+  `ASSERTION`, `EVIDENCE`, `ACTIVITY`), `STRUCTURAL` topology patterns,
+  `BELIEF` / `BELIEF SLOT` projection patterns, the independent `AS OF`
+  (`SEQ` / `TX`) and `FOR TIME` axes, `WITH EPISTEMIC`, multi-key `ORDER BY`,
+  and `CURSOR` paging.
+- **KML 2.0**: `MUTATE` transactions, `CREATE`/`UPSERT CONCEPT`,
+  `ENSURE PROPOSITION`, the `ASSERT` sugar with `SUPERSEDING`,
+  `CREATE EVIDENCE | ASSERTION | ACTIVITY`, generic `UPDATE`, the
+  `RETRACT` / `SUPERSEDE` / `CORRECT` / `TRANSITION` lifecycle,
+  `SET RETENTION`, `ARCHIVE`, `TOMBSTONE`, `PURGE ... CONFIRM "PURGE"`, and
+  `MERGE CONCEPT`, with `EXPECT VERSION` / `EXPECT STATE` preconditions.
+- **`UNSET STRUCTURAL`**, so every `SET` clause has its removal counterpart.
+- **META 2.0**: every `DESCRIBE`, `LIST`, `SEARCH`, `VERIFY`, `VALIDATE`,
+  `PREVIEW`, `HISTORY`, `CHANGES`, `SNAPSHOT` and `EXPORT CAPSULE` target.
+- **`validateExecutable(program)`**, chained into `diagnose`, so the editor
+  reports any statement that parses but cannot lower.
+- **`checkBudget` / `checkBatchBudget`** (`KIP_4002`) matching `anda_kip`'s
+  ceilings, so a command one engine refuses on size is refused by all.
+- `KipSyntaxError` carrying KIP error codes; `PARSER_VERSION` and
+  `KIP_SPEC_REVISION` exports.
+- `src/` ships alongside `dist/`, so published source maps resolve.
+
+### Changed
+
+- Keywords are ASCII case-insensitive and **contextual, not reserved** — `by`,
+  `mode`, `type`, `status` remain usable as object keys and dot-path steps.
+- The parser enforces the canonical KQL trailing-clause order (`AS OF`,
+  `FOR TIME`, `WITH EPISTEMIC`, `ORDER BY`, `LIMIT`, `CURSOR`) while still
+  recovering from a mistake.
+- Proposition subjects lower as Element references only; KML predicates and
+  structural edges take an exact quoted name or a `:parameter`, never a
+  `?variable`.
+- Unbound `MUTATE` references (`KIP_2102`) are errors, not warnings.
+- The formatter keeps a comment attached to the body clause it annotates.
+
+## 0.4.0 and earlier
+
+Implemented the KIP 1.0 command-text surface. See the git history.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 LDC Labs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,16 +1,51 @@
 # @ldclabs/kip-lang
 
-TypeScript toolkit for **KIP** (Knowledge Interaction Protocol) — a structured query language for knowledge graphs.
-
-Provides a full-featured **lexer → parser → AST → formatter / diagnostics** pipeline for `.kip` files.
-
-The parser targets the KIP v1.0-RC9 command-text syntax, including `:parameter`
-placeholders in full value positions, JSON-compatible object literals with
-unquoted identifier keys, predicate variables, multi-key `ORDER BY`, `SEARCH`
-retrieval modes and thresholds, `EXPECT VERSION`, `UPDATE`, `MERGE`, `EXPORT`,
-and batch-friendly multi-command source text. The `execute_kip` request envelope
-(`command`, `commands`, `parameters`, `dry_run`) is JSON outside the `.kip`
-language and should be validated by the caller.
+TypeScript toolkit for **KIP** (Knowledge Interaction Protocol) — the cognitive
+state protocol between an agent and a persistent Cognitive Nexus.
+
+Provides a full-featured **lexer → parser → AST → formatter / diagnostics**
+pipeline for `.kip` files, plus **`lower`**, which turns a syntax tree into the
+executable AST a KIP engine runs.
+
+This package targets the normative **KIP 2.0 draft** command-text surface. It
+implements the KQL, KML, and META syntax surfaces, but it does not claim a
+complete KIP conformance profile by itself: Schema resolution, Governance,
+transactions, persistence, projection, history, Capsules, and runtime
+envelopes remain engine responsibilities.
+
+The parser targets the KIP 2.0 command-text syntax — all three languages:
+**KQL** (`FIND`) to read, **KML** (`ASSERT`, `MUTATE`, `CREATE`, `UPSERT`,
+`UPDATE`, and the retract/supersede/archive/tombstone/purge lifecycle) to
+change cognition, and **META** (`DESCRIBE`, `LIST`, `SEARCH`, `VERIFY`,
+`VALIDATE`, `PREVIEW`, `HISTORY`, `CHANGES`, `SNAPSHOT`, `EXPORT CAPSULE`) to
+ground and introspect. That includes the five Core element kinds (Concept,
+Proposition, Assertion, Evidence, Activity), `BELIEF` / `BELIEF SLOT`
+projection patterns, `STRUCTURAL` topology patterns, facets, the independent
+`AS OF` / `FOR TIME` time axes, `:parameter` placeholders in full value
+positions, ASCII case-insensitive keywords, JSON-compatible object literals
+with unquoted identifier keys, and batch-friendly multi-command source text.
+The request envelope (`kip`, `request_id`, `space`, `execution`, `ingest`,
+`operations`) is JSON outside the `.kip` language and should be validated by
+the caller.
+
+## Two trees, two audiences
+
+`parse` returns a **syntax tree**: ranges, comments, quoting style, raw number
+text — everything a formatter or an editor needs to reproduce the source, and
+error recovery that keeps producing a tree after a mistake.
+
+`lower` returns the **executable AST**: every construct collapsed to the one
+shape it means, with the open-ended parts of the grammar closed. The `ASSERT`
+sugar desugars to exactly the `ENSURE PROPOSITION` + `CREATE ASSERTION`
+(+ `SUPERSEDE`) it stands for, and nothing more; a filter becomes a
+comparison, a logical node, a negation, or a call to a registered function; a
+variable becomes a name and a dot path. Anything the syntax admits but the
+language does not — an unknown filter function, an `UPSERT` identified by name
+alone, an `UPDATE` that reaches into Assertion epistemic payload — is rejected
+here, with a KIP error code.
+
+That split is why the two exist: an editor wants the loosest tree it can get,
+an engine wants the tightest.
 
 ## Installation
 
@@ -29,16 +64,20 @@ const tokens = tokenize('FIND(?x.name) WHERE { ?x {type: "Person"} }')
 // Token[] with type, value, line, column, offset
 ```
 
+Keywords are ASCII case-insensitive, so `find` and `FIND` produce the same
+token type; schema symbols and string values keep their own case-sensitive
+contracts.
+
 ### Parse
 
 ```ts
 import { parse } from '@ldclabs/kip-lang'
 
 const { ast, diagnostics } = parse(`
-FIND(?drug.name)
+FIND(?belief.status, ?tz)
 WHERE {
-  ?drug {type: "Drug"}
-  (?drug, "treats", {name: "Headache"})
+  ?person {type: "Person", name: "Alice"}
+  ?belief BELIEF (?person, "timezone", ?tz)
 }
 LIMIT 10
 `)
@@ -52,35 +91,35 @@ console.log(diagnostics)            // [] (no errors)
 ```ts
 import { format } from '@ldclabs/kip-lang'
 
-const source = `UPSERT { CONCEPT ?x { {type: "Drug", name: "Aspirin"} SET ATTRIBUTES { risk_level: 2 } } }`
+const source = `CREATE CONCEPT ?exp { TYPE "Experience" NAME "Deploy v2 failure" SET ATTRIBUTES { goal: :goal, outcome_status: "failure" } }`
 
 const formatted = format(source, {
-  indentSize: 4,       // default: 4
-  sortAttributes: true // default: true
+  indentSize: 4,        // default: 4
+  sortAttributes: false // default: false — author key order is preserved
 })
 ```
 
 Output:
 
 ```kip
-UPSERT {
-    CONCEPT ?x {
-        {type: "Drug", name: "Aspirin"}
-        SET ATTRIBUTES { risk_level: 2 }
-    }
+CREATE CONCEPT ?exp {
+    TYPE "Experience"
+    NAME "Deploy v2 failure"
+    SET ATTRIBUTES {goal: :goal, outcome_status: "failure"}
 }
 ```
 
-The formatter preserves comments and quoted/unquoted key styles from the original source.
-
-A lighter alternative `formatPreservingComments()` is also available — it normalizes indentation at the line level without AST reconstruction.
+The formatter preserves comments and quoted/unquoted key styles from the
+original source. `sortAttributes` alphabetizes `SET ATTRIBUTES` keys, and skips
+any block holding a comment, since reordering would detach the comment from
+its key.
 
 ### Diagnose
 
 ```ts
 import { diagnose } from '@ldclabs/kip-lang'
 
-const diagnostics = diagnose('UPSERT { CONCEPT ?x { {type: "Drug"} }')
+const diagnostics = diagnose('CREATE CONCEPT ?x { TYPE "Experience"')
 // [{ severity: "error", message: "Unclosed '{'", ... }]
 ```
 
@@ -90,35 +129,85 @@ Diagnostics cover:
 - Bracket matching (unclosed / mismatched braces, brackets, parentheses)
 - Parser errors (unexpected tokens, missing clauses)
 
+### Lower to the executable AST
+
+```ts
+import { parse, lower, checkBudget } from '@ldclabs/kip-lang'
+
+function command(source: string) {
+  checkBudget(source)                       // KIP_4002 before parsing recurses
+  const { ast, diagnostics } = parse(source)
+  const fatal = diagnostics.find((d) => d.severity === 'error')
+  if (fatal) throw new Error(`${fatal.code} ${fatal.message}`)
+  return lower(ast)                         // throws KipSyntaxError
+}
+
+command('FIND(?e.name) WHERE { ?e {type: "Experience"} } LIMIT 10')
+// → the executable Command AST
+```
+
+`lower` requires the program to hold exactly one command. In KML that means
+one statement — `MUTATE { ... }` is how several mutations become one
+all-or-nothing cognitive transition, so a transaction is still one command.
+Use `lowerAll` for source text that is genuinely a sequence of commands.
+
+The `diagnostics` check above is not decoration. `parse` recovers from a
+malformed string or an unquoted value by reading it leniently, and `lower`
+receives only the tree — so skipping the check lowers `"a\xb"` to `axb` and
+`{a: bare}` to `{a: "bare"}` without a word of complaint.
+
+`checkBudget` is not optional for a server. `parse` is recursive descent, so
+without a depth ceiling a hostile `[[[[…` overflows the JavaScript stack — and
+in a runtime that shares one stack across requests, that takes down more than
+the one request. The budgets match `anda_kip`'s, so a command one KIP engine
+refuses on size is refused by every other engine too.
+
 ## API Reference
 
-| Export                                       | Description                                              |
-| -------------------------------------------- | -------------------------------------------------------- |
-| `tokenize(source)`                           | Tokenize KIP source into `Token[]`                       |
-| `parse(source)`                              | Parse into `{ ast: Program, diagnostics: Diagnostic[] }` |
-| `format(source, options?)`                   | AST-based formatting with comment preservation           |
-| `formatPreservingComments(source, options?)` | Line-level indent normalization                          |
-| `diagnose(source)`                           | Return `Diagnostic[]` (lexer + bracket + parser errors)  |
-| `TokenType`                                  | Enum of all token types                                  |
-| `KEYWORDS` / `FUNCTIONS`                     | Maps of KIP keywords and built-in functions              |
+| Export                             | Description                                              |
+| ---------------------------------- | -------------------------------------------------------- |
+| `tokenize(source)`                 | Tokenize KIP source into `Token[]`                       |
+| `parse(source)`                    | Parse into `{ ast: Program, diagnostics: Diagnostic[] }` |
+| `format(source, options?)`         | AST-based formatting with comment preservation           |
+| `diagnose(source)`                 | Return syntax, static-semantic, and executable diagnostics |
+| `validateExecutable(program)`      | Check every parsed statement can lower to executable AST |
+| `analyzeSemantics(program)`        | Spec SHOULD/MUST checks decidable without a live schema  |
+| `lower(program)`                   | Lower one command to the executable `Command` AST        |
+| `lowerAll(program)`                | Lower every command in a multi-statement program         |
+| `lowerStatement(statement)`        | Lower a single statement                                 |
+| `checkBudget(source)`              | Enforce length and nesting ceilings (`KIP_4002`)         |
+| `checkBatchBudget(count)`          | Enforce the command-count ceiling (`KIP_4002`)           |
+| `KipSyntaxError`                   | Thrown by `lower` / `checkBudget`, carries a KIP code    |
+| `TokenType`                        | Enum of all token types                                  |
+| `KEYWORDS` / `FUNCTIONS`           | Keyword map and built-in function set                    |
+| `PARSER_VERSION` / `KIP_SPEC_REVISION` | Grammar version and the spec revision it targets     |
 
 ### AST Node Types
 
 All AST types are exported for downstream consumption:
 
-- **Statements**: `FindStatement`, `UpsertStatement`, `UpdateStatement`, `MergeStatement`, `DeleteStatement`, `DescribeStatement`, `SearchStatement`, `ExportStatement`
-- **Blocks**: `ConceptBlock`, `PropositionBlock`, `SetAttributes`, `SetMetadata`, `SetPropositions`, `WithMetadata`, `ExpectVersion`
-- **Patterns**: `ConceptPattern`, `PropositionPattern`, `FilterClause`, `NotClause`, `OptionalClause`, `UnionClause`
-- **Expressions**: `ObjectLiteral`, `ArrayLiteral`, `ObjectEntry`, `Expression`
+- **KQL**: `FindStatement`, with `AsOfClause`, `ForTimeClause`, `EpistemicClause`, `OrderByClause`, `LimitClause`, `CursorClause`
+- **KML**: `MutateStatement` plus every `MutationClause` — `CreateConceptStatement`, `UpsertConceptStatement`, `EnsurePropositionStatement`, `AssertStatement`, `CreateEvidenceStatement`, `CreateAssertionStatement`, `CreateActivityStatement`, `UpdateStatement`, `RetractAssertionStatement`, `SupersedeAssertionStatement`, `CorrectEvidenceStatement`, `TransitionActivityStatement`, `SetRetentionStatement`, `ArchiveStatement`, `TombstoneStatement`, `PurgeStatement`, `MergeConceptStatement`
+- **META**: `DescribeStatement`, `ListStatement`, `SearchStatement`, `VerifyStatement`, `ValidateStatement`, `PreviewStatement`, `HistoryStatement`, `ChangesStatement`, `SnapshotStatement`, `ExportCapsuleStatement`
+- **Clauses**: `TypeClause`, `ClientKeyClause`, `NameClause`, `MatchClause`, `SetFieldsClause`, `SetAttributesClause`, `SetFacetClause`, `UnsetAttributesClause`, `UnsetFacetClause`, `SetStructuralClause`, `UnsetStructuralClause`, `ExpectVersionClause`, `ExpectStateClause`
+- **Patterns**: `ConceptPattern`, `PropositionPattern`, `AssertionPattern`, `EvidencePattern`, `ActivityPattern`, `StructuralPattern`, `BeliefPattern`, `BeliefSlotPattern`, `FilterClause`, `NotClause`, `OptionalClause`, `UnionClause`
+- **Expressions**: `Expression`, `FieldAccess`, `AggregateExpr`, `FunctionCallExpr`, `ObjectLiteral`, `ObjectPattern`, `ArrayLiteral`, `ObjectEntry`
+
+The executable AST is exported separately, and names that collide with a
+syntax-tree node carry an `Exec` prefix.
 
-String-or-parameter statement fields such as `SearchStatement.termValue`,
-`SearchStatement.withTypeValue`, `SearchStatement.modeValue`, and
-`DescribeStatement.typeNameValue` preserve whether the original source used a
-quoted string or a `:parameter` placeholder.
+String-or-parameter operands stay `ScalarValue` nodes — `SearchStatement.term`,
+`SearchStatement.withType`, `DescribeStatement.value`, `ClientKeyClause.value`
+— so the tree still says whether the source wrote a quoted string or a
+`:parameter` placeholder.
 
 ## KIP Language
 
-KIP is a structured query and mutation language for knowledge graphs. See the [KIP Specification](../../SPECIFICATION.md) for full syntax details.
+KIP 2.0 is the cognitive state protocol between an agent and a persistent
+Cognitive Nexus: you read with KQL, change cognition with KML, and ground or
+introspect with META. See the [KIP 2.0 Specification](https://github.com/ldclabs/KIP/blob/main/v2/KIP-2.0-SPECIFICATION.md)
+for full syntax details, or the [syntax card](https://github.com/ldclabs/KIP/blob/main/v2/KIPSyntax.md) for a
+condensed reference.
 
 ## License