@fraczak/k 6.0.0 → 6.0.1

package/codecs/POLYMORPHIC_BINARY_FORMAT.html

@@ -0,0 +1,355 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
+<head>
+  <meta charset="utf-8" />
+  <meta name="generator" content="pandoc" />
+  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
+  <title>POLYMORPHIC_BINARY_FORMAT</title>
+  <style>
+    code{white-space: pre-wrap;}
+    span.smallcaps{font-variant: small-caps;}
+    div.columns{display: flex; gap: min(4vw, 1.5em);}
+    div.column{flex: auto; overflow-x: auto;}
+    div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
+    /* The extra [class] is a hack that increases specificity enough to
+       override a similar rule in reveal.js */
+    ul.task-list[class]{list-style: none;}
+    ul.task-list li input[type="checkbox"] {
+      font-size: inherit;
+      width: 0.8em;
+      margin: 0 0.8em 0.2em -1.6em;
+      vertical-align: middle;
+    }
+    .display.math{display: block; text-align: center; margin: 0.5rem auto;}
+  </style>
+  <link rel="stylesheet" href="pandoc.css" />
+</head>
+<body>
+<h1 id="k-polymorphic-binary-format">K Polymorphic Binary Format</h1>
+<h2 id="purpose">Purpose</h2>
+<p>This document defines a binary format for polymorphic values in
+<code>k</code>.</p>
+<p>The semantic object to be serialized is:</p>
+<pre class="text"><code>(P, v)</code></pre>
+<p>where:</p>
+<ul>
+<li><code>P</code> is a normalized pattern graph,</li>
+<li><code>v</code> is a typed value compatible with <code>P</code>.</li>
+</ul>
+<p>The binary format is designed so that:</p>
+<ul>
+<li>the pattern carries all field and tag names,</li>
+<li>the value payload carries no explicit labels or tags,</li>
+<li>the value payload is friendly to <code>.label</code> and
+<code>/tag</code>,</li>
+<li>closed typed values appear as the singleton-pattern special
+case.</li>
+</ul>
+<h2 id="design-principles">Design Principles</h2>
+<h3 id="pattern-is-the-structural-authority">1. Pattern is the
+structural authority</h3>
+<p>The pattern section is the authoritative description of the structure
+expected by the value section.</p>
+<p>It determines:</p>
+<ul>
+<li>whether a node is product or union,</li>
+<li>whether it is open or closed,</li>
+<li>which fields or tags are present explicitly,</li>
+<li>which subpositions point to the same pattern node,</li>
+<li>which leaves are exact types.</li>
+</ul>
+<h3 id="value-is-interpreted-relative-to-the-pattern">2. Value is
+interpreted relative to the pattern</h3>
+<p>The value payload is not self-describing. It is decoded relative to
+the pattern graph.</p>
+<p>This is why the payload does not need to repeat field names or tag
+names.</p>
+<h3 id="is-not-value-carrying">3. <code>(...)</code> is not
+value-carrying</h3>
+<p>The leaf pattern <code>(...)</code> is allowed in the pattern
+language, but it is <strong>not</strong> allowed to carry serialized
+value content in version 1 of this format.</p>
+<p>This restriction keeps explicit labels out of the value payload.
+Otherwise the missing structure would have to be reintroduced locally in
+the value itself.</p>
+<h3 id="value-dag-is-only-an-optimization">4. Value DAG is only an
+optimization</h3>
+<p>Semantically, the value is a tree. On the wire, it may be encoded as
+a DAG by collapsing equal occurrences relative to the pattern.</p>
+<h2 id="high-level-structure">High-Level Structure</h2>
+<p>The package layout is:</p>
+<pre class="text"><code>| magic | format_version | flags | symbol_table | pattern_section | value_section |</code></pre>
+<p>There is no separate <code>type_section</code>.</p>
+<h2 id="primitive-encodings">Primitive Encodings</h2>
+<h3 id="integers">Integers</h3>
+<ul>
+<li>fixed-width integers are unsigned and big-endian,</li>
+<li>variable-width integers use unsigned LEB128 and are written
+<code>uvarint</code>.</li>
+</ul>
+<h3 id="symbols">Symbols</h3>
+<p>A symbol is a UTF-8 string used as:</p>
+<ul>
+<li>a field name,</li>
+<li>a tag name,</li>
+<li>an exact type identifier when such identifiers are serialized as
+strings.</li>
+</ul>
+<p>Symbols are interned in a symbol table and referred to by symbol
+ID.</p>
+<h2 id="header">Header</h2>
+<pre class="text"><code>| magic:4 bytes | format_version:u8 | flags:u8 |</code></pre>
+<p>Current values:</p>
+<ul>
+<li><code>magic = "KPV2"</code></li>
+<li><code>format_version = 1</code></li>
+</ul>
+<p>All flag bits are reserved in version 1 and must be zero.</p>
+<h2 id="symbol-table">Symbol Table</h2>
+<pre class="text"><code>| symbol_count:uvarint | symbol_0 | ... | symbol_(N-1) |</code></pre>
+<p>Each symbol is:</p>
+<pre class="text"><code>| byte_length:uvarint | utf8_bytes... |</code></pre>
+<h3 id="canonical-ordering">Canonical ordering</h3>
+<p>Symbols are ordered canonically by:</p>
+<ol type="1">
+<li>ascending UTF-8 byte sequence,</li>
+<li>duplicates removed.</li>
+</ol>
+<p>The same symbol table is used by both the pattern and the value
+sections.</p>
+<h2 id="pattern-section">Pattern Section</h2>
+<p>The pattern section serializes a normalized rooted pattern graph.</p>
+<h3 id="pattern-node-kinds">Pattern node kinds</h3>
+<p>There are exactly five semantic node kinds:</p>
+<ul>
+<li><code>(...)</code></li>
+<li><code>{...}</code></li>
+<li><code>&lt;...&gt;</code></li>
+<li><code>{}</code></li>
+<li><code>&lt;&gt;</code></li>
+</ul>
+<p>These are constructor classes of pattern nodes, not leaf/internal
+tags. Only <code>(...)</code> is forced to have no outgoing edges. All
+other node kinds may have zero or more outgoing edges.</p>
+<p>Their meanings are:</p>
+<ul>
+<li><p><code>(...)</code> means unconstrained type,</p></li>
+<li><p><code>{...}</code> with edges
+<code>l1 -&gt; X1, ..., ln -&gt; Xn</code> means</p>
+<pre class="text"><code>{ X1 l1, ..., Xn ln, ... }</code></pre></li>
+<li><p><code>&lt;...&gt;</code> with edges
+<code>t1 -&gt; X1, ..., tn -&gt; Xn</code> means</p>
+<pre class="text"><code>&lt; X1 t1, ..., Xn tn, ... &gt;</code></pre></li>
+<li><p><code>{}</code> with edges
+<code>l1 -&gt; X1, ..., ln -&gt; Xn</code> means</p>
+<pre class="text"><code>{ X1 l1, ..., Xn ln }</code></pre></li>
+<li><p><code>&lt;&gt;</code> with edges
+<code>t1 -&gt; X1, ..., tn -&gt; Xn</code> means</p>
+<pre class="text"><code>&lt; X1 t1, ..., Xn tn &gt;</code></pre></li>
+</ul>
+<p>In particular:</p>
+<ul>
+<li><code>{}</code> with zero edges is the unit type,</li>
+<li><code>&lt;&gt;</code> with zero edges is the empty type,</li>
+<li><code>{...}</code> with zero edges matches any product type,</li>
+<li><code>&lt;...&gt;</code> with zero edges matches any union
+type.</li>
+</ul>
+<h3 id="canonical-node-numbering">Canonical node numbering</h3>
+<p>Pattern nodes are numbered by first discovery in rooted depth-first
+traversal:</p>
+<ol type="1">
+<li>start at the root,</li>
+<li>visit outgoing edges in ascending symbol ID order,</li>
+<li>assign a new node ID on first encounter,</li>
+<li>reuse the existing node ID on revisits.</li>
+</ol>
+<p>The root pattern node is always node <code>0</code>.</p>
+<h3 id="structure">Structure</h3>
+<pre class="text"><code>| pattern_node_count:uvarint | pattern_node_0 | ... | pattern_node_(P-1) |</code></pre>
+<h3 id="pattern-node-record">Pattern node record</h3>
+<p>Every pattern node record starts with:</p>
+<pre class="text"><code>| node_kind:u8 |</code></pre>
+<p>The codes are:</p>
+<ul>
+<li><code>0</code> = <code>(...)</code></li>
+<li><code>1</code> = <code>{...}</code></li>
+<li><code>2</code> = <code>&lt;...&gt;</code></li>
+<li><code>3</code> = <code>{}</code></li>
+<li><code>4</code> = <code>&lt;&gt;</code></li>
+</ul>
+<p>The remainder of every record is:</p>
+<pre class="text"><code>| edge_count:uvarint | edge_0 | ... | edge_(k-1) |</code></pre>
+<p>Each edge is:</p>
+<pre class="text"><code>| symbol_id:uvarint | target_pattern_node:uvarint |</code></pre>
+<p>Edges must be sorted by ascending <code>symbol_id</code> and have no
+duplicates.</p>
+<p>Validation rules for node kinds:</p>
+<ul>
+<li><code>(...)</code> must have <code>edge_count = 0</code>,</li>
+<li>all other node kinds may have any <code>edge_count ≥ 0</code>.</li>
+</ul>
+<p>This document therefore starts without type hashes or exact-type
+leaves. Closed typed values are represented simply as values whose root
+pattern happens to be singleton.</p>
+<h2 id="value-section">Value Section</h2>
+<p>The value section encodes the witness value tree relative to the
+pattern graph. On the wire, it is represented as a DAG of value
+occurrences decorated by pattern-node identity.</p>
+<h3 id="canonical-value-node-identity">Canonical value-node
+identity</h3>
+<p>The canonical key of a value node is:</p>
+<pre class="text"><code>(pattern_node_id, node_shape)</code></pre>
+<p>where:</p>
+<ul>
+<li>for product pattern nodes (<code>{...}</code> or <code>{}</code>),
+<code>node_shape</code> is the ordered list of child node IDs,</li>
+<li>for union pattern nodes (<code>&lt;...&gt;</code> or
+<code>&lt;&gt;</code>), <code>node_shape</code> is
+<code>(tag_ordinal, child_node_id)</code>,</li>
+<li><code>(...)</code> carries no value and therefore cannot appear in
+the value section.</li>
+</ul>
+<p>This is the criterion for canonical DAG sharing in the value
+payload.</p>
+<h3 id="encodability-restriction">Encodability restriction</h3>
+<p>Version 1 forbids serialized values from descending into an
+<code>any-leaf</code> pattern node.</p>
+<p>So if evaluation of a value against a pattern would require
+materializing a subtree at <code>(...)</code>, the value is <strong>not
+encodable</strong> in this format.</p>
+<h3 id="canonical-value-node-numbering">Canonical value-node
+numbering</h3>
+<p>Value nodes are numbered in canonical postorder:</p>
+<ol type="1">
+<li>children before parents,</li>
+<li>for product nodes, children in ascending pattern-edge symbol
+order,</li>
+<li>for union nodes, the selected child before the parent,</li>
+<li>a shared node is emitted only once, at the first completed postorder
+visit.</li>
+</ol>
+<p>The root value node is always the last node.</p>
+<h3 id="structure-1">Structure</h3>
+<pre class="text"><code>| value_node_count:uvarint | value_node_0 | ... | value_node_(V-1) |</code></pre>
+<h3 id="value-node-record">Value node record</h3>
+<p>Each value node starts with:</p>
+<pre class="text"><code>| pattern_node_id:uvarint | body... |</code></pre>
+<p>The body depends on the referenced pattern node kind.</p>
+<h4 id="product-value-node">Product value node</h4>
+<p>If the pattern node is <code>{...}</code> or <code>{}</code> with
+<code>k</code> outgoing edges in ascending symbol order:</p>
+<pre class="text"><code>| child_ref_0:uvarint | child_ref_1:uvarint | ... | child_ref_(k-1):uvarint |</code></pre>
+<p>The value carries exactly the children of the explicitly listed
+fields. Open product patterns do not permit the value section to
+introduce additional unlabeled fields.</p>
+<h4 id="union-value-node">Union value node</h4>
+<p>If the pattern node is <code>&lt;...&gt;</code> or
+<code>&lt;&gt;</code> with <code>m</code> outgoing edges in ascending
+symbol order:</p>
+<pre class="text"><code>| tag_ordinal:uvarint | child_ref:uvarint |</code></pre>
+<p><code>tag_ordinal</code> must be in <code>[0, m)</code>. Open union
+patterns do not permit the value section to introduce additional unnamed
+tags.</p>
+<h4 id="value-node"><code>(...)</code> value node</h4>
+<p>This case is forbidden in version 1.</p>
+<h2 id="child-references">Child References</h2>
+<p>Child references are encoded as back-distances:</p>
+<pre class="text"><code>child_ref = current_value_node_id - 1 - child_value_node_id</code></pre>
+<p>So:</p>
+<ul>
+<li><code>0</code> means the immediately preceding node,</li>
+<li><code>1</code> means two records back,</li>
+<li>and so on.</li>
+</ul>
+<p>The decoder reconstructs:</p>
+<pre class="text"><code>child_value_node_id = current_value_node_id - 1 - child_ref</code></pre>
+<h2 id="validation">Validation</h2>
+<p>After decoding, a package is valid iff all of the following hold.</p>
+<h3 id="pattern-section-1">Pattern section</h3>
+<ul>
+<li>all pattern node IDs referenced by edges are in range,</li>
+<li>all edge symbol IDs are in range,</li>
+<li>edge symbol IDs are strictly increasing within each node,</li>
+<li><code>(...)</code> has no outgoing edges,</li>
+<li>the root node exists.</li>
+</ul>
+<h3 id="value-section-1">Value section</h3>
+<ul>
+<li>every <code>pattern_node_id</code> is in range,</li>
+<li>every child reference points to an earlier value node,</li>
+<li>a product value node references a <code>{...}</code> or
+<code>{}</code> pattern node,</li>
+<li>a union value node references a <code>&lt;...&gt;</code> or
+<code>&lt;&gt;</code> pattern node,</li>
+<li>no value node references <code>(...)</code>,</li>
+<li><code>tag_ordinal</code> is valid for the referenced union pattern
+node,</li>
+<li>the root value node references pattern node <code>0</code>.</li>
+<li>no value node references <code>&lt;&gt;</code> with zero outgoing
+edges, since the empty type has no values.</li>
+</ul>
+<h2 id="example">Example</h2>
+<p>Consider:</p>
+<pre class="k"><code>pattern: &lt; X tag1, {} tag2, ... &gt; = X
+value:   {}|tag2|tag1|tag1</code></pre>
+<h3 id="symbols-1">Symbols</h3>
+<pre class="text"><code>0 -&gt; &quot;tag1&quot;
+1 -&gt; &quot;tag2&quot;</code></pre>
+<h3 id="pattern-graph">Pattern graph</h3>
+<pre class="text"><code>P0: &lt;...&gt;, edges(tag1 -&gt; P0, tag2 -&gt; P1)
+P1: {},    no edges
+root = P0</code></pre>
+<h3 id="value-dag">Value DAG</h3>
+<pre class="text"><code>V0: {}            at P1
+V1: V0 | tag2     at P0
+V2: V1 | tag1     at P0
+V3: V2 | tag1     at P0
+root = V3</code></pre>
+<p>This example has no repeated decorated subtrees, so the value DAG is
+the same as the value tree.</p>
+<h2 id="closed-value-specialization">Closed-Value Specialization</h2>
+<p>If the root pattern is singleton, then the whole package represents
+an ordinary closed typed value.</p>
+<p>In that case:</p>
+<ul>
+<li>the pattern section still exists and makes the package
+self-contained,</li>
+<li>a more compact specialization may omit the pattern section and use
+the dedicated closed-value format.</li>
+</ul>
+<p>So the closed-value format is a derived optimization, not the
+semantic foundation.</p>
+<h2 id="operational-properties">Operational Properties</h2>
+<p>The value payload is chosen to support efficient <code>k</code>
+navigation.</p>
+<h3 id="label"><code>.label</code></h3>
+<p>For a product value node:</p>
+<ol type="1">
+<li>inspect the referenced pattern node,</li>
+<li>map <code>label</code> to field ordinal by the pattern edge
+order,</li>
+<li>jump to the corresponding child reference.</li>
+</ol>
+<h3 id="tag"><code>/tag</code></h3>
+<p>For a union value node:</p>
+<ol type="1">
+<li>inspect the referenced pattern node,</li>
+<li>compare the stored <code>tag_ordinal</code> with the requested tag
+ordinal,</li>
+<li>on success follow the child reference.</li>
+</ol>
+<p>So field and tag names stay in the pattern graph, not in the value
+payload.</p>
+<h2 id="future-extensions">Future Extensions</h2>
+<p>Possible future extensions include:</p>
+<ul>
+<li>introducing exact-type leaf nodes and a pattern registry,</li>
+<li>allowing a controlled self-describing payload under
+<code>(...)</code>,</li>
+<li>auxiliary indexes for faster random access,</li>
+<li>optional textual debug envelopes,</li>
+<li>section compression that preserves the decoded semantic object.</li>
+</ul>
+</body>
+</html>