@fraczak/k 4.0.4 → 5.0.0

package/BIG_PICTURE.md

@@ -25,8 +25,10 @@ Types are defined by two constructs:
 
 Given a finite (possibly empty) set of types `t1, t2, ..., tn` and a set of pairwise different tags (labels) `l1, l2, ..., ln`,
 
-- by `{ t1 l1, t2 l2, ..., tn ln }`, we define a _product type_ with exactly `n` projection functions `.l1`, `.l2`, `...`, `.ln`,
-  mapping a value of type `{t1 l1, t2 l2, ..., tn ln}` into a value of type `t1`, `t2`, `...`, `tn`, respectively.
+- by `{ t1 l1, t2 l2, ..., tn ln }`, we define a _product type_ with exactly `n`
+  _projection_ functions: `.l1`, `.l2`, `...`, `.ln`.
+  Each projection function is mapping a value of type `{t1 l1, t2 l2, ..., tn ln}`
+  into a value of type `t1`, `t2`, `...`, `tn`, respectively.
 - by `< t1 l1, t2 l2, ..., tn ln >`, we define a _union type_ with exactly `n` projection functions `.l1`, `.l2`, `...`, `.ln`,
   mapping a value of type `<t1 l1, t2 l2, ..., tn ln>` into a value of type `t1`, `t2`, `...`, `tn`, respectively.
 
@@ -38,8 +40,8 @@ Some properties of these types:
 
 1. Singleton product and singleton union types, `{ t label }` and `< t label >`, are equivalent as they have
    the same set of values (and the same projection function `.label`). That's why, we not allow singleton product nor
-   singleton union types in the syntax of the language. The other benefit is that we will be able to use the same systax
-   for values of both product and union types, interpretting every singleton product-like value, e.g., `{ a_value label }`,
+   singleton union types in the syntax of the language. The other benefit is that we will be able to use the same syntax
+   for writing out values of both product and union types, interpretting every singleton product-like value, e.g., `{ a_value label }`,
    as a union type value (a variant).  
 2. The empty product type `{}` has a single value, called the _unit_ and also denoted by `{}`, which has no fields.
 3. The empty union type `<>` has no values.
@@ -50,7 +52,7 @@ A _partial function_ is a function that is not defined for every value of its do
 
 There are three ways of combining partial functions:
 
-1. __Composition__: we write a composition of `f` and `g` as `(f  g)`, meaning that if `f` is defined on valye `x` producing `y`,
+1. __Composition__: we write a composition of `f` and `g` as `(f  g)`, meaning that if `f` is defined on value `x` producing `y`,
    and `g` is defined on value `y` producing `z`, then `(f g)` is defined on value `x` producing `z`.
 2. __Union__: we write `< f1, f2 >` for a union of two functions `f1` and `f2`, meaning that
    if `f1` is defined on value `x` producing `y`, then `< f1, f2 >`
@@ -59,7 +61,7 @@ There are three ways of combining partial functions:
    are both defined on value `x` producing `y1` and `y2`, respectively, then `{f1 lab1, f2 lab2 }`
    is defined on value `x` producing `{y1 lab1, y2 lab2}`. Otherwise, `{f1 lab1, f2 lab2 }` is not defined for `x`.
 
-Actually, all of the above operations take any number of arguments, not only two. For example,
+Actually, composition, union, and product combine any number of partial functions, not only two. For example,
 empty composition, `()`, defines the identity function.
 
 ### Program
@@ -78,16 +80,21 @@ type_label_list ::= /* empty */ | ( type_expression label_name ',' ) + type_expr
 type_expression ::= type_name | type
 
 function_definition ::= function_name '=' expression ';'
-expression ::= projection | typing | composition | union | product | function_name
+expression ::= projection | typing | filter | composition | union | product | function_name
 projection ::= '.' label_name
-typing ::= '$' tape_expression 
+typing ::= '$' type_expression 
+filter ::= '?' filter_expression
+filter_expression ::= $ type_expression | filter_name 
+  | '{' filter_label_list '}' | '<' filter_label_list '>' | '(' filter_label_list ')'
+  | filter_expression '=' filter_name
+filter_label_list ::= /* empty */ | ( filter_expression label_name ',' ) * filter_expression label
 composition ::= '(' expression * ')'  
 union ::= '<' (expression ',') + expression '>' 
 product ::= '{' expression_label_list '}'
 expression_label_list ::= /* empty */ | ( expression label_name ',' ) * expression label
 ```
 
-We assume that the `type_name`, `function_name`, and `label_name` are identifiers (strings) such that:
+We assume that in a `kernel-core` program the `type_name`, `function_name`, and `label_name` are identifiers (strings) such that:
 
 1. All defined type names are distinct.
 2. All defined function names are distinct.
@@ -97,6 +104,7 @@ The `typing` expression acts as a type annotation and can be seen as a 'filter',
 function defined only for the values of the corresponding type.
 
 That's it.
+
 No `builtin` types, no `if` statement, no `loop`, no `throw`, no _closure_, no _annotations_, and no macros.
 Just types and partial functions.
 
@@ -104,22 +112,22 @@ Just types and partial functions.
 
 All type names are replaced by the hash of their canonical representation.
 
-We will also replace the function name by the hash of the normalized unfolded definition of the function.
-Obviously, two different definitions of the same function may have different hashes.
+We replace the function name by the hash of the normalized definition of the function as provided by the program defining the function.
+Obviously, a function may have many different definitions so the hash-based names are not canonical. By using hash-based function names we avoid the problem of name clashes when bringing together many `kernel-core` programs.
 
-This approach solves the problem of modules and imports and opens the door to a universal
+This approach solves the problem of modules, imports, etc., and opens the door to a universal
 registry of types (schemas) and functions.
 
 ## Typing and Polymorphism (codes and filters/patterns)
 
-Abstract Syntax Tree of a program after parsing consists of a dictionary of _type definitions_,
+Abstract Syntax Tree of a program consists of a dictionary of _type definitions_,
 a dictionary  of _function definitions_, and the final _main expression_.
 
 In principle, the initial form of the Abstract Syntax Tree, _raw_ AST, is enough to evaluate
 the program on an input.
 
 Our objective however is to compute canonicals representations for types and normalized
-representations for functions, so they could be effectively reused.
+representations for functions, so they could be reused.
 
 The normalization steps are:
 
@@ -127,7 +135,7 @@ The normalization steps are:
    as well as type expressions used in _function definitions_ and the _main expression_.
 2. Annotate the expressions with types or type patterns; every node of the AST is annotated with a pair of type patterns.
 3. Turn the _singleton_ type patterns into types and add them to the _type graph_. Go to (2) unless no change.
-4. Normalize the function definitions by rewriting them as the fix-point of a single equation.
+
 
 ### Example
 
@@ -169,7 +177,7 @@ and a type `byte` as a product, for simplicity, of four `bit`s.
 Functions `bit0`, `bit1`, and `zero` are "constant polymorphic functions", meaning:
 
 - constant: if defined, they always return exactly the same value;
-- polymorphic: they are defined for more than one pair of input and output type: functions
+- polymorphic: they are defined for more than one pair of input and output types: functions
   `bit0` and `bit1` are of type `?X -> $bit`, and `zero` is of type `?X -> $byte`, where `?X`
   denotes an unconstrained type pattern, also denoted as `?(...)`.
 
@@ -186,15 +194,15 @@ with following constraints:
    > `?( V i, ...)`;
 - `V` is unconstrained, denoted by `(...)`;
 
-We could write it as: `?($byte byte, ((...) i, ...) overflown, ...)`.
+We can write it as: `?($byte byte, ((...) i, ...) overflown, ...)`.
 
 The target type `${ byte byte, bit overflown }` was derived from pattern `?{ $byte byte, $bit overflown }`.
-Such a pettern is called _singleton pattern_ as only one type fits the pattern.
+Such a pattern is called _singleton pattern_, as only one type fits the pattern.
 
 ## Universal Schema Registry
 
 Since the normalization process for types is fast and deterministic, we can build a universal
-schema registry that will store all invented types. 
+schema registry that will store all invented types.
 The registry will be a key-value store, where the key is the hash of the normalized type,
 and the value is the normalized type itself.
 
@@ -203,8 +211,7 @@ definition and stored in the similar key-value store.
 
 Non-polymorphic functions can be easily indexed by the hashes of their input and output types
 so that we can quickly find the function we need.
-Searching for polymorphic functions is more complicated, as we need to find the function
-whose input and output type patterns fit given types.
+Indexing polymorphic functions seems complicated (TODO).
 
 ## Serialization and Compilation
 

package/TODO.md

@@ -0,0 +1,18 @@
+# To do list for canonical names
+
+- [ ] Write a script to search for conflicting names
+- [ ] Export a relation. i.e., rewrite it as a list of all
+      dependency definitions using canonical names
+- [ ] Find a way of adding 'comments' or some way of 'tagging' the definitions
+- [X] The definitions should be simplied by removing useless filters
+- [X] Definitions, which are just aliases to non built-in rels should not introduce a new canonical name
+- [ ] Can we inline some references w/o changing the canonical name?
+
+## Simplification rules
+
+It seems to me that after calculating input and output patterns, all
+filters and code expressions could be replaced by just input and output
+filters corresponding to the patterns. [DONE]
+
+The filer variables have to be renamed for normalization if they take part of 
+the canonical name calculation. [DONE]

package/TYPING.md

@@ -360,6 +360,6 @@ Repeat the procedure until the type pattern graph is not changing anymore.
 - [X] Redefine built-in CONS and SNOC to allign with no support for non-homogeneous lists
 - [X] Integrate filters into typePatters (execution is not affected)
 - [/] better type error repporting - the info exists in the pattern parent tree ... in progress
-- [ ] Introduce a space/time bounds for type derivation
+- [X] Introduce a space/time bounds for type derivation
 - [ ] Check for loops, e.g., `$x = < x f, ...>`, `$x = [x]`, `$x = {x f, ...}` -- this
-      looks a little arbitrary, but maybe this is actually an important part
+      looks a little arbitrary, but maybe this is actually an important part

package/export.mjs

@@ -0,0 +1,177 @@
+
+import { prettyRel, patterns2filters } from "./pretty.mjs";
+import { hash } from "./hash.mjs";
+
+function simplifyRel(relDef) {
+  // remove filters and codes
+  const prune = (rel) => {
+    const newRel = {...rel};
+    switch (rel.op) {
+      case "product":
+        newRel.product = rel.product.map(({label, exp}) => ({label, exp: prune(exp)}));
+        break;
+      case "union":
+        newRel.union = rel.union.map(exp => prune(exp));
+        break;
+      case "comp":
+        newRel.comp = rel.comp.reduce((list, exp) => {
+          const newExp = prune(exp);
+          if (newExp.op != "identity") 
+            list.push(newExp);
+          return list;
+        },[]);
+        if (newRel.comp.length == 0) 
+          newRel.op = "identity";
+        break;
+      case "vector":
+        newRel.vector = rel.vector.map(exp => prune(exp));
+        break;
+      case "caret":
+        newRel.caret = prune(rel.caret);
+        break;
+      case 'code':
+      case 'filter':
+        newRel.op = "identity";
+      default:
+        break;
+    };
+    return newRel;
+  };
+
+  const rel = relDef.def;
+  const filters = patterns2filters(relDef.typePatternGraph, ...rel.patterns).map( filter =>
+    ({op: "filter", filter: filter}));
+  const newRel = prune(rel);
+  let resultRel = {...newRel};
+  if (newRel.op == "identity")
+    return filters[0]; 
+
+  if (newRel.op == "comp") { 
+    resultRel.comp = [ filters[0], ...newRel.comp, filters[1] ];
+    return resultRel;
+  }
+  return {
+    op: "comp",
+    comp: [filters[0], newRel, filters[1] ]
+  };
+};
+
+
+function theID(alias, rel, scc, name) {
+  const sccNames = new Set(scc);
+  const auxNames = {[name]: "X0"};
+  // rename and remove filters and codes
+  const reNameX = (rel) => {
+    const newRel = {...rel};
+    switch (rel.op) {
+      case "product":
+        newRel.product = rel.product.map(({label, exp}) => ({label, exp: reNameX(exp)}));
+        break;
+      case "union":
+        newRel.union = rel.union.map(exp => reNameX(exp));
+        break;
+      case "comp":
+        newRel.comp = rel.comp.map(exp => reNameX(exp));
+        break;
+      case "vector":
+        newRel.vector = rel.vector.map(exp => reNameX(exp));
+        break;
+      case "caret":
+        newRel.caret = reNameX(rel.caret);
+        break;
+      case "ref": {
+        const n = rel.ref;
+        if (alias[n] != undefined) {
+          newRel.ref = alias[n];
+        } else if (auxNames[n] != undefined) {
+          newRel.ref = auxNames[n];
+        } else if (sccNames.has(n)) {
+          auxNames[n] = `X${Object.keys(auxNames).length}`;
+          newRel.ref = auxNames[n];
+        };
+      }; break;
+      default:
+        break;
+    };
+    return newRel;
+  };
+
+  const newRel = reNameX(rel);
+  const resultRelStr = prettyRel(newRel);
+  // console.log(` ${name} = ${resultRelStr}`);
+  const newName = hash(resultRelStr);
+  return newName;
+};
+
+function assignCanonicalNames(scc, rels, relAlias) {
+  const newAlias = scc.reduce( (newAlias,relName) => {
+    const relDef = rels[relName];
+    if (relDef.def.op == "ref" && rels[relDef.def.ref] != undefined) {
+      // inlining if direct alias to a non built-in relation
+      relDef.simplified = simplifyRel(rels[relDef.def.ref]);
+    } else {
+      relDef.simplified = simplifyRel(relDef);
+    }
+    newAlias[relName] = theID(relAlias, relDef.simplified, scc, relName);
+    return newAlias;
+  }, {});
+  const newNames = [...new Set(Object.values(newAlias))];
+  // console.log(` --- SCC: {${scc.join(",")}} ---`);
+
+  const sccCanonicalName = newNames.sort().join(":");
+  for (let relName in newAlias) {
+    if (newNames.length > 1)
+      relAlias[relName] = hash(newAlias[relName]+":"+sccCanonicalName);
+    else
+      relAlias[relName] = newAlias[relName];
+    // console.log(`  ${relName} -> ${relAlias[relName]}`);
+  };  
+}
+
+function exportRelation(rels, alias, name) {
+  const queue = [name];
+  const inQueue = { [name]: true };
+  const result = {};
+  const reName = (rel) => {
+    const newRel = {...rel};
+    switch (rel.op) {
+      case "product":
+        newRel.product = rel.product.map(({label, exp}) => ({label, exp: reName(exp)}));
+        break;
+      case "union":
+        newRel.union = rel.union.map(exp => reName(exp));
+        break;
+      case "comp":
+        newRel.comp = rel.comp.map(exp => reName(exp));
+        break;
+      case "vector":
+        newRel.vector = rel.vector.map(exp => reName(exp));
+        break;
+      case "caret":
+        newRel.caret = reName(rel.caret);
+        break;
+      case "ref": {
+        const n = rel.ref;
+        if (alias[n] == undefined) // built-in
+          return newRel; 
+        newRel.ref = alias[n];
+        if (! inQueue[n]) {
+          inQueue[n] = true;
+          queue.push(n);
+        }
+      }; break;
+      default:
+        break;
+    };
+    return newRel;
+  };
+  while (queue.length > 0 ) {
+    const name = queue.shift();
+    const canonicalName = alias[name];
+    result[canonicalName] = {...rels[name], def: reName(rels[name].def), name: name};
+  }
+  return result;
+}
+
+export default { assignCanonicalNames, exportRelation };
+export { assignCanonicalNames, exportRelation };

package/index.mjs

@@ -4,19 +4,7 @@ import run from "./run.mjs";
 import t from "./codes.mjs";
 
 function compile(script) {
-  try {
-    run.defs = annotate(script);
-  } catch (e) {
-    console.error(e.message);
-    console.error("WARN: Recompiling without type reconciliation due to the type error above.");
-    const { defs, exp } = parse(script);
-    const { codes, representatives } = t.finalize(defs.codes);
-    run.defs = {
-      rels: {...defs.rels, "__main__": {def: exp}}, 
-      codes, representatives
-    };
-    
-  }
+  run.defs = annotate(script);
   return run.bind(null, run.defs.rels.__main__.def);
 }
 
@@ -35,9 +23,9 @@ function annotate(script) {
   const representatives = t.register(defs.codes);
   const rels = {...defs.rels, "__main__": {def: exp}};
 
-  patterns(representatives, rels);
+  const relAlias = patterns(representatives, rels);
  
-  return {rels, representatives}
+  return {rels, representatives, relAlias};
 }
 annotate.doc = "Annotate all the script expressions with patterns";
 

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@fraczak/k",
-    "version": "4.0.4",
+    "version": "5.0.0",
     "description": "k-language for JSON-like data transformation",
     "main": "index.mjs",
     "scripts": {

package/patterns.mjs

@@ -3,6 +3,7 @@ import hash from "./hash.mjs";
 import { TypePatternGraph } from "./typing.mjs";
 import { Graph, sccs, topoOrder } from "./Graph.mjs";
 import codes from "./codes.mjs";
+import { assignCanonicalNames } from "./export.mjs";
 
 const unitCode = codes.unitCode;
 
@@ -101,6 +102,7 @@ function patterns(representatives, rels) {
   //   representatives:{"{}": "BG", ...}
   //   rels: {"rlz": [{op: comp,...}, ...], ...}
  
+  const relAlias = {};
 
   // 1 INITIALIZATION
   // 1.1 initialize patternNodes and rels
@@ -416,7 +418,7 @@ function patterns(representatives, rels) {
   for (const relName in rels) {
     const rootDef = rels[relName];
     rootDef.typePatternGraph = new TypePatternGraph();
-    rootDef.varRefs = []; // the list of non-built references as pointers to AST nodes
+    rootDef.varRefs = []; // the list of references to non-builtin rels as pointers to AST nodes
     rootDef.patternVars = {}; 
     augment(rootDef.def, rootDef);
     delete rootDef.patternVars;
@@ -453,12 +455,16 @@ function patterns(representatives, rels) {
 
   // 3. Topological sort of the DAG
 
-  const sccInOrder = topoOrder(DAG);
+  const sccInOrder = topoOrder(DAG).reverse();
 
   // 4. For each strongly connected component C in a bottom-up order
+  
+  //    we will also generate the canonical representation of the relations in C and replace 
+  //    the name to the relation by the hash of the canonical representation    
+
+  // console.log("----- SCC in order", sccInOrder);
 
-  for (const scc of sccInOrder.reverse()) {
-    // console.log(`SCC: { ${DAGnodes[scc].scc} }`);
+  for (const scc of sccInOrder) {
 
     const maxNumberOfIterations = 10;
     
@@ -497,9 +503,7 @@ function patterns(representatives, rels) {
             throw e;
           }
         }
-
         rels[relName] = compactRel(relDef,relName);
-        
         return relDefToString(rels[relName]); 
       }));
 
@@ -507,9 +511,19 @@ function patterns(representatives, rels) {
         // console.log("SUCCESS!");
         break;
       }
-    } // while
-      
+    } // end of iteration loop
+
+    DAGnodes[scc].scc.forEach( relName => {
+      const relDef = rels[relName];
+      const canonical = relDefToString(relDef);
+      const h = hash(canonical);
+      relAlias[relName] = h;
+    });
+
+    assignCanonicalNames(DAGnodes[scc].scc, rels, relAlias);
+
   }
+  return relAlias;
 }
 
 export default { patterns };

package/pretty.mjs

@@ -1,26 +1,22 @@
 import { find } from "./codes.mjs";
 import { TypePatternGraph } from "./typing.mjs";
 
-function compareAs(fn) {
-  return function (a, b) {
-    [a, b] = [a, b].map(fn);
-    if (a < b) return -1;
-    if (a > b) return 1;
-    return 0;
-  };
-}
-
 const nameRE = /^[a-zA-Z0-9_][a-zA-Z0-9_?!]*$/;
+function pLabel_(label) {
+  return nameRE.test(label) ? ` ${label}` : `${JSON.stringify(label)}`;
+}
+function pLabel(label) {
+  return nameRE.test(label) ? `${label}` : `${JSON.stringify(label)}`;
+}
 
 function prettyCode_labels (representatives, label_ref_map) {
   return Object.keys(label_ref_map)
-    .sort( compareAs((x) => x) )
+    .sort()
     .map( (label) => {
-      const plabel = nameRE.test(label) ? label : `'${label}'`;
       return `${prettyCode(representatives, {
         code: "ref",
         ref: label_ref_map[label],
-      })} ${plabel}`;
+      })}${pLabel_(label)}`;
     })
     .join(", ");
 };
@@ -46,10 +42,10 @@ function prettyCode (representatives, codeExp) {
   }
 };
 
-function prettyFilter (prettyCode, filter) {
+function prettyFilter ( filter ) {
   const fieldsStr = () => {
     const fields = Object.keys(filter.fields).map( (key) => 
-      `${prettyFilter(prettyCode, filter.fields[key])} ${key}` 
+      `${prettyFilter(filter.fields[key])}${pLabel_(key)}` 
     );
     if (filter.open) fields.push("...");
     return fields.join(", ");
@@ -57,11 +53,11 @@ function prettyFilter (prettyCode, filter) {
   switch (filter.type) {
     
     case "code":
-      return `$${prettyCode({code: "ref",ref: filter.code,})}`;
+      return `$${filter.code}`;
     case "name":
       return filter.name;
     case "vector":
-      return `[${prettyFilter(prettyCode, filter.vector)}]${filter.name ? "=" + filter.name : ""}`;
+      return `[${prettyFilter(filter.vector)}]${filter.name ? "=" + filter.name : ""}`;
     case null:
       if (filter.name && Object.keys(filter.fields).length == 0) 
         return filter.name;
@@ -74,12 +70,13 @@ function prettyFilter (prettyCode, filter) {
   throw new Error(`Unknown filter type ${filter.type} in ${JSON.stringify(filter)}`);
 }
 
-function prettyRel (prettyCode, exp) {
+function prettyRel (exp) {
   "use strict";
   const pretty = (exp) => {
     switch (exp.op) {
       case "filter":
-        return `?${prettyFilter(prettyCode, exp.filter)}`;
+        if (exp.filter.type == 'code') return `$${exp.filter.code}`;
+        return `?${prettyFilter(exp.filter)}`;
       case "pipe":
         return `|`;        
       case "caret":
@@ -101,27 +98,16 @@ function prettyRel (prettyCode, exp) {
       case "dot":
         if ("number" === typeof exp.dot) {
           return `.${exp.dot}`;
-        } else if (nameRE.test(exp.dot)) {
-          return `.${exp.dot}`;
-        } else {
-          return `'${exp.dot}'`;
-        }
-        break;
+        } else 
+          return `.${pLabel(exp.dot)}`;
       case "code":
-        return `$${prettyCode({
-          code: "ref",
-          ref: exp.code,
-        })}`;
+        return `$${exp.code}`;
       case "product":
         return (function (labelled) {
           return `{${labelled.join(", ")}}`;
         })(
           exp.product.map(function ({ label, exp }) {
-            if (nameRE.test(label)) {
-              return `${pretty(exp)} ${label}`;
-            } else {
-              return `${pretty(exp)} '${label}'`;
-            }
+              return `${pretty(exp)}${pLabel_(label)}`;
           })
         );
     }
@@ -148,25 +134,24 @@ function patterns2filters(typePatternGraph, ...patternIds) {
 
   // variables - variables which will have to be added as extra filters
   const variables = inDegree.reduce( (variables, degree, i) => {
-    if (degree > 1) variables[i] = "fresh";
+    if (degree > 1) variables[i] = true;
     return variables;
   }, {});
 
+  const filterVars = {};
   const buildFilter = (path, patternId, i) => {
     let named_filter = {};
     const pattern = newTypePatternGraph.get_pattern(patternId);
     if (variables[patternId] && pattern.pattern != 'type') {
-      const name = `X${patternId}`;
-      if (variables[patternId] == "done") 
-        return { type: 'name', name: name };
-      variables[patternId] = "done";
-      named_filter = { name: name };
+      if (filterVars[patternId]) 
+        return { type: 'name', name: filterVars[patternId] };
+      filterVars[patternId] = `X${Object.keys(filterVars).length}`;
+      named_filter = { name: filterVars[patternId] };
     }
     
-
     const edges = newTypePatternGraph.edges[patternId];
     const fields = () => 
-      Object.keys(edges).reduce( (fields, key) =>{
+      Object.keys(edges).sort().reduce( (fields, key) =>{
         fields[key] = buildFilter([...path, key], Object.values(edges[key])[0], i);
         return fields;
       }, {});