@muze-nl/simplystore 0.3.2 → 0.3.3

package/design/jsontag-selector.md

@@ -1,365 +0,0 @@
-# JSONTag-selector or query
-
-This document is now here for historical reference only. SimplyStore now uses the [npm package array-where-select](https://www.npmjs.com/package/array-where-select) as its default query or selector algorithm. In addition work is ongoing on a triplestore implementation with a datalog query engine, but this needs a lot of work to improve performance.
-
-Historical design document follows:
-
-<hr>
-
-The aim is to define a way to represent a selection of the JSONTag dataspace in a string. For JSON there are JSON Path and JSON Pointer. For HTML there is the CSS selector. JSONTag has properties of both HTML and JSON, so a mix of these is a logical solution.
-
-What must the JSONTag selector be capable of?
-
-- point to a single entity inside a JSONTag dataset
-- select multiple entities with something similar
-- combine two or more selectors
-
-In addition the selector must be usable within a graphql-like query syntax.
-
-## CSS Selectors and JSONTag data
-
-We have some experience with this, using the [JSON-CSS](https://github.com/simplyedit/json-css) library. This works reasonably well, but mostly because browsers have built-in CSS selector support that is highly optimized. Still let's see what JSONTag-selector would look like if it was just a CSS Selector.
-
-The JSONTag data would need to be converted to an XML structure that contains all data. The problem here is that JSONTag already contains tags, that would need to be converted somehow. This may actually be simpler than just JSON data, because we have tags for all types already.
-
-Given:
-```jsontag
-[
-	<object class="Task">{
-		"title": "This is a task",
-		"start": <date>"2023-01-18"
-	}
-]
-```
-
-This becomes:
-```xml
-<array>
-	<object class="Task" key="0">
-		<string key="title" value="This is a task" />
-		<date key="start" value="2023-01-18" />
-	</object>
-</array>
-```
-
-The only issue here is that there may be a conflict between attributes on the JSONTag tags, and attributes used to encode JSONTag values into XML attributes. This is solvable by introducing a namespace:
-
-```xml
-<array>
-	<object class="Task" js-key="0">
-		<string js-key="title" js-value="This is a task" />
-		<date js-key="start" js-value="2023-01-18" />
-	</object>
-</array>
-```
-
-The `-` is illegal in JSONTag, so this will never conflict.
-
-To query all tasks with a date after 2023-01-01, you could write this selector:
-
-```css
-.Task:has(> date[js-key="start"][js-value^="2023-01-"])
-```
-
-The problems with this approach are, among others:
-- you can only query tags and attributes, so all values must be encoded in attributes
-- you can only use text-like comparisons, there is no `attribute value is greater than` comparison, for example. There are no ranges.
-- the syntax is a bit cumbersome, although it could be improved by a simple compilation. The above syntax could be simplyfied to:
-
-```css
-.Task:has(> date[start^="2023-01-"])
-```
-
-The advantages are that there is a wealth of optimized code to apply CSS selectors to XML data, and CSS selectors are relatively well understood and documented.
-
-
-## JSONPath and JSONTag data
-
-JSONPath is a complete query and filter language for JSON data. However it is also fairly complex and cumbersome. The same query as above would be something like:
-
-```jsonpath
-$[?(@.start>'2023-01-01')]
-```
-
-There is no support for tag names or attributes, because JSON doesn't support those. So this needs to be added. One way is to encode these inside the JSON data, similar to what JSON-LD does. The JSON version of the JSONTag data then becomes:
-
-```json
-[
-	{
-		"_class": "Task",
-		"_tag": "object",
-		"title": "This is a task",
-		"start": {
-			"_tag": "date",
-			"_value": "2023-01-18"
-		}
-	}
-]
-```
-
-This is not ideal or consistent. I haven't converted the `title` value to an object, but have done so for the `start` value. There is probably a better encoding, but this shows the gist of it. The JSONPath selector now becomes:
-
-```jsonpath
-$[?(@._class='Task' && @.start._value>'2023-01-01')]
-```
-
-But the ideal solution would be to just extend the JSONPath language with tag and attribute query support and use the JSONTag data directly.
-
-See [JSONPath-plus](https://github.com/JSONPath-Plus/JSONPath) for a promising code base that has the potential to be extended for JSONTag.
-
-## GraphQL
-
-GraphQL not only allows for querying, but also mutations. Here we just consider the query syntax. Given the same data and selection, here's what this would look like in a GraphQL query:
-
-```
-Task(filter:{start:{greaterThen:"2023-01-01"}) {
-	title
-	start
-}
-```
-
-This is not entirely accurate, but will do for this purpose. The advantage of Graphql is that it allows you to specify the properties you are interested in, including properties of linked objects. This solves the problem of having to query a server many times to complete a dataset that contains links.
-
-For example, if we add assigned persons to a task, the dataset might look like this:
-
-```jsontag
-[
-	<object class="Person" id="john">{
-		"name": "John",
-		"dob": <date>"1972-09-20"
-	},
-	<object class="Task">{
-		"title": "This is a task",
-		"start": <date>"2023-01-18",
-		"assigned": [
-			<link>"#john"
-		]
-	}
-]
-```
-
-Then you could write this query:
-
-```
-Task(filter:{start:{greaterThen:"2023-01-01"}) {
-	title
-	start
-	assigned {
-		name
-	}
-}
-```
-
-And the result should be:
-
-```
-[
-	<object class="Task">{
-		"title": "This is a task",
-		"start": <date>"2023-01-18",
-		"assigned": [
-			<object class="Person" id="john">{
-				"name": "John"
-			}
-		]
-	}	
-]
-```
-
-The interesting part here is, whether the graphql query/filter syntax is the best fit, or perhaps can we replace that with the CSS selector or JSON Path? GraphQL is not quite designed for the purpose here, but I like the way you can select which data to return. The filter options are limited and non-standardized. Each graphql server has its own filter syntax and options. I also don't like the way the filter syntax is articifially limited to something that looks like JSON. I do like the focus on words instead of symbols to denote filter options and functions.
-
-GraphQL also allows you to specify aliases for resulting data, so instead of assigned you could write:
-
-```
-Task(filter:{start:{greaterThen:"2023-01-01"}) {
-	title
-	start
-	owner:assigned {
-		name
-	}
-}
-```
-
-```
-[
-	<object class="Task">{
-		"title": "This is a task",
-		"start": <date>"2023-01-18",
-		"owner": [
-			<object class="Person">{
-				"name": "John"
-			}
-		]
-	}	
-]
-```
-
-There is more you can do with aliases, but I'm not sure that this is an approach to copy. The main use for aliases is to prevent conflicts in result names.
-
-The main problem I see with the query syntax is that if I just want all the assigned persons, I'd query this way:
-
-```
-Task(filter:{start:{greaterThen:"2023-01-01"}) {
-	title
-	start
-	assigned
-}
-```
-
-But as it stands this would result in:
-
-```
-[
-	<object class="Task">{
-		"title": "This is a task",
-		"start": <date>"2023-01-18",
-		"assigned": [
-			<link>"/0/"
-		]
-	}	
-]
-```
-
-This requires extra queries again, which is precisely what we want to avoid.
-For this reason, the JSONTag Rest server always resolves arrays and literal values (strings, numbers and booleans). So the result should be:
-
-```
-[
-	<object class="Task">{
-		"title": "This is a task",
-		"start": <date>"2023-01-18",
-		"assigned": [
-			<object class="Person">{
-				"name": "John",
-				"dob": <date>"1972-09-20"
-			}
-		]
-	}	
-]
-```
-
-This does break the GrapQL paradigm where you will only ever get the exact properties that you request.
-
-## -Current approach- past approach
-
-For now I'm using [jsonpath-plus](https://github.com/JSONPath-Plus/JSONPath) which extends the default [JSONPath]() with among others a parent selector. This codebase looks relatively simple to add tagname and attribute selection support. This implementation allows for javascript methods to be called in filters, applied only to available data. The expressions are evaluated in a seperate 'vm', but security needs to be checked carefully.
-
-You can query the dataset by POSTing to a path, with the JSON Path query in the body. e.g.:
-
-```
-POST /persons/
-
-$[?(@.name==='Jane')]
-```
-
-This will result in:
-
-```jsontag
-[
-    <object class="Person">{
-        "name":"Jane",
-        "dob":<date>"1986-01-01"
-    }
-]
-```
-
-The next step is to add a bit of GraphQL syntax to describe which properties you want returned. e.g.
-
-```
-$.[?(@.name==='Jane')] {
-	name
-}
-```
-
-Add alias support, so you can rename properties:
-
-```
-$.[?(@.name==='Jane')] {
-	foo:name
-}
-```
-
-And finally add subqueries, where you can assign JSON Path search results to a property:
-
-```
-$.[?(@.name==='Jane')] {
-	name
-	tasks:$$..tasks[?(@.assigned==='Jane')]
-}
-```
-
-A problem here is whether the subquery should start at the global root, or the current object. The global root allows for much more features, but also adds complexity. It would be nice to be able to specify in the query, e.g. make the global root something like $$.
-
-JSON Path has no knowledge of JSONTag, so it has no support for tag names (types) or attributes. JSON Path uses these special characters already: `$.@?&=!<>*`. Lets use `#` for attributes, like `@` for properties. You could then do this:
-
-```
-$.[?(#.class==='Person')] {
-	name
-}
-```
-
-And we can add a tag name, like this:
-
-```
-$.[?(#.class==='Person' && #.tag-name==='object')] {
-	name
-}
-```
-
-By using 'tag-name' we are sure it will never conflict with an attribute name, since they can't contain `-`.
-
-## New Current Approach
-
-After reading [Datalog in Javascript](https://www.instantdb.com/essays/datalogjs) I've switched the query engine to Datalog. In fact the query engine is a javascript interpreter running in [VM2](https://github.com/patriksimek/vm2), with access to the query function of the triple store compiled from the jsontag input data.
-
-So you can now POST to the /query endpoint with just a simple query:
-
-```javascript
-query({
-	find: ["?name"],
-	where: [
-		["?person", "dob", "1972-09-20"],
-		["?person", "name", "?name"]
-	]
-})
-```
-
-And the result will be:
-```
-[
-	["John"]
-]
-```
-
-Or you can add your own function to match values:
-
-```javascript
-function lessThan(pattern) {
-	return function(match) {
-		if (match<pattern) {
-			return match
-		}
-		return null
-	}
-}
-
-query({
-	find: ["?name"],
-	where: [
-		["?person", "dob", lessThan("1980")],
-		["?person", "name", "?name"]
-	]
-})
-```
-
-You can also return complete objects:
-```javascript
-query({
-	find: ["?person"],
-	where: [
-		["?person", "dob", lessThan("1980")]
-	]
-})
-```
-
-There is as yet no way to transform the result, other than adding your own bespoke javascript to do that.

package/design/multitasking.md

@@ -1,13 +0,0 @@
-# multitasking
-
-Node itself is (almost) a single process. The server must be able to handle multiple requests simultaneously, maximizing cpu core usage.
-
-The query endpoint is the only part that must be multitasking. Instead of using an external system like PM2, the server should use worker threads. Threads use the same memory, but since the query endpoints are all using the immutable dataset only, that is no worry here.
-
-The update endpoint needs a single seperate worker. This allows for the main process to be limited to managing the workers only. This way there is less chance of a problem crashing the whole server.
-
-Each worker starts a single VM, which runs all the code. The isolated-vm package allows for more control than VM2, but also has no easy way to share the immutable dataset. So for now we stick to VM2.
-Update: isolated-vm may be able to do this using the [ivm.Reference](https://github.com/laverdet/isolated-vm#class-reference-transferable) function.
-
-https://www.digitalocean.com/community/tutorials/how-to-use-multithreading-in-node-js
-https://www.npmjs.com/package/piscina

package/design/thoughts.md

@@ -1,32 +0,0 @@
-JSONTag
-
-- streaming parser (handrolled for performance)
-- option: documentURL
-	automatically enhance id's as urls with baseURL the documentURL
-	do this for <link> as well as id attributes
-- option: parent index
-	when parsing, keep track of the parent for each value, and add it to a WeakMap index.parents, this contains all objects with a reference to the object used as key.
-- object id and link value should always be URI's, and should be identical. If the link value starts with a '#', then the id should also start with a '#'. 
-- fast-parse doesn't enforce parsing rules for specific tags yet
-
-JSONTag REST Server
-
-- think of a better name (iets met live?) simplyStore?
-- add a set of default functions / standard library to walk/map/reduce over the data. Like the arc/tree methods, e.g. dive. Note: JSONTag data is a graph, so it can contain circular references, make sure you don't walk into an infinite loop. Keep track of which objects have already been walked over.
-- create extra indexes to speed up 'common' searches, e.g. give me all objects with class X that are contained/child/descendant of Y. (like the JSONPath `$..book` expression)
-
-JSONTag REST Server Commands
-- create endpoint POST /update/ for example
-- syntax of commands should be simple and easily parsible
-- create a log of all commands, with a unique id per command
-- commands are javascript methods defined serverside in a commands.js file / commands object
-- each command is a transaction, it succeeds wholly or fails wholly
-- commands may have preconditions, if not met, they fail
-- when sending a command, you get back the generated command id immediately
-- you can (long) poll for the status of a command (pending, success, failed)
-	GET /update/{uuid}
-- there should be an SSE event bus endpoint that allows you to listen wihtout polling
-
-
-
-

package/docs/docker.md

@@ -1,129 +0,0 @@
-# Docker Image
-
-A minimal Docker image is available for the `jsontag-rest-server`.
-
-<!-- toc -->
-
-- [Installation](#installation)
-  - [Building the image](#building-the-image)
-  - [Pulling the image](#pulling-the-image)
-- [Usage](#usage)
-  - [Docker User](#docker-user)
-  - [Environment Variables](#environment-variables)
-  - [Extending the image](#extending-the-image)
-
-<!-- tocstop -->
-
-The docker image is created from the [`Dockerfile`](../Dockerfile) in the root of the repository.
-
-The image is based on [alpine][1] with [NodeJS installed][2], rather than using the official NodeJS image. This is done to keep the image size down (from more than 200MB to less than 100MB).
-
-If more functionality is needed, the image can be extended by creating a new `Dockerfile` that uses the `jsontag-rest-server` image as a base.
-
-## Installation
-
-The image can be build from the Dockerfile provide by this project or pulled from the GitHub Container Registry.
-
-### Building the image
-
-The image can be built using the following command:
-
-```bash
-docker build --tag jsontag-rest-server .
-```
-
-### Pulling the image
-
-The image can be pulled from the GitHub Container Registry using the following command:
-
-```bash
-docker pull ghcr.io/poef/jsontag-rest-server
-```
-
-## Usage
-
-To use the `jsontag-rest-server` container, two things are needed:
-
-1. **The port the server is listening on needs to be exposed.**<br>
-   This can be done by using the `--expose` flag when running the container. By default, this is port 3000. The port can be changed by setting the `NODE_PORT` environment variable (see the "Environment Variables" section). 
-
-2. **A data file needs to be mounted into the container.**<br>
-   The image comes with [a default dataset][3], loaded from `/app/data.jsontag` file that can be used to test the server. To use a different dataset, mount it into the container using the `--volume` flag when running the container.
-
-To change the default for either the port or the data file, see the "Environment Variables" section.
-
-An example of running the server (with the default data file and port):
-
-```bash
-docker run \
-    --expose 3000 \
-    --interactive \
-    --name=jsontag-rest-server \
-    --rm \
-    --tty \
-    --volume "$PWD/my-data.json:/app/data.jsontag"
-    jsontag-rest-server
-```
-
-### Docker User
-
-Similar to the official NodeJS image, the image is set up to run as a non-root user. The user and group are both `node`. The user is created with a UID of 1000 and a GID of 1000.
-
-### Environment Variables
-
-The runtime environment variables that can be set are:
-
-- `DATAFILE`:  The file that is loaded into the REST server.<br>
-  Defaults to `data.jsontag`.
-- `NODE_ENV`:  The environment the server is running in.<br>
-  Defaults to `production`.
-- `NODE_PORT`: The port the server will listen on.<br>
-  Defaults to `3000`.
-
-These variables can be set when _building_ the Docker image:
-
-```bash
-docker build --tag jsontag-rest-server\
-    --build-arg DATAFILE=my-data.jsontag \
-    --build-arg NODE_ENV=development \
-    --build-arg NODE_PORT=8080 \
-    .
-```
-
-or when _running_ the Docker container
-    
-```bash
-docker run \
-    --env DATAFILE=my-data.jsontag \
-    --env NODE_ENV=development \
-    --env NODE_PORT=8080 \
-    --expose 8080 \
-    --volume "$PWD/my-data.json:/app/my-data.jsontag"    
-    jsontag-rest-server
-```
-### Extending the image
-
-The image can be extended by creating a new `Dockerfile` that uses the `jsontag-rest-server` image as a base.
-
-For example, to add a custom data file to the image:
-
-```dockerfile
-FROM node:lts-buster-slim
-
-# ... (other instructions)
-
-COPY --from=ghcr.io/poef/jsontag-rest-server /app/ /usr/src/app/jsontag-rest-server/
-COPY my-data.jsontag /usr/src/app/jsontag-rest-server/data.jsontag
-
-# ... (other instructions)
-
-# The jsontag-rest-server is assumed to be started from entrypoint.sh
-CMD ["entrypoint.sh"]
-``` 
-
-
-
-
-[1]: https://hub.docker.com/_/alpine
-[2]: https://pkgs.alpinelinux.org/package/edge/main/x86/nodejs
-[3]: ../data.jsontag

package/www/assets/css/page.css

@@ -1,34 +0,0 @@
-	:root {
-		--ds-primary: var(--ds-simplystore);
-		--ds-primary-contrast: var(--ds-simplystore-contrast);
-		--ds-primary-gradient: var(--ds-simplystore-gradient);
-		--ds-primary-gradient-bump: var(--ds-simplystore-gradient-bump);
-	}
-	html, body {
-		height: 100%;
-		margin: 0;
-		background: var(--ds-black);
-		color: var(--ds-white);
-		overflow: hidden;
-	}
-	body {
-		display: flow-root;
-		overflow: auto;
-	}
-	a:hover, a:active {
-		text-decoration: underline;
-		color: var(--ds-simplystore-light);
-	}
-	a:link {
-		text-decoration: none;
-		color: var(--ds-simplystore);
-	}
-	a:visited {
-		text-decoration: none;
-		color: var(--ds-simplystore-dark);
-	}
-	pre {
-		background-color: var(--ds-grey-80);
-		border-radius: 8px;
-		padding: 8px;
-	}

package/www/help/index.html

@@ -1,94 +0,0 @@
-<!doctype html>
-<link rel="stylesheet" href="/assets/css/style.css">
-<link rel="stylesheet" href="/assets/css/page.css">
-<section class="ds-space">
-	<h1>Help</h1>
-	<nav>
-		<ul> 
-			<li><a href="#intro">Introduction</a></li>
-			<li><a href="#queries">Queries</a></li>
-			<li><a href="#jsontag">JSONTag</a></li>
-			<li><a href="#fixtures">Fixtures</a></li>
-			<li><a href="#about">About</a></li>
-		</ul>
-	</nav>
-	<h2 id="introduction">Introduction</h2>
-	<p>
-		SimplyStore is a datastore that allows you to query data with javascript. The data is stored and retrieved as JSONTag, an enhanced version of JSON that has support for metadata (type, attributes) and links.
-	</p><p>
-		SimplyStore has a single dataspace, named <code>data</code>. You can enter a query on the left and press &lt;Control+Enter&gt; to execute it. If you haven't written a query, the right data pane shows an overview of the available data in the dataspace.
-	</p>
-	<h2 id="queries">Queries</h2>
-	<p>SimplyStore queries are just javascript, so this works:</p>
-	<pre class="javascript">data.filter(p => p.name=='John')</pre>
-	<p>SimplyStore uses the <a href="//npmjs.com/package/array-where-select" target="_blank">array-where-select library</a> to allow for shorter, more expressive queries, e.g:</p>
-	<pre class="javascript">from(data)
-.where({
-	name: 'John'
-})
-.select({
-	name: _,
-	dob: _
-})</pre>
-	<p>The <code>_</code> is a placeholder that selects the property value corresponding with the property name on the left.</p>
-	<p>This query results in output like this:</p>
-	<pre class="JSONTag">[
-    {
-        "name":"John",
-        "dob":&lt;date&gt;"1972-09-20"
-    },
-    {
-        "name":"Jane",
-        "dob":&lt;date&gt;"1986-01-01"
-    }
-]</pre>
-	<p>See the array-where-select library to learn more about this.</p>
-	<h2 id="jsontag">JSONTag</h2>
-	<p>JSONtag is backwards compatible with JSON. It adds two important things: links and tags.</p>
-	<p>JSON cannot represent data structures with internal references. JSONTag adds a <code>&lt;link&gt;</code> tag to represent these, like this:</p>
-	<pre class="JSONTag">{
-		"name": "John",
-		"friends": [
-			&lt;link&gt;"/persons/1/"
-		]
-	}</pre>
-	<p>Here you also see the first tag pop up. Tags in JSONTag are similar to HTML opening tags. There are no closing tags. Each tag starts with a tagName, which denotes the basic type of the JSON value to the right.</p>
-	<p>Available types (tag names) are:</p>
-	<h3>JSON</h3>
-	<ul>
-		<li>boolean</li>
-		<li>number</li>
-		<li>string</li>
-		<li>array</li>
-		<li>object</li>
-	</ul>
-	<h3>Scalars</h3>
-	<ul>
-	<li>text</li>
-	<li>blob</li>
-    <li>int (uint, int8, uint8, int16, uint16, int32, uint32, int64, uint64)</li>
-    <li>float (float32, float64)</li>
-	</ul>
-	<h3>Semantic types</h3>
-	<ul>
-	<li>color</li>
-	<li>date</li>
-	<li>decimal</li>
-	<li>email</li>
-	<li>hash</li>
-	<li>interval</li>
-	<li>link</li>
-	<li>money</li>
-	<li>phone</li>
-	<li>range</li>
-	<li>time</li>
-	<li>timestamp</li>
-	<li>url</li>
-	<li>uuid</li>
-	</ul>
-	<p>Read <a href="//npmjs.com/package/@muze-nl/jsontag" target="_blank">more about JSONTag here</a>.
-	<h3 id="fixtures">Fixtures</h3>
-	<p>The SimplyStore UI allows you to save and restore queries. These are saved in your browser on your local machine. Because queries are just javascript, you can add your own helper functions. To prevent having to add these to all your seperate queries, the UI has a special 'Fixtures' query. The contents of this query are always prepended to whatever query you send.</p>
-	<h3 id="about">About</h3>
-	<p>SimplyStore is made by <a href="//muze.nl/" target="_blank">muze.nl</a> and is part of the <a href="//simplyedit.io/" target="_blank">SimplyEdit</a> product range. SimplyStore's source code is open source (MIT) and can be found on <a href="//github.com/SimplyEdit/SimplyStore" target="_blank">//github.com/SimplyEdit/SimplyStore</a></p>
-</section>