querysub 0.432.0 → 0.436.0

package/.eslintrc.js

@@ -1,51 +1,51 @@
-module.exports = {
-    "root": true,
-    "env": {
-        "browser": true,
-        "es6": true,
-        "node": true
-    },
-    "extends": [],
-    "globals": {
-        "Atomics": "readonly",
-        "SharedArrayBuffer": "readonly"
-    },
-    "overrides": [
-        {
-            "files": [
-                "**/*.ts",
-                "**/*.tsx"
-            ]
-        }
-    ],
-    "parser": "@typescript-eslint/parser",
-    "parserOptions": {
-        "ecmaVersion": 2018,
-        "sourceType": "module",
-        "project": "./tsconfig.json",
-        "tsconfigRootDir": __dirname,
-    },
-    "plugins": [
-        "@typescript-eslint"
-    ],
-    "rules": {
-        "@typescript-eslint/unbound-method": [
-            "error",
-            {
-                "ignoreStatic": true
-            }
-        ],
-        "@typescript-eslint/no-floating-promises": [
-            "error"
-        ],
-        "quotes": [
-            "error",
-            "double",
-            {
-                "allowTemplateLiterals": true
-            }
-        ],
-        "eqeqeq": "error",
-        "semi": "error"
-    }
+module.exports = {
+    "root": true,
+    "env": {
+        "browser": true,
+        "es6": true,
+        "node": true
+    },
+    "extends": [],
+    "globals": {
+        "Atomics": "readonly",
+        "SharedArrayBuffer": "readonly"
+    },
+    "overrides": [
+        {
+            "files": [
+                "**/*.ts",
+                "**/*.tsx"
+            ]
+        }
+    ],
+    "parser": "@typescript-eslint/parser",
+    "parserOptions": {
+        "ecmaVersion": 2018,
+        "sourceType": "module",
+        "project": "./tsconfig.json",
+        "tsconfigRootDir": __dirname,
+    },
+    "plugins": [
+        "@typescript-eslint"
+    ],
+    "rules": {
+        "@typescript-eslint/unbound-method": [
+            "error",
+            {
+                "ignoreStatic": true
+            }
+        ],
+        "@typescript-eslint/no-floating-promises": [
+            "error"
+        ],
+        "quotes": [
+            "error",
+            "double",
+            {
+                "allowTemplateLiterals": true
+            }
+        ],
+        "eqeqeq": "error",
+        "semi": "error"
+    }
 };

package/costsBenefits.txt

@@ -1,115 +1,115 @@
-Concepts
-    Naming
-        PathValue is the database
-        FunctionRunner is a tool which makes remote function calling easy
-        Querysub is the interface which untrusted parts of apps use to sync data and commit calls
-            - Quadratic sieve: `The quadratic sieve algorithm (QS) is an integer factorization algorithm and, in practice, the second fastest method known (after the general number field sieve). It is still the fastest for integers under 100 decimal digits or so, and is considerably simpler than the number field sieve`
-            - Quorum sensing: `In biology, quorum sensing or quorum signalling (QS)[1] is the ability to detect and respond to cell population density by gene regulation.`
-            - Query subscriptions (https://www.apollographql.com/docs/react/data/subscriptions/): In graphQL you have to choose between them, with Querysub you get both. `Like queries, subscriptions enable you to fetch data. Unlike queries, subscriptions are long-lasting operations that can change their result over time. They can maintain an active connection to your GraphQL server (most commonly via WebSocket), enabling the server to push updates to the subscription's result.` and the BS justification for why graphQL doesn't support automatic subscriptions: `In the majority of cases, your client should not use subscriptions to stay up to date with your backend. Instead, you should poll intermittently with queries, or re-execute queries on demand when a user performs a relevant action (such as clicking a button).`, which is true, but automatic subscriptions should still be easy to enable with 100% accuracy, instead of being hacked kludge to fill the massive missing piece of graphQL.
-
-    NOTE: The whole idea is "fix later" locks, where we invalidate writes later on, if we realized our lock wasn't ACID.
-
-    NOTE: We don't need to hang onto lock chains forever, because we require servers to keep their data fresh by constantly receiving the latest values. This means if a piece of data is old enough, we know it can be relied upon, because no one can go back in time far enough to remove it.
-        - We still need locks, because have changed relatively recently, but... we don't need to follow it's locks back, or even remember that it is supposed to be valid!
-
-    NOTE: Functions don't exist explicitly, but due to identical readLocks you can tell what happened from the same function. This also ensures writes are atomic, as if they have the same readLocks, either they are all accepted, or all rejected.
-
-    NOTE: We don't actually lock things such as Object.keys(). In practice we will get pretty close, but our handling of it is simply not fully ACID.
-
-    NOTE: In regards to FunctionRunners. FunctionRunner rerun any calls which have no result (included if their result was written, then invalidate). The first one to run will get the value, with the others having their ReadLock on the result (from epoch to their LATER writeTime) invalidated.
-        - Although, in practice, the writeTimes will be close, as they are based on requested writeTimes, made unique by creatorId. So... if two FunctionRunners run the same call, one will consistently create the accepted value. So... just don't have two FunctionRunners run the same call!
-
-    NOTE: "Bad" predictions are possibly by if you know the path some change will write to (ie, the call result path). You can create a prediction that depends on that path being undefined (in some range that includes the eventual remote change, but not the prediction), simply watch that path, and when the change eventually happens your prediction will automatically be invalidated.
-        - This is not suitable for trusted nodes, as it means all of our dependent values on that prediction will be only acceptable to us (as our prediction won't really be able to be propagated, as the change might never happen, in which case our prediction has to go away, which is impossible to do if we propagate it), but... it is very useful for clients to quickly predict writes (trusted nodes can just write actual values, not trying to get some remote to make a change).
-        - AND, if you only depend on that path there is no jitter as you reject old values, as there is no cascading! This DOES allow values to drift out of sync with the server (if they are CONSTANTLY written to), but... that is another problem, for games, and just one of many issues games have to overcome.
-
-    NOTE: The reason that we have our main database servers resolving all the values before they write to disk, is simple. They need to resolve the values, because somebody needs to and if the watchers resolve it it's extremely inefficient. We could have them resolve it but then pass the full stream to another server which then resolves it before writing to disk. But then any cases where this fixes an issue of lost data would mean the disk is now inconsistent with what our watchers have. So therefore, splitting off another server to record the full stream of data and having that be all it does and then resolving the data afterwards isn't any faster, and doesn't make our servers appear more stable.
-        - However, splitting off the data stream might be a good idea for the purposes of auditing, instead of speed, or reliability.
-
-
-Summary of benefits/costs
-
-    Benefits
-        1) Decreased code size / decreased development time / decreased complexity
-            - Adding a new value doesn't require new endpoints
-            - Even for a non-syncing application the code size is about as small as it can get
-        2) Minimum possible clientside delay (so everything feels fast)
-            - If you just explicitly input predict values, and then clobber the predicted value when the call comes back, you run into a case where multiple predictions get clobber by an old callback. Which is bad.
-            - If you explicitly input predict values, and only clobber the latest, that works... until you have reasonably complex data.
-            - Explicit endpoints per data path is a non-starter, because when your user interfaces migrates it won't overlap, and then you will have to explicitly calling multiple endpoints after a mutate call, and it will become your entire job.
-            - The only way to do input prediction at a framework level is to track all accesses, and have a serverside which exposes path bases values, AND to have the same mutate functions clientside as you use serverside (otherwise you have to rewrite all mutate functions twice).
-                - AND THEN... if other users mutate data, you NEED the server to tell you when it changes. If you don't, 
-                - In theory you can download data every time you need it... but this can't scale to reasonably large applications. AND, if you don't the user won't just see stale data, they will predict stale data, which will be really bad. So... the server needs to synchronize the paths as well
-        3) Excellent sharding / scaling via very lax sharding
-            - PathValue.locks are a simple locking mechanism, and very efficient (less efficient with high contention)
-            - Transactions can easily be done cross shard, and are still fully ACID, which most other databases have a hard time doing (unless they use virtually the same locking scheme)
-
-    Database flowchart
-        (NOTE: Large static assets should always be stored in some object storage database, like S3, even if you are using another system for other data.)
-        (NOTE: If you want an ultra responsive UI you can often use a write-then-reader wrapper for your database (which might require extra work to specify what should update, and what should be read). Or... just use query-sub)
-
-        - Do you want a very responsive user interface (via input prediction)
-            = Yes
-                - Is it easy to specify exactly what the writes are (with high accuracy) for most server write operations
-                    (As in, are you using complex writes that involve functions with many lines of code, or is pretty much just KVP writes.)
-                    = Yes
-                        - Will the user often experience write contention
-                            = YES
-                                - query-sub
-                            = NO
-                                - query-sub
-                                - OR Run again without a very responsive user interface, and then slap a write-then-reread wrapper onto your database, and then tell it what values it should predict.
-                    = No
-                        - query-sub
-            = NO
-                - Is % chance contention * impact of contention low (contention free)?
-                = Yes
-                    Is the data static enough where stale data can be shown? (ex, search results, non-user part of youtube, twitter?)
-                    = Yes
-                        - Any NoSQL implementation (S3 for large data, MongoDB for complex data, otherwise BigTable, or even ElasticSearch?)
-                    = No
-                        - Do you want clientside data syncing
-                        = Yes
-                            - firebase
-                        = No
-                            - A fast writing NoSQL implementation (probably MongoDB)
-                = No
-                    - Do you want clientside data syncing
-                    = Yes
-                        - firebase + firebase cloud functions
-                    = No
-                        - Are writes simple and only affect a single collection
-                        = Yes
-                            - MongoDB
-                        = No
-                            - MongoDB / SQL (likely with high transaction isolation levels)
-
-        Conclusion
-            - Realistically, you want a responsive user interface (who wouldn't?)
-
-    NOTE: For applications with very static data (so not just mostly reads, but the data actually changes so infrequently that users can be shown stale data and they won't even notice), using MongoDB is just as good (unless you need really complex transactions).
-        - You will still run into issues at huge scales with other databases though, as eventually the amount of transactions you want will exceed what one server can handle, and then you need to shard, and if you don't or can't shard well, everything will get slow.
-
-    Timestamped writes / clientside reproducible writes
-        BENEFIT: Allows clients to compare against incoming writes and know if they write is newer, or older, allowing predictions to never get out of sync.
-            ALTERNATIVE: Reading state after each write also allows this. HOWEVER, if the write is a function mutation, the impetus for the write might be small, but the data mutated could be large. For example, a paint application, where a write mutates the canvas, which is large, so we don't want to download the entire canvas after each write.
-                - AND, even if the values are small, this requires increased network usage
-        COST: Is slower. Is only useful with with KVP writes (if you can't map from clientside write to read paths this can't be used to predict values).
-        ADDITIONAL BENEFITS: Makes debugging easier, via keeping track of a history of operations
-    
-    Watching/syncing
-        BENEFIT: Greatly reduces read count needed to keep data up to date
-        COST: Writes have more overhead, requiring everything to be notified
-
-    Lock tracking / valid tracking / contention invalidation / eventually consistent transactions
-        BENEFIT: ACID safety that is fast even with large collections.
-            - Also allows ACID safety even for servers that are off the main network, allowing anyone to add transactions without worrying about inserting an inconsistent write (worst case the write will be rejected).
-            - Also, ACID safety is a lot easier to implement with this technique
-            ALTERNATIVE: Use a single server, or a single server per collection and don't provide ACID cross collection.
-            ALTERNATIVE: Collection locking, which if you have enough servers using the same collection degenerates to single server performance.
-            ALTERNATIVE: Smart evenly sub-collection sharding. This can be very efficient, HOWEVER, it is extremely difficult to anticipate the access patterns. Accurately predicting all access patterns would be akin to a universal storage system that can run any search query in optimal time (either O(1), or O(logN)).
-            ALTERNATIVE: No ACID safety. This is pretty much only viable if the write interface is purely KVP based. Otherwise it is easy to create functions which depend on ACID safety, and give unpredictable results when it is not kept (ex, contention causes really bad breaking of things). This makes it impossible to implement things such as financial transactions, most games, etc.
-                - And, of course, we can have lock tracking support, but just commit writes with no readLocks, which will produce much faster writes, making the database based degenerate to a NoSQL database, with theoretically the same speed.
-        COST: Slower to compute writes, and writes become MUCH larger.
-            - Lock/eventually consistency transactions ONLY work if the client readers are syncing. If they aren't, they will get the wrong data and KEEP the wrong data, which would break pretty much any application.
-        ADDITIONAL BENEFITS: Makes debugging easier, via making it possible to trace the read source used to generate any write (so to track data through the system).
+Concepts
+    Naming
+        PathValue is the database
+        FunctionRunner is a tool which makes remote function calling easy
+        Querysub is the interface which untrusted parts of apps use to sync data and commit calls
+            - Quadratic sieve: `The quadratic sieve algorithm (QS) is an integer factorization algorithm and, in practice, the second fastest method known (after the general number field sieve). It is still the fastest for integers under 100 decimal digits or so, and is considerably simpler than the number field sieve`
+            - Quorum sensing: `In biology, quorum sensing or quorum signalling (QS)[1] is the ability to detect and respond to cell population density by gene regulation.`
+            - Query subscriptions (https://www.apollographql.com/docs/react/data/subscriptions/): In graphQL you have to choose between them, with Querysub you get both. `Like queries, subscriptions enable you to fetch data. Unlike queries, subscriptions are long-lasting operations that can change their result over time. They can maintain an active connection to your GraphQL server (most commonly via WebSocket), enabling the server to push updates to the subscription's result.` and the BS justification for why graphQL doesn't support automatic subscriptions: `In the majority of cases, your client should not use subscriptions to stay up to date with your backend. Instead, you should poll intermittently with queries, or re-execute queries on demand when a user performs a relevant action (such as clicking a button).`, which is true, but automatic subscriptions should still be easy to enable with 100% accuracy, instead of being hacked kludge to fill the massive missing piece of graphQL.
+
+    NOTE: The whole idea is "fix later" locks, where we invalidate writes later on, if we realized our lock wasn't ACID.
+
+    NOTE: We don't need to hang onto lock chains forever, because we require servers to keep their data fresh by constantly receiving the latest values. This means if a piece of data is old enough, we know it can be relied upon, because no one can go back in time far enough to remove it.
+        - We still need locks, because have changed relatively recently, but... we don't need to follow it's locks back, or even remember that it is supposed to be valid!
+
+    NOTE: Functions don't exist explicitly, but due to identical readLocks you can tell what happened from the same function. This also ensures writes are atomic, as if they have the same readLocks, either they are all accepted, or all rejected.
+
+    NOTE: We don't actually lock things such as Object.keys(). In practice we will get pretty close, but our handling of it is simply not fully ACID.
+
+    NOTE: In regards to FunctionRunners. FunctionRunner rerun any calls which have no result (included if their result was written, then invalidate). The first one to run will get the value, with the others having their ReadLock on the result (from epoch to their LATER writeTime) invalidated.
+        - Although, in practice, the writeTimes will be close, as they are based on requested writeTimes, made unique by creatorId. So... if two FunctionRunners run the same call, one will consistently create the accepted value. So... just don't have two FunctionRunners run the same call!
+
+    NOTE: "Bad" predictions are possibly by if you know the path some change will write to (ie, the call result path). You can create a prediction that depends on that path being undefined (in some range that includes the eventual remote change, but not the prediction), simply watch that path, and when the change eventually happens your prediction will automatically be invalidated.
+        - This is not suitable for trusted nodes, as it means all of our dependent values on that prediction will be only acceptable to us (as our prediction won't really be able to be propagated, as the change might never happen, in which case our prediction has to go away, which is impossible to do if we propagate it), but... it is very useful for clients to quickly predict writes (trusted nodes can just write actual values, not trying to get some remote to make a change).
+        - AND, if you only depend on that path there is no jitter as you reject old values, as there is no cascading! This DOES allow values to drift out of sync with the server (if they are CONSTANTLY written to), but... that is another problem, for games, and just one of many issues games have to overcome.
+
+    NOTE: The reason that we have our main database servers resolving all the values before they write to disk, is simple. They need to resolve the values, because somebody needs to and if the watchers resolve it it's extremely inefficient. We could have them resolve it but then pass the full stream to another server which then resolves it before writing to disk. But then any cases where this fixes an issue of lost data would mean the disk is now inconsistent with what our watchers have. So therefore, splitting off another server to record the full stream of data and having that be all it does and then resolving the data afterwards isn't any faster, and doesn't make our servers appear more stable.
+        - However, splitting off the data stream might be a good idea for the purposes of auditing, instead of speed, or reliability.
+
+
+Summary of benefits/costs
+
+    Benefits
+        1) Decreased code size / decreased development time / decreased complexity
+            - Adding a new value doesn't require new endpoints
+            - Even for a non-syncing application the code size is about as small as it can get
+        2) Minimum possible clientside delay (so everything feels fast)
+            - If you just explicitly input predict values, and then clobber the predicted value when the call comes back, you run into a case where multiple predictions get clobber by an old callback. Which is bad.
+            - If you explicitly input predict values, and only clobber the latest, that works... until you have reasonably complex data.
+            - Explicit endpoints per data path is a non-starter, because when your user interfaces migrates it won't overlap, and then you will have to explicitly calling multiple endpoints after a mutate call, and it will become your entire job.
+            - The only way to do input prediction at a framework level is to track all accesses, and have a serverside which exposes path bases values, AND to have the same mutate functions clientside as you use serverside (otherwise you have to rewrite all mutate functions twice).
+                - AND THEN... if other users mutate data, you NEED the server to tell you when it changes. If you don't, 
+                - In theory you can download data every time you need it... but this can't scale to reasonably large applications. AND, if you don't the user won't just see stale data, they will predict stale data, which will be really bad. So... the server needs to synchronize the paths as well
+        3) Excellent sharding / scaling via very lax sharding
+            - PathValue.locks are a simple locking mechanism, and very efficient (less efficient with high contention)
+            - Transactions can easily be done cross shard, and are still fully ACID, which most other databases have a hard time doing (unless they use virtually the same locking scheme)
+
+    Database flowchart
+        (NOTE: Large static assets should always be stored in some object storage database, like S3, even if you are using another system for other data.)
+        (NOTE: If you want an ultra responsive UI you can often use a write-then-reader wrapper for your database (which might require extra work to specify what should update, and what should be read). Or... just use query-sub)
+
+        - Do you want a very responsive user interface (via input prediction)
+            = Yes
+                - Is it easy to specify exactly what the writes are (with high accuracy) for most server write operations
+                    (As in, are you using complex writes that involve functions with many lines of code, or is pretty much just KVP writes.)
+                    = Yes
+                        - Will the user often experience write contention
+                            = YES
+                                - query-sub
+                            = NO
+                                - query-sub
+                                - OR Run again without a very responsive user interface, and then slap a write-then-reread wrapper onto your database, and then tell it what values it should predict.
+                    = No
+                        - query-sub
+            = NO
+                - Is % chance contention * impact of contention low (contention free)?
+                = Yes
+                    Is the data static enough where stale data can be shown? (ex, search results, non-user part of youtube, twitter?)
+                    = Yes
+                        - Any NoSQL implementation (S3 for large data, MongoDB for complex data, otherwise BigTable, or even ElasticSearch?)
+                    = No
+                        - Do you want clientside data syncing
+                        = Yes
+                            - firebase
+                        = No
+                            - A fast writing NoSQL implementation (probably MongoDB)
+                = No
+                    - Do you want clientside data syncing
+                    = Yes
+                        - firebase + firebase cloud functions
+                    = No
+                        - Are writes simple and only affect a single collection
+                        = Yes
+                            - MongoDB
+                        = No
+                            - MongoDB / SQL (likely with high transaction isolation levels)
+
+        Conclusion
+            - Realistically, you want a responsive user interface (who wouldn't?)
+
+    NOTE: For applications with very static data (so not just mostly reads, but the data actually changes so infrequently that users can be shown stale data and they won't even notice), using MongoDB is just as good (unless you need really complex transactions).
+        - You will still run into issues at huge scales with other databases though, as eventually the amount of transactions you want will exceed what one server can handle, and then you need to shard, and if you don't or can't shard well, everything will get slow.
+
+    Timestamped writes / clientside reproducible writes
+        BENEFIT: Allows clients to compare against incoming writes and know if they write is newer, or older, allowing predictions to never get out of sync.
+            ALTERNATIVE: Reading state after each write also allows this. HOWEVER, if the write is a function mutation, the impetus for the write might be small, but the data mutated could be large. For example, a paint application, where a write mutates the canvas, which is large, so we don't want to download the entire canvas after each write.
+                - AND, even if the values are small, this requires increased network usage
+        COST: Is slower. Is only useful with with KVP writes (if you can't map from clientside write to read paths this can't be used to predict values).
+        ADDITIONAL BENEFITS: Makes debugging easier, via keeping track of a history of operations
+    
+    Watching/syncing
+        BENEFIT: Greatly reduces read count needed to keep data up to date
+        COST: Writes have more overhead, requiring everything to be notified
+
+    Lock tracking / valid tracking / contention invalidation / eventually consistent transactions
+        BENEFIT: ACID safety that is fast even with large collections.
+            - Also allows ACID safety even for servers that are off the main network, allowing anyone to add transactions without worrying about inserting an inconsistent write (worst case the write will be rejected).
+            - Also, ACID safety is a lot easier to implement with this technique
+            ALTERNATIVE: Use a single server, or a single server per collection and don't provide ACID cross collection.
+            ALTERNATIVE: Collection locking, which if you have enough servers using the same collection degenerates to single server performance.
+            ALTERNATIVE: Smart evenly sub-collection sharding. This can be very efficient, HOWEVER, it is extremely difficult to anticipate the access patterns. Accurately predicting all access patterns would be akin to a universal storage system that can run any search query in optimal time (either O(1), or O(logN)).
+            ALTERNATIVE: No ACID safety. This is pretty much only viable if the write interface is purely KVP based. Otherwise it is easy to create functions which depend on ACID safety, and give unpredictable results when it is not kept (ex, contention causes really bad breaking of things). This makes it impossible to implement things such as financial transactions, most games, etc.
+                - And, of course, we can have lock tracking support, but just commit writes with no readLocks, which will produce much faster writes, making the database based degenerate to a NoSQL database, with theoretically the same speed.
+        COST: Slower to compute writes, and writes become MUCH larger.
+            - Lock/eventually consistency transactions ONLY work if the client readers are syncing. If they aren't, they will get the wrong data and KEEP the wrong data, which would break pretty much any application.
+        ADDITIONAL BENEFITS: Makes debugging easier, via making it possible to trace the read source used to generate any write (so to track data through the system).

package/deploy.ts

@@ -1,3 +1,3 @@
-import "./src/diagnostics/satSchema";
-import "./src/test/test";
+import "./src/diagnostics/satSchema";
+import "./src/test/test";
 import "./src/4-dom/qreactTest";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "querysub",
-  "version": "0.432.0",
+  "version": "0.436.0",
   "main": "index.js",
   "license": "MIT",
   "note1": "note on node-forge fork, see https://github.com/digitalbazaar/forge/issues/744 for details",