querysub 0.311.0 → 0.313.0

package/.cursorrules

@@ -43,7 +43,7 @@ Never use em or rem. Only use px or vw/vh/%.
 
 Errors should almost always use a template string to include the expected value, and the actual value, as in: "throw new Error(`Expected X, was ${Y}`);"
 
-We use a dark theme.
+Don't add font colors unless asked to add styling. Don't add any aethetics beyond hbox/vbox/pad2 unless asked to add styling. Don't add fontSize unless asked to add styling. If you believe styling is possible, just tell the user, I can add styling, but won't do it unless you ask me to. 
 
 Don't use switch statements. Use if statements instead.
 

package/costsBenefits.txt

@@ -16,13 +16,16 @@ Concepts
 
     NOTE: We don't actually lock things such as Object.keys(). In practice we will get pretty close, but our handling of it is simply not fully ACID.
 
-    NOTE: In regards to FunctionRunners. FunctionRunner rerun any calls which seem to not be run. The first one to run will get the value, with the others having their ReadLock on the result (from epoch to their LATER writeTime) invalidated.
+    NOTE: In regards to FunctionRunners. FunctionRunner rerun any calls which have no result (included if their result was written, then invalidate). The first one to run will get the value, with the others having their ReadLock on the result (from epoch to their LATER writeTime) invalidated.
         - Although, in practice, the writeTimes will be close, as they are based on requested writeTimes, made unique by creatorId. So... if two FunctionRunners run the same call, one will consistently create the accepted value. So... just don't have two FunctionRunners run the same call!
 
     NOTE: "Bad" predictions are possibly by if you know the path some change will write to (ie, the call result path). You can create a prediction that depends on that path being undefined (in some range that includes the eventual remote change, but not the prediction), simply watch that path, and when the change eventually happens your prediction will automatically be invalidated.
         - This is not suitable for trusted nodes, as it means all of our dependent values on that prediction will be only acceptable to us (as our prediction won't really be able to be propagated, as the change might never happen, in which case our prediction has to go away, which is impossible to do if we propagate it), but... it is very useful for clients to quickly predict writes (trusted nodes can just write actual values, not trying to get some remote to make a change).
         - AND, if you only depend on that path there is no jitter as you reject old values, as there is no cascading! This DOES allow values to drift out of sync with the server (if they are CONSTANTLY written to), but... that is another problem, for games, and just one of many issues games have to overcome.
 
+    NOTE: The reason that we have our main database servers resolving all the values before they write to disk, is simple. They need to resolve the values, because somebody needs to and if the watchers resolve it it's extremely inefficient. We could have them resolve it but then pass the full stream to another server which then resolves it before writing to disk. But then any cases where this fixes an issue of lost data would mean the disk is now inconsistent with what our watchers have. So therefore, splitting off another server to record the full stream of data and having that be all it does and then resolving the data afterwards isn't any faster, and doesn't make our servers appear more stable.
+        - However, splitting off the data stream might be a good idea for the purposes of auditing, instead of speed, or reliability.
+
 
 Summary of benefits/costs
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "querysub",
-  "version": "0.311.0",
+  "version": "0.313.0",
   "main": "index.js",
   "license": "MIT",
   "note1": "note on node-forge fork, see https://github.com/digitalbazaar/forge/issues/744 for details",
@@ -35,7 +35,8 @@
     "servershardedtest": "yarn server --authority ./pathremain.json & yarn server --authority ./patha.json & yarn server --authority ./pathb.json & yarn server --authority ./pathc.json & yarn server --authority ./pathd.json",
     "type": "yarn tsc --noEmit",
     "depend": "yarn --silent depcruise src --include-only \"^src\" --config --output-type dot | dot -T svg > dependency-graph.svg",
-    "test": "yarn typenode ./src/test/test.tsx --local",
+    "test": "yarn typenode ./test.ts",
+    "test3": "yarn typenode ./src/test/test.tsx --local",
     "test2": "yarn typenode ./src/4-dom/qreactTest.tsx --local"
   },
   "bin": {

package/spec.txt

@@ -52,23 +52,24 @@ More corruption resistance files
                 - and... we might as well add support for short, int, float, and uint (uint is a good way to store a guid, via storing 8 uint variables).
 
 Schema/binary PathValues accesses
-        0) First our stream/schema wrappers. Which at first just convert to raw path reads/writes, but once we add schemas they can efficiently create/use those
-        1) PathValue.path (string[]), need to be converted to an opaque type `Path`
-            - Helpers to manipulate it, and go back to string[], string, etc.
-        2) Have Path be able to store itself in schema + variables mode
-            - Where schema is shared across all Path instances
-            { schema: { path: (string|Variable)[] }; variables: (string | number)[]; }
-            - Add functions to access the schema and variable values
-        3) Create an efficient `Map<Path, T>`, via using schemas from the paths
-            - By requiring Path as the input we can directly use the schema to narrow down the cases, and then within that we just have to lookup values by variables
-        4) Use Path everywhere, replacing PathValue.path, using our efficient lookup to manage it.
-            - I think a lookup should handle all the cases. We should be able to nest them as well?
-        5) This SHOULD let us entirely get rid of path joining, which should be SO much faster.
-        6) Update serialization to keep schemas, instead of converting them back to paths. This should be more efficient, and a lot smaller.
-            - Values serialization won't change, and we still need to encode the variables, but... it should still be a lot faster.
-        7) Try to remove as many Path conversions to string (and also to a lesser degree string[]), as possible, by making them schema aware.
-        8) Investigate further optimizations
-            - Replacing variables with numbers, so we our internals Maps can be replaced with arrays / typed arrays
+    PER Path, not per value. Values will likely be lazily deserialized anyways, making them either pointers, or just numbers, stored in unchanging objects, which... should be quite fast.
+    0) First our stream/schema wrappers. Which at first just convert to raw path reads/writes, but once we add schemas they can efficiently create/use those
+    1) PathValue.path (string[]), need to be converted to an opaque type `Path`
+        - Helpers to manipulate it, and go back to string[], string, etc.
+    2) Have Path be able to store itself in schema + variables mode
+        - Where schema is shared across all Path instances
+        { schema: { path: (string|Variable)[] }; variables: (string | number)[]; }
+        - Add functions to access the schema and variable values
+    3) Create an efficient `Map<Path, T>`, via using schemas from the paths
+        - By requiring Path as the input we can directly use the schema to narrow down the cases, and then within that we just have to lookup values by variables
+    4) Use Path everywhere, replacing PathValue.path, using our efficient lookup to manage it.
+        - I think a lookup should handle all the cases. We should be able to nest them as well?
+    5) This SHOULD let us entirely get rid of path joining, which should be SO much faster.
+    6) Update serialization to keep schemas, instead of converting them back to paths. This should be more efficient, and a lot smaller.
+        - Values serialization won't change, and we still need to encode the variables, but... it should still be a lot faster.
+    7) Try to remove as many Path conversions to string (and also to a lesser degree string[]), as possible, by making them schema aware.
+    8) Investigate further optimizations
+        - Replacing variables with numbers, so we our internals Maps can be replaced with arrays / typed arrays
 
     IMPORTANT! Actually... a lot of the following is wrong. We should have this PER atomic value, and perhaps ONLY for paths!
 
@@ -1108,4 +1109,8 @@ Very short expiry times on thread certificates, finding some way to automaticall
 - Our system for the client to update it's cert after the fact can likely be reused to trigger an update of credentials?
     - I assume outstanding connections won't be killed if their certs expire, so... we would actually want to
         add an additional check to close a connection if it's cert isn't updated? Or... we could just call renegotiate, which
-        I think serverside TLS connections can do?
+        I think serverside TLS connections can do?
+
+Public release?
+Get benchmarks on https://benchant.com/ranking/database-ranking
+    - It seems like the throughput can be easily beat. We trade massive data-loss and memory efficiency for speed, so after our path optimizations we should be able to easily beat all others servers (again, because we lose data easily, and store everything in memory...)

package/src/-0-hooks/hooks.ts

@@ -21,7 +21,7 @@ function createHookFunction<Fnc extends (...args: any[]) => void>(debugName: str
     }, 1000);
     function flushQueued() {
         if (!declaration) {
-            throw new Error(`Hook function ${debugName} not declared`);
+            throw new Error(`Hook function ${debugName} not declared, but the hook was used. Make sure to include the code somewhere after using the hook.`);
         }
         let queued = queuedCalls;
         queuedCalls = [];

package/src/-a-archives/archives.ts

@@ -7,9 +7,7 @@ import { magenta } from "socket-function/src/formatting/logColors";
 import debugbreak from "debugbreak";
 import { isClient } from "../config2";
 import { wrapArchivesWithCache } from "./archiveCache";
-// https://www.backblaze.com/apidocs/b2-start-large-file
-// https://www.backblaze.com/apidocs/b2-upload-part
-// https://www.backblaze.com/apidocs/b2-finish-large-file
+import { Args } from "socket-function/src/types";
 
 export interface Archives {
     getDebugName(): string;
@@ -58,6 +56,19 @@ export interface Archives {
     assertPathValid(path: string): Promise<void>;
 
     getBaseArchives?: () => { parentPath: string; archives: Archives; } | undefined;
+
+    /** Get a download URL for a file. Only implemented for some archive types (like Backblaze). */
+    getURL?: (path: string) => Promise<string>;
+
+    // NOTE: This isn't part of getURL, as it is much more efficient to call it once and reuse the authorization. This is only for backblaze as of right now.
+    // NOTE: This also gives authorization for ALL files, so... ONLY use it for internal download links. If we need external links, we should use a public folder with hash links, or... make a new endpoint.
+    getDownloadAuthorization?: (config: {
+        validDurationInSeconds: number;
+    }) => Promise<{
+        bucketId: string;
+        fileNamePrefix: string;
+        authorizationToken: string;
+    }>;
 }
 
 export function nestArchives(path: string, archives: Archives): Archives {
@@ -101,6 +112,8 @@ export function nestArchives(path: string, archives: Archives): Archives {
             }
             return result;
         },
+        getURL: archives.getURL ? (filePath: string) => archives.getURL!(path + stripFilePrefix(filePath)) : undefined,
+        getDownloadAuthorization: archives.getDownloadAuthorization && ((...args: Args<typeof archives.getDownloadAuthorization>) => archives.getDownloadAuthorization!(...args)),
     };
 }
 

package/src/-a-archives/archivesBackBlaze.ts

@@ -342,6 +342,22 @@ const getAPI = lazy(async () => {
         fileId: string;
     }, {}>("b2_cancel_large_file", "POST", "noAccountId");
 
+    const getDownloadAuthorization = createB2Function<{
+        bucketId: string;
+        fileNamePrefix: string;
+        validDurationInSeconds: number;
+        b2ContentDisposition?: string;
+        b2ContentLanguage?: string;
+        b2Expires?: string;
+        b2CacheControl?: string;
+        b2ContentEncoding?: string;
+        b2ContentType?: string;
+    }, {
+        bucketId: string;
+        fileNamePrefix: string;
+        authorizationToken: string;
+    }>("b2_get_download_authorization", "POST", "noAccountId");
+
     async function getDownloadURL(path: string) {
         if (!path.startsWith("/")) {
             path = "/" + path;
@@ -349,6 +365,7 @@ const getAPI = lazy(async () => {
         return auth.downloadUrl + path;
     }
 
+
     return {
         createBucket,
         updateBucket,
@@ -363,6 +380,7 @@ const getAPI = lazy(async () => {
         uploadPart,
         finishLargeFile,
         cancelLargeFile,
+        getDownloadAuthorization,
         getDownloadURL,
     };
 });
@@ -402,14 +420,15 @@ export class ArchivesBackblaze {
             cacheTime = 86400 * 1000;
         }
 
-        let desiredCorsRules = this.config.public ? [{
+        // ALWAYS set access control, as we can make urls for private buckets with getDownloadAuthorization
+        let desiredCorsRules = [{
             corsRuleName: "allowAll",
             allowedOrigins: ["https"],
             allowedOperations: ["b2_download_file_by_id", "b2_download_file_by_name"],
             allowedHeaders: ["range"],
             exposeHeaders: ["x-bz-content-sha1"],
             maxAgeSeconds: cacheTime / 1000,
-        }] : [];
+        }];
         let bucketInfo: Record<string, unknown> = {};
         if (cacheTime) {
             bucketInfo["cache-control"] = `max-age=${cacheTime / 1000}`;
@@ -858,7 +877,7 @@ export class ArchivesBackblaze {
         return this.move({ ...config, copyInstead: true });
     }
 
-    public async getDownloadURL(path: string) {
+    public async getURL(path: string) {
         return await this.apiRetryLogic(async (api) => {
             if (path.startsWith("/")) {
                 path = path.slice(1);
@@ -866,6 +885,29 @@ export class ArchivesBackblaze {
             return await api.getDownloadURL("file/" + this.bucketName + "/" + path);
         });
     }
+
+    public async getDownloadAuthorization(config: {
+        fileNamePrefix?: string;
+        validDurationInSeconds: number;
+        b2ContentDisposition?: string;
+        b2ContentLanguage?: string;
+        b2Expires?: string;
+        b2CacheControl?: string;
+        b2ContentEncoding?: string;
+        b2ContentType?: string;
+    }): Promise<{
+        bucketId: string;
+        fileNamePrefix: string;
+        authorizationToken: string;
+    }> {
+        return await this.apiRetryLogic(async (api) => {
+            return await api.getDownloadAuthorization({
+                bucketId: this.bucketId,
+                fileNamePrefix: config.fileNamePrefix ?? "",
+                ...config,
+            });
+        });
+    }
 }
 
 /*
@@ -880,6 +922,12 @@ Names should be a UTF-8 string up to 1024 bytes with the following exceptions:
 export const getArchivesBackblaze = cache((domain: string) => {
     return new ArchivesBackblaze({ bucketName: domain });
 });
+export const getArchivesBackblazePrivateImmutable = cache((domain: string) => {
+    return new ArchivesBackblaze({
+        bucketName: domain + "-private-immutable",
+        immutable: true
+    });
+});
 export const getArchivesBackblazePublicImmutable = cache((domain: string) => {
     return new ArchivesBackblaze({
         bucketName: domain + "-public-immutable",

package/src/-a-archives/archivesLimitedCache.ts

@@ -0,0 +1,175 @@
+import { formatNumber } from "socket-function/src/formatting/format";
+import { Archives } from "./archives";
+import { cache } from "socket-function/src/caching";
+
+interface FileInfo {
+    writeTime: number;
+    accessTime: number;
+    size: number;
+}
+
+class ArchivesLimitedCache {
+    private baseArchives: Archives;
+    private maxFiles: number;
+    private maxSize: number;
+    private cache = new Map<string, FileInfo>();
+    private initialized = false;
+
+    constructor(baseArchives: Archives, config: { maxFiles: number; maxSize: number }) {
+        this.baseArchives = baseArchives;
+        this.maxFiles = config.maxFiles;
+        this.maxSize = config.maxSize;
+        this.initOptionalMethods();
+    }
+
+    public getDebugName(): string {
+        return `limitedCache(${this.maxFiles}files,${Math.round(this.maxSize / (1024 * 1024))}MB)/${this.baseArchives.getDebugName()}`;
+    }
+
+    private async ensureInitialized(): Promise<void> {
+        if (this.initialized) return;
+
+        // Load all files on first access
+        const allFiles = await this.baseArchives.findInfo("", { type: "files" });
+        for (const file of allFiles) {
+            this.cache.set(file.path, {
+                writeTime: file.createTime,
+                accessTime: file.createTime,
+                size: file.size,
+            });
+        }
+        this.initialized = true;
+
+        // Cleanup if we're already over limits
+        await this.applyLimits();
+    }
+
+    private async applyLimits(): Promise<void> {
+        const files = Array.from(this.cache.entries()).map(([path, info]) => ({
+            path,
+            ...info,
+        }));
+
+        const totalSize = files.reduce((sum, file) => sum + file.size, 0);
+        const totalFiles = files.length;
+
+        if (totalFiles <= this.maxFiles && totalSize <= this.maxSize) {
+            return; // No cleanup needed
+        }
+
+        // Sort by access time (oldest first)
+        files.sort((a, b) => a.accessTime - b.accessTime);
+
+        let currentSize = totalSize;
+        let currentFiles = totalFiles;
+
+        // Remove files until we're under both limits
+        for (const file of files) {
+            if (currentFiles <= this.maxFiles && currentSize <= this.maxSize) {
+                break;
+            }
+
+            console.warn(`Deleting file ${file.path} to maintain limits. Currently have ${formatNumber(currentFiles)} files and ${formatNumber(currentSize)}B`);
+            try {
+                await this.baseArchives.del(file.path);
+                this.cache.delete(file.path);
+                currentSize -= file.size;
+                currentFiles--;
+            } catch (error) {
+                // File might already be deleted, continue
+                console.warn(`Failed to delete file ${file.path}:`, error);
+                this.cache.delete(file.path); // Remove from cache anyway
+                currentSize -= file.size;
+                currentFiles--;
+            }
+        }
+    }
+
+    private updateAccessTime(path: string): void {
+        const info = this.cache.get(path);
+        if (info) {
+            info.accessTime = Date.now();
+        }
+    }
+
+    public async get(path: string, config?: { range?: { start: number; end: number; }; retryCount?: number }): Promise<Buffer | undefined> {
+        await this.ensureInitialized();
+        this.updateAccessTime(path);
+        return this.baseArchives.get(path, config);
+    }
+
+    public async set(path: string, data: Buffer): Promise<void> {
+        await this.ensureInitialized();
+        // Limit before, not after, so we don't remove the file we just added
+        await this.applyLimits();
+        await this.baseArchives.set(path, data);
+        const now = Date.now();
+        this.cache.set(path, {
+            writeTime: now,
+            accessTime: now,
+            size: data.length,
+        });
+    }
+
+    public async del(path: string): Promise<void> {
+        await this.ensureInitialized();
+        await this.baseArchives.del(path);
+        this.cache.delete(path);
+    }
+
+    public async getInfo(path: string): Promise<{ writeTime: number; size: number; } | undefined> {
+        await this.ensureInitialized();
+        this.updateAccessTime(path);
+        return this.baseArchives.getInfo(path);
+    }
+
+    public async setLargeFile(config: { path: string; getNextData(): Promise<Buffer | undefined>; }): Promise<void> {
+        throw new Error(`setLargeFile in archivesLimitedCache is not supported`);
+    }
+
+    public async find(prefix: string, config?: { shallow?: boolean; type: "files" | "folders" }): Promise<string[]> {
+        await this.ensureInitialized();
+        return this.baseArchives.find(prefix, config);
+    }
+
+    public async findInfo(prefix: string, config?: { shallow?: boolean; type: "files" | "folders" }): Promise<{ path: string; createTime: number; size: number; }[]> {
+        await this.ensureInitialized();
+        return this.baseArchives.findInfo(prefix, config);
+    }
+
+    public async move(config: { path: string; target: Archives; targetPath: string }): Promise<void> {
+        throw new Error(`Move in archivesLimitedCache is not supported`);
+    }
+
+    public async copy(config: { path: string; target: Archives; targetPath: string }): Promise<void> {
+        throw new Error(`Copy in archivesLimitedCache is not supported`);
+    }
+
+    public enableLogging(): void {
+        this.baseArchives.enableLogging();
+    }
+
+    public async assertPathValid(path: string): Promise<void> {
+        await this.baseArchives.assertPathValid(path);
+    }
+
+    public getBaseArchives?: () => { parentPath: string; archives: Archives; } | undefined = () => {
+        const base = this.baseArchives.getBaseArchives?.();
+        if (base) {
+            return base;
+        }
+        return { parentPath: "", archives: this.baseArchives };
+    };
+
+    public getURL?: (path: string) => Promise<string>;
+    public getDownloadAuthorization?: (config: { validDurationInSeconds: number }) => Promise<{ bucketId: string; fileNamePrefix: string; authorizationToken: string; }>;
+
+    private initOptionalMethods(): void {
+        this.getURL = this.baseArchives.getURL;
+        this.getDownloadAuthorization = this.baseArchives.getDownloadAuthorization;
+    }
+}
+
+export function createArchivesLimitedCache(baseArchives: Archives, config: { maxFiles: number; maxSize: number }): Archives {
+    return new ArchivesLimitedCache(baseArchives, config);
+}