sliftutils 1.7.10 → 1.7.12

package/spec.txt

@@ -1,95 +1,5 @@
 
 
-IMPORTANT! modified time PLUS size is what's important. Store both!
 
 
-1) Tests locally, running it in Slift utils and running some code in CYOA to access it. 
-2) Set up a bootstrapper that we can chain with another command that will set up secrets, cloudflare key, etc
-    yarn exposesecrets && yarn storageserver --url https://storage.vidgridweb.com:4446 --folder ~/storagetest
-    - The bootstrapper needs to write to "cloudflare.json" and "backblaze.json"
-        - Using our sync server secrets. Not sure if they're stored in there, but, you know, easy enough...
 
-3) Verify backblaze works, verify basically all the features work. 
-
-
-
-
-3) Support sharding in synchronization sources as well, adding new field
-    route: {
-        range: [number, number];
-        level: number;
-    }
-    - Route level is required as if two things have different route ranges, they obviously aren't going to be pulling from each other. So we need another way to stack redundancy, and the route level can do that. If two things are at the same route level, it means they aren't using each other for redundancy. We could automatically detect this, but it's more clear if it's explicit.
-
-3.1) We will almost always set up copy files on our different shards, but of course, these will only be considered depending on the valid window. 
-3.2) Of course, having our site configuration support sharding as well. So you don't need to fiddle with the shards. You kind of just say there are a certain number of shards. And if you want to change the shards, it will automatically set them up in the future, applying a default switch time of a day that you can also configure yourself, etc. 
-
-
-
-
-5) Shard retry writes - Basically, they give us part of a path, and part of it we can fill in ourselves. We try to keep track of which shards are down and if a shard is down, we will keep changing that random value until we find something that doesn't match that shard Until we hit a certain percentage of shards being down, probably greater than 99%. And then we return the path that we ended up using. 
-    - This solves the problem of not being able to write to the backup sources. As it'll mean you'll be able to read if a shard is down, and you'll also be able to write if a shard is down. So it'll be like everything's fine. 
-    - We only need to use this for certain important things like uploading images. pretty much all of the AI stuff, we can probably use it. 
-    HMMM!!! This causes problems if we want it to be an image or a video. So we should probably have it so by default we try to put nothing there, So it'll be the exact path the user requested. That way, if they try to do the same right multiple times, it'll clobber itself.
-        - Although realistically, I don't think it'll really matter.  because we're not generating the same content most of the time. 
-6) geographical sharding as well. Which tries to change the hash so it's the server that we're closest to
-    - This can massively reduce the write latency, Which is important because we don't want to have to send the data around all the time, we just want to send around the URLs. Well then this requires waiting for the remote to confirm that we can read back. which will be purely limited really on latency if it's close to us. 
-
-7) readerDiskLimit. Only when our first sync source has copyFiles is false and cacheReads is true (and required true). Once we copy over a certain amount, we start deleting the least recently used from our local cache. This allows us to cache large sources without getting overwhelmed. 
-
-
-
-
-
-
-
-
-
-
-2) Use our machine configuration to setup a storage service
-
-2.1) cyoa page to setup services + VIEW their storagerouting.JSON
-    WE DO NOT want to configure it via the interface! We want the configuration to always be in code. That way, we can update one place and all of our buckets will update!
-    OH! Make something that will import all of our code just like in a deploy, and then we'll gather all of the bucket configurations that we created using our helper functions so we register them all, and then we'll touch them all to make sure the configuration is updated on all of them!
-    - We should probably make a factory so we can easily create buckets without having to find the full chain every single time, and so we can change the full chain easily in one place. 
-    - We should allow easily chaining them for redundancy
-    - We might even have some UI that shows all of the storage services together, allowing you to easily configure how they chain, etc. 
-    - We should also have it poll for the status of like scanning file stats, etc. 
-
-2.2) Test some of these custom buckets on CYOA (local dev), testing redundancy, speed, etc.
-    - If we run into a lot of issues that we need to debug, we can always host the storage server locally. 
-
-    custom => backblaze
-        - So internal redundancy and then also backblaze at the very end
-2.3) Test using our thing as a front end for back plays with no copy, which should mean there's really no setup. 
-    - Just test reading images or something. 
-
-
-
-
-
-
-
-ArchivesRemote
-    bucketName
-    public
-    fast
-    writeDelay
-
-ArchivesFactory
-    host: https://storage.vidgridweb.com:4444/storagerouting.json
-    redundantHosts:
-        - https://storage2.vidgridweb.com:4445/storagerouting.json
-        - https://f002.backblazeb2.com/file/querysubtest-com-public-immutable/storage/storagerouting.json
-
-HostStorageServerConfig
-    host: https://storage.vidgridweb.com:4444/storagerouting.json
-    redundantHosts:
-        - https://storage.vidgridweb.com:4444/storagerouting.json
-        - https://storage2.vidgridweb.com:4445/storagerouting.json
-        - https://f002.backblazeb2.com/file/querysubtest-com-public-immutable/storage/storagerouting.json
-    folder: os.homedir() + "/storage"
-    cloudflareApiToken: ...
-
-
-https://f002.backblazeb2.com/file/querysubtest-com-public-immutable/image/img_00BloAFyIevA_1024x1536.jpg

package/storage/ArchivesDisk.d.ts

@@ -16,6 +16,7 @@ export declare class ArchivesDisk implements IArchives {
     };
     getDebugName(): string;
     getConfig(): Promise<ArchivesConfig>;
+    hasWriteAccess(): Promise<boolean>;
     private filePath;
     set(key: string, data: Buffer, config?: {
         lastModified?: number;
@@ -35,6 +36,7 @@ export declare class ArchivesDisk implements IArchives {
     }): Promise<{
         data: Buffer;
         writeTime: number;
+        size: number;
     } | undefined>;
     getInfo(key: string): Promise<{
         writeTime: number;

package/storage/ArchivesDisk.ts

@@ -132,6 +132,10 @@ export class ArchivesDisk implements IArchives {
         return {};
     }
 
+    public async hasWriteAccess(): Promise<boolean> {
+        return true;
+    }
+
     private filePath(key: string): string {
         let result = path.join(this.filesDir, key);
         if (!result.startsWith(this.filesDir + path.sep)) {
@@ -181,7 +185,7 @@ export class ArchivesDisk implements IArchives {
         return result && result.data || undefined;
     }
 
-    public async get2(key: string, config?: { range?: { start: number; end: number } }): Promise<{ data: Buffer; writeTime: number } | undefined> {
+    public async get2(key: string, config?: { range?: { start: number; end: number } }): Promise<{ data: Buffer; writeTime: number; size: number } | undefined> {
         await this.init();
         let range = config?.range;
         let filePath = this.filePath(key);
@@ -197,13 +201,13 @@ export class ArchivesDisk implements IArchives {
             let size = stats.size;
             let start = range && Math.min(range.start, size) || 0;
             let end = range && Math.min(range.end, size) || size;
-            if (end <= start) return { data: Buffer.alloc(0), writeTime: stats.mtimeMs };
+            if (end <= start) return { data: Buffer.alloc(0), writeTime: stats.mtimeMs, size };
             let buffer = Buffer.alloc(end - start);
             let { bytesRead } = await handle.read(buffer, 0, buffer.length, start);
             if (bytesRead !== buffer.length) {
                 throw new Error(`Expected ${buffer.length} bytes at ${filePath}:${start}, read ${bytesRead}`);
             }
-            return { data: buffer, writeTime: stats.mtimeMs };
+            return { data: buffer, writeTime: stats.mtimeMs, size };
         });
     }