braidfs 0.0.57 → 0.0.58

package/README.md

@@ -1,88 +1,146 @@
 # BraidFS: Braid your Filesystem with the Web
 
-Provides interoperability between **web pages** via http and your **OS filesystem**.
-  - Using collaborative CRDTs and the Braid extensions for HTTP.
+Synchronize ***WWW Pages*** with your ***OS Filesystem***.
 
-The `braidfs` daemon performs bi-directional synchronization between remote Braid-HTTP resources and your local filesystem.
+The `braidfs` daemon performs bi-directional synchronization between remote
+[Braided](https://braid.org) HTTP resources and your local filesystem.  It uses
+the [braid-text](https://github.com/braid-org/braid-text) library for
+high-performance, peer-to-peer collaborative text synchronization over HTTP,
+and keeps your filesystem two-way synchronized with the fully-consistent CRDT.
 
-### Web resources map onto your filesystem
+### Sync the web into your `~/http` folder
 
-It creates a `~/http` folder in your home directory to synchronize webpages with:
+Braidfs synchronizes webpages to a local `~/http` folder:
 
 ```
 ~/http/
-  ├── example.com/
-  │   ├── document.html
-  │   ├── notes.md
-  │   └── assets/
-  │       └── style.css
-  └── braid.org/
-      └── meeting-53
+  ├── braid.org/
+  |   └── meeting-53
+  └── example.com/
+      ├── document.html
+      ├── notes.md
+      └── assets/
+          └── style.css
 ```
 
+Add a new page with the `braidfs sync <url>` command:
 
-https://github.com/user-attachments/assets/4fb36208-e0bd-471b-b47b-bbeee20e4f3f
+![](https://braid.org/files/braidfs-demo1.webp)
 
+Unsync a page with `braidfs unsync <url>`.
 
+### Edit remote state as a local file—and vice versa
 
-### Two-way sync edits between files and web
+Any synced page can be edited with your favorite local text editor—like
+VSCode, Emacs, Vim, Sublime, TextWrangler, BBEdit, Pico, Nano, or Notepad—and
+edits propagate live to the web, and vice-versa.
 
-As long as `braidfs` is running, it will keep the file and web resources in
-sync!
+Here's a demo of VSCode editing [braid.org/braidfs](https://braid.org/braidfs):
 
- - Edit to the file → web resource
- - Edit to the web resource → file
+![](https://braid.org/files/braidfs-demo2.webp)
 
-Each edit to the file is diffed and sent as a CRDT patch, so you can edit
-files offline, and even collaboratively edit them, with one caveat:
+After VSCode saves the file, braidfs immediately computes a diff of the edits
+and sends them as patches over Braid HTTP to https://braid.org/braidfs:
 
-#### Caveat
+```
+PUT /braidfs
+Version: "n0j5kg9g23-100"
+Parents: "ercurwxmz7g-37"
+Content-Length: 9
+Content-Range: text [32:32]
 
-For the period of time that you have edited the file in your editor but not
-saved it, external edits cannot be integrated.
+ with the
 
-## Installation
+```
 
-Install braidfs globally using npm:
+Conversely, remote edits instantly update the local filesystem, and thus the
+editor.
 
 ```
-npm install -g braidfs
+HTTP 200 OK
+Version: "ercurwxmz7g-41"
+Parents: "n0j5kg9g23-100"
+Content-Length: 4
+Content-Range: text [41:41]
+
+ Web
+
 ```
 
-## Usage
+Edits are formatted as simple
+[Braid-HTTP](https://github.com/braid-org/braid-spec).  To participate in the
+network, you can even author these messages by hand, from your own code, using
+simple GETs and PUTs.
+
+### Conflict-Free Collaborative Editing
+
+Braidfs has a full [braid-text](https://github.com/braid-org/braid-text) peer
+within it, providing high-performance collaborative text editing on the
+diamond-types CRDT over the Braid HTTP protocol, guaranteeing conflict-free
+editing with multiple editors, whether online or offline.
+
+A novel trick using [Time Machines](https://braid.org/time-machines) lets us
+making regular text editors conflict-free, as well, without speaking CRDT!
+This means that you can edit a file in Emacs, even while other people edit the
+same file, without write conflicts, and without adding CRDT code to Emacs.
+(Still under development.)
+
+# Installation and Usage
+
+Install the `braidfs` command onto your computer with npm:
+
+```
+npm install -g braidfs
+```
 
-To start the braidfs daemon:
+Then you can start the braidfs daemon with:
 
 ```
 braidfs run
 ```
 
-To sync a URL:
+To run it automatically as a background service on MacOS, use:
 
 ```
-braidfs sync <url>
+ # Todo: fix this.  Not working yet.
+ # launchctl submit -l org.braid.braidfs -- braidfs run
 ```
 
-When you run `braidfs sync <url>`, it creates a local file mirroring the content at that URL. The local file path is determined by the URL structure. For example:
+### Adding and removing URLs
 
-- If you sync `https://example.com/path/file.txt`
-- It creates a local file at `~/http/example.com/path/file.txt`
+Sync a URL with:
 
-To unsync a URL:
+```
+braidfs sync <url>
+```
+
+Unsync a URL with:
 
 ```
 braidfs unsync <url>
 ```
 
-## Configuration
+URLs map to files with a simple pattern:
 
-braidfs looks for a configuration file at `~/http/.braidfs/config`, or creates it if it doesn't exist. You can set the following options:
+- url: `https://<domain>/<path>`
+- file: `~/http/<domain>/<path>`
 
-- `sync`: An object where the keys are URLs to sync, and the values are simply `true`
-- `cookies`: An object for setting domain-specific cookies for authentication
-- `port`: The port number for the internal daemon (default: 45678)
 
-Example `config`:
+Examples:
+
+| URL | File |
+| --- | --- |
+| `https://example.com/path/file.txt` | `~/http/example.com/path/file.txt` |
+| `https://braid.org:8939/path` | `~/http/braid.org:8939/path` |
+| `https://braid.org/` | `~/http/braid.org/index` |
+
+If you sync a URL path to a directory containing items within it, the
+directory will be named `/index`.
+
+
+### Configuration
+
+The config file lives at `~/http/.braidfs/config`.  It looks like this:
 
 ```json
 {
@@ -91,14 +149,34 @@ Example `config`:
     "https://example.com/document2.txt": true
   },
   "cookies": {
-    "example.com": "secret_pass"
+    "example.com": "secret_pass",
+    "braid.org": "client=hsu238s88adhab3afhalkj3jasdhfdf"
   },
   "port": 45678
 }
 ```
 
-The `cookies` configuration allows you to set authentication cookies for specific domains. In the example above, any requests to `example.com` will include the specified cookie value.
+These are the options:
+- `sync`: A set of URLs to synchronize.  Each one maps to `true`.
+- `cookies`: Braidfs will use these cookie when connecting to the domains.
+  - Put your login session cookies here.
+  - To find your cookie for a website:
+    - Log into the website with a browser
+    - Open the Javascript console and run `document.cookie`
+    - That's your cookie.  Copy paste it into  your config file.
+- `port`: The port number for the internal daemon (default: 45678)
+
+This config file live-updates, too.  Changes are automatically detected and
+applied to the running braidfs daemon.  The only exception is the `port`
+setting, which requires restarting the daemon after a change.
+
 
-## Security
+## Limitations & Future Work
 
-braidfs is designed to run locally and only accepts connections from localhost (127.0.0.1 or ::1) for security reasons. The `cookies` configuration enables secure communication with servers that require authentication by allowing you to set domain-specific cookie values.
+- Doesn't sync binary yet.  Just text text mime-types:
+  `text/*`, `application/html`, and `application/json`
+  - Binary blob support would be pretty easy and nice to add.
+  - Contact us if you'd like to add it!
+- Doesn't update your editor's text with remote updates until you save
+  - It's not hard to make it live-update, though, so that you can see your edits integrated with others' before you save.
+  - Contact us if you'd like to help!  It would be a fun project!

package/index.js

@@ -1,14 +1,9 @@
 #!/usr/bin/env node
 
-console.log(`braidfs version: ${require(`${__dirname}/package.json`).version}`)
-
 var { diff_main } = require(`${__dirname}/diff.js`),
     braid_text = require("braid-text"),
     braid_fetch = require('braid-http').fetch
 
-process.on("unhandledRejection", (x) => console.log(`unhandledRejection: ${x.stack}`))
-process.on("uncaughtException", (x) => console.log(`uncaughtException: ${x.stack}`))
-
 var proxy_base = `${require('os').homedir()}/http`,
     braidfs_config_dir = `${proxy_base}/.braidfs`,
     braidfs_config_file = `${braidfs_config_dir}/config`,
@@ -57,6 +52,41 @@ let to_run_in_background = process.platform === 'darwin' ? `
 To run daemon in background:
     launchctl submit -l org.braid.braidfs -- braidfs run` : ''
 let argv = process.argv.slice(2)
+
+if (argv[0] === 'editing') {
+    return (async () => {
+        var filename = argv[1]
+        if (!require('path').isAbsolute(filename))
+            filename = require('path').resolve(process.cwd(), filename)
+        var input_string = await new Promise(done => {
+            const chunks = []
+            process.stdin.on('data', chunk => chunks.push(chunk))
+            process.stdin.on('end', () => done(Buffer.concat(chunks).toString()))
+        })
+
+        var r = await fetch(`http://localhost:${config.port}/.braidfs/get_version/${encodeURIComponent(filename)}/${encodeURIComponent(sha256(input_string))}`)
+        if (!r.ok) throw new Error(`bad status: ${r.status}`)
+        console.log(await r.text())
+    })()
+} else if (argv[0] === 'edited') {
+    return (async () => {
+        var filename = argv[1]
+        if (!require('path').isAbsolute(filename))
+            filename = require('path').resolve(process.cwd(), filename)
+        var parent_version = argv[2]
+        var input_string = await new Promise(done => {
+            const chunks = []
+            process.stdin.on('data', chunk => chunks.push(chunk))
+            process.stdin.on('end', () => done(Buffer.concat(chunks).toString()))
+        })
+        var r = await fetch(`http://localhost:${config.port}/.braidfs/set_version/${encodeURIComponent(filename)}/${encodeURIComponent(parent_version)}`, { method: 'PUT', body: input_string })
+        if (!r.ok) throw new Error(`bad status: ${r.status}`)
+        console.log(await r.text())
+    })()
+}
+
+console.log(`braidfs version: ${require(`${__dirname}/package.json`).version}`)
+
 if (argv.length === 1 && argv[0].match(/^(run|serve)$/)) {
     return main()
 } else if (argv.length && argv.length % 2 == 0 && argv.every((x, i) => i % 2 != 0 || x.match(/^(sync|unsync)$/))) {
@@ -94,28 +124,82 @@ You can run it with:
 }
 
 async function main() {
+    process.on("unhandledRejection", (x) => console.log(`unhandledRejection: ${x.stack}`))
+    process.on("uncaughtException", (x) => console.log(`uncaughtException: ${x.stack}`))
     require('http').createServer(async (req, res) => {
-        console.log(`${req.method} ${req.url}`)
+        try {
+            console.log(`${req.method} ${req.url}`)
 
-        if (req.url === '/favicon.ico') return
+            if (req.url === '/favicon.ico') return
 
-        if (req.socket.remoteAddress !== '127.0.0.1' && req.socket.remoteAddress !== '::1') {
-            res.writeHead(403, { 'Content-Type': 'text/plain' })
-            return res.end('Access denied: only accessible from localhost')
-        }
+            if (req.socket.remoteAddress !== '127.0.0.1' && req.socket.remoteAddress !== '::1') {
+                res.writeHead(403, { 'Content-Type': 'text/plain' })
+                return res.end('Access denied: only accessible from localhost')
+            }
 
-        // Free the CORS
-        free_the_cors(req, res)
-        if (req.method === 'OPTIONS') return
+            // Free the CORS
+            free_the_cors(req, res)
+            if (req.method === 'OPTIONS') return
 
-        var url = req.url.slice(1)
+            var url = req.url.slice(1)
 
-        if (url !== '.braidfs/config' && url !== '.braidfs/errors') {
-            res.writeHead(404, { 'Content-Type': 'text/html' })
-            return res.end('Nothing to see here. You can go to <a href=".braidfs/config">.braidfs/config</a> or <a href=".braidfs/errors">.braidfs/errors</a>')
-        }
+            var m = url.match(/^\.braidfs\/get_version\/([^\/]*)\/([^\/]*)/)
+            if (m) {
+                var fullpath = decodeURIComponent(m[1])
+                var hash = decodeURIComponent(m[2])
+
+                var path = require('path').relative(proxy_base, fullpath)
+                var proxy = await proxy_url.cache[normalize_url(path)]
+                var version = proxy?.hash_to_version_cache.get(hash)?.version
+                return res.end(JSON.stringify(version ?? null))
+            }
+
+            var m = url.match(/^\.braidfs\/set_version\/([^\/]*)\/([^\/]*)/)
+            if (m) {
+                var fullpath = decodeURIComponent(m[1])
+                var path = require('path').relative(proxy_base, fullpath)
+                var proxy = await proxy_url.cache[normalize_url(path)]
+
+                var parents = JSON.parse(decodeURIComponent(m[2]))
+                var parent_text = (await braid_text.get(proxy.url,
+                    { parents })).body
+
+                var text = await new Promise(done => {
+                    const chunks = []
+                    req.on('data', chunk => chunks.push(chunk))
+                    req.on('end', () => done(Buffer.concat(chunks).toString()))
+                })
+
+                var patches = diff(parent_text, text)
 
-        braid_text.serve(req, res, { key: normalize_url(url) })
+                if (patches.length) {
+                    var peer = Math.random().toString(36).slice(2)
+                    var char_count = patches_to_code_points(patches, parent_text)
+                    var version = [peer + "-" + char_count]
+                    await braid_text.put(proxy.url, { version, parents, patches, peer, merge_type: 'dt' })
+
+                    // may be able to do this more efficiently.. we want to make sure we're capturing a file write that is after our version was written.. there may be a way we can avoid calling file_needs_writing here
+                    var stat = await new Promise(done => {
+                        proxy.file_written_cbs.push(done)
+                        proxy.signal_file_needs_writing()
+                    })
+
+                    res.writeHead(200, { 'Content-Type': 'application/json' })
+                    return res.end(stat.mtimeMs.toString())
+                } else return res.end('null')
+            }
+
+            if (url !== '.braidfs/config' && url !== '.braidfs/errors') {
+                res.writeHead(404, { 'Content-Type': 'text/html' })
+                return res.end('Nothing to see here. You can go to <a href=".braidfs/config">.braidfs/config</a> or <a href=".braidfs/errors">.braidfs/errors</a>')
+            }
+
+            braid_text.serve(req, res, { key: normalize_url(url) })
+        } catch (e) {
+            console.log(`e = ${e.stack}`)
+            res.writeHead(500, { 'Error-Message': '' + e })
+            res.end('' + e)
+        }
     }).listen(config.port, () => {
         console.log(`daemon started on port ${config.port}`)
         if (!config.allow_remote_access) console.log('!! only accessible from localhost !!')
@@ -263,6 +347,9 @@ async function scan_files() {
     scan_files.timeout = setTimeout(scan_files, config.scan_interval_ms ?? (20 * 1000))
 
     async function f(fullpath) {
+        path = require('path').relative(proxy_base, fullpath)
+        if (skip_file(path)) return
+
         let stat = await require('fs').promises.stat(fullpath, { bigint: true })
         if (stat.isDirectory()) {
             let found
@@ -270,9 +357,6 @@ async function scan_files() {
                 found ||= await f(`${fullpath}/${file}`)
             return found
         } else {
-            path = require('path').relative(proxy_base, fullpath)
-            if (skip_file(path)) return
-
             var proxy = await proxy_url.cache[normalize_url(path)]
             if (!proxy) return await trash_file(fullpath, path)
 
@@ -343,7 +427,7 @@ async function proxy_url(url) {
         }
         await old_unproxy
 
-        var self = {}
+        var self = {url}
 
         console.log(`proxy_url: ${url}`)
 
@@ -373,6 +457,23 @@ async function proxy_url(url) {
         var file_needs_reading = true,
             file_needs_writing = null,
             file_loop_pump_lock = 0
+        self.file_written_cbs = []
+
+        // store a recent mapping of content-hashes to their versions,
+        // to support the command line: braidfs editing filename < file
+        self.hash_to_version_cache = new Map()
+        function add_to_version_cache(text, version) {
+            var hash = sha256(text)
+            self.hash_to_version_cache.delete(hash)
+            self.hash_to_version_cache.set(hash, { version, time: Date.now() })
+
+            var too_old = Date.now() - 30000
+            for (var [key, value] of self.hash_to_version_cache) {
+                if (value.time > too_old ||
+                    self.hash_to_version_cache.size <= 1) break
+                self.hash_to_version_cache.delete(key)
+            }
+        }
 
         self.signal_file_needs_reading = () => {
             if (freed) return
@@ -380,7 +481,7 @@ async function proxy_url(url) {
             file_loop_pump()
         }
 
-        function signal_file_needs_writing() {
+        self.signal_file_needs_writing = () => {
             if (freed) return
             file_needs_writing = true
             file_loop_pump()
@@ -434,7 +535,7 @@ async function proxy_url(url) {
                         file_needs_writing = !v_eq(file_last_version, (await braid_text.get(url, {})).version)
 
                         // sanity check
-                        if (file_last_digest && require('crypto').createHash('sha256').update(self.file_last_text).digest('base64') != file_last_digest) throw new Error('file_last_text does not match file_last_digest')
+                        if (file_last_digest && sha256(self.file_last_text) != file_last_digest) throw new Error('file_last_text does not match file_last_digest')
                     } else if (await require('fs').promises.access(fullpath).then(() => 1, () => 0)) {
                         // file exists, but not meta file
                         file_last_version = []
@@ -479,12 +580,16 @@ async function proxy_url(url) {
                             var parents = file_last_version
                             file_last_version = version
 
+                            add_to_version_cache(text, version)
+
                             send_out({ version, parents, patches, peer })
 
                             await braid_text.put(url, { version, parents, patches, peer, merge_type: 'dt' })
 
-                            await require('fs').promises.writeFile(meta_path, JSON.stringify({ version: file_last_version, digest: require('crypto').createHash('sha256').update(self.file_last_text).digest('base64') }))
+                            await require('fs').promises.writeFile(meta_path, JSON.stringify({ version: file_last_version, digest: sha256(self.file_last_text) }))
                         } else {
+                            add_to_version_cache(text, file_last_version)
+
                             console.log(`no changes found in: ${fullpath}`)
                             if (stat_eq(stat, self.file_last_stat)) {
                                 if (Date.now() > (self.file_ignore_until ?? 0))
@@ -511,6 +616,8 @@ async function proxy_url(url) {
 
                             console.log(`writing file ${fullpath}`)
 
+                            add_to_version_cache(body, version)
+
                             try { if (await is_read_only(fullpath)) await set_read_only(fullpath, false) } catch (e) { }
 
                             file_last_version = version
@@ -521,8 +628,7 @@ async function proxy_url(url) {
 
                             await require('fs').promises.writeFile(meta_path, JSON.stringify({
                                 version: file_last_version,
-                                digest: require('crypto').createHash('sha256')
-                                    .update(self.file_last_text).digest('base64')
+                                digest: sha256(self.file_last_text)
                             }))
                         }
 
@@ -532,6 +638,9 @@ async function proxy_url(url) {
                         }
 
                         self.file_last_stat = await require('fs').promises.stat(fullpath, { bigint: true })
+
+                        for (var cb of self.file_written_cbs) cb(self.file_last_stat)
+                        self.file_written_cbs = []
                     }
                 }
             })
@@ -568,7 +677,7 @@ async function proxy_url(url) {
                         console.log(`  editable = ${res.headers.get('editable')}`)
 
                         self.file_read_only = res.headers.get('editable') === 'false'
-                        signal_file_needs_writing()
+                        self.signal_file_needs_writing()
                     }
                 },
                 heartbeats: 120,
@@ -593,7 +702,7 @@ async function proxy_url(url) {
                     await braid_text.put(url, { ...update, peer, merge_type: 'dt' })
 
 
-                    signal_file_needs_writing()
+                    self.signal_file_needs_writing()
                     finish_something()
                 }, e => (e?.name !== "AbortError") && crash(e))
             }).catch(e => (e?.name !== "AbortError") && crash(e))
@@ -628,20 +737,23 @@ async function proxy_url(url) {
                     merge_type: 'dt',
                     peer,
                     subscribe: async (u) => {
-                        if (u.version.length) chain = chain.then(() => send_out({...u, peer}))
+                        if (u.version.length) {
+                            self.signal_file_needs_writing()
+                            chain = chain.then(() => send_out({...u, peer}))
+                        }
                     },
                 })
             } catch (e) {
                 if (e?.name !== "AbortError") crash(e)
             }
-            finish_something()            
+            finish_something()
         }
 
         // for config and errors file, listen for web changes
         if (!is_external_link) braid_text.get(url, braid_text_get_options = {
             merge_type: 'dt',
             peer,
-            subscribe: signal_file_needs_writing,
+            subscribe: self.signal_file_needs_writing,
         })
 
         return self
@@ -818,3 +930,7 @@ async function file_exists(fullpath) {
         return x.isFile()
     } catch (e) { }
 }
+
+function sha256(x) {
+    return require('crypto').createHash('sha256').update(x).digest('base64')
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "braidfs",
-  "version": "0.0.57",
+  "version": "0.0.58",
   "description": "braid technology synchronizing files and webpages",
   "author": "Braid Working Group",
   "repository": "braid-org/braidfs",