hypercore-fetch 8.3.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 RangerMauve
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,226 @@
+# hypercore-fetch
+
+Implementation of Fetch that uses the Hyper SDK for loading p2p content
+
+`npm install --save hypercore-fetch`
+
+```javascript
+const fetch = require('hypercore-fetch')()
+
+const someURL = `hyper://blog.mauve.moe`
+
+const response = await fetch(`${someURL}/index.json`)
+
+const json = await response.json()
+
+console.log(json)
+```
+
+You can also use the bundled CLI
+
+```
+npm i -g hypercore-fetch
+
+hypercore-fetch hyper://somethingorother
+
+# Or
+
+npx hypercore-fetch hyper://somethingorother
+```
+
+## API
+
+### `makeFetch({Hyperdrive, resolveURL, base, session, writable}) => fetch()`
+
+Creates a hypercore-fetch instance.
+
+The `base` parameter can be used to specify what the base URL is for relative paths like `fetch('./dat.json')`.
+
+You can pass in options for the [Dat SDK](https://github.com/datproject/sdk) to have it be auto-created,
+or you can pass in both a function matching  `const archive = Hyperdrive(key)` and a `const resolved = await resolveName(url)` function (where `resolved` is an instance of URL, uses hyper-dns by default).
+
+Set `session` to your Electron session if you want to enable setting the `body` of fetch requests to Electron's [UploadData](https://www.electronjs.org/docs/api/structures/upload-data) API in their protocol handlers.
+
+If you don't want to allow write access to archives, pass in `writable: false`.
+
+Typically, you don't need to pass in any of these and they're there for more advanced users.
+
+After you've created it, `fetch` will be have like it does in [browsers](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API).
+
+### `await fetch.close()`
+
+Closes resources for the Dat SDK. This does nothing if you specified the Hyperdrive and `resolveName` options.
+
+### Common Headers
+
+Each response will contain a header for the canonical URL represented as a `Link` header with `rel=canonical`.
+
+Each response will also contain the `Allow` header of all the methods currently allowed. If the archive is writable, this will contain `PUT`.
+
+There is also an `ETag` header which will be a JSON string containging the drive's current `version`. This will change only when the drive has gotten an update of some sort and is monotonically incrementing.
+
+### `fetch('hyper://NAME/example.txt', {method: 'GET'})`
+
+This will attempt to load `example.txt` from the archive labeled by `NAME`.
+
+It will also load `index.html` files automatically for a folder.
+You can find the details about how resolution works in the [resolve-dat-path](https://github.com/RangerMauve/resolve-dat-path/blob/master/index.js#L3) module.
+
+`NAME` can either be the 64 character hex key for an archive, a domain to parse with [dat-dns](https://www.npmjs.com/package/dat-dns), or a name for an archive which allows you to write to it.
+
+The response headers will contain `X-Blocks` for the number of blocks of data this file represents on disk, and `X-Blocks-Downloaded` which is the number of blocks from this file that have been downloaded locally.
+
+### `fetch('hyper://NAME/.well-known/dat', {method: 'GET'})`
+
+This is used by the dat-dns module for resoving dns domains to `dat://` URLs.
+
+This will return some text which will have a `dat://` URL of your archive, followed by a newline and a TTL for the DNS record.
+
+### `fetch('hyper://NAME/example/', {method: 'GET'})`
+
+When doing a `GET` on a directory, you will get a directory listing.
+
+By default it will return a JSON array of files and folders in that directory.
+
+You can differentiate a folder from files by the fact that it ends with a `/`.
+
+You can set the `Accept` header to `text/html` in order to have it return a basic HTML page with links to files and folders in that directory.
+
+e.g.
+
+```json
+["example.txt", "posts/", "example2.md"]
+```
+
+Files in the directory will be listed under their name, sub-directories will have a `/` appended to them.
+
+`NAME` can either be the 64 character hex key for an archive, a domain to parse with [dat-dns](https://www.npmjs.com/package/dat-dns), or a name for an archive which allows you to write to it.
+
+### `fetch('hyper://NAME/example/?noResolve', {method: 'GET'})`
+
+Adding `?noResolve` to a URL will prevent resolving `index.html` files and will attempt to load the path as is.
+This can be useful for list files in a directory that would normally render as a page.
+
+`NAME` can either be the 64 character hex key for an archive, a domain to parse with [dat-dns](https://www.npmjs.com/package/dat-dns), or a name for an archive which allows you to write to it.
+
+The response headers will contain `X-Blocks` for the number of blocks of data this file represents on disk, and `X-Blocks-Downloaded` which is the number of blocks from this file that have been downloaded locally.
+
+### `fetch('hyper://NAME/', {headers: {'Accept': 'text/event-stream'}})`
+
+Using the `text/event-stream` content type in the `Accept` header will get back an event stream full of `change` events for every time a file at that path changes.
+
+This can be useful if you want to trigger a download every time a file changes.
+The `data` for the event will contain the version at the time of the change.
+
+This stream of data can be used with the `EventSource` in browsers.
+
+Currently there's no way to watch for changes to specific files, so that should be handled at the application level.
+
+You can also watch for the `download` and `upload` events which will be emitted whenever you download or upload blocks from the hyperdrive.
+
+The `data` for the event will contain a JSON encoded object with the `index` of the block, and the `source` which is the public key of the hypercore (either the metadata of the hyperdrive, or the content feed).
+
+### `fetch('hyper://NAME/example.txt', {method: 'PUT', body: 'Hello World'})`
+
+You can add files to archives using a `PUT` method along with a `body`.
+
+The `body` can be either a `String`, an `ArrayBuffer`, a `Blob`, a WHATWG `ReadableStream`, a Node.js `Stream`, or electron's [UploadData](https://www.electronjs.org/docs/api/structures/upload-data) object (make sure to specify the `session` argument in the `makeFetch` function for electron support).
+
+`NAME` can either be the 64 character hex key for an archive, a domain to parse with [dat-dns](https://www.npmjs.com/package/dat-dns), or a name for an archive which allows you to write to it.
+
+Your `NAME` will likely be a `name` in most cases to ensure you have a writeable archive.
+
+### `fetch('hyper://NAME/example.txt', {method: 'DELETE'})`
+
+You can delete a file in an archive by using the `DELETE` method.
+
+You cannot delete directories if they are not empty.
+
+`NAME` can either be the 64 character hex key for an archive, a domain to parse with [dat-dns](https://www.npmjs.com/package/dat-dns), or a name for an archive which allows you to write to it.
+
+### `fetch('hyper://NAME/example.txt', {method: 'GET', headers: {'x-download': 'cache'}})`
+
+You can download a file or an entire folder to the local cache using the `x-download` header set to `cache` in a `GET` request.
+
+`NAME` can either be the 64 character hex key for an archive, a domain to parse with [dat-dns](https://www.npmjs.com/package/dat-dns), or a name for an archive which allows you to write to it.
+
+You can use `/` for the path to download the entire contents
+
+### `fetch('hyper://NAME/example.txt', {method: 'DELETE', headers: {'x-clear': 'cache'}})`
+
+You can clear the data stored in the local cache for a file or folder using the `x-clear` header set to `cache` in a `DELETE` request..
+
+This is like the opposite of using `x-download` to download data.
+
+This does not delete data, it only deletes the cached data from disk.
+
+`NAME` can either be the 64 character hex key for an archive, a domain to parse with [dat-dns](https://www.npmjs.com/package/dat-dns), or a name for an archive which allows you to write to it.
+
+You can use `/` for the path to clear all data for the archive.
+
+### `fetch('hyper://NAME/$/tags/TAG_NAME', {method: 'PUT'})`
+
+You can add a tag a version of the archive with a human readable name (like SPAGHETTI), in the example represented as `tagName` by doing a PUT into the special `/$/tags/` folder.
+
+Afterwards you can load the archive at that given version with `hyper://NAME+TAG_NAME`.
+
+E.g.
+
+`PUT hyper://123kjh213kjh123/$/tags/v4.20`
+`GET hyper://123kjh213kjh123+v4.20/example.txt`
+
+### `fetch('hyper://NAME/$/tags/', {method: 'GET'})`
+
+You can get a list of all tags by doing a `GET` on the `/$/tags/` folder.
+
+The response will be a JSON object which maps tag names to archive versions.
+
+Use `await response.json()` to get the data out.
+
+e.g.
+
+```json
+{
+  "tagOne": 1,
+  "example": 100000
+}
+```
+
+### `fetch('hyper://NAME/$/tags/TAG_NAME', {method: 'DELETE'})`
+
+You can delete a given tag with the `DELETE` method on a name within the special `$/tags/` folder.
+
+Specify the tag you want in the URL, and it'll be removed from the tags list.
+
+### `fetch('hyper://NAME/$/extensions/')`
+
+You can list the current [hypercore extensions](https://github.com/hypercore-protocol/hypercore#ext--feedregisterextensionname-handlers) that are enabled by doing a `GET` on the `/$/extensions/` directory.
+
+This will give you a directory listing with the names of all the extensions.
+
+### `fetch('hyper://NAME/$/extensions/EXTENSION_NAME')`
+
+You can list the peers that you are replication with which have registered this extension by doing a `GET` to the directory for the extension.
+
+This is also how you can register an extension that hasn't been registered yet.
+
+The list will be a JSON array with objects that contain the fields `remotePublicKey`, `remoteAddress`, `remoteType`, and `stats`
+
+### `fetch('hyper://NAME/$/extensions/', {headers: {'Accept': 'text/event-stream'}})`
+
+Using the `text/event-stream` content type in the `Accept` header will get back an event stream with the extension events.
+
+The `event` will be the name of the extension you got the data for, the `id` (accessible by `e.lastEventId` in EventSource) will be set to the ID of the peer that sent it.
+
+### `fetch('hyper://NAME/$/extensions/EXTENSION_NAME', {method: 'POST', body: 'Example'})`
+
+You can broadcast an extension message to all peers that are replicating that extension type with a `POST` to the extension's URL.
+
+The `body` of the request will be used as the payload. Please note that only utf8 encoded text is currently supported due to limitations of the event-stream encoding.
+
+### `fetch('hyper://NAME/$/extensions/EXTENSION_NAME/REMOTE_PUBLIC_KEY', {method: 'POST', body: 'Example'})`
+
+You can send an extension message to a specific peer by doing a `POST` to the extension with their remote public key ID.
+
+The `body` of the request will be used as the payload. Please note that only utf8 encoded text is currently supported due to limitations of the event-stream encoding.
+

package/bin.js

@@ -0,0 +1,27 @@
+#!/usr/bin/env node
+
+const eosp = require('end-of-stream-promise')
+const { Readable } = require('streamx')
+
+run()
+  .catch((e) => process.nextTick(() => {
+    throw e
+  }))
+
+async function run () {
+  const fetch = require('./')()
+
+  try {
+    const url = process.argv[2]
+
+    const response = await fetch(url)
+
+    const stream = Readable.from(response.body)
+
+    stream.pipe(process.stdout)
+
+    await eosp(stream)
+  } finally {
+    fetch.close()
+  }
+}

package/index.js

@@ -0,0 +1,722 @@
+const resolveDatPath = require('resolve-dat-path')
+const Headers = require('fetch-headers')
+const mime = require('mime/lite')
+const SDK = require('hyper-sdk')
+const parseRange = require('range-parser')
+const makeDir = require('make-dir')
+const { Readable } = require('streamx')
+const makeFetch = require('make-fetch')
+const { EventIterator } = require('event-iterator')
+
+const DEFAULT_TIMEOUT = 5000
+
+const NUMBER_REGEX = /^\d+$/
+const PROTOCOL_REGEX = /^\w+:\/\//
+const NOT_WRITABLE_ERROR = 'Archive not writable'
+
+const READABLE_ALLOW = ['GET', 'HEAD']
+const WRITABLE_ALLOW = ['PUT', 'POST', 'DELETE']
+const ALL_ALLOW = READABLE_ALLOW.concat(WRITABLE_ALLOW)
+
+const SPECIAL_FOLDER = '/$/'
+const TAGS_FOLDER_NAME = 'tags/'
+const TAGS_FOLDER = SPECIAL_FOLDER + TAGS_FOLDER_NAME
+const EXTENSIONS_FOLDER_NAME = 'extensions/'
+const EXTENSIONS_FOLDER = SPECIAL_FOLDER + EXTENSIONS_FOLDER_NAME
+const EXTENSION_EVENT = 'extension-message'
+
+// TODO: Add caching support
+const { resolveURL: DEFAULT_RESOLVE_URL } = require('hyper-dns')
+
+module.exports = function makeHyperFetch (opts = {}) {
+  let {
+    Hyperdrive,
+    resolveURL = DEFAULT_RESOLVE_URL,
+    base,
+    timeout = DEFAULT_TIMEOUT,
+    writable = false
+  } = opts
+
+  let sdk = null
+  let gettingSDK = null
+  let onClose = async () => undefined
+
+  const isSourceDat = base && base.startsWith('hyper://')
+
+  const fetch = makeFetch(hyperFetch)
+
+  fetch.close = () => onClose()
+
+  function getExtension (archive, name) {
+    const existing = archive.metadata.extensions.get(name)
+    if (existing) return existing
+
+    const extension = archive.registerExtension(name, {
+      encoding: 'utf8',
+      onmessage: (content, peer) => {
+        archive.emit(EXTENSION_EVENT, name, content, peer)
+      }
+    })
+
+    return extension
+  }
+
+  function getExtensionPeers (archive, name) {
+    // List peers with this extension
+    const allPeers = archive.peers
+    return allPeers.filter((peer) => {
+      const { remoteExtensions } = peer
+
+      if (!remoteExtensions) return false
+
+      const { names } = remoteExtensions
+
+      if (!names) return false
+
+      return names.includes(name)
+    })
+  }
+
+  function listExtensionNames (archive) {
+    return archive.metadata.extensions.names()
+  }
+
+  async function loadArchive (key) {
+    const Hyperdrive = await getHyperdrive()
+    return Hyperdrive(key)
+  }
+
+  return fetch
+
+  async function hyperFetch ({ url, headers: rawHeaders, method, signal, body }) {
+    const isHyperURL = url.startsWith('hyper://')
+    const urlHasProtocol = url.match(PROTOCOL_REGEX)
+
+    const shouldIntercept = isHyperURL || (!urlHasProtocol && isSourceDat)
+
+    if (!shouldIntercept) throw new Error('Invalid protocol, must be hyper://')
+
+    const headers = new Headers(rawHeaders || {})
+
+    const responseHeaders = {}
+    responseHeaders['Access-Control-Allow-Origin'] = '*'
+    responseHeaders['Allow-CSP-From'] = '*'
+    responseHeaders['Access-Control-Allow-Headers'] = '*'
+
+    try {
+      let { pathname: path, key, version, searchParams } = parseDatURL(url)
+      if (!path) path = '/'
+      if (!path.startsWith('/')) path = '/' + path
+
+      try {
+        const resolvedURL = await resolveURL(`hyper://${key}`)
+        key = resolvedURL.hostname
+      } catch (e) {
+        // Probably a domain that couldn't resolve
+        if (key.includes('.')) throw e
+      }
+
+      let archive = await loadArchive(key)
+
+      if (!archive) {
+        return {
+          statusCode: 404,
+          headers: responseHeaders,
+          data: intoAsyncIterable('Unknown drive')
+        }
+      }
+
+      await archive.ready()
+
+      if (!archive.version) {
+        if (!archive.peers.length) {
+          await new Promise((resolve, reject) => {
+            setTimeout(() => reject(new Error('Timed out looking for peers')), timeout)
+            archive.once('peer-open', resolve)
+          })
+        }
+        await new Promise((resolve, reject) => {
+          archive.metadata.update({ ifAvailable: true }, (err) => {
+            if (err) reject(err)
+            else resolve()
+          })
+        })
+      }
+
+      if (version) {
+        if (NUMBER_REGEX.test(version)) {
+          archive = await archive.checkout(version)
+        } else {
+          archive = await archive.checkout(await archive.getTaggedVersion(version))
+        }
+        await archive.ready()
+      }
+
+      const canonical = `hyper://${archive.key.toString('hex')}${path || ''}`
+      responseHeaders.Link = `<${canonical}>; rel="canonical"`
+
+      const isWritable = writable && archive.writable
+      const allowHeaders = isWritable ? ALL_ALLOW : READABLE_ALLOW
+      responseHeaders.Allow = allowHeaders.join(', ')
+
+      // We can say the file hasn't changed if the drive version hasn't changed
+      responseHeaders.ETag = `"${archive.version}"`
+
+      if (path.startsWith(SPECIAL_FOLDER)) {
+        if (path === SPECIAL_FOLDER) {
+          const files = [
+            TAGS_FOLDER_NAME,
+            EXTENSIONS_FOLDER_NAME
+          ]
+
+          const data = renderFiles(headers, responseHeaders, url, path, files)
+          if (method === 'HEAD') {
+            return {
+              statusCode: 204,
+              headers: responseHeaders,
+              data: intoAsyncIterable('')
+            }
+          } else {
+            return {
+              statusCode: 200,
+              headers: responseHeaders,
+              data
+            }
+          }
+        } else if (path.startsWith(TAGS_FOLDER)) {
+          if (method === 'GET') {
+            if (path === TAGS_FOLDER) {
+              responseHeaders['x-is-directory'] = 'true'
+              const tags = await archive.getAllTags()
+              const tagsObject = Object.fromEntries(tags)
+              const json = JSON.stringify(tagsObject, null, '\t')
+
+              responseHeaders['Content-Type'] = 'application/json; charset=utf-8'
+
+              return {
+                statusCode: 200,
+                headers: responseHeaders,
+                data: intoAsyncIterable(json)
+              }
+            } else {
+              const tagName = path.slice(TAGS_FOLDER.length)
+              try {
+                const tagVersion = await archive.getTaggedVersion(tagName)
+
+                return {
+                  statusCode: 200,
+                  headers: responseHeaders,
+                  data: intoAsyncIterable(`${tagVersion}`)
+                }
+              } catch {
+                return {
+                  statusCode: 404,
+                  headers: responseHeaders,
+                  data: intoAsyncIterable('Tag Not Found')
+                }
+              }
+            }
+          } else if (method === 'DELETE') {
+            checkWritable(archive)
+            const tagName = path.slice(TAGS_FOLDER.length)
+            await archive.deleteTag(tagName || version)
+            responseHeaders.ETag = `"${archive.version}"`
+
+            return {
+              statusCode: 200,
+              headers: responseHeaders,
+              data: intoAsyncIterable('')
+            }
+          } else if (method === 'PUT') {
+            checkWritable(archive)
+            const tagName = path.slice(TAGS_FOLDER.length)
+            const tagVersion = archive.version
+
+            await archive.createTag(tagName, tagVersion)
+            responseHeaders['Content-Type'] = 'text/plain; charset=utf-8'
+            responseHeaders.ETag = `"${archive.version}"`
+
+            return {
+              statusCode: 200,
+              headers: responseHeaders,
+              data: intoAsyncIterable(`${tagVersion}`)
+            }
+          } else if (method === 'HEAD') {
+            return {
+              statusCode: 204,
+              headers: responseHeaders,
+              data: intoAsyncIterable('')
+            }
+          } else {
+            return {
+              statusCode: 405,
+              headers: responseHeaders,
+              data: intoAsyncIterable('Method Not Allowed')
+            }
+          }
+        } else if (path.startsWith(EXTENSIONS_FOLDER)) {
+          if (path === EXTENSIONS_FOLDER) {
+            if (method === 'GET') {
+              const accept = headers.get('Accept') || ''
+              if (!accept.includes('text/event-stream')) {
+                responseHeaders['x-is-directory'] = 'true'
+
+                const extensions = listExtensionNames(archive)
+                const data = renderFiles(headers, responseHeaders, url, path, extensions)
+
+                return {
+                  statusCode: 204,
+                  headers: responseHeaders,
+                  data
+                }
+              }
+
+              const events = new EventIterator(({ push }) => {
+                function onMessage (name, content, peer) {
+                  const id = peer.remotePublicKey.toString('hex')
+                  // TODO: Fancy verification on the `name`?
+                  // Send each line of content separately on a `data` line
+                  const data = content.split('\n').map((line) => `data:${line}\n`).join('')
+                  push(`id:${id}\nevent:${name}\n${data}\n`)
+                }
+                archive.on(EXTENSION_EVENT, onMessage)
+                return () => {
+                  archive.removeListener('extension-message', onMessage)
+                }
+              })
+
+              responseHeaders['Content-Type'] = 'text/event-stream'
+
+              return {
+                statusCode: 200,
+                headers: responseHeaders,
+                data: events
+              }
+            } else {
+              return {
+                statusCode: 405,
+                headers: responseHeaders,
+                data: intoAsyncIterable('Method Not Allowed')
+              }
+            }
+          } else {
+            let extensionName = path.slice(EXTENSIONS_FOLDER.length)
+            let extensionPeer = null
+            if (extensionName.includes('/')) {
+              const split = extensionName.split('/')
+              extensionName = split[0]
+              if (split[1]) extensionPeer = split[1]
+            }
+            if (method === 'POST') {
+              const extension = getExtension(archive, extensionName)
+              if (extensionPeer) {
+                const peers = getExtensionPeers(archive, extensionName)
+                const peer = peers.find(({ remotePublicKey }) => remotePublicKey.toString('hex') === extensionPeer)
+                if (!peer) {
+                  return {
+                    statusCode: 404,
+                    headers: responseHeaders,
+                    data: intoAsyncIterable('Peer Not Found')
+                  }
+                }
+                extension.send(await collect(body), peer)
+              } else {
+                extension.broadcast(await collect(body))
+              }
+              return {
+                statusCode: 200,
+                headers: responseHeaders,
+                data: intoAsyncIterable('')
+              }
+            } else if (method === 'GET') {
+              const accept = headers.get('Accept') || ''
+              if (!accept.includes('text/event-stream')) {
+                // Load up the extension into memory
+                getExtension(archive, extensionName)
+
+                const extensionPeers = getExtensionPeers(archive, extensionName)
+                const finalPeers = formatPeers(extensionPeers)
+
+                const json = JSON.stringify(finalPeers, null, '\t')
+
+                return {
+                  statusCode: 200,
+                  header: responseHeaders,
+                  data: intoAsyncIterable(json)
+                }
+              }
+            } else {
+              return {
+                statusCode: 405,
+                headers: responseHeaders,
+                data: intoAsyncIterable('Method Not Allowed')
+              }
+            }
+          }
+        } else {
+          return {
+            statusCode: 404,
+            headers: responseHeaders,
+            data: intoAsyncIterable('Not Found')
+          }
+        }
+      }
+
+      if (method === 'PUT') {
+        checkWritable(archive)
+        if (path.endsWith('/')) {
+          await makeDir(path, { fs: archive })
+        } else {
+          const parentDir = path.split('/').slice(0, -1).join('/')
+          if (parentDir) {
+            await makeDir(parentDir, { fs: archive })
+          }
+
+          const source = Readable.from(body)
+          const destination = archive.createWriteStream(path)
+          // The sink is needed because Hyperdrive's write stream is duplex
+
+          source.pipe(destination)
+
+          await Promise.race([
+            once(source, 'error'),
+            once(destination, 'error'),
+            once(source, 'end')
+          ])
+        }
+        responseHeaders.ETag = `"${archive.version}"`
+
+        return {
+          statusCode: 200,
+          headers: responseHeaders,
+          data: intoAsyncIterable('')
+        }
+      } else if (method === 'DELETE') {
+        if (headers.get('x-clear') === 'cache') {
+          await archive.clear(path)
+          return {
+            statusCode: 200,
+            headers: responseHeaders,
+            data: intoAsyncIterable('')
+          }
+        } else {
+          checkWritable(archive)
+
+          const stats = await archive.stat(path)
+          // Weird stuff happening up in here...
+          const stat = Array.isArray(stats) ? stats[0] : stats
+
+          if (stat.isDirectory()) {
+            await archive.rmdir(path)
+          } else {
+            await archive.unlink(path)
+          }
+          responseHeaders.ETag = `"${archive.version}"`
+
+          return {
+            statusCode: 200,
+            headers: responseHeaders,
+            data: intoAsyncIterable('')
+          }
+        }
+      } else if ((method === 'GET') || (method === 'HEAD')) {
+        if (method === 'GET' && headers.get('Accept') === 'text/event-stream') {
+          const contentFeed = await archive.getContent()
+          const events = new EventIterator(({ push, fail }) => {
+            const watcher = archive.watch(path, () => {
+              const event = 'change'
+              const data = archive.version
+              push({ event, data })
+            })
+            watcher.on('error', fail)
+            function onDownloadMetadata (index) {
+              const event = 'download'
+              const source = archive.metadata.key.toString('hex')
+              const data = { index, source }
+              push({ event, data })
+            }
+            function onUploadMetadata (index) {
+              const event = 'download'
+              const source = archive.metadata.key.toString('hex')
+              const data = { index, source }
+              push({ event, data })
+            }
+
+            function onDownloadContent (index) {
+              const event = 'download'
+              const source = contentFeed.key.toString('hex')
+              const data = { index, source }
+              push({ event, data })
+            }
+            function onUploadContent (index) {
+              const event = 'download'
+              const source = contentFeed.key.toString('hex')
+              const data = { index, source }
+              push({ event, data })
+            }
+
+            // TODO: Filter out indexes that don't belong to files?
+
+            archive.metadata.on('download', onDownloadMetadata)
+            archive.metadata.on('upload', onUploadMetadata)
+            contentFeed.on('download', onDownloadContent)
+            contentFeed.on('upload', onUploadMetadata)
+            return () => {
+              watcher.destroy()
+              archive.metadata.removeListener('download', onDownloadMetadata)
+              archive.metadata.removeListener('upload', onUploadMetadata)
+              contentFeed.removeListener('download', onDownloadContent)
+              contentFeed.removeListener('upload', onUploadContent)
+            }
+          })
+          async function * startReader () {
+            for await (const { event, data } of events) {
+              yield `event:${event}\ndata:${JSON.stringify(data)}\n\n`
+            }
+          }
+
+          responseHeaders['Content-Type'] = 'text/event-stream'
+
+          return {
+            statusCode: 200,
+            headers: responseHeaders,
+            data: startReader()
+          }
+        }
+
+        let stat = null
+        let finalPath = path
+
+        if (headers.get('x-download') === 'cache') {
+          await archive.download(path)
+        }
+
+        // Legacy DNS spec from Dat protocol: https://github.com/datprotocol/DEPs/blob/master/proposals/0005-dns.md
+        if (finalPath === '/.well-known/dat') {
+          const { key } = archive
+          const entry = `dat://${key.toString('hex')}\nttl=3600`
+          return {
+            statusCode: 200,
+            headers: responseHeaders,
+            data: intoAsyncIterable(entry)
+          }
+        }
+
+        // New spec from hyper-dns https://github.com/martinheidegger/hyper-dns
+        if (finalPath === '/.well-known/hyper') {
+          const { key } = archive
+          const entry = `hyper://${key.toString('hex')}\nttl=3600`
+          return {
+            statusCode: 200,
+            headers: responseHeaders,
+            data: intoAsyncIterable(entry)
+          }
+        }
+        try {
+          if (searchParams.has('noResolve')) {
+            const stats = await archive.stat(path)
+            stat = stats[0]
+          } else {
+            const resolved = await resolveDatPath(archive, path)
+            finalPath = resolved.path
+            stat = resolved.stat
+          }
+        } catch (e) {
+          responseHeaders['Content-Type'] = 'text/plain; charset=utf-8'
+          return {
+            statusCode: 404,
+            headers: responseHeaders,
+            data: intoAsyncIterable(e.stack)
+          }
+        }
+
+        responseHeaders['Content-Type'] = getMimeType(finalPath)
+        responseHeaders['Last-Modified'] = stat.mtime.toUTCString()
+
+        let data = null
+        const isRanged = headers.get('Range') || headers.get('range')
+        let statusCode = 200
+
+        if (stat.isDirectory()) {
+          responseHeaders['x-is-directory'] = 'true'
+          const stats = await archive.readdir(finalPath, { includeStats: true })
+          const files = stats.map(({ stat, name }) => (stat.isDirectory() ? `${name}/` : name))
+
+          // Add special directory
+          if (finalPath === '/') files.unshift('$/')
+
+          data = renderFiles(headers, responseHeaders, url, path, files)
+        } else {
+          responseHeaders['Accept-Ranges'] = 'bytes'
+
+          try {
+            const { blocks, downloadedBlocks } = await archive.stats(finalPath)
+            responseHeaders['X-Blocks'] = `${blocks}`
+            responseHeaders['X-Blocks-Downloaded'] = `${downloadedBlocks}`
+          } catch (e) {
+            // Don't worry about it, it's optional.
+          }
+
+          const { size } = stat
+          responseHeaders['Content-Length'] = `${size}`
+
+          if (isRanged) {
+            const ranges = parseRange(size, isRanged)
+            if (ranges && ranges.length && ranges.type === 'bytes') {
+              statusCode = 206
+              const [{ start, end }] = ranges
+              const length = (end - start + 1)
+              responseHeaders['Content-Length'] = `${length}`
+              responseHeaders['Content-Range'] = `bytes ${start}-${end}/${size}`
+              if (method !== 'HEAD') {
+                data = archive.createReadStream(finalPath, {
+                  start,
+                  end
+                })
+              }
+            } else {
+              if (method !== 'HEAD') {
+                data = archive.createReadStream(finalPath)
+              }
+            }
+          } else if (method !== 'HEAD') {
+            data = archive.createReadStream(finalPath)
+          }
+        }
+
+        if (method === 'HEAD') {
+          return {
+            statusCode: 204,
+            headers: responseHeaders,
+            data: intoAsyncIterable('')
+          }
+        } else {
+          return {
+            statusCode,
+            headers: responseHeaders,
+            data
+          }
+        }
+      } else {
+        return {
+          statusCode: 405,
+          headers: responseHeaders,
+          data: intoAsyncIterable('Method Not Allowed')
+        }
+      }
+    } catch (e) {
+      const isUnauthorized = (e.message === NOT_WRITABLE_ERROR)
+      const statusCode = isUnauthorized ? 403 : 500
+      const statusText = isUnauthorized ? 'Not Authorized' : 'Server Error'
+      return {
+        statusCode,
+        statusText,
+        headers: responseHeaders,
+        data: intoAsyncIterable(e.stack)
+      }
+    }
+  }
+
+  function getHyperdrive () {
+    if (Hyperdrive) return Hyperdrive
+    return getSDK().then(({ Hyperdrive }) => Hyperdrive)
+  }
+
+  function getSDK () {
+    if (sdk) return Promise.resolve(sdk)
+    if (gettingSDK) return gettingSDK
+    gettingSDK = SDK(opts).then((gotSDK) => {
+      sdk = gotSDK
+      gettingSDK = null
+      onClose = async () => sdk.close()
+      Hyperdrive = sdk.Hyperdrive
+
+      return sdk
+    })
+
+    return gettingSDK
+  }
+
+  function checkWritable (archive) {
+    if (!writable) throw new Error(NOT_WRITABLE_ERROR)
+    if (!archive.writable) {
+      throw new Error(NOT_WRITABLE_ERROR)
+    }
+  }
+}
+
+function parseDatURL (url) {
+  const parsed = new URL(url)
+  let key = parsed.hostname
+  let version = null
+  if (key.includes('+')) [key, version] = key.split('+')
+
+  parsed.key = key
+  parsed.version = version
+
+  return parsed
+}
+
+async function * intoAsyncIterable (data) {
+  yield Buffer.from(data)
+}
+
+function getMimeType (path) {
+  let mimeType = mime.getType(path) || 'text/plain; charset=utf-8'
+  if (mimeType.startsWith('text/')) mimeType = `${mimeType}; charset=utf-8`
+  return mimeType
+}
+
+function renderDirectory (url, path, files) {
+  return `<!DOCTYPE html>
+<title>${url}</title>
+<meta name="viewport" content="width=device-width, initial-scale=1" />
+<h1>Index of ${path}</h1>
+<ul>
+  <li><a href="../">../</a></li>${files.map((file) => `
+  <li><a href="${file}">./${file}</a></li>
+`).join('')}
+</ul>
+`
+}
+
+function renderFiles (headers, responseHeaders, url, path, files) {
+  if (headers.get('Accept') && headers.get('Accept').includes('text/html')) {
+    const page = renderDirectory(url, path, files)
+    responseHeaders['Content-Type'] = 'text/html; charset=utf-8'
+    return intoAsyncIterable(page)
+  } else {
+    const json = JSON.stringify(files, null, '\t')
+    responseHeaders['Content-Type'] = 'application/json; charset=utf-8'
+    return intoAsyncIterable(json)
+  }
+}
+
+function once (ee, name) {
+  return new Promise((resolve, reject) => {
+    const isError = name === 'error'
+    const cb = isError ? reject : resolve
+    ee.once(name, cb)
+  })
+}
+
+async function collect (source) {
+  let buffer = ''
+
+  for await (const chunk of source) {
+    buffer += chunk
+  }
+
+  return buffer
+}
+
+function formatPeers (peers) {
+  return peers.map(({ remotePublicKey, remoteAddress, remoteType, stats }) => {
+    return {
+      remotePublicKey: remotePublicKey.toString('hex'),
+      remoteType,
+      remoteAddress,
+      stats
+    }
+  })
+}

package/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "hypercore-fetch",
+  "version": "8.3.2",
+  "description": "Implementation of Fetch that uses the Dat SDK for loading p2p content",
+  "bin": {
+    "hypercore-fetch": "bin.js"
+  },
+  "main": "index.js",
+  "scripts": {
+    "test": "node test"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/RangerMauve/hypercore-fetch.git"
+  },
+  "keywords": [
+    "dat",
+    "fetch"
+  ],
+  "author": "RangerMauve",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/RangerMauve/hypercore-fetch/issues"
+  },
+  "homepage": "https://github.com/RangerMauve/hypercore-fetch#readme",
+  "dependencies": {
+    "event-iterator": "^2.0.0",
+    "fetch-headers": "^2.0.0",
+    "hyper-dns": "^0.12.0",
+    "hyper-sdk": "^3.0.8",
+    "make-dir": "^3.1.0",
+    "make-fetch": "^2.2.1",
+    "mime": "^2.4.4",
+    "range-parser": "^1.2.1",
+    "resolve-dat-path": "^2.0.0",
+    "sodium-universal": "^3.0.2",
+    "streamx": "^2.10.0"
+  },
+  "devDependencies": {
+    "random-access-memory": "^3.1.1",
+    "tape": "^5.2.2"
+  }
+}

package/test.js

@@ -0,0 +1,261 @@
+const SDK = require('hyper-sdk')
+const test = require('tape')
+
+runTests()
+
+const SAMPLE_CONTENT = 'Hello World'
+
+async function runTests () {
+  const { Hyperdrive, close } = await SDK({
+    persist: false
+  })
+
+  const { Hyperdrive: Hyperdrive2, close: close2 } = await SDK({
+    persist: false
+  })
+
+  const fetch = require('./')({
+    Hyperdrive,
+    writable: true
+  })
+
+  const fetch2 = require('./')({
+    Hyperdrive: Hyperdrive2,
+    writable: true
+  })
+
+  test.onFinish(() => {
+    close()
+    close2()
+  })
+
+  test('Read index.html', async (t) => {
+    const archive = Hyperdrive('example1')
+
+    const FILE_LOCATION = '/index.html'
+    const FILE_DATA = '<h1>Hello World!</h1>'
+
+    await archive.writeFile(FILE_LOCATION, FILE_DATA)
+
+    const url = `hyper://${archive.key.toString('hex')}${FILE_LOCATION}`
+
+    t.pass('Prepped archive ' + url)
+
+    const response = await fetch(url)
+
+    t.pass('got response')
+
+    const text = await response.text()
+
+    t.pass('got response text')
+
+    const contentType = response.headers.get('content-type')
+
+    t.equal(contentType, 'text/html; charset=utf-8')
+    t.equal(text, FILE_DATA)
+    t.pass('Headers ' + [...response.headers.entries()])
+  })
+
+  test('GET .well-known/dat', async (t) => {
+    const response = await fetch('hyper://example/.well-known/dat')
+    t.ok(response, 'Got response')
+    t.equal(response.status, 200, 'Got OK response code')
+    const text = await response.text()
+    t.ok(text.startsWith('dat://'), 'Returned dat URL')
+  })
+
+  test('GET .well-known/hyper', async (t) => {
+    const response = await fetch('hyper://example/.well-known/hyper')
+    t.ok(response, 'Got response')
+    t.equal(response.status, 200, 'Got OK response code')
+    const text = await response.text()
+    t.ok(text.startsWith('hyper://'), 'Returned dat URL')
+  })
+
+  test('PUT file', async (t) => {
+    const response1 = await fetch('hyper://example/checkthis.txt', { method: 'PUT', body: SAMPLE_CONTENT })
+
+    t.equal(response1.status, 200, 'Got OK response on write')
+
+    const response2 = await fetch('hyper://example/checkthis.txt')
+
+    t.equal(response2.status, 200, 'Got OK response on read')
+
+    t.equal(await response2.text(), SAMPLE_CONTENT, 'Read back written data')
+  })
+
+  test('PUT directory', async (t) => {
+    const response1 = await fetch('hyper://example/foo/bar/', { method: 'PUT' })
+
+    t.equal(response1.status, 200, 'Got OK response on directory creation')
+  })
+
+  test('PUT file in new directory', async (t) => {
+    const response1 = await fetch('hyper://example/fizz/buzz/example.txt', { method: 'PUT', body: SAMPLE_CONTENT })
+
+    t.equal(response1.status, 200, 'Got OK response on directory/file creation')
+  })
+
+  test('PUT to overwrite a file', async (t) => {
+    const response1 = await fetch('hyper://example/baz/index.html', { method: 'PUT', body: SAMPLE_CONTENT })
+    t.ok(response1.ok)
+    const response2 = await fetch('hyper://example/baz/index.html', { method: 'PUT', body: SAMPLE_CONTENT })
+
+    t.equal(response2.status, 200, 'Got OK response on file overwrite')
+  })
+
+  test('DELETE file', async (t) => {
+    const response1 = await fetch('hyper://example/test.txt', { method: 'PUT', body: SAMPLE_CONTENT })
+    t.ok(response1.ok)
+
+    const response2 = await fetch('hyper://example/test.txt', { method: 'DELETE' })
+
+    t.equal(response2.status, 200, 'Got OK response on file delete')
+
+    const response3 = await fetch('hyper://example/test.txt', { method: 'GET' })
+
+    t.equal(response3.status, 404, 'Got not found on deleted file')
+  })
+
+  test('GET index.html', async (t) => {
+    const response1 = await fetch('hyper://example/baz')
+
+    t.equal(await response1.text(), SAMPLE_CONTENT, 'Got index.html content')
+
+    const response2 = await fetch('hyper://example/baz?noResolve')
+
+    t.equal(response2.headers.get('content-type'), 'application/json; charset=utf-8', 'noResolve flag yields JSON by default')
+    t.deepEqual(await response2.json(), ['index.html'], 'Listed directory')
+
+    const response3 = await fetch('hyper://example/baz?noResolve')
+    t.equal(response3.headers.get('content-type'), 'application/json; charset=utf-8', 'noResolve flag yields JSON by default')
+    t.deepEqual(await response3.json(), ['index.html'], 'Listed directory')
+  })
+
+  test('Create and read tags', async (t) => {
+    await fetch('hyper://example/test.txt', { method: 'PUT', body: SAMPLE_CONTENT })
+
+    const response2 = await fetch('hyper://example/$/tags/tag1', { method: 'PUT' })
+    t.ok(response2.ok, 'Able to create tag')
+
+    const version = await response2.json()
+
+    const response3 = await fetch('hyper://example/$/tags/')
+
+    t.ok(response3.ok, 'Able to ask for tags')
+    t.deepEqual(await response3.json(), { tag1: version }, 'Tag got created')
+
+    // Insert a file which won't be available with the old tag
+    await fetch('hyper://example/notaccessible.txt', { method: 'PUT', body: 'test' })
+
+    const response4 = await fetch('hyper://example+tag1/notaccessible.txt')
+
+    t.equal(response4.status, 404, 'Newer file not found in older tag')
+
+    const response5 = await fetch('hyper://example/$/tags/tag1', { method: 'DELETE' })
+
+    t.ok(response5.ok, 'Able to delete tag')
+
+    const response6 = await fetch('hyper://example/$/tags/')
+
+    t.deepEqual(await response6.json(), {}, 'No tags left after delete')
+  })
+
+  test('Load Mauve\'s blog', async (t) => {
+    const response = await fetch('hyper://blog.mauve.moe/')
+
+    t.ok(response.ok, 'Succesfully loaded homepage')
+  })
+
+  test('Watch for changes', async (t) => {
+    const response = await fetch('hyper://example/', {
+      headers: {
+        Accept: 'text/event-stream'
+      }
+    })
+
+    t.ok(response.ok, 'Able to open request')
+    t.equal(response.headers.get('Content-Type'), 'text/event-stream', 'Response is event stream')
+
+    const reader = await response.body.getReader()
+
+    const [data] = await Promise.all([
+      reader.read(),
+      fetch('hyper://example/example4.txt', { method: 'PUT', body: 'Hello World' })
+    ])
+
+    t.ok(data.value, 'Got eventsource data after writing')
+    t.ok(data.value.includes('event:change'), 'Eventsource data represents a change event')
+    t.ok(data.value.endsWith('\n\n'), 'Ends with two newlines')
+
+    await reader.cancel()
+  })
+
+  test('Send extension from one peer to another', async (t) => {
+    const domainResponse = await fetch('hyper://example/.well-known/hyper')
+    const domain = (await domainResponse.text()).split('\n')[0]
+
+    const extensionURL = `${domain}/$/extensions/example`
+    const extensionListURL = `${domain}/$/extensions/`
+
+    // Load up extension message on peer 1
+    await fetch(extensionURL)
+    // Load up extension message on peer 2
+    await fetch2(extensionURL)
+
+    t.pass('Able to initialize extensions')
+
+    const extensionListRequest = await fetch(extensionListURL)
+    const extensionList = await extensionListRequest.json()
+
+    // Extension list will always be alphabetically sorted
+    t.deepEqual(extensionList, ['example', 'hypertrie'], 'Got expected list of extensions')
+
+    // Wait a bit for them to connect
+    // TODO: Peers API
+    await delay(2000)
+
+    const peerResponse1 = await fetch(extensionURL)
+    const peerList1 = await peerResponse1.json()
+
+    t.equal(peerList1.length, 1, 'Got one peer for extension message on peer1')
+
+    const peerResponse2 = await fetch2(extensionURL)
+    const peerList2 = await peerResponse2.json()
+
+    t.equal(peerList2.length, 1, 'Got one peer for extension message on peer2')
+
+    const eventRequest = await fetch(extensionListURL, {
+      headers: {
+        Accept: 'text/event-stream'
+      }
+    })
+
+    t.ok(eventRequest.ok, 'Able to open request')
+    t.equal(eventRequest.headers.get('Content-Type'), 'text/event-stream', 'Response is event stream')
+
+    const reader = await eventRequest.body.getReader()
+
+    const toRead = reader.read()
+
+    await delay(500)
+
+    const broadcastRequest = await fetch2(extensionURL, { method: 'POST', body: 'Hello World' })
+
+    t.ok(broadcastRequest.ok, 'Able to broadcast to peers')
+
+    const data = await toRead
+
+    t.ok(data.value, 'Got eventsource data after writing')
+    t.ok(data.value.includes('event:example\n'), 'EventSource data represents an example event')
+    t.ok(data.value.includes('data:Hello World\n'), 'EventSource data contains expected body')
+    t.ok(data.value.includes('id:'), 'EventSource data contains an ID')
+    t.ok(data.value.endsWith('\n\n'), 'Ends with two newlines')
+
+    await reader.cancel()
+  })
+}
+
+function delay (time) {
+  return new Promise((resolve) => setTimeout(resolve, time))
+}