@helia/verified-fetch 0.0.0-f58d467 → 1.0.0

package/README.md

@@ -4,18 +4,35 @@
   </a>
 </p>
 
+# @helia/verified-fetch
+
 [![ipfs.tech](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](https://ipfs.tech)
 [![Discuss](https://img.shields.io/discourse/https/discuss.ipfs.tech/posts.svg?style=flat-square)](https://discuss.ipfs.tech)
-[![codecov](https://img.shields.io/codecov/c/github/ipfs/helia.svg?style=flat-square)](https://codecov.io/gh/ipfs/helia)
-[![CI](https://img.shields.io/github/actions/workflow/status/ipfs/helia/main.yml?branch=main\&style=flat-square)](https://github.com/ipfs/helia/actions/workflows/main.yml?query=branch%3Amain)
+[![codecov](https://img.shields.io/codecov/c/github/ipfs/helia-verified-fetch.svg?style=flat-square)](https://codecov.io/gh/ipfs/helia-verified-fetch)
+[![CI](https://img.shields.io/github/actions/workflow/status/ipfs/helia-verified-fetch/js-test-and-release.yml?branch=main\&style=flat-square)](https://github.com/ipfs/helia-verified-fetch/actions/workflows/js-test-and-release.yml?query=branch%3Amain)
 
-> A fetch-like API for obtaining verified & trustless IPFS content on the web.
+> A fetch-like API for obtaining verified & trustless IPFS content on the web
 
 # About
 
+<!--
+
+!IMPORTANT!
+
+Everything in this README between "# About" and "# Install" is automatically
+generated and will be overwritten the next time the doc generator is run.
+
+To make changes to this section, please update the @packageDocumentation section
+of src/index.js or src/index.ts
+
+To experiment with formatting, please run "npm run docs" from the root of this
+repo and examine the changes made.
+
+-->
+
 `@helia/verified-fetch` provides a [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API)-like API for retrieving content from the [IPFS](https://ipfs.tech/) network.
 
-All content is retrieved in a [trustless manner](https://www.techopedia.com/definition/trustless), and the integrity of all bytes are verified by comparing hashes of the data.
+All content is retrieved in a [trustless manner](https://www.techopedia.com/definition/trustless), and the integrity of all bytes are verified by comparing hashes of the data. By default, CIDs are retrieved over HTTP from [trustless gateways](https://specs.ipfs.tech/http-gateways/trustless-gateway/).
 
 This is a marked improvement over `fetch` which offers no such protections and is vulnerable to all sorts of attacks like [Content Spoofing](https://owasp.org/www-community/attacks/Content_Spoofing), [DNS Hijacking](https://en.wikipedia.org/wiki/DNS_hijacking), etc.
 
@@ -45,7 +62,7 @@ const json = await resp.json()
 import { verifiedFetch } from '@helia/verified-fetch'
 import { CID } from 'multiformats/cid'
 
-const cid = CID.parse('bafyFoo') // some image file
+const cid = CID.parse('bafyFoo') // some json file
 const response = await verifiedFetch(cid)
 const json = await response.json()
 ```
@@ -68,7 +85,7 @@ document.body.appendChild(image)
 import { verifiedFetch } from '@helia/verified-fetch'
 
 const response = await verifiedFetch('ipns://mydomain.com/path/to/very-long-file.log')
-const bigFileStreamReader = await response.body.getReader()
+const bigFileStreamReader = await response.body?.getReader()
 ```
 
 ## Configuration
@@ -85,8 +102,8 @@ It's possible to override these by passing `gateways` and `routers` keys to the
 import { createVerifiedFetch } from '@helia/verified-fetch'
 
 const fetch = await createVerifiedFetch({
- gateways: ['https://trustless-gateway.link'],
- routers: ['http://delegated-ipfs.dev']
+  gateways: ['https://trustless-gateway.link'],
+  routers: ['http://delegated-ipfs.dev']
 })
 
 const resp = await fetch('ipfs://bafy...')
@@ -110,13 +127,13 @@ import { createVerifiedFetch } from '@helia/verified-fetch'
 
 const fetch = await createVerifiedFetch(
   await createHeliaHTTP({
-      blockBrokers: [
-        trustlessGateway({
-          gateways: ['https://mygateway.example.net', 'https://trustless-gateway.link']
-        })
-      ],
-      routers: ['http://delegated-ipfs.dev'].map((routerUrl) => delegatedHTTPRouting(routerUrl))
-    })
+    blockBrokers: [
+      trustlessGateway({
+        gateways: ['https://mygateway.example.net', 'https://trustless-gateway.link']
+      })
+    ],
+    routers: ['http://delegated-ipfs.dev'].map((routerUrl) => delegatedHTTPRouting(routerUrl))
+  })
 )
 
 const resp = await fetch('ipfs://bafy...')
@@ -126,7 +143,7 @@ const json = await resp.json()
 
 ### Custom content-type parsing
 
-By default, `@helia/verified-fetch` sets the `Content-Type` header as `application/octet-stream` - this is because the `.json()`, `.text()`, `.blob()`, and `.arrayBuffer()` methods will usually work as expected without a detailed content type.
+By default, if the response can be parsed as JSON, `@helia/verified-fetch` sets the `Content-Type` header as `application/json`, otherwise it sets it as `application/octet-stream` - this is because the `.json()`, `.text()`, `.blob()`, and `.arrayBuffer()` methods will usually work as expected without a detailed content type.
 
 If you require an accurate content-type you can provide a `contentTypeParser` function as an option to `createVerifiedFetch` to handle parsing the content type.
 
@@ -139,13 +156,256 @@ import { createVerifiedFetch } from '@helia/verified-fetch'
 import { fileTypeFromBuffer } from '@sgtpooki/file-type'
 
 const fetch = await createVerifiedFetch({
- gateways: ['https://trustless-gateway.link'],
- routers: ['http://delegated-ipfs.dev'],
- contentTypeParser: async (bytes) => {
-   // call to some magic-byte recognition library like magic-bytes, file-type, or your own custom byte recognition
-   return fileTypeFromBuffer(bytes)?.mime
- }
+  gateways: ['https://trustless-gateway.link'],
+  routers: ['http://delegated-ipfs.dev']
+}, {
+  contentTypeParser: async (bytes) => {
+    // call to some magic-byte recognition library like magic-bytes, file-type, or your own custom byte recognition
+    const result = await fileTypeFromBuffer(bytes)
+    return result?.mime
+  }
+})
+```
+
+### Custom DNS resolvers
+
+If you don't want to leak DNS queries to the default resolvers, you can provide your own list of DNS resolvers to `createVerifiedFetch`.
+
+Note that you do not need to provide both a DNS-over-HTTPS and a DNS-over-JSON resolver, and you should prefer `dnsJsonOverHttps` resolvers for usage in the browser for a smaller bundle size. See <https://github.com/ipfs/helia/tree/main/packages/ipns#example---using-dns-json-over-https> for more information.
+
+## Example - Customizing DNS resolvers
+
+```typescript
+import { createVerifiedFetch } from '@helia/verified-fetch'
+import { dnsJsonOverHttps, dnsOverHttps } from '@helia/ipns/dns-resolvers'
+
+const fetch = await createVerifiedFetch({
+  gateways: ['https://trustless-gateway.link'],
+  routers: ['http://delegated-ipfs.dev'],
+  dnsResolvers: [
+    dnsJsonOverHttps('https://my-dns-resolver.example.com/dns-json'),
+    dnsOverHttps('https://my-dns-resolver.example.com/dns-query')
+  ]
+})
+```
+
+### IPLD codec handling
+
+IPFS supports several data formats (typically referred to as codecs) which are included in the CID. `@helia/verified-fetch` attempts to abstract away some of the details for easier consumption.
+
+#### DAG-PB
+
+[DAG-PB](https://ipld.io/docs/codecs/known/dag-pb/) is the codec we are most likely to encounter, it is what [UnixFS](https://github.com/ipfs/specs/blob/main/UNIXFS.md) uses under the hood.
+
+##### Using the DAG-PB codec as a Blob
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://Qmfoo')
+const blob = await res.blob()
+
+console.info(blob) // Blob { size: x, type: 'application/octet-stream' }
+```
+
+##### Using the DAG-PB codec as an ArrayBuffer
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://Qmfoo')
+const buf = await res.arrayBuffer()
+
+console.info(buf) // ArrayBuffer { [Uint8Contents]: < ... >, byteLength: x }
+```
+
+##### Using the DAG-PB codec as a stream
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://Qmfoo')
+const reader = res.body?.getReader()
+
+if (reader == null) {
+  throw new Error('Could not create reader from response body')
+}
+
+while (true) {
+  const next = await reader.read()
+
+  if (next?.done === true) {
+    break
+  }
+
+  if (next?.value != null) {
+    console.info(next.value) // Uint8Array(x) [ ... ]
+  }
+}
+```
+
+##### Content-Type
+
+When fetching `DAG-PB` data, the content type will be set to `application/octet-stream` unless a custom content-type parser is configured.
+
+#### JSON
+
+The JSON codec is a very simple codec, a block parseable with this codec is a JSON string encoded into a `Uint8Array`.
+
+##### Using the JSON codec
+
+```typescript
+import * as json from 'multiformats/codecs/json'
+
+const block = new TextEncoder().encode('{ "hello": "world" }')
+const obj = json.decode(block)
+
+console.info(obj) // { hello: 'world' }
+```
+
+##### Content-Type
+
+When the `JSON` codec is encountered, the `Content-Type` header of the response will be set to `application/json`.
+
+### DAG-JSON
+
+[DAG-JSON](https://ipld.io/docs/codecs/known/dag-json/) expands on the `JSON` codec, adding the ability to contain [CID](https://docs.ipfs.tech/concepts/content-addressing/)s which act as links to other blocks, and byte arrays.
+
+`CID`s and byte arrays are represented using special object structures with a single `"/"` property.
+
+Using `DAG-JSON` has two important caveats:
+
+1. Your `JSON` structure cannot contain an object with only a `"/"` property, as it will be interpreted as a special type.
+2. Since `JSON` has no technical limit on number sizes, `DAG-JSON` also allows numbers larger than `Number.MAX_SAFE_INTEGER`. JavaScript requires use of `BigInt`s to represent numbers larger than this, and `JSON.parse` does not support them, so precision will be lost.
+
+Otherwise this codec follows the same rules as the `JSON` codec.
+
+##### Using the DAG-JSON codec
+
+```typescript
+import * as dagJson from '@ipld/dag-json'
+
+const block = new TextEncoder().encode(`{
+  "hello": "world",
+  "cid": {
+    "/": "baeaaac3imvwgy3zao5xxe3de"
+  },
+  "buf": {
+    "/": {
+      "bytes": "AAECAwQ"
+    }
+  }
+}`)
+
+const obj = dagJson.decode(block)
+
+console.info(obj)
+// {
+// hello: 'world',
+// cid: CID(baeaaac3imvwgy3zao5xxe3de),
+// buf: Uint8Array(5) [ 0, 1, 2, 3, 4 ]
+// }
+```
+
+##### Content-Type
+
+When the `DAG-JSON` codec is encountered in the requested CID, the `Content-Type` header of the response will be set to `application/json`.
+
+`DAG-JSON` data can be parsed from the response by using the `.json()` function, which will return `CID`s/byte arrays as plain `{ "/": ... }` objects:
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+import * as dagJson from '@ipld/dag-json'
+
+const res = await verifiedFetch('ipfs://bafyDAGJSON')
+
+// either:
+const obj = await res.json()
+console.info(obj.cid) // { "/": "baeaaac3imvwgy3zao5xxe3de" }
+console.info(obj.buf) // { "/": { "bytes": "AAECAwQ" } }
+```
+
+Alternatively, it can be decoded using the `@ipld/dag-json` module and the `.arrayBuffer()` method, in which case you will get `CID` objects and `Uint8Array`s:
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+import * as dagJson from '@ipld/dag-json'
+
+const res = await verifiedFetch('ipfs://bafyDAGJSON')
+
+// or:
+const obj = dagJson.decode<any>(await res.arrayBuffer())
+console.info(obj.cid) // CID(baeaaac3imvwgy3zao5xxe3de)
+console.info(obj.buf) // Uint8Array(5) [ 0, 1, 2, 3, 4 ]
+```
+
+#### DAG-CBOR
+
+[DAG-CBOR](https://ipld.io/docs/codecs/known/dag-cbor/) uses the [Concise Binary Object Representation](https://cbor.io/) format for serialization instead of JSON.
+
+This supports more datatypes in a safer way than JSON and is smaller on the wire to boot so is usually preferable to JSON or DAG-JSON.
+
+##### Content-Type
+
+Not all data types supported by `DAG-CBOR` can be successfully turned into JSON and back into the same binary form.
+
+When a decoded block can be round-tripped to JSON, the `Content-Type` will be set to `application/json`. In this case the `.json()` method on the `Response` object can be used to obtain an object representation of the response.
+
+When it cannot, the `Content-Type` will be `application/octet-stream` - in this case the `@ipld/dag-json` module must be used to deserialize the return value from `.arrayBuffer()`.
+
+##### Detecting JSON-safe DAG-CBOR
+
+If the `Content-Type` header of the response is `application/json`, the `.json()` method may be used to access the response body in object form, otherwise the `.arrayBuffer()` method must be used to decode the raw bytes using the `@ipld/dag-cbor` module.
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+import * as dagCbor from '@ipld/dag-cbor'
+
+const res = await verifiedFetch('ipfs://bafyDagCborCID')
+let obj
+
+if (res.headers.get('Content-Type') === 'application/json') {
+  // DAG-CBOR data can be safely decoded as JSON
+  obj = await res.json()
+} else {
+  // response contains non-JSON friendly data types
+  obj = dagCbor.decode(await res.arrayBuffer())
+}
+
+console.info(obj) // ...
+```
+
+## The `Accept` header
+
+The `Accept` header can be passed to override certain response processing, or to ensure that the final `Content-Type` of the response is the one that is expected.
+
+If the final `Content-Type` does not match the `Accept` header, or if the content cannot be represented in the format dictated by the `Accept` header, or you have configured a custom content type parser, and that parser returns a value that isn't in the accept header, a [406: Not Acceptable](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/406) response will be returned:
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://bafyJPEGImageCID', {
+  headers: {
+    accept: 'image/png'
+  }
+})
+
+console.info(res.status) // 406 - the image was a JPEG but we specified PNG as the accept header
+```
+
+It can also be used to skip processing the data from some formats such as `DAG-CBOR` if you wish to handle decoding it yourself:
+
+```typescript
+import { verifiedFetch } from '@helia/verified-fetch'
+
+const res = await verifiedFetch('ipfs://bafyDAGCBORCID', {
+  headers: {
+    accept: 'application/octet-stream'
+  }
 })
+
+console.info(res.headers.get('accept')) // application/octet-stream
+const buf = await res.arrayBuffer() // raw bytes, not processed as JSON
 ```
 
 ## Comparison to fetch
@@ -165,7 +425,7 @@ This library supports the following methods of fetching web3 content from IPFS:
 2. IPNS protocol: `ipns://<peerId>` & `ipns://<publicKey>` & `ipns://<hostUri_Supporting_DnsLink_TxtRecords>`
 3. CID instances: An actual CID instance `CID.parse('bafy...')`
 
-As well as support for pathing & params for item 1 & 2 above according to [IPFS - Path Gateway Specification](https://specs.ipfs.tech/http-gateways/path-gateway) & [IPFS - Trustless Gateway Specification](https://specs.ipfs.tech/http-gateways/trustless-gateway/). Further refinement of those specifications specifically for web-based scenarios can be found in the [Web Pathing Specification IPIP](https://github.com/ipfs/specs/pull/453).
+As well as support for pathing & params for items 1 & 2 above according to [IPFS - Path Gateway Specification](https://specs.ipfs.tech/http-gateways/path-gateway) & [IPFS - Trustless Gateway Specification](https://specs.ipfs.tech/http-gateways/trustless-gateway/). Further refinement of those specifications specifically for web-based scenarios can be found in the [Web Pathing Specification IPIP](https://github.com/ipfs/specs/pull/453).
 
 If you pass a CID instance, it assumes you want the content for that specific CID only, and does not support pathing or params for that CID.
 
@@ -277,7 +537,7 @@ Loading this module through a script tag will make it's exports available as `He
 
 # API Docs
 
-- <https://ipfs.github.io/helia/modules/_helia_verified_fetch.html>
+- <https://ipfs.github.io/helia-verified-fetch/modules/_helia_verified_fetch.html>
 
 # License
 
@@ -288,7 +548,7 @@ Licensed under either of
 
 # Contribute
 
-Contributions welcome! Please check out [the issues](https://github.com/ipfs/helia/issues).
+Contributions welcome! Please check out [the issues](https://github.com/ipfs/helia-verified-fetch/issues).
 
 Also see our [contributing document](https://github.com/ipfs/community/blob/master/CONTRIBUTING_JS.md) for more information on how we work, and about contributing in general.