@speckle/objectloader 2.1.1 → 2.3.17-alpha.2819

package/.eslintrc.cjs

@@ -0,0 +1,16 @@
+/**
+ * Extends repo root config, only put changes here that are scoped to this specific package
+ * (if you already are - evaluate whether you really need package scoped linting rules)
+ */
+
+/** @type {import("eslint").Linter.Config} */
+const config = {
+  env: {
+    browser: true
+  },
+  parserOptions: {
+    sourceType: 'module'
+  }
+}
+
+module.exports = config

package/LICENSE

@@ -0,0 +1,208 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2020 AEC Systems
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+   
+   NOTICE: Unless otherwise described, the code in this repository is 
+   licensed under the license above. Some modules, extensions or code herein 
+   might be otherwise licensed. This is indicated either in the root of the 
+   containing folder under a different license file, or in the respective 
+   file's header. If you have any questions, don't hesitate to get in touch 
+   with us via [email](mailto:hello@speckle.systems).

package/examples/browser/index.html

@@ -1,19 +1,21 @@
 <!DOCTYPE html>
 <html lang="en" class="no-js">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width" />
 
-<head>
-  <meta charset="UTF-8">
-  <meta name="viewport" content="width=device-width">
+    <title>Object Loader Test</title>
+  </head>
 
-  <title>Object Loader Test</title>
-</head>
+  <body>
+    <h1>This is a test.</h1>
+    <p>All the magic is in the console.</p>
+    <p>
+      To ensure this example runs correctly, please serve this file from a local http
+      server - if you manually open the file in a browser, it might not work.
+    </p>
+    <button onclick="loadData()">PRESS ME</button>
+  </body>
 
-<body>
-  <h1>This is a test.</h1>
-  <p>All the magic is in the console.</p>
-  <p>To ensure this example runs correctly, please serve this file from a local http server - if you manually open the file in a browser, it might not work.</p>
-  <button onclick="loadData()">PRESS ME</button>
-</body>
-  
   <script src="script.js" type="module"></script>
-</html>
+</html>

package/examples/browser/script.js

@@ -9,25 +9,23 @@ window.ObjectLoader = ObjectLoader
 
 // https://latest.speckle.dev/streams/92b620fb17/objects/7cd9d41b5b5f3c8908536aec2a05f1a1
 // let loader = new ObjectLoader({
-//   serverUrl:"https://latest.speckle.dev", 
-//   streamId:"92b620fb17", 
+//   serverUrl:"https://latest.speckle.dev",
+//   streamId:"92b620fb17",
 //   objectId:"878c426bb213ddb4d580da74922a2b16"
 // })
 
 // https://latest.speckle.dev/streams/3ed8357f29/objects/0408ab9caaa2ebefb2dd7f1f671e7555
-let loader = new ObjectLoader({
-  serverUrl:"https://latest.speckle.dev", 
-  streamId:"3ed8357f29", 
-  objectId:"0408ab9caaa2ebefb2dd7f1f671e7555"
+const loader = new ObjectLoader({
+  serverUrl: 'https://latest.speckle.dev',
+  streamId: '3ed8357f29',
+  objectId: '0408ab9caaa2ebefb2dd7f1f671e7555'
 })
 
-
 window.loadData = async function loadData() {
-  
-  let obj = await loader.getAndConstructObject((e) =>{
+  const obj = await loader.getAndConstructObject((e) => {
     console.log(e) // log progress!
   })
 
   console.log('Done!')
-  console.log( obj )
-}
+  console.log(obj)
+}

package/examples/node/script.js

@@ -1 +1,22 @@
-// NOTE: This lib is not working in node, because node-fetch returns node-native readable streams - we need a workaround first. 
+// Since Node v<18 does not provide fetch, we need to pass it in the options object. Note that fetch must return a WHATWG compliant stream, so cross-fetch won't work, but node/undici's implementation will.
+
+import { fetch } from 'undici'
+import ObjectLoader from '../../index.js'
+
+const loader = new ObjectLoader({
+  serverUrl: 'https://latest.speckle.dev',
+  streamId: '3ed8357f29',
+  objectId: '0408ab9caaa2ebefb2dd7f1f671e7555',
+  options: { enableCaching: false, excludeProps: [], fetch }
+})
+
+const loadData = async function loadData() {
+  const obj = await loader.getAndConstructObject((e) => {
+    console.log(e) // log progress!
+  })
+
+  console.log('Done!')
+  console.log(obj)
+}
+
+loadData()

package/index.js

@@ -1,34 +1,48 @@
 /**
- * Simple client that streams object info from a Speckle Server. 
+ * Simple client that streams object info from a Speckle Server.
  * TODO: Object construction progress reporting is weird.
  */
 
-
-
 export default class ObjectLoader {
-
   /**
    * Creates a new object loader instance.
-   * @param {*} param0 
+   * @param {*} param0
    */
-  constructor( { serverUrl, streamId, token, objectId, options = { fullyTraverseArrays: false, excludeProps: [ ] } } ) {
+  constructor({
+    serverUrl,
+    streamId,
+    token,
+    objectId,
+    options = {
+      enableCaching: true,
+      fullyTraverseArrays: false,
+      excludeProps: [],
+      fetch: null
+    }
+  }) {
     this.INTERVAL_MS = 20
     this.TIMEOUT_MS = 180000 // three mins
 
     this.serverUrl = serverUrl || window.location.origin
     this.streamId = streamId
     this.objectId = objectId
-    this.token = token || localStorage.getItem( 'AuthToken' )
+    console.log('Object loader constructor called!')
+    try {
+      this.token = token || localStorage.getItem('AuthToken')
+    } catch (error) {
+      // Accessing localStorage may throw when executing on sandboxed document, ignore.
+    }
 
     this.headers = {
-      'Accept': 'text/plain'
+      Accept: 'text/plain'
     }
 
-    if( token ) {
+    if (this.token) {
       this.headers['Authorization'] = `Bearer ${this.token}`
     }
 
-    this.requestUrl = `${this.serverUrl}/objects/${this.streamId}/${this.objectId}`
+    this.requestUrlRootObj = `${this.serverUrl}/objects/${this.streamId}/${this.objectId}/single`
+    this.requestUrlChildren = `${this.serverUrl}/api/getobjects/${this.streamId}`
     this.promises = []
     this.intervals = {}
     this.buffer = []
@@ -36,180 +50,478 @@ export default class ObjectLoader {
     this.totalChildrenCount = 0
     this.traversedReferencesCount = 0
     this.options = options
+    this.options.numConnections = this.options.numConnections || 4
+
+    this.cacheDB = null
+
+    this.lastAsyncPause = Date.now()
+    this.existingAsyncPause = null
+
+    // we can't simply bind fetch to this.fetch, so instead we have to do some acrobatics:
+    // https://stackoverflow.com/questions/69337187/uncaught-in-promise-typeerror-failed-to-execute-fetch-on-workerglobalscope#comment124731316_69337187
+    this.preferredFetch = options.fetch
+    this.fetch = function (...args) {
+      const currentFetch = this.preferredFetch || fetch
+      return currentFetch(...args)
+    }
+  }
+
+  async asyncPause() {
+    // Don't freeze the UI
+    // while ( this.existingAsyncPause ) {
+    //   await this.existingAsyncPause
+    // }
+    if (Date.now() - this.lastAsyncPause >= 100) {
+      this.lastAsyncPause = Date.now()
+      this.existingAsyncPause = new Promise((resolve) => setTimeout(resolve, 0))
+      await this.existingAsyncPause
+      this.existingAsyncPause = null
+      if (Date.now() - this.lastAsyncPause > 500)
+        console.log('Loader Event loop lag: ', Date.now() - this.lastAsyncPause)
+    }
   }
 
   dispose() {
     this.buffer = []
-    this.intervals.forEach( i => clearInterval( i.interval ) )
+    this.intervals.forEach((i) => clearInterval(i.interval))
   }
 
   /**
    * Use this method to receive and construct the object. It will return the full, de-referenced and de-chunked original object.
-   * @param {*} onProgress 
-   * @returns 
+   * @param {*} onProgress
+   * @returns
    */
-  async getAndConstructObject( onProgress ) {
-    
-    ;( await this.downloadObjectsInBuffer( onProgress ) ) // Fire and forget; PS: semicolon of doom
-    
-    let rootObject = await this.getObject( this.objectId )
-    return this.traverseAndConstruct( rootObject, onProgress )
+  async getAndConstructObject(onProgress) {
+    await this.downloadObjectsInBuffer(onProgress) // Fire and forget; PS: semicolon of doom
+
+    const rootObject = await this.getObject(this.objectId)
+    return this.traverseAndConstruct(rootObject, onProgress)
   }
 
   /**
    * Internal function used to download all the objects in a local buffer.
-   * @param {*} onProgress 
+   * @param {*} onProgress
    */
-  async downloadObjectsInBuffer( onProgress ) {
+  async downloadObjectsInBuffer(onProgress) {
     let first = true
     let downloadNum = 0
-    
-    for await ( let obj of this.getObjectIterator() ) {
-      if( first ) {
+
+    for await (const obj of this.getObjectIterator()) {
+      if (first) {
         this.totalChildrenCount = obj.totalChildrenCount
         first = false
         this.isLoading = true
       }
       downloadNum++
-      if( onProgress ) onProgress( { stage: 'download', current: downloadNum, total: this.totalChildrenCount } )
+      if (onProgress)
+        onProgress({
+          stage: 'download',
+          current: downloadNum,
+          total: this.totalChildrenCount
+        })
     }
     this.isLoading = false
   }
 
   /**
    * Internal function used to recursively traverse an object and populate its references and dechunk any arrays.
-   * @param {*} obj 
-   * @param {*} onProgress 
-   * @returns 
+   * @param {*} obj
+   * @param {*} onProgress
+   * @returns
    */
-  async traverseAndConstruct( obj, onProgress ) {
-    if( !obj ) return
-    if ( typeof obj !== 'object' ) return obj
+  async traverseAndConstruct(obj, onProgress) {
+    if (!obj) return
+    if (typeof obj !== 'object') return obj
 
     // Handle arrays
-    if ( Array.isArray( obj ) &&  obj.length !== 0 ) {
-      let arr = []
-      for ( let element of obj ) {
-        if ( typeof element !== 'object' && ! this.options.fullyTraverseArrays ) return obj
-        
+    if (Array.isArray(obj) && obj.length !== 0) {
+      const arr = []
+      for (const element of obj) {
+        if (typeof element !== 'object' && !this.options.fullyTraverseArrays) return obj
+
         // Dereference element if needed
-        let deRef = element.referencedId ? await this.getObject( element.referencedId ) : element
-        if( element.referencedId && onProgress ) onProgress( { stage: 'construction', current: ++this.traversedReferencesCount > this.totalChildrenCount ? this.totalChildrenCount : this.traversedReferencesCount, total: this.totalChildrenCount } ) 
-        
+        const deRef = element.referencedId
+          ? await this.getObject(element.referencedId)
+          : element
+        if (element.referencedId && onProgress)
+          onProgress({
+            stage: 'construction',
+            current:
+              ++this.traversedReferencesCount > this.totalChildrenCount
+                ? this.totalChildrenCount
+                : this.traversedReferencesCount,
+            total: this.totalChildrenCount
+          })
+
         // Push the traversed object in the array
-        arr.push( await this.traverseAndConstruct( deRef, onProgress ) )
+        arr.push(await this.traverseAndConstruct(deRef, onProgress))
       }
-      
+
       // De-chunk
-      if( arr[0]?.speckle_type?.toLowerCase().includes('datachunk') ) { 
-        return arr.reduce( ( prev, curr ) => prev.concat( curr.data ), [] )
+      if (arr[0]?.speckle_type?.toLowerCase().includes('datachunk')) {
+        return arr.reduce((prev, curr) => prev.concat(curr.data), [])
       }
-      
+
       return arr
-     }
+    }
 
     // Handle objects
     // 1) Purge ignored props
-    for( let ignoredProp of this.options.excludeProps ) {
-      delete obj[ ignoredProp ]
+    for (const ignoredProp of this.options.excludeProps) {
+      delete obj[ignoredProp]
     }
-    
+
     // 2) Iterate through obj
-    for( let prop in obj ) {      
-      if( typeof obj[prop] !== 'object' ) continue // leave alone primitive props
-      
-      if( obj[prop].referencedId ) {
-        obj[prop] = await this.getObject( obj[prop].referencedId )
-        if( onProgress ) onProgress( { stage: 'construction', current: ++this.traversedReferencesCount > this.totalChildrenCount ? this.totalChildrenCount : this.traversedReferencesCount, total: this.totalChildrenCount } )
+    for (const prop in obj) {
+      if (typeof obj[prop] !== 'object' || obj[prop] === null) continue // leave alone primitive props
+
+      if (obj[prop].referencedId) {
+        obj[prop] = await this.getObject(obj[prop].referencedId)
+        if (onProgress)
+          onProgress({
+            stage: 'construction',
+            current:
+              ++this.traversedReferencesCount > this.totalChildrenCount
+                ? this.totalChildrenCount
+                : this.traversedReferencesCount,
+            total: this.totalChildrenCount
+          })
       }
 
-      obj[prop] = await this.traverseAndConstruct( obj[prop], onProgress )
+      obj[prop] = await this.traverseAndConstruct(obj[prop], onProgress)
     }
 
-     return obj
+    return obj
   }
 
   /**
    * Internal function. Returns a promise that is resolved when the object id is loaded into the internal buffer.
-   * @param {*} id 
-   * @returns 
+   * @param {*} id
+   * @returns
    */
-  async getObject( id ){
-    if ( this.buffer[id] ) return this.buffer[id]
+  async getObject(id) {
+    if (this.buffer[id]) return this.buffer[id]
 
-    let promise = new Promise( ( resolve, reject ) => {
-      this.promises.push( { id, resolve, reject } )
+    const promise = new Promise((resolve, reject) => {
+      this.promises.push({ id, resolve, reject })
       // Only create a new interval checker if none is already present!
-      if ( this.intervals[id] ) {
+      if (this.intervals[id]) {
         this.intervals[id].elapsed = 0 // reset elapsed
       } else {
-        let intervalId = setInterval( this.tryResolvePromise.bind( this ), this.INTERVAL_MS, id )
+        const intervalId = setInterval(
+          this.tryResolvePromise.bind(this),
+          this.INTERVAL_MS,
+          id
+        )
         this.intervals[id] = { interval: intervalId, elapsed: 0 }
       }
-    } )
+    })
     return promise
   }
 
-  tryResolvePromise( id ) {
+  tryResolvePromise(id) {
     this.intervals[id].elapsed += this.INTERVAL_MS
-    if ( this.buffer[id] ) {
-      for ( let p of this.promises.filter( p => p.id === id ) ) {
-        p.resolve( this.buffer[id] )
+    if (this.buffer[id]) {
+      for (const p of this.promises.filter((p) => p.id === id)) {
+        p.resolve(this.buffer[id])
       }
 
-      clearInterval( this.intervals[id].interval )
+      clearInterval(this.intervals[id].interval)
       delete this.intervals[id]
       // this.promises = this.promises.filter( p => p.id !== p.id ) // clearing out promises too early seems to nuke loading
       return
     }
 
-    if ( this.intervals[id].elapsed > this.TIMEOUT_MS ) {
-      console.warn( `Timeout resolving ${id}. HIC SVNT DRACONES.` )
-      clearInterval( this.intervals[id].interval )
-      this.promises.filter( p => p.id === id ).forEach( p => p.reject() )
-      this.promises = this.promises.filter( p => p.id !== p.id ) // clear out
+    if (this.intervals[id].elapsed > this.TIMEOUT_MS) {
+      console.warn(`Timeout resolving ${id}. HIC SVNT DRACONES.`)
+      clearInterval(this.intervals[id].interval)
+      this.promises.filter((p) => p.id === id).forEach((p) => p.reject())
+      this.promises = this.promises.filter((p) => p.id !== p.id) // clear out
     }
   }
 
-  async * getObjectIterator(  ) {
-    for await ( let line of this.getRawObjectIterator() ) {
-      let { id, obj } = this.processLine( line )
-      this.buffer[ id ] = obj
+  async *getObjectIterator() {
+    const t0 = Date.now()
+    let count = 0
+    for await (const line of this.getRawObjectIterator()) {
+      const { id, obj } = this.processLine(line)
+      this.buffer[id] = obj
+      count += 1
       yield obj
     }
+    console.log(`Loaded ${count} objects in: ${(Date.now() - t0) / 1000}`)
   }
 
-  processLine( chunk ) {
-    var pieces = chunk.split( '\t' )
-    return { id: pieces[0], obj: JSON.parse( pieces[1] ) }
+  processLine(chunk) {
+    const pieces = chunk.split('\t')
+    return { id: pieces[0], obj: JSON.parse(pieces[1]) }
   }
 
-  async * getRawObjectIterator() {
-    const decoder = new TextDecoder()
-    const response = await fetch( this.requestUrl, { headers: this.headers } )
-    const reader = response.body.getReader()
-    let { value: chunk, done: readerDone } = await reader.read()
-    chunk = chunk ? decoder.decode( chunk ) : ''
-
-    let re = /\r\n|\n|\r/gm
-    let startIndex = 0
-
-    while ( true ) {
-      let result = re.exec( chunk )
-      if ( !result ) {
-        if ( readerDone ) break
-        let remainder = chunk.substr( startIndex ) 
-        ;( { value: chunk, done: readerDone } = await reader.read() ) // PS: semicolon of doom
-        chunk = remainder + ( chunk ? decoder.decode( chunk ) : '' )
-        startIndex = re.lastIndex = 0
+  async *getRawObjectIterator() {
+    if (this.options.enableCaching && window.indexedDB && this.cacheDB === null) {
+      await safariFix()
+      const idbOpenRequest = indexedDB.open('speckle-object-cache', 1)
+      idbOpenRequest.onupgradeneeded = () =>
+        idbOpenRequest.result.createObjectStore('objects')
+      this.cacheDB = await this.promisifyIdbRequest(idbOpenRequest)
+    }
+
+    const rootObjJson = await this.getRawRootObject()
+    // console.log("Root in: ", Date.now() - tSTART)
+
+    yield `${this.objectId}\t${rootObjJson}`
+
+    const rootObj = JSON.parse(rootObjJson)
+    if (!rootObj.__closure) return
+
+    let childrenIds = Object.keys(rootObj.__closure).sort(
+      (a, b) => rootObj.__closure[a] - rootObj.__closure[b]
+    )
+    if (childrenIds.length === 0) return
+
+    let splitHttpRequests = []
+
+    if (childrenIds.length > 50) {
+      // split into 5%, 15%, 40%, 40% (5% for the high priority children: the ones with lower minDepth)
+      const splitBeforeCacheCheck = [[], [], [], []]
+      let crtChildIndex = 0
+
+      for (; crtChildIndex < 0.05 * childrenIds.length; crtChildIndex++) {
+        splitBeforeCacheCheck[0].push(childrenIds[crtChildIndex])
+      }
+      for (; crtChildIndex < 0.2 * childrenIds.length; crtChildIndex++) {
+        splitBeforeCacheCheck[1].push(childrenIds[crtChildIndex])
+      }
+      for (; crtChildIndex < 0.6 * childrenIds.length; crtChildIndex++) {
+        splitBeforeCacheCheck[2].push(childrenIds[crtChildIndex])
+      }
+      for (; crtChildIndex < childrenIds.length; crtChildIndex++) {
+        splitBeforeCacheCheck[3].push(childrenIds[crtChildIndex])
+      }
+
+      console.log('Cache check for: ', splitBeforeCacheCheck)
+
+      const newChildren = []
+      let nextCachePromise = this.cacheGetObjects(splitBeforeCacheCheck[0])
+
+      for (let i = 0; i < 4; i++) {
+        const cachedObjects = await nextCachePromise
+        if (i < 3) nextCachePromise = this.cacheGetObjects(splitBeforeCacheCheck[i + 1])
+
+        const sortedCachedKeys = Object.keys(cachedObjects).sort(
+          (a, b) => rootObj.__closure[a] - rootObj.__closure[b]
+        )
+        for (const id of sortedCachedKeys) {
+          yield `${id}\t${cachedObjects[id]}`
+        }
+        const newChildrenForBatch = splitBeforeCacheCheck[i].filter(
+          (id) => !(id in cachedObjects)
+        )
+        newChildren.push(...newChildrenForBatch)
+      }
+
+      if (newChildren.length === 0) return
+
+      if (newChildren.length <= 50) {
+        // we have almost all of children in the cache. do only 1 requests for the remaining new children
+        splitHttpRequests.push(newChildren)
+      } else {
+        // we now set up the batches for 4 http requests, starting from `newChildren` (already sorted by priority)
+        splitHttpRequests = [[], [], [], []]
+        crtChildIndex = 0
+
+        for (; crtChildIndex < 0.05 * newChildren.length; crtChildIndex++) {
+          splitHttpRequests[0].push(newChildren[crtChildIndex])
+        }
+        for (; crtChildIndex < 0.2 * newChildren.length; crtChildIndex++) {
+          splitHttpRequests[1].push(newChildren[crtChildIndex])
+        }
+        for (; crtChildIndex < 0.6 * newChildren.length; crtChildIndex++) {
+          splitHttpRequests[2].push(newChildren[crtChildIndex])
+        }
+        for (; crtChildIndex < newChildren.length; crtChildIndex++) {
+          splitHttpRequests[3].push(newChildren[crtChildIndex])
+        }
+      }
+    } else {
+      // small object with <= 50 children. check cache and make only 1 request
+      const cachedObjects = await this.cacheGetObjects(childrenIds)
+      const sortedCachedKeys = Object.keys(cachedObjects).sort(
+        (a, b) => rootObj.__closure[a] - rootObj.__closure[b]
+      )
+      for (const id of sortedCachedKeys) {
+        yield `${id}\t${cachedObjects[id]}`
+      }
+      childrenIds = childrenIds.filter((id) => !(id in cachedObjects))
+      if (childrenIds.length === 0) return
+
+      // only 1 http request with the remaining children ( <= 50 )
+      splitHttpRequests.push(childrenIds)
+    }
+
+    // Starting http requests for batches in `splitHttpRequests`
+
+    const decoders = []
+    const readers = []
+    const readPromises = []
+    const startIndexes = []
+    const readBuffers = []
+    const finishedRequests = []
+
+    for (let i = 0; i < splitHttpRequests.length; i++) {
+      decoders.push(new TextDecoder())
+      readers.push(null)
+      readPromises.push(null)
+      startIndexes.push(0)
+      readBuffers.push('')
+      finishedRequests.push(false)
+
+      this.fetch(this.requestUrlChildren, {
+        method: 'POST',
+        headers: { ...this.headers, 'Content-Type': 'application/json' },
+        body: JSON.stringify({ objects: JSON.stringify(splitHttpRequests[i]) })
+      }).then((crtResponse) => {
+        const crtReader = crtResponse.body.getReader()
+        readers[i] = crtReader
+        const crtReadPromise = crtReader.read().then((x) => {
+          x.reqId = i
+          return x
+        })
+        readPromises[i] = crtReadPromise
+      })
+    }
+
+    while (true) {
+      const validReadPromises = readPromises.filter((x) => !!x)
+      if (validReadPromises.length === 0) {
+        // Check if all requests finished
+        if (finishedRequests.every((x) => x)) {
+          break
+        }
+        // Sleep 10 ms
+        await new Promise((resolve) => {
+          setTimeout(resolve, 10)
+        })
         continue
       }
-      yield chunk.substring( startIndex, result.index )
-      startIndex = re.lastIndex
+
+      // Wait for data on any running request
+      const data = await Promise.any(validReadPromises)
+      // eslint-disable-next-line prefer-const
+      let { value: crtDataChunk, done: readerDone, reqId } = data
+      finishedRequests[reqId] = readerDone
+
+      // Replace read promise on this request with a new `read` call
+      if (!readerDone) {
+        const crtReadPromise = readers[reqId].read().then((x) => {
+          x.reqId = reqId
+          return x
+        })
+        readPromises[reqId] = crtReadPromise
+      } else {
+        // This request finished. "Flush any non-newline-terminated text"
+        if (readBuffers[reqId].length > 0) {
+          yield readBuffers[reqId]
+          readBuffers[reqId] = ''
+        }
+        // no other read calls for this request
+        readPromises[reqId] = null
+      }
+
+      if (!crtDataChunk) continue
+
+      crtDataChunk = decoders[reqId].decode(crtDataChunk)
+      const unprocessedText = readBuffers[reqId] + crtDataChunk
+      const unprocessedLines = unprocessedText.split(/\r\n|\n|\r/)
+      const remainderText = unprocessedLines.pop()
+      readBuffers[reqId] = remainderText
+
+      for (const line of unprocessedLines) {
+        yield line
+      }
+      this.cacheStoreObjects(unprocessedLines)
+    }
+  }
+
+  async getRawRootObject() {
+    const cachedRootObject = await this.cacheGetObjects([this.objectId])
+    if (cachedRootObject[this.objectId]) return cachedRootObject[this.objectId]
+    const response = await this.fetch(this.requestUrlRootObj, { headers: this.headers })
+    const responseText = await response.text()
+    this.cacheStoreObjects([`${this.objectId}\t${responseText}`])
+    return responseText
+  }
+
+  promisifyIdbRequest(request) {
+    return new Promise((resolve, reject) => {
+      request.oncomplete = request.onsuccess = () => resolve(request.result)
+      request.onabort = request.onerror = () => reject(request.error)
+    })
+  }
+
+  async cacheGetObjects(ids) {
+    if (!this.options.enableCaching || !window.indexedDB) {
+      return {}
+    }
+
+    const ret = {}
+
+    for (let i = 0; i < ids.length; i += 500) {
+      const idsChunk = ids.slice(i, i + 500)
+
+      const store = this.cacheDB
+        .transaction('objects', 'readonly')
+        .objectStore('objects')
+      const idbChildrenPromises = idsChunk.map((id) =>
+        this.promisifyIdbRequest(store.get(id)).then((data) => ({ id, data }))
+      )
+      const cachedData = await Promise.all(idbChildrenPromises)
+
+      // console.log("Cache check for : ", idsChunk.length, Date.now() - t0)
+
+      for (const cachedObj of cachedData) {
+        if (!cachedObj.data)
+          // non-existent objects are retrieved with `undefined` data
+          continue
+        ret[cachedObj.id] = cachedObj.data
+      }
     }
 
-    if ( startIndex < chunk.length ) {
-      yield chunk.substr( startIndex )
+    return ret
+  }
+
+  cacheStoreObjects(objects) {
+    if (!this.options.enableCaching || !window.indexedDB) {
+      return {}
     }
+
+    const store = this.cacheDB
+      .transaction('objects', 'readwrite')
+      .objectStore('objects')
+    for (const obj of objects) {
+      const idAndData = obj.split('\t')
+      store.put(idAndData[1], idAndData[0])
+    }
+
+    return this.promisifyIdbRequest(store.transaction)
   }
 }
+
+// Credits and more info: https://github.com/jakearchibald/safari-14-idb-fix
+function safariFix() {
+  const isSafari =
+    !navigator.userAgentData &&
+    /Safari\//.test(navigator.userAgent) &&
+    !/Chrom(e|ium)\//.test(navigator.userAgent)
+
+  // No point putting other browsers or older versions of Safari through this mess.
+  if (!isSafari || !indexedDB.databases) return Promise.resolve()
+
+  let intervalId
+
+  return new Promise((resolve) => {
+    const tryIdb = () => indexedDB.databases().finally(resolve)
+    intervalId = setInterval(tryIdb, 100)
+    tryIdb()
+  }).finally(() => clearInterval(intervalId))
+}

package/jsconfig.json

@@ -0,0 +1,7 @@
+{
+  "extends": "../../jsconfig.base.json",
+  "compilerOptions": {
+    "module": "es2020"
+  },
+  "include": ["index.js", "examples"]
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@speckle/objectloader",
-  "version": "2.1.1",
+  "version": "2.3.17-alpha.2819",
   "description": "Simple API helper to stream in objects from the Speckle Server.",
   "main": "index.js",
   "homepage": "https://speckle.systems",
@@ -11,9 +11,21 @@
     "directory": "packages/objectloader"
   },
   "scripts": {
-    "test": "echo \"Error: no test specified\" && exit 1"
+    "test": "echo \"Error: no test specified\" && exit 1",
+    "lint": "eslint . --ext .js,.ts"
   },
-  "keywords": ["speckle", "aec", "speckle api"],
+  "keywords": [
+    "speckle",
+    "aec",
+    "speckle api"
+  ],
   "author": "AEC Systems",
-  "license": "Apache-2.0"
+  "license": "Apache-2.0",
+  "devDependencies": {
+    "eslint": "^8.11.0",
+    "eslint-config-prettier": "^8.5.0",
+    "prettier": "^2.5.1",
+    "undici": "^4.14.1"
+  },
+  "gitHead": "78dc2b89ab4864e3e97988417bab06c906ef454a"
 }

package/readme.md

@@ -10,9 +10,11 @@ Comprehensive developer and user documentation can be found in our:
 
 ## Getting started
 
-This is a small utility class that helps you stream an object and all its sub-components from the Speckle Server API. It is inteded to be used in contexts where you want to "download" the whole object, or iteratively traverse its whole tree.
+This is a small utility class that helps you stream an object and all its sub-components from the Speckle Server API. It is intended to be used in contexts where you want to "download" the whole object, or iteratively traverse its whole tree.
 
-Here's a sample way on how to use it, pfilfered from the [3d viewer package](../viewer):
+### In the browser
+
+Here's a sample way on how to use it, pilfered from the [3d viewer package](../viewer):
 
 ```js
 
@@ -36,23 +38,36 @@ async load( { serverUrl, token, streamId, objectId } ) {
 
 ```
 
-If you do not want to process the objects one by one as they are streamed to you, you can use the `getAndConstructObject()` method. Here's an example: 
+If you do not want to process the objects one by one as they are streamed to you, you can use the `getAndConstructObject()` method. Here's an example:
 
-```js
+````js
 
 let loader = new ObjectLoader( {
-  serverUrl: "https://latest.speckle.dev", 
-  streamId: "3ed8357f29", 
+  serverUrl: "https://latest.speckle.dev",
+  streamId: "3ed8357f29",
   objectId: "0408ab9caaa2ebefb2dd7f1f671e7555",
   options: {
     fullyTraverseArrays: false, // Default: false. By default, if an array starts with a primitive type, it will not be traversed. Set it to true if you want to capture scenarios in which lists can have intersped objects and primitives, e.g. [ 1, 2, "a", { important object } ]
-    excludeProps: [ 'displayValue', 'displayMesh', '__closure' ] // Default: []. Any prop names that you pass in here will be ignored from object construction traversal. 
+    excludeProps: [ 'displayValue', 'displayMesh', '__closure' ] // Default: []. Any prop names that you pass in here will be ignored from object construction traversal.
   }
 } )
 
 let obj = await loader.getAndConstructObject( ( e ) => console.log( 'Progress', e ) )
 
-```
+### On the server
+
+Since Node.js does not yet support the [`fetch API`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch), you'll need to provide your own `fetch` function in the options object. Note that `fetch` must return a [Web Stream](https://nodejs.org/api/webstreams.html), so [node-fetch](https://github.com/node-fetch/node-fetch) won't work, but [node/undici's](https://undici.nodejs.org/) implementation will.
+
+```js
+import { fetch } from 'undici'
+
+let loader = new ObjectLoader({
+  serverUrl: 'https://latest.speckle.dev',
+  streamId: '3ed8357f29',
+  objectId: '0408ab9caaa2ebefb2dd7f1f671e7555',
+  options: { enableCaching: false, excludeProps: [], fetch },
+})
+````
 
 ## Community