startgg-helper 2.3.1 → 2.4.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "startgg-helper",
-  "version": "2.3.1",
+  "version": "2.4.0",
   "description": "A set of functions and classes useful to communicate with the start.gg API, using any client (YOU NEED TO PROVIDE A CLIENT YOURSELF, SEE README)",
   "main": "main.js",
   "module": "main.js",

package/readme.md

@@ -1,16 +1,102 @@
 # start.gg Helper
 
-A set of functions and classes useful to interact with the start.gg GraphQL API.  
+A set of functions and classes useful to interact with the [start.gg GraphQL API](https://developer.start.gg/docs/intro/). It does NOT provide abstractions for the actual data retrieved from the API (events, players, sets, etc), only eases making the queries. An understanding of [GraphQL](https://graphql.org/learn/introduction/), and [start.gg's GraphQL Schema](https://smashgg-schema.netlify.app/reference/query.doc.html) are necessary to leverage the API through this package. 
 
-## You need to provide a client to interact with the API
-The functions in this package usually take a "client" argument, which is a GraphQL client object. You need to handle this object yourself, which can be done using the `graphql` package, which is not a dependency of `startgg-helper` and must be installed manually.
+## You need to provide a client to interact with the API  
+tl;dr : you're probably looking for `startgg-helper-node` or `startgg-helper-browser`.
 
-### Why do this ?
+To interact with the API, the functions in this package need (and take as argument) a "client" object, able to send requests to the GraphQL API. Such a client **is not provided by this package**. This is to ensure that this package is usable not only in a node ecosystem, but also in browser front-end code : sending requests to an API is done differently in browser-oriented code and usual NodeJS code, meaning that providing a client in this package would make it unfit for at least some purposes.  
+Two packages exist to solve that issue : 
+- `startgg-helper-node`, which includes this one and provides a client using the `graphql` package : this is for your NodeJS projects
+- `startgg-helper-browser`, which includes this ont and provides a simple client relying on the `fetch` API : this is for your web projects, to be run by a browser (using a tool like `browserify` to make your node package usable on browser)
 
-The purpose of this package is to be usable not onlyin a node ecosystem but also in a browser project, using `browserify`. The `graphql` package does not work well with browserify, so I decided to let the user provide a client depending on their environment.
+### I still want to use this package and provide my own client
+OK ! A client is actually a very simple thing : all it needs is a `request(schema, variables)` method, taking a GraphQL schema as a string and a collection of variables as an object. As long as your object exposes this method, and it correctly returns the result of the desired GraphQL request, it can be passed to `startgg-helper` functions.
 
-### Ok actually you can just use startgg-helper-node or startgg-helper-browser
+## Quick Doc
+[Full focumentation here](./doc.md)
 
-If you don't want to handle the client yourself, you can simply install
-- `startgg-helper-node` if you're on Node, it includes `graphql` and handles the client using this package
-- `startgg-helper-browser` if you're doing a browser-based app using `browserify`, which provides a custom client class based on the `fetch` API.
+The basic feature of this package is the `Query` object, which represents a GraphQL Schema/Document. Its methods allow to execute the query (i.e. make a GraphQL request with the defined schema) with different variables, with automatic retries in case of failure.
+
+```js
+const schema = `
+query Test($slug: String, $page: Int, $perPage: Int){
+    event(slug: $slug) {
+        sets(page: $page, perPage: $perPage){
+            nodes {
+                id
+                state
+            }
+        }
+    }
+}
+`
+
+const query = new Query(schema, 3); //3 is the default number of retries
+
+let res = await Promise.all([
+    "tournament/my-tournament-1/event/ult-singles",
+    "tournament/my-tournament-2/event/ult-singles",
+].map(async slug => await query.execute(client, {slug, page: 1})))
+```
+
+### Paginated collections and queries
+Paginated collections are collections not represented by a GraphQL Array, but a page system, where each query needs to specify the index and size of the page it's fecthing. See the start.gg API for more information.  
+A paginated collection of type T is represented by a field with a `page` and `perPage` parameter returning a Connection-style type : 
+```graphql
+field(page: Int, perPage: Int): TConnection
+ 
+type TConnection {
+    pageInfo: PageInfo
+
+    nodes: [T]
+}
+ 
+type PageInfo {
+  total: Int
+  totalPages: Int
+  page: Int
+  perPage: Int 
+}
+```
+
+The `Query.executePaginated` method can be used to query entire paginated collections, by qerying repeatedly while increasing a certain parameter of the query, which must be used as the page argument of paginated collection field in the query, which you need to point to using the path parameter, and agregating the elements of each page into one single array. The page argument will be controlled entirely by the loop and doesn't need to be included with other graphQL variables. If the pageInfo.totalPages field is included in your schema, it will be used to determine the last page ; if not, this method will stop once it receives an empty page. 
+
+Example : 
+
+```graphql
+query Example($page: Int){
+  topField {
+    field2(page: $page){
+      pageInfo {
+        totalPages
+      }
+      nodes {
+        ...
+      }
+    }
+  }
+}
+```
+```js
+query.executePaginated(client, {}, "topfield.field2")
+``` 
+
+### Limiters : dealing with the API rate limit
+The start.gg API has a rate limit of (as of writing this) 80 requests per minute per API key. To avoid exceeding this limit, `startgg-helper` provide a client-side rate-limiting mechanism, that comes in the form of objects you can pass to request-sending functions. 
+
+```js
+const limiter = new StartGGDelayQueryLimiter(); //it is important to only create and use one in the entire program.
+
+for (let i = 0; i < 200; i++){
+    query.execute(client, {slug: `tournament/my-tournament-${i}/event/ult-singles`}, limiter);
+}
+//queries will be delayed to avoid exceeding start.gg's rate limit
+```
+
+there are a handful of limiter classes but only a few are useful to you (some others are here only for legacy)
+- DelayQueryLimiter(rpm) : allows up to `rpm` request per minute
+- StartGGDelayQueryLimiter : allows up to 60 request per minute. This is intentionally lower than the actual server-side limit, to prevent network timing mishaps from making us accidentally going over. 
+
+### Etc
+This package also provides lots of utility functions ; see the [full doc](./doc.md)

package/src/jsUtil.js

@@ -12,11 +12,30 @@ function processObjectPath(path){
 }//jsutil
 
 /**
+ * Traverses nested object following a path and returns what's at the end, without throwing an error if an intermediary object-property is not found.  
+ * The "path" argument works like a JS object access expression, starting with a property from the initial object (first parameter)   
+ * ```js
+ * const obj = {a: {b: {c: 12}}};
+ * obj.a.b.c; //value : 12
+ * deep_get(obj, "a.b.c"); //returns 12
  * 
- * @param {{}} obj 
- * @param {string} path 
- * @param {*} def 
- * @returns 
+ * obj.a.d.c; //ERROR : Cannot read properties of undefined (reading 'c')
+ * deep_get(obj, "a.d.c"); //returns null, no error
+ * deep_get(obj, "a.d.c", 15);//returns 15, default value
+ * ```
+ * 
+ * This function supports numbers as property names, which **works with arrays**.
+ * ```js
+ * const obj = {a: [{}, {}, {b: 12}]};
+ * 
+ * obj.a[2].b; value: 12
+ * obj.a.2.b; Syntax ERROR
+ * deep_get(obj, "a.2.b"); //returns 12
+ * ```
+ * 
+ * @param {Object | Array} obj Object or array
+ * @param {string} path See above
+ * @param {*} def Value returned if the path cannot be followed to the end
  */
 export function deep_get(obj, path, def = null){
     //https://stackoverflow.com/a/8817473
@@ -29,6 +48,13 @@ export function deep_get(obj, path, def = null){
     return obj;
 };
 
+/**
+ * Traverses nest objects following a path and sets the final property to a given vallue.  
+ * Traversing works the same as with deep_get()
+ * @param {{}} obj 
+ * @param {string} path 
+ * @param {*} value 
+ */
 export function deep_set(obj, path, value){
     path = processObjectPath(path);
 
@@ -49,8 +75,8 @@ export function generateUniqueID(){
 }//jsutil
 
 /**
- * 
- * @param {(() => any)[]} fArray 
+ * Returns an array containing the results of an array of functions, called without parameters. Any "undefined" result is ignored, meaning the resulting array can be smaller than the function array.
+ * @param {(()=>any)[]} fArray 
  */
 export function fResultsArray(fArray){
     let result = [];
@@ -65,7 +91,7 @@ export function fResultsArray(fArray){
 } 
 
 /**
- * 
+ * Returns an array containing the results of all parameters, treated as functions, called without parameters. See fResultsArray
  * @param  {...(() => any)} functions 
  */
 export function fResults(...functions){
@@ -73,20 +99,26 @@ export function fResults(...functions){
 }
 
 /**
- * 
+ * Serializes a value to JSON text like JSON.stringify does. (this function is just a very thin wrapper only useful for the "pretty" parameter)
  * @param {any} data 
- * @param {boolean} pretty 
+ * @param {boolean} pretty If true, the resulting JSON will be made to be human-readable, with 4-space indentation
  */
 export function toJSON(data, pretty){
     return JSON.stringify(data, null, pretty ? 4 : undefined);
 }//jsutil
 
+/**
+ * Take a wild guess
+ * @param {any} n 
+ */
 export function isNumber(n){
     return typeof n == "number";
 }
 
 /**
- * 
+ * Converts a data to a UNIX timestamp, i.e. number of seconds since 1/1/1970 00:00. Accepts : 
+ * - JS Date objects
+ * - strings and numbers : will be converted inta a Date like `new Date(d)` does. (number treated as UNIX timestamps with milisecond granularity, more complicated for strings)
  * @param {number | Date | string} d 
  */
 export function toUNIXTimestamp(d){

package/src/query.js

@@ -5,14 +5,27 @@ function isConnection(val){
     return val instanceof Object && val.nodes instanceof Array
 }
 
+export class PageResult {
+    /**
+     * @param {any} result 
+     * @param {boolean} stop 
+     */
+    constructor(result, stop){
+        this.result = result;
+        this.stop = stop;
+    }
+}
+
+/**
+ * A GraphQL Document/Schema. Allows easily batch-requesting with the same schema but different parameters. 
+ */
 export class Query {
     #schema;
     #maxTries;
 
     /**
-     * 
-     * @param {string} schema 
-     * @param {number?} maxTries 
+     * @param {string} schema The GraphQL [Schema] 
+     * @param {number?} maxTries How many retries will be performed when requesting using this query
      */
     constructor (schema, maxTries = null){
         this.#schema = schema;
@@ -25,9 +38,11 @@ export class Query {
      * @param {{[varName: string]: value}} params 
      * @returns 
      */
-    #getLog(logName, params){
-        if (!this.log) return null;
-        let log = this.log[logName];
+    #getLog(logName, params, logsOverride = null){
+        let log;
+        if (logsOverride) log = logsOverride[logName];
+        if (this.log) log = this.log[logName] ?? log;
+
         if (log){
             if (typeof log == "string"){
                 return log;
@@ -38,20 +53,24 @@ export class Query {
         return null;
     }
 
+    #getPaginatedLogOverride(params){
+        if (!this.paginatedLog) return null;
+        return this.paginatedLog(params);
+    }
+
     /**
      * 
      * @param {GraphQLClient} client 
      * @param {{[varName: string]: value}} params 
      * @param {number} tries How many tries in are we 
      * @param {TimedQuerySemaphore} limiter 
-     * @param {boolean} silentErrors 
+     * @param {boolean} silentErrors legacy parameter, does nothing
      * @param {number} maxTries Overrides this.#maxTries
-     * @returns 
      */
-    async #execute_(client, params, tries, limiter = null, silentErrors = false, maxTries = null){
+    async #execute_(client, params, tries, limiter = null, silentErrors = false, maxTries = null, logsOverride = null){
         maxTries = maxTries || this.#maxTries || 1
 
-        console.log((this.#getLog("query", params) || "Querying ...") + " Try " + (tries + 1));
+        console.log((this.#getLog("query", params, logsOverride) || "Querying ...") + " Try " + (tries + 1));
         try {
             let data = await ( limiter ? limiter.execute(client, this.#schema, params) : client.request(this.#schema, params));
             
@@ -62,22 +81,21 @@ export class Query {
                 console.error("Maximum number of tries reached. Throwing.", e);
                 throw e;
             }
-            console.error((this.#getLog("error", params) || "Request failed.") + ` Retrying (try ${tries + 1}). Error : `, e);
-            return this.#execute_(client, params, tries + 1, limiter, silentErrors, maxTries);
+            console.error((this.#getLog("error", params, logsOverride) || "Request failed.") + ` Retrying (try ${tries + 1}). Error : `, e);
+            return this.#execute_(client, params, tries + 1, limiter, silentErrors, maxTries, logsOverride);
         }
     }
 
     /**
-     * Executes the query with given parameters and client
-     * @param {GraphQLClient} client 
-     * @param {{[varName: string]: value}} params 
-     * @param {TimedQuerySemaphore} limiter 
-     * @param {boolean} silentErrors 
-     * @param {number} maxTries Overrides the default maximum tries count for this query
+     * @param {GraphQLClient} client A client object ; not provided by this package, either use startgg-helper-node (or-browser) or refer to README.md
+     * @param {{[varName: string]: value}} params GraphQL variables
+     * @param {TimedQuerySemaphore} limiter A request limiter object; see TimedQuerySemaphore
+     * @param {boolean} silentErrors No effect, exists only for legacy purposes
+     * @param {number} maxTries Overrides the default maximum tries count for this query.
      * @returns 
      */
-    async execute(client, params, limiter = null, silentErrors = false, maxTries = null){
-        return await this.#execute_(client, params, 0, limiter, silentErrors, maxTries);
+    async execute(client, params, limiter = null, silentErrors = false, maxTries = null, logsOverride = null){
+        return await this.#execute_(client, params, 0, limiter, silentErrors, maxTries, logsOverride);
     }
 
     static IWQModes = {
@@ -87,16 +105,26 @@ export class Query {
         OUT: 3
     }
 
+    /** @typedef {(localResult: T[], currentResult: T[], pageIndex: number) => (T[]|PageResult|boolean)?} PageCallback*/
+
     /**
-     * Executes a query containing a paginated collection (of a *Connection type) repeatedly, increasing the page index each time until nothing is returned, returning an aggregation of all the pages.
-     * @param {GraphQLClient} client 
-     * @param {{[varName: string]: value}} params 
-     * @param {string} connectionPathInQuery JSON path to the paginated collection that must aggregated in the query (JSON path : property names separated by dots)
-     * @param {TimedQuerySemaphore} limiter 
-     * @param {{pageParamName?: string, perPageParamName?: string, perPage?: number, delay?: number, maxElements?: number, includeWholeQuery?: number, callback: (localResult: T[], page: number, totalPages: number) => T[]?}} config 
-     * @param {boolean} silentErrors 
+     * Queries a whole paginated collection (of a *Connection type). See the start.gg API doc or this package's documentation for more info about pagniated collections. This is done through executing the query repeatedly while increasing the page index each time. 
+     * The target collection must be pointed to by the "path" argument, and will be agregated in a single array. The schema must have a variable (whose name can be specified in parameters) that is used as the page index of the paginated field.
+     * 
+     * 
+     * @param {GraphQLClient} client A client object ; not provided by this package, either use startgg-helper-node (or-browser) or refer to README.md
+     * @param {{[varName: string]: value}} params GraphQL variables ; does not include the page index variable. 
+     * @param {string} connectionPathInQuery JSON path to the paginated collection that must be aggregated in the query (JSON path : property names separated by dots, see deep_get())
+     * @param {TimedQuerySemaphore} limiter A request limiter object; see TimedQuerySemaphore
+     * @param {{pageParamName?: string, perPageParamName?: string, perPage?: number, delay?: number, maxElements?: number, includeWholeQuery?: number, callback: PageCallback?}} config 
+     * @param  config.pageParamName name of the variable representing the page index. This variable must exist in your query, and be used as an argument in a paginated collection field
+     * @param  config.delay number of miliseconds to wait for between each query. No delay if absent.
+     * @param  config.maxElements if present, queries will stop once this many elements have been fetched
+     * @param  config.includeWholeQuery controls the structure of the result. If 0, only an array is returned, if 1 the whole query is returned with the `nodes` field replaced with the full array, if 3 this functions returns a tuple containing the aggregated collection and the rest of the query, 2 does both 1 and 3. 
+     * @param config.callback a callback function called for each fetched page, with the page elements and the page index. It should return an array itself, which will be treated as the actual page (allowing users to transform the pages on the fly) ; if it does not, the paginated execution is stopped.
+     * @param {boolean} silentErrors No effect, exists only for legacy purposes
      * @param {number} maxTries 
-     * @returns 
+     * @returns See config.includeWholeQuery
      */
     async executePaginated(client, params, connectionPathInQuery, limiter = null, config = {}, silentErrors = false, maxTries = null){
         let result = [];
@@ -113,14 +141,16 @@ export class Query {
         params[pageParamName] = currentPage;
         params[perPageParamName] = perPage;
 
+        const logsOverride = this.#getPaginatedLogOverride(params);
+
         let data;
         while (true){
             if (result.length >= maxElements) break;
 
             console.log("Querying page", params[pageParamName], `(${result.length} elements loaded)`);
-            data = await this.execute(client, params, limiter, silentErrors, maxTries);
+            data = await this.execute(client, params, limiter, silentErrors, maxTries, logsOverride);
 
-            if (!data) throw (this.#getLog("error", params) ?? "Request failed.") + "(in paginated execution, at page " + params[pageParamName] + ")";
+            if (!data) throw (this.#getLog("error", params, logsOverride) ?? "Request failed.") + "(in paginated execution, at page " + params[pageParamName] + ")";
 
             let connection = deep_get(data, connectionPathInQuery);
 
@@ -133,29 +163,58 @@ export class Query {
 
             let localResult = connection.nodes;
 
-            if (connection.pageInfo && connection.pageInfo.totalPages){
+            if (connection.pageInfo && connection.pageInfo.totalPages){ //if the query contains pageInfo we use that to determine if we're at the last page
                 let totalPages = connection.pageInfo.totalPages;
-                if (!totalPages || currentPage >= totalPages) {
+                if (currentPage >= totalPages) {
                     if (config.callback){
-                        let cbRes = config.callback(localResult, currentPage);
-                        if (cbRes) break;
-                        localResult = cbRes;
+                        let cbRes = config.callback(localResult, currentResult, pageIndex)
+
+                        if (cbRes instanceof PageResult){
+                            if (cbRes.result){ //If the user gives us a result we take it ofc
+                                result = cbRes.result;
+                            } else {
+                                result = result.concat(localResult);
+                            }
+                        } else if (cbRes && cbRes !== true) { //cbres isn't one that means something loop-control-related (it's over anyways so we don't care!)
+                            result = result.concat(cbRes);
+                        }
+                    } else {
+                        result = result.concat(localResult);
                     }
-                    result = result.concat(localResult);
                     break;
                 }
-            } else {
+            } else { //if not, we only know when we get an empty page
                 if (localResult.length < 1) break;
             }
 
             if (config.callback){
                 let cbRes = config.callback(localResult, currentPage);
-                if (!cbRes) break;
-                localResult = cbRes;
+
+                if (cbRes instanceof PageResult){
+                    if (cbRes.result){ //If the user gives us a result we take it ofc
+                        result = cbRes.result;
+                    } else {
+                        result = result.concat(localResult);
+                    }
+                    if (cbRes.stop){
+                        break;
+                    }
+                    currentPage++;
+                } else if (!cbRes) { //"normal case" : basic callback that doesn't touch the flow of the pexecution, we concat and increment normally
+                    result = result.concat(localResult)
+                    currentPage++;
+                } else if (cbRes === true){ //callback is asking us to stop by returning true
+                    break;
+                } else { //callback returns a non-boolean value : it's the local result (and we increment normally)
+                    result = result.concat(cbRes);
+                    currentPage++;
+                }
+
+            } else {
+                result = result.concat(localResult);
+                currentPage++;
             }
 
-            result = result.concat(localResult);
-            currentPage++;
             params[pageParamName] = currentPage;
 
             if (delay)
@@ -180,6 +239,8 @@ export class Query {
     }
 
     /**
+     * Deprecated, use executePaginated instead.
+     * 
      * Executes a query containing a paginated collection, repeatedly, increasing the page index each time until nothing is returned, returning an aggregation of all the pages.
      * @param {GraphQLClient} client 
      * @param {{[varName: string]: value}} params 

package/src/queryLimiter.js

@@ -1,7 +1,10 @@
 //i am a god of JS
 
-//voir la diff avec la version de sgghelper ?
-
+/**
+ * Provides timing control for requests. If passed to a Query method, it will delay its execution to stay withing a defined limit.  
+ * 
+ * This is implemented like a semaphore automatically releasing tokens after a certain delay
+ */
 export class TimedQuerySemaphore {
     /**
      * @typedef {{client: any, schema: string, 
@@ -25,8 +28,8 @@ export class TimedQuerySemaphore {
 
     /**
      * 
-     * @param {number} size Semaphore counter initial value
-     * @param {number} delay 
+     * @param {number} size Semaphore counter initial value : how many requests we can send before having to wait
+     * @param {number} delay Delay before each taken token is released ;  how much time we wait before making new requests
      */
     constructor(size, delay){
         this.#counter = size;
@@ -67,7 +70,8 @@ export class TimedQuerySemaphore {
     }
 
     /**
-     * Executes the given query with the given GraphQL Client
+     * Executes the given query with the given GraphQL Client. 
+     * Note that this method probably shouldn't be called by anything else than Query.execute
      * @param {any} client 
      * @param {string} schema 
      * @param {{[varName: string]: value}} params 
@@ -97,6 +101,9 @@ export class TimedQuerySemaphore {
         }
     }
 
+    /**
+     * Stops the timing manager ; always call this at the end of your program, if you dont node will think there is still work to be done and won't exit
+     */
     stop(){
         for (let timer of this.#timers){
             clearTimeout(timer);
@@ -104,19 +111,22 @@ export class TimedQuerySemaphore {
     }
 }
 
+/**TimedQuerySemaphore that allows one request before waiting. You should use the DelayQueryLimiter instead.*/
 export class ClockQueryLimiter extends TimedQuerySemaphore {
-    /** @param {number} rpm */
+    /** @param {number} rpm Requests per minute*/
     constructor(rpm){
         super(1, 60000 / rpm);
     }
 }
 
+/** ClockQueryLimiter configured to stay within the start.gg API requests limit (60 per minute, so 1 second delay). You should use the StartGGDelayQueryLimiter instead*/
 export class StartGGClockQueryLimiter extends ClockQueryLimiter {
     constructor(){
         super(60);
     }
 }
 
+/**TimedQuerySemaphore that allows x requests before waiting, and waits for one minute. */
 export class DelayQueryLimiter extends TimedQuerySemaphore {
     /** @param {number} rpm */
     constructor(rpm){
@@ -124,6 +134,7 @@ export class DelayQueryLimiter extends TimedQuerySemaphore {
     }
 }
 
+/** ClockQueryLimiter configured to stay within the start.gg API requests limit (60 per minute)*/
 export class StartGGDelayQueryLimiter extends DelayQueryLimiter {
     constructor(){
         super(60)