@foxtware/mineral 0.1.39 → 1.0.0

package/AGENTS.md

@@ -2,6 +2,11 @@
 
 Notes for AI assistants working in this directory. **Read this file at the start of mineral tasks** and follow anything here that isn't overridden by the user in chat.
 
+## Related docs
+
+- Utils landmarks (`Processor`, `canFinish`, `Getter`, …): [`api/UTILS.md`](api/UTILS.md)
+- Function layout templates: `api/_example.js`, `api/[platform]/_example*.js`
+
 ## Misc
 - When working in shopify, refer to gids (gid://shopify/Page/12345678) explicitly as "gids", and use "id" to refer to the number itself (12345678). "id" may still be required for use of the GraphQL API but semantically we should make this distinction.
 - funcApiConfig should not list `options` in `argNames` — it is optional. When present on the request body, `funcApi` appends it after the named args.

package/README.md

@@ -1,21 +1,24 @@
 # mineral
 
-The [bedrock](https://github.com/GorgonFreeman/bedrock) middleware, refactored for compatibility with other projects and more consistent standards.
+An everything middleware, centred around easy-to-bring credentials and instantly useful curl commands. Now importable into your project.
 
-## Problems with bedrock
+## How to get mineralling
 
-- **Inconsistent response format**
-  - `success: true` vs `success: false` for calls that succeed but return no data
-  - No distinction between technical failures and valid business errors
-  - Mixed use of `error` vs `errors` in responses
-- **Portability** — `.creds.yml` and related structures resist reuse outside the repo
-- **Auth** — helpers can fail without retrying, especially when using creds from Upstash
-- **Architecture** — boundaries between core and private functions could be firmer
+1. From your project, install mineral: `npm i @foxtware/mineral`.
+2. Set up a `.creds.yml` in your project directory, based on mineral's `.creds.yml.sample`. Add the credentials for the platform you want to use.
+3. Add the `dev` command to your project's `package.json`:
+```
+"dev": "node --watch --watch-path=./api --watch-path=./.creds.yml --watch-path=./.env --watch-path=./hosting node_modules/@foxtware/mineral/server.js --workspace . --api_dirs api"
+```
+4. Run `npm run dev` to start a server.
+5. Use an example curl command from the file whose function you want to use, with the creds path you set up.
 
-## The plan
+## Key ideas
+### The response format to rule them all
 
-- **The response format to rule them all**
-  ```typescript
+All functions should return data in this format.
+
+```typescript
   interface Response {
     ok: boolean
     data?: unknown
@@ -32,28 +35,44 @@ The [bedrock](https://github.com/GorgonFreeman/bedrock) middleware, refactored f
     }
     results?: Response[]
   }
-  ```
-  - `ok` — `true` if the intent executed; `false` otherwise
-  - `data` — any info returned
-  - `error` — when present, multiple errors go in `error.details`
-  - `meta` — pagination, total items, etc.
-  - `results` — array of responses from a queue or batch of calls
-  - [Examples](_docs/standard_response_examples.md)
-- **BYO Creds**
-  ```
-  {
-    credsPath,
-    shopifyCredsPath,
-    credsObject,
-    credsProvider, // function that returns a credsObject
-  }
-  ```
-- **Monorepo structure**
-Mineral gets pushed to from a larger repo that can also contain private functions. Mineral should be strictly useful stuff for the public, and can be used standalone, but needs to be instantiated for serving, setting stuff like which creds file to use. This allows it to be used as part of another repo in the same HTTP/curl way as by itself. Pass `--workspace` to locate `.creds.yml` and `--api_dirs` to serve additional function directories.
+```
+- `ok` — `true` if the intent executed; `false` otherwise
+- `data` — any info returned
+- `error` — when present, multiple errors go in `error.details`
+- `meta` — pagination, total items, etc.
+- `results` — array of responses from a queue or batch of calls
+- [Examples](_docs/standard_response_examples.md)
+
+### BYO creds
+
+Each function accepts an object like this, supplying auth info for each platform it contacts:
+
+```
+{
+  credsPath,
+  credsObject,
+  credsProvider, // function that returns a credsObject
+}
+```
+
+You can set up a `.creds.yml` in your workspace directory and refer to it using `credsPath` > `platform.account`, or supply credentials inline using `credsObject`.
+
+### Monorepo context
+
+Mineral gets pushed to from a larger repo. Private functions are contained in a sibling project, which imports mineral to power all of the core functionalities, and layers certain business logic on top - e.g. known creds paths. 
+
+Mineral is strictly core functionalities, useful for anyone.
+
+Pass `--workspace` to locate `.creds.yml` and `--api_dirs` to serve additional function directories.
+
+### Hosting
+
+For cloud deploy, workspaces use `hosting/.hosting.yml` and `npm run host` (same `--workspace` / `--api_dirs` flags as dev/serve). See `hosting/.hosting.yml.sample`. 
+
+In the hosting YML, you can use wrappers which are additional layers to your hosted function. For instance, you may want to require an API key in requests to your middleware while it's live, or, require a hash in the headers that matches a hash of the body. 
+
+The decoupled nature of these desires to what the function actually does and their broad application is why they're implemented in the hosting config itself, rather than in a custom instance of the function.
 
-For cloud deploy, workspaces use `hosting/.hosting.yml` and `npm run host` (same `--workspace` / `--api_dirs` flags as dev/serve). See `hosting/.hosting.yml.sample`.
+### AI use
 
-## What the thang do
-- Server makes functions available from the api/ route, where an export matches the filename. Run `npm run serve`, and they're all curlable.
-- .creds.yml is copied into .env when deploying, so creds can be accessed while hosted. Locally, it reads from the file directly.
-- Cloud deploy reads `hosting/.hosting.yml` for per-function config — `before_wrappers` / `after_wrappers` like `requireHostedApiKey`, `max_instances`, schedules — and deploys each function to Google Cloud.
+Not gonna lie, I have absolutely composed some of these platforms with AI. Some of them may not work, some may not stick closely to the core principles, some may use awkward auth methods - some error payloads are definitely not optimised. However, know that I have hand-coded precursors to this repo, that do largely the same thing, and that any additions benefit (or suffer) from the structure that is laid out in the earlier platforms.

package/api/UTILS.md

@@ -0,0 +1,52 @@
+# Mineral utils landmarks
+
+Cheat sheet for [`utils.js`](utils.js). Prefer this over cold-grepping the file. Exports are listed near the bottom of `utils.js` (`module.exports`).
+
+## Validation and args
+
+| Export | Use when |
+|--------|----------|
+| `ArgsWarden` | Validate required function args. Do **not** list `options` in `argNames` — `funcApi` appends body `options` after named args. |
+| `credsFromPayload` | Resolve `{ credsPath }` / `{ credsObject }` / `{ credsProvider }` into creds. |
+| `valueProvided` | Default ArgsWarden singleton validator (“anything truthy/present”). |
+| `objHasAny` / `objHasAll` | Object-shape validators (e.g. product identifier has `productId` or `handle`). |
+
+## Queues and batching
+
+| Export | Use when |
+|--------|----------|
+| `Processor` | Drain a mutable pile with concurrency. Constructor options include `canFinish` (default `true`), `maxInFlightRequests`, `logFlavourText`, `onDone`. Set `this.canFinish = false` (or pass `canFinish: false`) to keep waiting when the pile is empty until you flip it true — useful when producers still fill the pile. |
+| `oneTrickProcessor` | Fire-and-forget: pile of arg arrays → `func(...args)` via a `Processor`. |
+| `actionSingleOrMultiple` | Single resource action that also accepts an array (or cartesian product of arrays) via `OperationQueue`. |
+| `Operation` / `OperationQueue` | Lower-level queue primitives behind `actionSingleOrMultiple`. |
+| `Getter` | Paginated list fetch: `paginator` + `digester`; export both `platformThingGet` and `platformThingGetter` with `.bind`-style wrappers. Call `getter.end()` to stop paging early. `run({ verbose })` defaults to quiet when `HOSTED`. |
+| `FakeGetter` | Wrap a one-shot fetch (e.g. Peoplevox report) so it emits `items` / `done` like a `Getter`. |
+| `ThresholdActioner` | Call an action once N `increment()` calls have happened (e.g. unlock a tagger after processors finish). |
+| `MultiDex` | Index items by multiple primary keys and merge partial records (useful for cross-store joins). |
+
+## Shopify ids
+
+| Export | Use when |
+|--------|----------|
+| `gidToId` | Strip `gid://shopify/.../123` → `"123"`. Prefer numeric **id** inputs on mineral functions; use **gid** only when the GraphQL API requires it. |
+
+## HTTP client
+
+| Export | Use when |
+|--------|----------|
+| `FetchClient` | Platform HTTP client base (pipeline steps, base URL, auth). |
+| `customFetch` | Shared fetch used inside clients; already sets `Content-Type: application/json` when there is a body — don’t set it again. |
+| `fetchClient` option | Many handlers accept `fetchClient` in `options` so geode (or tests) can inject a wrapped client. |
+
+## Diff / inspect (sweeps)
+
+| Export | Use when |
+|--------|----------|
+| `surveyObject` | Summarise / compare object fields for assess steps. |
+| `diffObjects` | Field-level diffs between source and target. |
+| `logDeep` | Deep console dump (local inspect). |
+| `askQuestion` | Interactive confirm when `!HOSTED`. |
+
+## Small helpers often needed
+
+`ensureArray`, `arrayToChunks`, `groupObjectsByFields`, `arrayPartition`, `wait`, `timeMs`, `responseArrayToResponse`, `responseResultsByOutcome`, `normalise`, `objectDigNodeAtPath`.

package/api/shopify/shopifyProductUpdateTrigger.js

@@ -6,32 +6,41 @@ const { ArgsWarden } = require('../utils');
 const { shopifyProductGet, productIdentifierValidator } = require('../shopify/shopifyProductGet');
 const { shopifyProductUpdate } = require('../shopify/shopifyProductUpdate');
 
+const productPayloadValidator = (productPayload) => {
+  const { productIdentifier, productTitle } = productPayload;
+  return productIdentifierValidator(productIdentifier);
+};
+
 const argsWarden = new ArgsWarden([
   ['credsPayload', credsValidator],
-  ['productIdentifier', productIdentifierValidator],
+  ['productPayload', productPayloadValidator],
 ]);
 
 const shopifyProductUpdateTrigger = async (
   credsPayload,
-  productIdentifier,
+  productPayload,
   {
     apiVersion,
-    productTitle,
   } = {},
 ) => {
 
   const rejectResponse = await argsWarden.responseIfRejectingArgs({
     credsPayload,
-    productIdentifier,
+    productPayload,
   });
   if (rejectResponse) {
     return rejectResponse;
   }
 
+  const { 
+    productIdentifier,
+    productTitle,
+  } = productPayload;
+
   if (!productTitle) {
     const productResponse = await shopifyProductGet(
       credsPayload,
-      productIdentifier,
+      productPayload,
       {
         attrs: 'title',
       },
@@ -83,14 +92,13 @@ module.exports = {
     -H "Content-Type: application/json" \
     -d '{
       "credsPayload": { "credsPath": "shopify.au" },
-      "productIdentifier": { "productId": "1234567890" }
+      "productPayload": { "productIdentifier": { "productId": "1234567890" } }
     }'
 
   curl -X POST "http://localhost:8000/shopifyProductUpdateTrigger" \
     -H "Content-Type: application/json" \
     -d '{
       "credsPayload": { "credsPath": "shopify.au" },
-      "productIdentifier": { "productId": "1234567890" },
-      "options": { "productTitle": "Ultra Strength Freeze Ray" }
+      "productPayload": { "productIdentifier": { "productId": "1234567890" }, "productTitle": "Ultra Strength Freeze Ray" },
     }'
 */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@foxtware/mineral",
-  "version": "0.1.39",
+  "version": "1.0.0",
   "bin": {
     "mineral": "bin/mineral.js"
   },