@kairos-es/feed-algolia 0.0.0-stage → 0.0.0

package/dist/dts-esm/sink.d.ts

@@ -0,0 +1,220 @@
+/**
+ * `makeAlgoliaSink` — a `FeedSink` onto one served Algolia index, with both optional
+ * parts: a `FeedResync` built on Algolia's index copy and move, and a
+ * `FeedReconcile` built on `browse`.
+ *
+ * ## The steady state
+ *
+ * `apply` is ONE `batch` to the served index (`records.ts` owns what goes in it and
+ * which records are refused before it is sent), and its receipt is the index and
+ * the task Algolia returned. `confirm` polls `getTask` on that index until the task
+ * is `published`, which is the moment Algolia serves the batch — the moment the
+ * port's `confirm` promises. The poll runs on an Effect `Schedule`: it backs off
+ * from `taskPolling.initialDelay`, doubling to `taskPolling.maxDelay`, and no poll
+ * STARTS after `taskPolling.timeout` has passed since the wait began — the last
+ * wait is cut short to land on that deadline rather than overrun it by up to
+ * `maxDelay`. A task still unpublished then is a `SinkRetryable`: the driver
+ * resends the batch, which is safe, rather than waiting on a task without bound.
+ * A poll that has started can still outlast the deadline by its own request
+ * timeout (`rest.ts`). A `Schedule` rather than a hand-written loop because the
+ * curve is then one declared value, run under the `Clock`, so a test's
+ * `TestClock` drives it like any other wait.
+ *
+ * ## Resync: a fresh index, swapped in whole
+ *
+ * Algolia's `operationIndex` move replaces the destination index atomically, which
+ * is exactly the swap `FeedResync` asks for. So:
+ *
+ * - `begin(generation)` names the target `<served>__kairos_g<generation>_<attempt>`,
+ *   where `<attempt>` is drawn from Effect's `Random` afresh on every call, and
+ *   then COPIES the served index into it with `scope` limited to settings, synonyms
+ *   and rules. The fresh index then ranks and matches exactly as the served one does,
+ *   and holds none of its records. A served index that has never been written has no
+ *   scope to copy, and that is not an error: `begin` reads the served index's
+ *   settings first and makes no copy when it does not exist. Algolia would copy a
+ *   missing source as an empty index, but the copy's task is polled on the served
+ *   index, and `getTask` on an index that does not exist answers 404, so skipping
+ *   the copy is what keeps that wait off a missing index. The name is the ATTEMPT's, not the
+ *   generation's, because two workers can be resyncing one generation at once —
+ *   one of them having lost its lease without knowing it yet — and a target they
+ *   shared is one either could publish half-built; `runFeedSync.ts` owns the
+ *   argument. Random rather than the lease holder's id, because the port hands a
+ *   sink no lease, and a draw needs none.
+ * - `begin` then MARKS the target as its attempt's own: it writes the target's
+ *   `userData` setting as whatever the copy brought from the served index, plus one
+ *   key, `kairosResyncAttempt`, holding the attempt. Algolia creates an index on a
+ *   settings write, so this is also what makes the target EXIST whatever the copy
+ *   did — over a served index never written, and for a rebuild that loads nothing.
+ *   The mark is what `publish` checks, below. It goes in `userData` because that
+ *   is the one setting Algolia keeps for the application's own data and reads
+ *   nothing from, so the mark changes nothing about how the target ranks or
+ *   matches.
+ * - `begin` first drops every OTHER resync index of this served index it can list,
+ *   whatever its generation, settings snapshots included. Each was left by an attempt that will never publish
+ *   it now — one that crashed, one abandoned because the generation moved, or one
+ *   a worker that lost its lease is still loading — and every such index holds
+ *   records against the application's quota; an ephemeral feed, whose generation
+ *   is new on every boot, would otherwise leave one behind per interrupted boot.
+ *   Dropping the one a stale worker is loading is also what makes that worker's
+ *   `publish` fail rather than land. Listing needs the key to carry Algolia's
+ *   `listIndexes` ACL, and the sink REQUIRES it rather than skipping the drop when
+ *   it is missing: without the drop, a stale worker's `publish` could land and
+ *   abandoned targets would pile up. A key without it is refused like any other
+ *   4xx, a `SinkHalted` at the start of the resync, before the copy, so nothing is
+ *   created.
+ * - `load` is `apply`, against the target. It does not check the mark: that would
+ *   cost a settings read per page and still race the drop, and `publish` is where
+ *   a target is served, so `publish` is where the check has to hold.
+ * - `publish` reads the target's settings and refuses, `SinkRetryable`, a target
+ *   that is missing or does not carry this attempt's mark; only then does it move
+ *   the target over the served index and wait on the move's task. Then it reads the
+ *   SERVED index's settings and reports success only if the served index now
+ *   carries this attempt's mark, which only this move could have put there;
+ *   otherwise it fails `SinkRetryable`. That read is the authority, and the
+ *   move's own answer is not: Algolia ignores a move of a missing index "without
+ *   returning an error", so a target dropped after the check gets a task back, the
+ *   task publishes, and the served index still holds the old copy. Taking that for
+ *   success would let the driver save its cursor at the new generation over an
+ *   index that never got it. For the same
+ *   reason a 404 on the move's task poll — `getTask` on an index that no longer
+ *   exists — ends the wait rather than halting, and the read decides. An empty rebuild
+ *   is a move like any other, of a target that holds no records, and it leaves the
+ *   served index empty. A target can go missing before its publish in two ways: a
+ *   newer `begin` dropped it, or something moved it over the served index first.
+ *   Only the second could ever land a half-built target, and the per-attempt name
+ *   is what rules it out: the one `publish` that names this target is this
+ *   attempt's own, so no other worker's can move it. What is left is the first,
+ *   and it has a second form, which is why the mark exists: a worker that lost its
+ *   lease without knowing it goes on LOADING into its dropped target, and Algolia
+ *   creates an index on a batch write, so the target comes back holding the stale
+ *   worker's later pages alone. A name alone cannot tell that index from the one
+ *   `begin` made; the missing mark can, since only `begin` writes it. Either way
+ *   `publish` fails `SinkRetryable` and the resync starts again. It never touches
+ *   the served index on that account, since the worker that finds its target gone
+ *   is the one most likely to have lost the lease. An index a stale load brought
+ *   back is left where it is, and the next `begin` drops it with the rest.
+ * - Before it marks the target, `begin` takes a SNAPSHOT of the served index's
+ *   settings into `<target>__settings`, a settings-scoped copy, or an empty
+ *   settings write when the served index does not exist. After the move has landed,
+ *   `publish` takes the mark back off the served index by copying the snapshot's
+ *   settings over it, scoped to settings, then drops the snapshot. The drop is a
+ *   best effort of its own, and runs whether or not the restore landed. A copy is
+ *   the one write that REMOVES a setting: Algolia stores a `null` written to `userData` as
+ *   the value `null`, and `{}` or `resetToDefault` leave an empty object, so a
+ *   rebuild of an index that had no `userData` would otherwise leave one behind.
+ *   The snapshot is a second copy of the same served index's settings, made just
+ *   after the target's, so the restore changes nothing else unless those settings
+ *   changed between the two copies, when it applies the later ones. Its scope is
+ *   SETTINGS ONLY, and must stay so: the snapshot holds no synonyms or rules, so a
+ *   wider restore would wipe the served index's. Records are untouched, and so are the
+ *   served index's replicas (see below). A mark an earlier failed restore left on the
+ *   served index is taken off the snapshot as it is made, so it cannot come
+ *   back; when it was all the `userData` there was, a settings write is the only
+ *   way to take it off, and the `null` Algolia stores for it is what remains.
+ *   The move and the restore are two steps, so a search index briefly carries the
+ *   key, which nothing reads. The restore is BEST-EFFORT: the swap has
+ *   already landed, and failing `publish` now would tell the driver it had not. A
+ *   snapshot that is gone (a newer `begin` dropped it) is not copied, since a copy
+ *   from a missing index changes nothing and its task never publishes; the mark
+ *   then stays until the next rebuild's restore, and a warning says so. One that
+ *   goes between that check and the copy is reported the same way, once the wait
+ *   on the copy's task ends: at the first poll, if `getTask` on the missing index
+ *   answers 404 as above, and at `pollTimeout` if it does not. The swap has landed
+ *   by then, so the worst cost is a `publish` that returns late, not a wrong one.
+ *
+ * Check-then-move leaves no gap a stale load can use: the only worker that loads
+ * into this target is the one publishing it, and its loads have finished. A drop
+ * between the check and the move makes the move a no-op, which the read of the
+ * served index after it turns into a `SinkRetryable`, as above. Taking the mark
+ * off the TARGET before the move would open one: a drop between that write and
+ * the move, and the write itself would bring back an empty target for the move to
+ * land.
+ *
+ * Replicas are the one thing this does not settle by construction: a copy with
+ * `scope` is not documented to carry a primary's replicas to the target, nor a
+ * move over a primary, or the restore's settings-scoped copy over it, to keep
+ * them. Measured against a real application, all three leave them where they
+ * were, and the package's live test checks it after `publish` when credentials
+ * are present.
+ *
+ * ## Reconcile
+ *
+ * `listKeys` browses the served index page by page, retrieving `objectID` and nothing
+ * else, so listing a large index costs the ids and not the records.
+ *
+ * ## The key's ACLs
+ *
+ * A scoped key needs `addObject` (record writes, and the copy and the move),
+ * `deleteObject`, `deleteIndex` (the drop), `settings` (the mark check),
+ * `editSettings` (writing and removing the mark), `browse` (reconcile) and
+ * `listIndexes` (the drop's listing), on the served index and its resync indices.
+ * The README states the same set, ACL by ACL, for a consumer.
+ *
+ * ## What the sink holds
+ *
+ * Nothing between calls but its options and the captured `HttpClient`: no queue, no
+ * cursor, no buffered batch. Every piece of progress is the driver's, which is what
+ * lets a resent batch and a restarted resync be safe.
+ */
+import { type FeedSink, type Generation, type SinkName } from '@kairos-es/feed';
+import { Duration, Effect, type Schema } from 'effect';
+import type { HttpClient } from 'effect/http';
+import { AlgoliaIndexName, type AlgoliaTaskId } from './identifiers.js';
+import { type AlgoliaConnection } from './rest.js';
+/** What `apply` and `load` return: the index the batch went to, and its task. */
+export interface AlgoliaReceipt {
+    readonly index: AlgoliaIndexName;
+    readonly taskID: AlgoliaTaskId;
+}
+/** How `confirm`, and every wait on an index operation, polls `getTask`. */
+export interface AlgoliaTaskPolling {
+    /** The first wait between polls. Default 100 ms. */
+    readonly initialDelay?: Duration.Input;
+    /** The cap the wait doubles up to. Default 2 seconds. */
+    readonly maxDelay?: Duration.Input;
+    /** How long to poll before giving up, retryably. Default 60 seconds. */
+    readonly timeout?: Duration.Input;
+}
+/** Options for {@link makeAlgoliaSink}. */
+export interface AlgoliaSinkOptions<D, I> extends AlgoliaConnection {
+    /** The sink's name: one cursor per feed and sink. */
+    readonly name: SinkName;
+    /** The SERVED index: the one search reads. */
+    readonly index: AlgoliaIndexName;
+    /** The document's schema; a record is its encoded form plus `objectID`. */
+    readonly doc: Schema.Codec<D, I>;
+    /**
+     * Records larger than this, in bytes of their JSON with `objectID`, are refused
+     * (parked) before sending. Default 10 000 bytes — {@link DEFAULT_MAX_RECORD_BYTES}
+     * — which is the record size limit of Algolia's smallest plan, so the default
+     * refuses nothing Algolia would take on any plan.
+     *
+     * On a plan with a larger limit, raise this to that limit (100 000 for a plan
+     * whose records may be 100 KB, say); records between the two are then sent
+     * rather than parked. Set it no higher than the plan's own limit: a record over
+     * the plan's limit that passes this check costs a round trip, which Algolia
+     * refuses whole, before it is parked by the answer naming it.
+     */
+    readonly maxRecordBytes?: number;
+    readonly taskPolling?: AlgoliaTaskPolling;
+    /** Ids per `browse` page when listing keys. Default 1 000, Algolia's maximum. */
+    readonly browsePageSize?: number;
+}
+/**
+ * The resync target's name for one attempt at `generation`. PACKAGE-INTERNAL, and
+ * kept off the entry point deliberately: `attempt` is what `begin` draws for each
+ * call, so a real target's name is the one `begin` returns, and a consumer has no
+ * use for composing one.
+ *
+ * It does not check `attempt`. Every caller here passes lower-case base-36
+ * digits, which `begin` draws and the tests write by hand, and that shape is all
+ * `resyncGenerationOf` below recognises: a suffix of any other shape would name an
+ * index that `begin` never drops and `publish` never accepts.
+ */
+export declare const resyncIndexName: (served: AlgoliaIndexName, generation: Generation, attempt: string) => AlgoliaIndexName;
+/**
+ * Build an Algolia sink over the `HttpClient` in context; see the module doc.
+ * Dies at once on options out of range.
+ */
+export declare const makeAlgoliaSink: <D, I>(options: AlgoliaSinkOptions<D, I>) => Effect.Effect<FeedSink<D, AlgoliaReceipt>, never, HttpClient.HttpClient>;
+//# sourceMappingURL=sink.d.ts.map

package/dist/dts-esm/sink.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"sink.d.ts","sourceRoot":"","sources":["../../src/sink.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6JG;AACH,OAAO,EAIL,KAAK,QAAQ,EACb,KAAK,UAAU,EAGf,KAAK,QAAQ,EAEd,MAAM,iBAAiB,CAAA;AACxB,OAAO,EAEL,QAAQ,EACR,MAAM,EAIN,KAAK,MAAM,EAEZ,MAAM,QAAQ,CAAA;AACf,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAA;AAE7C,OAAO,EAAE,gBAAgB,EAAE,KAAK,aAAa,EAAE,MAAM,eAAe,CAAA;AAEpE,OAAO,EAAE,KAAK,iBAAiB,EAAmB,MAAM,QAAQ,CAAA;AAEhE,iFAAiF;AACjF,MAAM,WAAW,cAAc;IAC7B,QAAQ,CAAC,KAAK,EAAE,gBAAgB,CAAA;IAChC,QAAQ,CAAC,MAAM,EAAE,aAAa,CAAA;CAC/B;AAED,4EAA4E;AAC5E,MAAM,WAAW,kBAAkB;IACjC,oDAAoD;IACpD,QAAQ,CAAC,YAAY,CAAC,EAAE,QAAQ,CAAC,KAAK,CAAA;IACtC,yDAAyD;IACzD,QAAQ,CAAC,QAAQ,CAAC,EAAE,QAAQ,CAAC,KAAK,CAAA;IAClC,wEAAwE;IACxE,QAAQ,CAAC,OAAO,CAAC,EAAE,QAAQ,CAAC,KAAK,CAAA;CAClC;AAED,2CAA2C;AAC3C,MAAM,WAAW,kBAAkB,CAAC,CAAC,EAAE,CAAC,CAAE,SAAQ,iBAAiB;IACjE,qDAAqD;IACrD,QAAQ,CAAC,IAAI,EAAE,QAAQ,CAAA;IACvB,8CAA8C;IAC9C,QAAQ,CAAC,KAAK,EAAE,gBAAgB,CAAA;IAChC,2EAA2E;IAC3E,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAA;IAChC;;;;;;;;;;;OAWG;IACH,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,CAAA;IAChC,QAAQ,CAAC,WAAW,CAAC,EAAE,kBAAkB,CAAA;IACzC,iFAAiF;IACjF,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,CAAA;CACjC;AAED;;;;;;;;;;GAUG;AACH,eAAO,MAAM,eAAe,GAC1B,QAAQ,gBAAgB,EACxB,YAAY,UAAU,EACtB,SAAS,MAAM,KACd,gBACmE,CAAA;AA2JtE;;;GAGG;AACH,eAAO,MAAM,eAAe,GAAI,CAAC,EAAE,CAAC,EAClC,SAAS,kBAAkB,CAAC,CAAC,EAAE,CAAC,CAAC,KAChC,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,EAAE,cAAc,CAAC,EAAE,KAAK,EAAE,UAAU,CAAC,UAAU,CAwUtE,CAAA"}

package/dist/esm/classify.js

@@ -0,0 +1,145 @@
+/**
+ * How an Algolia answer becomes one of the feed's three sink failures.
+ *
+ * The driver's whole behaviour on failure follows from which of the three a sink
+ * names (`@kairos-es/feed`'s `errors.ts` owns that), so this mapping is the most
+ * consequential thing in the package and it is kept pure, in one place, and tested
+ * on its own. The rules:
+ *
+ * - **A 429 is `SinkRetryable`**, carrying `Retry-After` when Algolia sends one,
+ *   so the driver backs off no less than Algolia asked — UNLESS its message names
+ *   a QUOTA rather than a rate: Algolia answers 429 for an application over its
+ *   record quota or its indexing file-size allowance too, and neither lifts by
+ *   waiting, so retrying would only hold the feed until supervision reported it
+ *   stalled, with Algolia's reason lost. Such a 429 is `SinkHalted`, carrying
+ *   Algolia's message. The wordings recognised are {@link QUOTA_429}; a 429 whose
+ *   wording is not among them stays retryable, which is the safe side to err on.
+ * - **Every 5xx is `SinkRetryable`, and so is a 1xx or 3xx**, a status no call
+ *   here expects and one the transport has already tried on every host, as the
+ *   official client does. A transport failure — no answer at all — is retryable
+ *   too: the request may never have arrived, and resending is safe because every
+ *   write this sink makes is a full-record replace or a delete.
+ * - **A 400 that names an `objectID` in the batch is `SinkRejected` for that key.**
+ *   Algolia refuses a whole batch for one bad record, and names the record in its
+ *   message (`… objectID=<id> …`). Parking just that key lets the driver send the
+ *   rest again without it, which is the difference between one record missing from
+ *   search and the whole feed stopping. The key has to be one the batch carried: an
+ *   id the batch did not send is not a record the driver can park, so such an
+ *   answer falls through to the next rule. The message does not delimit the id, and
+ *   a feed key may hold a space, a comma or anything else, so the id is not parsed
+ *   out of the message: the batch's own keys are matched against what follows
+ *   `objectID=` (`objectIdNamedIn`).
+ * - **Every other 4xx is `SinkHalted`.** Bad credentials, a key without the right
+ *   ACL, an index name Algolia will not accept, a request this package built wrong:
+ *   none of them is mended by sending again, so the worker stops and supervision
+ *   reports it.
+ *
+ * Nothing here reads the request, and nothing here sees a header but `Retry-After`,
+ * so no classification can carry the admin key into a reason string.
+ */
+import { SinkHalted, SinkRejected, SinkRetryable } from '@kairos-es/feed';
+import { DateTime, Duration, Option } from 'effect';
+const OBJECT_ID_MARKER = 'objectID=';
+/** What may follow an id in Algolia's message: the end, a space or punctuation. */
+const ID_ENDS = /^(?:$|[\s,;.:)])/;
+/**
+ * The key of `batchKeys` an Algolia error message names, when it names one.
+ *
+ * Algolia's record-level refusals all carry `objectID=<id>` (the record-too-big
+ * message is the common one), with nothing marking where the id ends. So each
+ * batch key is tried against the text after each `objectID=`, and it matches when
+ * that text starts with the key and the key is followed by the end of the message,
+ * a space or punctuation; the longest key that matches wins, so of `big` and
+ * `big one` the message `objectID=big one is too big` names the second.
+ */
+export const objectIdNamedIn = (message, batchKeys) => {
+  let named;
+  let from = message.indexOf(OBJECT_ID_MARKER);
+  while (from >= 0) {
+    const after = message.slice(from + OBJECT_ID_MARKER.length);
+    for (const key of batchKeys) {
+      if (after.startsWith(key) && ID_ENDS.test(after.slice(key.length)) && (named === undefined || key.length > named.length)) {
+        named = key;
+      }
+    }
+    from = message.indexOf(OBJECT_ID_MARKER, from + 1);
+  }
+  return Option.fromUndefinedOr(named);
+};
+/**
+ * The 429 wordings that name a quota, not a rate, matched case-insensitively
+ * anywhere in Algolia's message: an application over its record quota, or over
+ * its allowance of indexing by file size. See the module doc for why each halts.
+ */
+export const QUOTA_429 = [/quota/i, /file size/i];
+/**
+ * `Retry-After` as a duration, given the current time.
+ *
+ * The header is either a whole number of seconds or an HTTP date. A date is turned
+ * into a wait against `now`, which the caller reads from `Clock` so a test's
+ * `TestClock` governs it; a date already past is a wait of zero. Anything else is
+ * ignored rather than guessed at, and the driver's own back-off applies.
+ */
+export const parseRetryAfter = (header, nowMillis) => {
+  const trimmed = header.trim();
+  if (/^\d+$/.test(trimmed)) {
+    return Option.some(Duration.seconds(Number(trimmed)));
+  }
+  // A bare number with a fraction or a sign is neither form; do not let the date
+  // parser read it as a year.
+  if (/^[-+\d.]+$/.test(trimmed)) {
+    return Option.none();
+  }
+  return Option.map(DateTime.make(trimmed), at => Duration.millis(Math.max(0, DateTime.toEpochMillis(at) - nowMillis)));
+};
+/**
+ * Classify an Algolia error answer; see the module doc for the rules.
+ *
+ * `batchKeys` is the set of keys the request carried, empty for a call that sends
+ * no records, so only a write can ever produce a `SinkRejected`.
+ */
+export const classifyAnswer = (call, answer, batchKeys, nowMillis) => {
+  const reason = `Algolia answered ${answer.status} to ${call.method} ${call.path}: ${answer.message}`;
+  if (answer.status === 429 && QUOTA_429.some(wording => wording.test(answer.message))) {
+    return new SinkHalted({
+      reason
+    });
+  }
+  if (answer.status === 429 || answer.status >= 500 || answer.status < 200 || answer.status >= 300 && answer.status < 400) {
+    const retryAfter = Option.flatMap(answer.retryAfter, header => parseRetryAfter(header, nowMillis));
+    return new SinkRetryable({
+      reason,
+      ...(Option.isSome(retryAfter) ? {
+        retryAfter: retryAfter.value
+      } : {})
+    });
+  }
+  if (answer.status === 400) {
+    const named = objectIdNamedIn(answer.message, batchKeys);
+    if (Option.isSome(named)) {
+      const key = named.value;
+      return new SinkRejected({
+        keys: [key],
+        reasons: new Map([[key, answer.message]])
+      });
+    }
+  }
+  return new SinkHalted({
+    reason
+  });
+};
+/**
+ * A request that got no answer at all. Retryable: see the module doc for why a
+ * resend is safe.
+ */
+export const transportFailure = (call, detail) => new SinkRetryable({
+  reason: `no answer from Algolia to ${call.method} ${call.path}: ${detail}`
+});
+/**
+ * An answer this package cannot read: a success whose body is not the shape
+ * Algolia documents. Halted, because sending again will get the same body back.
+ */
+export const unreadableAnswer = (call, detail) => new SinkHalted({
+  reason: `unreadable answer from Algolia to ${call.method} ${call.path}: ${detail}`
+});
+//# sourceMappingURL=classify.js.map

package/dist/esm/classify.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"classify.js","names":["SinkHalted","SinkRejected","SinkRetryable","DateTime","Duration","Option","OBJECT_ID_MARKER","ID_ENDS","objectIdNamedIn","message","batchKeys","named","from","indexOf","after","slice","length","key","startsWith","test","undefined","fromUndefinedOr","QUOTA_429","parseRetryAfter","header","nowMillis","trimmed","trim","some","seconds","Number","none","map","make","at","millis","Math","max","toEpochMillis","classifyAnswer","call","answer","reason","status","method","path","wording","retryAfter","flatMap","isSome","value","keys","reasons","Map","transportFailure","detail","unreadableAnswer"],"sources":["../../src/classify.ts"],"sourcesContent":[null],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAyCA,SAASA,UAAU,EAAEC,YAAY,EAAEC,aAAa,QAAQ,iBAAiB;AACzE,SAASC,QAAQ,EAAEC,QAAQ,EAAEC,MAAM,QAAQ,QAAQ;AAoBnD,MAAMC,gBAAgB,GAAG,WAAW;AAEpC;AACA,MAAMC,OAAO,GAAG,kBAAkB;AAElC;;;;;;;;;;AAUA,OAAO,MAAMC,eAAe,GAAGA,CAC7BC,OAAe,EACfC,SAA+B,KACL;EAC1B,IAAIC,KAA0B;EAC9B,IAAIC,IAAI,GAAGH,OAAO,CAACI,OAAO,CAACP,gBAAgB,CAAC;EAC5C,OAAOM,IAAI,IAAI,CAAC,EAAE;IAChB,MAAME,KAAK,GAAGL,OAAO,CAACM,KAAK,CAACH,IAAI,GAAGN,gBAAgB,CAACU,MAAM,CAAC;IAC3D,KAAK,MAAMC,GAAG,IAAIP,SAAS,EAAE;MAC3B,IACEI,KAAK,CAACI,UAAU,CAACD,GAAG,CAAC,IACrBV,OAAO,CAACY,IAAI,CAACL,KAAK,CAACC,KAAK,CAACE,GAAG,CAACD,MAAM,CAAC,CAAC,KACpCL,KAAK,KAAKS,SAAS,IAAIH,GAAG,CAACD,MAAM,GAAGL,KAAK,CAACK,MAAM,CAAC,EAClD;QACAL,KAAK,GAAGM,GAAG;MACb;IACF;IACAL,IAAI,GAAGH,OAAO,CAACI,OAAO,CAACP,gBAAgB,EAAEM,IAAI,GAAG,CAAC,CAAC;EACpD;EACA,OAAOP,MAAM,CAACgB,eAAe,CAACV,KAAK,CAAC;AACtC,CAAC;AAED;;;;;AAKA,OAAO,MAAMW,SAAS,GAA0B,CAAC,QAAQ,EAAE,YAAY,CAAC;AAExE;;;;;;;;AAQA,OAAO,MAAMC,eAAe,GAAGA,CAC7BC,MAAc,EACdC,SAAiB,KACmB;EACpC,MAAMC,OAAO,GAAGF,MAAM,CAACG,IAAI,EAAE;EAC7B,IAAI,OAAO,CAACR,IAAI,CAACO,OAAO,CAAC,EAAE;IACzB,OAAOrB,MAAM,CAACuB,IAAI,CAACxB,QAAQ,CAACyB,OAAO,CAACC,MAAM,CAACJ,OAAO,CAAC,CAAC,CAAC;EACvD;EACA;EACA;EACA,IAAI,YAAY,CAACP,IAAI,CAACO,OAAO,CAAC,EAAE;IAC9B,OAAOrB,MAAM,CAAC0B,IAAI,EAAE;EACtB;EACA,OAAO1B,MAAM,CAAC2B,GAAG,CAAC7B,QAAQ,CAAC8B,IAAI,CAACP,OAAO,CAAC,EAAGQ,EAAE,IAC3C9B,QAAQ,CAAC+B,MAAM,CAACC,IAAI,CAACC,GAAG,CAAC,CAAC,EAAElC,QAAQ,CAACmC,aAAa,CAACJ,EAAE,CAAC,GAAGT,SAAS,CAAC,CAAC,CACrE;AACH,CAAC;AAED;;;;;;AAMA,OAAO,MAAMc,cAAc,GAAGA,CAC5BC,IAAsB,EACtBC,MAA0B,EAC1B/B,SAA+B,EAC/Be,SAAiB,KACC;EAClB,MAAMiB,MAAM,GAAG,oBAAoBD,MAAM,CAACE,MAAM,OAAOH,IAAI,CAACI,MAAM,IAAIJ,IAAI,CAACK,IAAI,KAAKJ,MAAM,CAAChC,OAAO,EAAE;EACpG,IACEgC,MAAM,CAACE,MAAM,KAAK,GAAG,IACrBrB,SAAS,CAACM,IAAI,CAAEkB,OAAO,IAAKA,OAAO,CAAC3B,IAAI,CAACsB,MAAM,CAAChC,OAAO,CAAC,CAAC,EACzD;IACA,OAAO,IAAIT,UAAU,CAAC;MAAE0C;IAAM,CAAE,CAAC;EACnC;EACA,IACED,MAAM,CAACE,MAAM,KAAK,GAAG,IACrBF,MAAM,CAACE,MAAM,IAAI,GAAG,IACpBF,MAAM,CAACE,MAAM,GAAG,GAAG,IAClBF,MAAM,CAACE,MAAM,IAAI,GAAG,IAAIF,MAAM,CAACE,MAAM,GAAG,GAAI,EAC7C;IACA,MAAMI,UAAU,GAAG1C,MAAM,CAAC2C,OAAO,CAACP,MAAM,CAACM,UAAU,EAAGvB,MAAM,IAC1DD,eAAe,CAACC,MAAM,EAAEC,SAAS,CAAC,CACnC;IACD,OAAO,IAAIvB,aAAa,CAAC;MACvBwC,MAAM;MACN,IAAIrC,MAAM,CAAC4C,MAAM,CAACF,UAAU,CAAC,GAAG;QAAEA,UAAU,EAAEA,UAAU,CAACG;MAAK,CAAE,GAAG,EAAE;KACtE,CAAC;EACJ;EACA,IAAIT,MAAM,CAACE,MAAM,KAAK,GAAG,EAAE;IACzB,MAAMhC,KAAK,GAAGH,eAAe,CAACiC,MAAM,CAAChC,OAAO,EAAEC,SAAS,CAAC;IACxD,IAAIL,MAAM,CAAC4C,MAAM,CAACtC,KAAK,CAAC,EAAE;MACxB,MAAMM,GAAG,GAAGN,KAAK,CAACuC,KAAK;MACvB,OAAO,IAAIjD,YAAY,CAAC;QACtBkD,IAAI,EAAE,CAAClC,GAAG,CAAC;QACXmC,OAAO,EAAE,IAAIC,GAAG,CAAC,CAAC,CAACpC,GAAG,EAAEwB,MAAM,CAAChC,OAAO,CAAC,CAAC;OACzC,CAAC;IACJ;EACF;EACA,OAAO,IAAIT,UAAU,CAAC;IAAE0C;EAAM,CAAE,CAAC;AACnC,CAAC;AAED;;;;AAIA,OAAO,MAAMY,gBAAgB,GAAGA,CAC9Bd,IAAsB,EACtBe,MAAc,KAEd,IAAIrD,aAAa,CAAC;EAChBwC,MAAM,EAAE,6BAA6BF,IAAI,CAACI,MAAM,IAAIJ,IAAI,CAACK,IAAI,KAAKU,MAAM;CACzE,CAAC;AAEJ;;;;AAIA,OAAO,MAAMC,gBAAgB,GAAGA,CAC9BhB,IAAsB,EACtBe,MAAc,KAEd,IAAIvD,UAAU,CAAC;EACb0C,MAAM,EAAE,qCAAqCF,IAAI,CAACI,MAAM,IAAIJ,IAAI,CAACK,IAAI,KAAKU,MAAM;CACjF,CAAC","ignoreList":[]}

package/dist/esm/identifiers.js

@@ -0,0 +1,31 @@
+/**
+ * The Algolia sink's branded identifiers.
+ *
+ * Three values cross this package's boundary that would otherwise be bare
+ * strings and numbers, and each is easy to pass in the place of another: an
+ * application id where an index name belongs, a feed key where an index name
+ * belongs, a task id from one index polled against another. Branding them is the
+ * repository's standing rule for a domain identifier, and here it is also what
+ * keeps a `Receipt` — an index and a task id — from being assembled by hand out of
+ * values that never belonged together.
+ */
+import { Schema } from 'effect';
+/**
+ * The Algolia application id: the first half of the credentials, and the name of
+ * the application's hosts. Not a secret — it is sent beside every search a browser
+ * makes — so it is a plain branded string, unlike the admin key.
+ */
+export const AlgoliaAppId = /*#__PURE__*/Schema.String.pipe(/*#__PURE__*/Schema.check(/*#__PURE__*/Schema.isNonEmpty()), /*#__PURE__*/Schema.brand('AlgoliaAppId'));
+/**
+ * The name of an Algolia index. The sink's SERVED index is one; a resync builds into
+ * another, named after the served one, the generation it rebuilds and the attempt.
+ */
+export const AlgoliaIndexName = /*#__PURE__*/Schema.String.pipe(/*#__PURE__*/Schema.check(/*#__PURE__*/Schema.isNonEmpty()), /*#__PURE__*/Schema.brand('AlgoliaIndexName'));
+/**
+ * An Algolia task id: what every write returns, and what `getTask` is polled with
+ * until the write is published. A task belongs to the index it was issued on, and
+ * is polled there and nowhere else — which is why a receipt carries the index
+ * beside it.
+ */
+export const AlgoliaTaskId = /*#__PURE__*/Schema.Number.pipe(/*#__PURE__*/Schema.check(/*#__PURE__*/Schema.isInt(), /*#__PURE__*/Schema.isGreaterThanOrEqualTo(0)), /*#__PURE__*/Schema.brand('AlgoliaTaskId'));
+//# sourceMappingURL=identifiers.js.map

package/dist/esm/identifiers.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"identifiers.js","names":["Schema","AlgoliaAppId","String","pipe","check","isNonEmpty","brand","AlgoliaIndexName","AlgoliaTaskId","Number","isInt","isGreaterThanOrEqualTo"],"sources":["../../src/identifiers.ts"],"sourcesContent":[null],"mappings":"AAAA;;;;;;;;;;;AAWA,SAASA,MAAM,QAAQ,QAAQ;AAE/B;;;;;AAKA,OAAO,MAAMC,YAAY,gBAAGD,MAAM,CAACE,MAAM,CAACC,IAAI,cAC5CH,MAAM,CAACI,KAAK,cAACJ,MAAM,CAACK,UAAU,EAAE,CAAC,eACjCL,MAAM,CAACM,KAAK,CAAC,cAAc,CAAC,CAC7B;AAGD;;;;AAIA,OAAO,MAAMC,gBAAgB,gBAAGP,MAAM,CAACE,MAAM,CAACC,IAAI,cAChDH,MAAM,CAACI,KAAK,cAACJ,MAAM,CAACK,UAAU,EAAE,CAAC,eACjCL,MAAM,CAACM,KAAK,CAAC,kBAAkB,CAAC,CACjC;AAGD;;;;;;AAMA,OAAO,MAAME,aAAa,gBAAGR,MAAM,CAACS,MAAM,CAACN,IAAI,cAC7CH,MAAM,CAACI,KAAK,cAACJ,MAAM,CAACU,KAAK,EAAE,eAAEV,MAAM,CAACW,sBAAsB,CAAC,CAAC,CAAC,CAAC,eAC9DX,MAAM,CAACM,KAAK,CAAC,eAAe,CAAC,CAC9B","ignoreList":[]}

package/dist/esm/index.js

@@ -0,0 +1,38 @@
+/**
+ * @kairos-es/feed-algolia — a `FeedSink` onto an Algolia index, with both of the
+ * port's optional parts: a `FeedResync` that rebuilds into a fresh index and moves
+ * it over the served one, and a `FeedReconcile` that lists the served index's ids.
+ *
+ * It talks to Algolia's REST API through `effect`'s own `HttpClient`, so it peers
+ * `effect` and `@kairos-es/feed` and nothing else, and the application chooses the
+ * HTTP transport by providing the client — `FetchHttpClient.layer` from
+ * `effect/http`, say.
+ *
+ * Where each part's argument lives:
+ *
+ * - `makeAlgoliaSink` (`sink.ts`): the steady state, resync and reconcile, and how
+ *   `confirm` waits.
+ * - Encoding and the size pre-check (`records.ts`): why only full-record writes,
+ *   and why oversized records are refused before they are sent.
+ * - Error classification (`classify.ts`): which Algolia answers are retryable,
+ *   which refuse one record, and which stop the worker.
+ * - The REST calls (`rest.ts`): why no SDK, the host failover kept from it, and
+ *   how the admin key is kept out of anything printed.
+ *
+ * Wiring:
+ *
+ * ```ts
+ * const sink = yield* makeAlgoliaSink({
+ *   name: SinkName.make('search'),
+ *   appId: AlgoliaAppId.make(config.appId),
+ *   adminKey: config.adminKey, // a Redacted<string>, from the application's config
+ *   index: AlgoliaIndexName.make('courses'),
+ *   doc: CourseDoc,
+ * }).pipe(Effect.provide(FetchHttpClient.layer))
+ * yield* Effect.forkScoped(runFeedSync({ source, sink, store }))
+ * ```
+ */
+export { AlgoliaAppId, AlgoliaIndexName, AlgoliaTaskId } from "./identifiers.js";
+export { DEFAULT_MAX_RECORD_BYTES } from "./records.js";
+export { makeAlgoliaSink } from "./sink.js";
+//# sourceMappingURL=index.js.map

package/dist/esm/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","names":["AlgoliaAppId","AlgoliaIndexName","AlgoliaTaskId","DEFAULT_MAX_RECORD_BYTES","makeAlgoliaSink"],"sources":["../../src/index.ts"],"sourcesContent":[null],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAmCA,SACEA,YAAY,EACZC,gBAAgB,EAChBC,aAAa,QACR,kBAAe;AACtB,SAASC,wBAAwB,QAAQ,cAAW;AAEpD,SAIEC,eAAe,QACV,WAAQ","ignoreList":[]}

package/dist/esm/package.json

@@ -0,0 +1,4 @@
+{
+  "type": "module",
+  "sideEffects": []
+}

package/dist/esm/records.js

@@ -0,0 +1,96 @@
+/**
+ * Turning a batch of feed changes into Algolia batch operations, and refusing the
+ * records Algolia would refuse BEFORE they are sent.
+ *
+ * ## Full-record replace, and nothing partial
+ *
+ * An upsert is `addObject` with the record's `objectID` set to the feed key, which
+ * replaces whatever Algolia held under that id; a delete is `deleteObject`. Both
+ * leave the index the same however many times they are sent, which is the
+ * idempotence the driver relies on when it resends a batch it could not confirm.
+ * Algolia's partial operations (`partialUpdateObject`, the built-in `Increment`
+ * and its kin) do not: applied twice they apply twice. They are never used.
+ *
+ * The key is written LAST, over the encoded document, so a document that carries an
+ * `objectID` field of its own cannot point a record at another key.
+ *
+ * ## Why oversized records are refused here
+ *
+ * Algolia refuses a whole `batch` when one record in it is over the plan's record
+ * size limit. Its answer does name the record, and `classify.ts` turns that into a
+ * rejection of that one key — but only after a round trip that wrote nothing, and
+ * only for the first oversized record the batch held. Checking every record's size
+ * up front rejects all of them at once, so the driver's resend of the rest goes
+ * through first time. The 400 path stays as the backstop for a limit configured
+ * higher than the plan's.
+ *
+ * The size is the record's JSON in UTF-8 bytes, `objectID` included, which is how
+ * Algolia measures it. The default limit is the smallest Algolia plan's (10 KB);
+ * an application on a larger plan raises it.
+ *
+ * ## A document that does not encode
+ *
+ * The document schema is the read model's, and a document it cannot encode — or one
+ * that encodes to something other than a JSON object, which is all an Algolia
+ * record can be — is a defect in the read model, not a record the sink refuses. It
+ * dies, loudly, rather than being parked where nobody looks.
+ */
+import { SinkRejected } from '@kairos-es/feed';
+import { Array as Arr, Effect, Schema } from 'effect';
+/** The smallest Algolia plan's record size limit, in bytes. */
+export const DEFAULT_MAX_RECORD_BYTES = 10_000;
+const utf8 = /*#__PURE__*/new TextEncoder();
+/** A record's size as Algolia measures it: its JSON, in UTF-8 bytes. */
+export const recordBytes = record => utf8.encode(JSON.stringify(record)).length;
+const isJsonObject = u => typeof u === 'object' && u !== null && !Array.isArray(u);
+/**
+ * Encode a batch, failing `SinkRejected` with EVERY oversized key when there is
+ * one; see the module doc.
+ */
+export const encodeBatch = (doc, maxRecordBytes, batch) => Effect.gen(function* () {
+  const encode = Schema.encodeEffect(doc);
+  const operations = [];
+  const keys = new Set();
+  const reasons = new Map();
+  for (const change of batch) {
+    keys.add(change.key);
+    if (change._tag === 'Delete') {
+      operations.push({
+        action: 'deleteObject',
+        body: {
+          objectID: change.key
+        }
+      });
+      continue;
+    }
+    const encoded = yield* Effect.orDie(encode(change.doc));
+    if (!isJsonObject(encoded)) {
+      return yield* Effect.die(new Error(`feed-algolia: the document for key '${change.key}' encodes to ${encoded === null ? 'null' : typeof encoded}, and an Algolia record must be a JSON object`));
+    }
+    const record = {
+      ...encoded,
+      objectID: change.key
+    };
+    const size = recordBytes(record);
+    if (size > maxRecordBytes) {
+      reasons.set(change.key, `the record is ${size} bytes, over the configured maxRecordBytes of ${maxRecordBytes}`);
+      continue;
+    }
+    operations.push({
+      action: 'addObject',
+      body: record
+    });
+  }
+  const rejected = Array.from(reasons.keys());
+  if (Arr.isReadonlyArrayNonEmpty(rejected)) {
+    return yield* Effect.fail(new SinkRejected({
+      keys: rejected,
+      reasons
+    }));
+  }
+  return {
+    operations,
+    keys
+  };
+});
+//# sourceMappingURL=records.js.map

package/dist/esm/records.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"records.js","names":["SinkRejected","Array","Arr","Effect","Schema","DEFAULT_MAX_RECORD_BYTES","utf8","TextEncoder","recordBytes","record","encode","JSON","stringify","length","isJsonObject","u","isArray","encodeBatch","doc","maxRecordBytes","batch","gen","encodeEffect","operations","keys","Set","reasons","Map","change","add","key","_tag","push","action","body","objectID","encoded","orDie","die","Error","size","set","rejected","from","isReadonlyArrayNonEmpty","fail"],"sources":["../../src/records.ts"],"sourcesContent":[null],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAqCA,SAAwCA,YAAY,QAAQ,iBAAiB;AAC7E,SAASC,KAAK,IAAIC,GAAG,EAAEC,MAAM,EAAEC,MAAM,QAAQ,QAAQ;AAIrD;AACA,OAAO,MAAMC,wBAAwB,GAAG,MAAM;AAE9C,MAAMC,IAAI,gBAAG,IAAIC,WAAW,EAAE;AAE9B;AACA,OAAO,MAAMC,WAAW,GACtBC,MAAyC,IAC9BH,IAAI,CAACI,MAAM,CAACC,IAAI,CAACC,SAAS,CAACH,MAAM,CAAC,CAAC,CAACI,MAAM;AAQvD,MAAMC,YAAY,GAAIC,CAAU,IAC9B,OAAOA,CAAC,KAAK,QAAQ,IAAIA,CAAC,KAAK,IAAI,IAAI,CAACd,KAAK,CAACe,OAAO,CAACD,CAAC,CAAC;AAE1D;;;;AAIA,OAAO,MAAME,WAAW,GAAGA,CACzBC,GAAuB,EACvBC,cAAsB,EACtBC,KAA2C,KAE3CjB,MAAM,CAACkB,GAAG,CAAC,aAAS;EAClB,MAAMX,MAAM,GAAGN,MAAM,CAACkB,YAAY,CAACJ,GAAG,CAAC;EACvC,MAAMK,UAAU,GAA0B,EAAE;EAC5C,MAAMC,IAAI,GAAG,IAAIC,GAAG,EAAW;EAC/B,MAAMC,OAAO,GAAG,IAAIC,GAAG,EAAmB;EAC1C,KAAK,MAAMC,MAAM,IAAIR,KAAK,EAAE;IAC1BI,IAAI,CAACK,GAAG,CAACD,MAAM,CAACE,GAAG,CAAC;IACpB,IAAIF,MAAM,CAACG,IAAI,KAAK,QAAQ,EAAE;MAC5BR,UAAU,CAACS,IAAI,CAAC;QACdC,MAAM,EAAE,cAAc;QACtBC,IAAI,EAAE;UAAEC,QAAQ,EAAEP,MAAM,CAACE;QAAG;OAC7B,CAAC;MACF;IACF;IACA,MAAMM,OAAO,GAAY,OAAOjC,MAAM,CAACkC,KAAK,CAAC3B,MAAM,CAACkB,MAAM,CAACV,GAAG,CAAC,CAAC;IAChE,IAAI,CAACJ,YAAY,CAACsB,OAAO,CAAC,EAAE;MAC1B,OAAO,OAAOjC,MAAM,CAACmC,GAAG,CACtB,IAAIC,KAAK,CACP,uCAAuCX,MAAM,CAACE,GAAG,gBAAgBM,OAAO,KAAK,IAAI,GAAG,MAAM,GAAG,OAAOA,OAAO,+CAA+C,CAC3J,CACF;IACH;IACA,MAAM3B,MAAM,GAAG;MAAE,GAAG2B,OAAO;MAAED,QAAQ,EAAEP,MAAM,CAACE;IAAG,CAAE;IACnD,MAAMU,IAAI,GAAGhC,WAAW,CAACC,MAAM,CAAC;IAChC,IAAI+B,IAAI,GAAGrB,cAAc,EAAE;MACzBO,OAAO,CAACe,GAAG,CACTb,MAAM,CAACE,GAAG,EACV,iBAAiBU,IAAI,iDAAiDrB,cAAc,EAAE,CACvF;MACD;IACF;IACAI,UAAU,CAACS,IAAI,CAAC;MAAEC,MAAM,EAAE,WAAW;MAAEC,IAAI,EAAEzB;IAAM,CAAE,CAAC;EACxD;EACA,MAAMiC,QAAQ,GAAGzC,KAAK,CAAC0C,IAAI,CAACjB,OAAO,CAACF,IAAI,EAAE,CAAC;EAC3C,IAAItB,GAAG,CAAC0C,uBAAuB,CAACF,QAAQ,CAAC,EAAE;IACzC,OAAO,OAAOvC,MAAM,CAAC0C,IAAI,CAAC,IAAI7C,YAAY,CAAC;MAAEwB,IAAI,EAAEkB,QAAQ;MAAEhB;IAAO,CAAE,CAAC,CAAC;EAC1E;EACA,OAAO;IAAEH,UAAU;IAAEC;EAAI,CAAE;AAC7B,CAAC,CAAC","ignoreList":[]}