@lde/distribution-probe 0.1.13 → 0.2.1

package/README.md

@@ -24,14 +24,48 @@ Sends `POST` with the configured query (default `SELECT * { ?s ?p ?o } LIMIT 1`)
 
 ### Data dumps
 
-Sends `HEAD` with `Accept: <distribution.mimeType>` and `Accept-Encoding: identity`. If `Content-Length` is missing or ≤ 10 KB, retries with `GET` to validate the body – this also catches servers that return `0` from `HEAD`.
+#### Reachability (the default)
+
+Sends `HEAD` with `Accept: <distribution.mimeType>` and `Accept-Encoding: identity`. A successful `HEAD` settles reachability and gathers metadata (`Content-Length`, `Last-Modified`) **without reading the body**. If `HEAD` is unsuccessful — e.g. a server that returns `405`/`501` because it does not implement `HEAD` — the probe falls back to a body-less `GET` to confirm the endpoint is up. The body is never downloaded.
+
+This is deliberately cheap: reading a body forces a slow, generate-on-the-fly endpoint (a TriplyDB dump, a SPARQL `CONSTRUCT` export) to start producing its export, which a `HEAD` does not.
 
 - **Content-Type is checked as a soft warning, not a hard failure.** If the server’s Content-Type disagrees with the distribution’s declared `mimeType`, a message is appended to `result.warnings` but `isSuccess()` stays `true`. Compression wrappers (`application/gzip`, `application/x-gzip`, `application/octet-stream`) are skipped so a gzipped Turtle file doesn’t trigger a warning.
-- **Body is parse-validated only for Turtle, N-Triples, and N-Quads** (Content-Type starting with `text/turtle`, `application/n-triples`, or `application/n-quads`). Empty bodies and parse errors fail the probe. Other RDF serializations (RDF/XML, JSON-LD, TriG, …) are not parse-validated – only HTTP status and headers are checked.
-- Bodies larger than 10 KB are not fetched; only `HEAD` metadata is inspected.
+
+#### Content validation (opt-in)
+
+Set `validateRdfContent: true` to additionally confirm that a dump actually carries RDF. It applies only to distributions whose **declared** `mimeType` is an RDF serialization (`text/turtle`, `application/n-triples`, `application/n-quads`, `application/trig`, `text/n3`, `application/ld+json`, `application/rdf+xml`); non-RDF and undeclared-type distributions stay reachability-only.
+
+When on, the probe `GET`s the dump — **regardless of size** — and reads only a **bounded prefix** (256 KiB), never the whole body:
+
+- It settles on the **first triple** and stops, so a large dump is validated from its opening chunk. The line/statement-oriented serializations and RDF/XML stream a triple out of the prefix; **JSON-LD is not streamable** (its parser needs the whole document), so a JSON-LD dump is only validated when it fits the prefix in full — a larger one is reported reachable but unvalidated.
+- A gzip body that `fetch` did not decompress (a `.gz` dump, or one served with a non-standard `Content-Encoding`) is inflated in-place; a gzip that will not inflate when the **complete** compressed body was read fails as `Distribution is not valid gzip`.
+- Empty bodies (`Distribution is empty`) and bodies that parse to **zero** triples (`Distribution contains no RDF triples`) fail the probe. A deliberately truncated prefix is never mistaken for either — it is inconclusive.
+- **Reachability is settled by the response, so validation never turns a reachable dump into a failure.** If no triple surfaces within `rdfValidationBudgetMs` (default `min(timeoutMs, 2000)`, clamped to `timeoutMs`), the read is aborted and the distribution is reported reachable but unvalidated (no `failureReason`). This bounds the extra latency content validation adds on slow, generate-on-the-fly endpoints.
 
 ### Network errors
 
 A thrown exception from `fetch` (DNS failure, connection refused, socket reset, TLS error, timeout after the configured `timeoutMs` – default 5 000 ms) is a connection-level failure. The probe retries these up to `retries` times (default 2) with a short backoff before giving up and returning a `NetworkError`. This turns a transient transport blip into a reliable single measurement without looking backward across checks. A genuine outage still resolves to a `NetworkError` on the current check – every attempt fails – but note each attempt gets its own `timeoutMs`, so an endpoint that fails only by timing out takes up to `(retries + 1) × timeoutMs` (plus backoff) to be reported down. HTTP error responses (4xx/5xx) and content-validation failures are real ‘down’ states and are **never** retried.
 
 `NetworkError.message` includes the underlying `error.cause` (e.g. `ECONNRESET`, `UND_ERR_SOCKET “other side closed”`) when Node wraps one, so observations record what actually failed rather than a bare ‘fetch failed’.
+
+## Probing many distributions
+
+`probeMany` probes an array of distributions concurrently and returns one result per input, in input order. Each distribution is probed once with `probe`, so every behaviour above applies per distribution; like `probe`, `probeMany` never throws – a probe that fails is reported as a `NetworkError` in its slot.
+
+```ts
+import { probeMany } from '@lde/distribution-probe';
+
+const results = await probeMany(distributions, {
+  concurrency: 20, // max probes in flight across all hosts (default 20)
+  perHostConcurrency: 4, // max probes in flight against one host (default 4)
+  validateRdfContent: true, // any ProbeOptions are forwarded to each probe
+});
+```
+
+Two caps bound the batch:
+
+- **`concurrency`** bounds the total fan-out, so a large catalogue does not exhaust sockets or buffer too many response bodies at once.
+- **`perHostConcurrency`** bounds the burst any one server sees, keeping the batch a polite client: a catalogue that declares many distributions on a single host (e.g. a download endpoint per named graph) will not trip that server’s rate limiter (HTTP 429). Distributions sharing a host (by `accessUrl`) contend for the same budget; a probe whose host is saturated waits while probes for other hosts proceed, so one busy host never idles the global pool.
+
+All other `ProbeOptions` (`timeoutMs`, `retries`, `validateRdfContent`, and the rest) are forwarded unchanged to every probe.

package/dist/index.d.ts

@@ -1,2 +1,2 @@
-export { probe, NetworkError, SparqlProbeResult, DataDumpProbeResult, type ProbeOptions, type ProbeResultType, } from './probe.js';
+export { probe, probeMany, NetworkError, SparqlProbeResult, DataDumpProbeResult, type ProbeOptions, type ProbeManyOptions, type ProbeResultType, } from './probe.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,EACL,YAAY,EACZ,iBAAiB,EACjB,mBAAmB,EACnB,KAAK,YAAY,EACjB,KAAK,eAAe,GACrB,MAAM,YAAY,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,EACL,SAAS,EACT,YAAY,EACZ,iBAAiB,EACjB,mBAAmB,EACnB,KAAK,YAAY,EACjB,KAAK,gBAAgB,EACrB,KAAK,eAAe,GACrB,MAAM,YAAY,CAAC"}

package/dist/index.js

@@ -1 +1 @@
-export { probe, NetworkError, SparqlProbeResult, DataDumpProbeResult, } from './probe.js';
+export { probe, probeMany, NetworkError, SparqlProbeResult, DataDumpProbeResult, } from './probe.js';

package/dist/probe.d.ts

@@ -28,6 +28,49 @@ export interface ProbeOptions {
      * the default; negative values are clamped to `0`.
      */
     retries?: number;
+    /**
+     * Validate the body content of data-dump distributions whose declared media
+     * type is an RDF serialization, by reading a bounded prefix and confirming it
+     * carries at least one triple. When `false` (the default) a data dump is only
+     * checked for reachability (a `HEAD`, with a body-less `GET` fallback if `HEAD`
+     * is unsupported) and its body is never read. When `true`, every declared-RDF
+     * dump — regardless of size — is fetched and validated; non-RDF and
+     * undeclared-type distributions are still reachability-only. Validation is
+     * opt-in because reading a body forces a slow, generate-on-the-fly endpoint to
+     * start producing its export, which a `HEAD` does not.
+     */
+    validateRdfContent?: boolean;
+    /**
+     * Soft deadline, in milliseconds, for finding the first triple when
+     * {@link validateRdfContent} is on. Reachability is settled by the response
+     * itself; if no triple has surfaced within this budget the read is aborted and
+     * the distribution is reported reachable but unvalidated (no `failureReason`),
+     * never failed. This bounds the extra latency content validation adds on slow,
+     * generate-on-the-fly endpoints. Clamped to {@link timeoutMs} (a longer budget
+     * is meaningless — the request times out first). Defaults to
+     * `min(timeoutMs, 2000)`.
+     */
+    rdfValidationBudgetMs?: number;
+}
+/**
+ * Options for {@link probeMany}: the per-probe {@link ProbeOptions} plus the
+ * concurrency budgets that bound the batch.
+ */
+export interface ProbeManyOptions extends ProbeOptions {
+    /**
+     * Maximum number of probes to run at once across all hosts. Bounds the batch’s
+     * total fan-out so a large catalogue does not exhaust sockets or buffer too many
+     * response bodies at once. Default 20.
+     */
+    concurrency?: number;
+    /**
+     * Maximum number of probes to run at once against a single host. Bounds the
+     * burst any one server sees, so a catalogue that declares many distributions on
+     * one host (e.g. a download endpoint per named graph) does not trip its rate
+     * limiter (HTTP 429). A probe whose host is at this cap waits while probes for
+     * other hosts proceed, so this never idles the global pool. Default 4.
+     */
+    perHostConcurrency?: number;
 }
 /**
  * Result of a network error during probing.
@@ -80,10 +123,25 @@ export type ProbeResultType = SparqlProbeResult | DataDumpProbeResult | NetworkE
  *
  * For SPARQL endpoints, issues the configured SPARQL query (default: a
  * minimal `SELECT`). For data dumps, issues `HEAD` (with a `GET` fallback
- * for small or unknown-size bodies).
+ * for small or unknown-size bodies, reading only a bounded prefix so a large
+ * streamed dump is never downloaded in full).
  *
  * Returns a pure result object; never throws.
  */
 export declare function probe(distribution: Distribution, options?: ProbeOptions): Promise<ProbeResultType>;
+/**
+ * Probe many distributions concurrently, bounded by a global cap and a per-host
+ * cap, returning one result per input in input order. Like {@link probe}, this
+ * never throws: a probe that somehow fails is reported as a {@link NetworkError}
+ * in its slot.
+ *
+ * The per-host cap keeps the batch a polite client. Distributions sharing a host
+ * (by {@link Distribution.accessUrl}) contend for the same budget, so no single
+ * server is hit by the full global pool at once — the burst that trips a rate
+ * limiter (HTTP 429). When the next queued probe’s host is saturated it is
+ * skipped in favour of a later probe on a different host, so one busy host never
+ * idles the global pool (no head-of-line blocking).
+ */
+export declare function probeMany(distributions: readonly Distribution[], options?: ProbeManyOptions): Promise<ProbeResultType[]>;
 export {};
 //# sourceMappingURL=probe.d.ts.map

package/dist/probe.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"probe.d.ts","sourceRoot":"","sources":["../src/probe.ts"],"names":[],"mappings":"AAAA,OAAO,EAAyB,YAAY,EAAE,MAAM,cAAc,CAAC;AAInE;;GAEG;AACH,MAAM,WAAW,YAAY;IAC3B,0DAA0D;IAC1D,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB;;;OAGG;IACH,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB;;;;;OAKG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;;;;;;OASG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AASD;;GAEG;AACH,qBAAa,YAAY;aAEL,GAAG,EAAE,MAAM;aACX,OAAO,EAAE,MAAM;aACf,cAAc,EAAE,MAAM;gBAFtB,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,MAAM,EACf,cAAc,EAAE,MAAM;CAEzC;AAED;;GAEG;AACH,uBAAe,WAAW;aAUN,GAAG,EAAE,MAAM;IAT7B,SAAgB,UAAU,EAAE,MAAM,CAAC;IACnC,SAAgB,UAAU,EAAE,MAAM,CAAC;IACnC,SAAgB,YAAY,EAAE,IAAI,GAAG,IAAI,CAAQ;IACjD,SAAgB,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;IAC3C,SAAgB,aAAa,EAAE,MAAM,GAAG,IAAI,CAAC;IAC7C,SAAgB,QAAQ,EAAE,MAAM,EAAE,CAAM;IACxC,SAAgB,cAAc,EAAE,MAAM,CAAC;gBAGrB,GAAG,EAAE,MAAM,EAC3B,QAAQ,EAAE,QAAQ,EAClB,cAAc,EAAE,MAAM,EACtB,aAAa,GAAE,MAAM,GAAG,IAAW;IAa9B,SAAS,IAAI,OAAO;CAO5B;AAqBD;;GAEG;AACH,qBAAa,iBAAkB,SAAQ,WAAW;IAChD;;;;;OAKG;IACH,SAAgB,oBAAoB,EAAE,SAAS,MAAM,EAAE,CAAC;gBAGtD,GAAG,EAAE,MAAM,EACX,QAAQ,EAAE,QAAQ,EAClB,cAAc,EAAE,MAAM,EACtB,oBAAoB,EAAE,MAAM,GAAG,SAAS,MAAM,EAAE,EAChD,aAAa,GAAE,MAAM,GAAG,IAAW;IAS5B,SAAS,IAAI,OAAO;CAQ9B;AAED;;GAEG;AACH,qBAAa,mBAAoB,SAAQ,WAAW;IAClD,SAAgB,WAAW,EAAE,MAAM,GAAG,IAAI,CAAQ;gBAGhD,GAAG,EAAE,MAAM,EACX,QAAQ,EAAE,QAAQ,EAClB,cAAc,EAAE,MAAM,EACtB,aAAa,GAAE,MAAM,GAAG,IAAW;CAQtC;AAED,MAAM,MAAM,eAAe,GACvB,iBAAiB,GACjB,mBAAmB,GACnB,YAAY,CAAC;AAIjB;;;;;;;;GAQG;AACH,wBAAsB,KAAK,CACzB,YAAY,EAAE,YAAY,EAC1B,OAAO,CAAC,EAAE,YAAY,GACrB,OAAO,CAAC,eAAe,CAAC,CAqD1B"}
+{"version":3,"file":"probe.d.ts","sourceRoot":"","sources":["../src/probe.ts"],"names":[],"mappings":"AAAA,OAAO,EAAyB,YAAY,EAAE,MAAM,cAAc,CAAC;AAKnE;;GAEG;AACH,MAAM,WAAW,YAAY;IAC3B,0DAA0D;IAC1D,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB;;;OAGG;IACH,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB;;;;;OAKG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;;;;;;OASG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;;;;;;;;OAUG;IACH,kBAAkB,CAAC,EAAE,OAAO,CAAC;IAC7B;;;;;;;;;OASG;IACH,qBAAqB,CAAC,EAAE,MAAM,CAAC;CAChC;AAED;;;GAGG;AACH,MAAM,WAAW,gBAAiB,SAAQ,YAAY;IACpD;;;;OAIG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;;;OAMG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;CAC7B;AAkCD;;GAEG;AACH,qBAAa,YAAY;aAEL,GAAG,EAAE,MAAM;aACX,OAAO,EAAE,MAAM;aACf,cAAc,EAAE,MAAM;gBAFtB,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,MAAM,EACf,cAAc,EAAE,MAAM;CAEzC;AAED;;GAEG;AACH,uBAAe,WAAW;aAUN,GAAG,EAAE,MAAM;IAT7B,SAAgB,UAAU,EAAE,MAAM,CAAC;IACnC,SAAgB,UAAU,EAAE,MAAM,CAAC;IACnC,SAAgB,YAAY,EAAE,IAAI,GAAG,IAAI,CAAQ;IACjD,SAAgB,WAAW,EAAE,MAAM,GAAG,IAAI,CAAC;IAC3C,SAAgB,aAAa,EAAE,MAAM,GAAG,IAAI,CAAC;IAC7C,SAAgB,QAAQ,EAAE,MAAM,EAAE,CAAM;IACxC,SAAgB,cAAc,EAAE,MAAM,CAAC;gBAGrB,GAAG,EAAE,MAAM,EAC3B,QAAQ,EAAE,QAAQ,EAClB,cAAc,EAAE,MAAM,EACtB,aAAa,GAAE,MAAM,GAAG,IAAW;IAa9B,SAAS,IAAI,OAAO;CAO5B;AAqBD;;GAEG;AACH,qBAAa,iBAAkB,SAAQ,WAAW;IAChD;;;;;OAKG;IACH,SAAgB,oBAAoB,EAAE,SAAS,MAAM,EAAE,CAAC;gBAGtD,GAAG,EAAE,MAAM,EACX,QAAQ,EAAE,QAAQ,EAClB,cAAc,EAAE,MAAM,EACtB,oBAAoB,EAAE,MAAM,GAAG,SAAS,MAAM,EAAE,EAChD,aAAa,GAAE,MAAM,GAAG,IAAW;IAS5B,SAAS,IAAI,OAAO;CAQ9B;AAED;;GAEG;AACH,qBAAa,mBAAoB,SAAQ,WAAW;IAClD,SAAgB,WAAW,EAAE,MAAM,GAAG,IAAI,CAAQ;gBAGhD,GAAG,EAAE,MAAM,EACX,QAAQ,EAAE,QAAQ,EAClB,cAAc,EAAE,MAAM,EACtB,aAAa,GAAE,MAAM,GAAG,IAAW;CAQtC;AAED,MAAM,MAAM,eAAe,GACvB,iBAAiB,GACjB,mBAAmB,GACnB,YAAY,CAAC;AAIjB;;;;;;;;;GASG;AACH,wBAAsB,KAAK,CACzB,YAAY,EAAE,YAAY,EAC1B,OAAO,CAAC,EAAE,YAAY,GACrB,OAAO,CAAC,eAAe,CAAC,CAqD1B;AAED;;;;;;;;;;;;GAYG;AACH,wBAAsB,SAAS,CAC7B,aAAa,EAAE,SAAS,YAAY,EAAE,EACtC,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,eAAe,EAAE,CAAC,CA4B5B"}

package/dist/probe.js

@@ -1,9 +1,32 @@
 import { compressionMediaTypes } from '@lde/dataset';
 import { rdfParser } from 'rdf-parse';
 import { Readable } from 'node:stream';
+import { createGunzip } from 'node:zlib';
 const DEFAULT_SPARQL_QUERY = 'SELECT * { ?s ?p ?o } LIMIT 1';
 const DEFAULT_TIMEOUT_MS = 5000;
 const DEFAULT_RETRIES = 2;
+const DEFAULT_PROBE_CONCURRENCY = 20;
+const DEFAULT_PROBE_PER_HOST_CONCURRENCY = 4;
+/**
+ * Default soft deadline for finding the first triple when content validation is
+ * on (capped at `timeoutMs`). Two seconds comfortably covers a static file
+ * server's first chunk while keeping the extra wait bounded on a slow,
+ * generate-on-the-fly endpoint.
+ */
+const DEFAULT_RDF_VALIDATION_BUDGET_MS = 2000;
+/** Sentinel: the validation budget elapsed before a triple surfaced. */
+const VALIDATION_TIMED_OUT = Symbol('rdf-validation-timed-out');
+/**
+ * Maximum number of body bytes the data-dump probe reads before it stops and
+ * releases the connection. Reachability needs only that the endpoint answered
+ * with a success status and produced bytes; a large dump must never be
+ * downloaded in full within the probe's timeout budget. 256 KiB comfortably
+ * surfaces the first RDF triple — the signal {@link validateBody} needs — while
+ * bounding the read regardless of the dump's true size, chunked transfer, or
+ * compression. Applied to both the raw read and, for a gzip body, the inflated
+ * output.
+ */
+const MAX_PROBE_BODY_BYTES = 256 * 1024;
 /** Base backoff between retries; the nth retry waits `n × base`. */
 const RETRY_BACKOFF_MS = 250;
 /**
@@ -107,7 +130,8 @@ export class DataDumpProbeResult extends ProbeResult {
  *
  * For SPARQL endpoints, issues the configured SPARQL query (default: a
  * minimal `SELECT`). For data dumps, issues `HEAD` (with a `GET` fallback
- * for small or unknown-size bodies).
+ * for small or unknown-size bodies, reading only a bounded prefix so a large
+ * streamed dump is never downloaded in full).
  *
  * Returns a pure result object; never throws.
  */
@@ -147,6 +171,96 @@ export async function probe(distribution, options) {
     // real cost of a down endpoint.
     return new NetworkError(url, describeNetworkError(lastError), Math.round(performance.now() - overallStart));
 }
+/**
+ * Probe many distributions concurrently, bounded by a global cap and a per-host
+ * cap, returning one result per input in input order. Like {@link probe}, this
+ * never throws: a probe that somehow fails is reported as a {@link NetworkError}
+ * in its slot.
+ *
+ * The per-host cap keeps the batch a polite client. Distributions sharing a host
+ * (by {@link Distribution.accessUrl}) contend for the same budget, so no single
+ * server is hit by the full global pool at once — the burst that trips a rate
+ * limiter (HTTP 429). When the next queued probe’s host is saturated it is
+ * skipped in favour of a later probe on a different host, so one busy host never
+ * idles the global pool (no head-of-line blocking).
+ */
+export async function probeMany(distributions, options) {
+    // Clamp the budgets to a positive integer, mirroring how probe() treats an
+    // invalid retries value: a zero, negative, fractional, or NaN limit would
+    // otherwise stall the scheduler (no task ever starts, so the promise never
+    // resolves) or overrun the cap, so fall back to the default rather than trust
+    // the caller.
+    const globalLimit = positiveIntOrDefault(options?.concurrency, DEFAULT_PROBE_CONCURRENCY);
+    const perHostLimit = positiveIntOrDefault(options?.perHostConcurrency, DEFAULT_PROBE_PER_HOST_CONCURRENCY);
+    // Probes contend per host. An authority-less URL (e.g. urn:, file:) has an
+    // empty host, so it falls back to its full href and never shares a budget with
+    // an unrelated one.
+    const hostKeys = distributions.map((distribution) => distribution.accessUrl.host || distribution.accessUrl.href);
+    return mapHostLimited(distributions, hostKeys, globalLimit, perHostLimit, (distribution) => probe(distribution, options));
+}
+/**
+ * Coerce an optional concurrency budget to a usable value: a positive integer is
+ * taken as-is; undefined, zero, negative, fractional, or NaN falls back to the
+ * default. Matches probe()’s treatment of an invalid retries value.
+ */
+function positiveIntOrDefault(value, fallback) {
+    return value !== undefined && Number.isInteger(value) && value >= 1
+        ? value
+        : fallback;
+}
+/**
+ * Run `task` over `items` with two concurrency caps — a global cap and a per-host
+ * cap keyed by `hostKeys[index]` — resolving to results in input order. When the
+ * next queued item’s host is at the per-host cap it is skipped for a later item on
+ * a different host, so a saturated host never idles the global pool (no head-of-line
+ * blocking); the skipped host always has a task in flight, whose completion re-runs
+ * the scheduler, so the queue always drains. `task` must not reject — callers wrap
+ * failures into a result value — as a rejection would leave the promise pending.
+ */
+function mapHostLimited(items, hostKeys, globalLimit, perHostLimit, task) {
+    const results = new Array(items.length);
+    const perHostInFlight = new Map();
+    const pending = items.map((_unused, index) => index);
+    let globalInFlight = 0;
+    let settledCount = 0;
+    const adjustHost = (host, delta) => {
+        perHostInFlight.set(host, (perHostInFlight.get(host) ?? 0) + delta);
+    };
+    return new Promise((resolve) => {
+        const schedule = () => {
+            let cursor = 0;
+            while (cursor < pending.length && globalInFlight < globalLimit) {
+                const index = pending[cursor];
+                const host = hostKeys[index];
+                if ((perHostInFlight.get(host) ?? 0) >= perHostLimit) {
+                    cursor++; // Host saturated; leave it queued and try a later, different host.
+                    continue;
+                }
+                pending.splice(cursor, 1);
+                globalInFlight++;
+                adjustHost(host, 1);
+                void task(items[index]).then((result) => {
+                    results[index] = result;
+                    globalInFlight--;
+                    adjustHost(host, -1);
+                    settledCount++;
+                    if (settledCount === items.length) {
+                        resolve(results);
+                    }
+                    else {
+                        schedule();
+                    }
+                });
+                // pending[cursor] now holds the next queued item; do not advance cursor.
+            }
+        };
+        schedule();
+        // Resolve immediately when there is nothing to settle (empty input); a
+        // non-empty run resolves via the task completion above.
+        if (settledCount === items.length)
+            resolve(results);
+    });
+}
 function delay(milliseconds) {
     return new Promise((resolve) => setTimeout(resolve, milliseconds));
 }
@@ -186,6 +300,9 @@ function resolveOptions(options) {
         retries: retries === undefined || !Number.isInteger(retries)
             ? DEFAULT_RETRIES
             : Math.max(0, retries),
+        validateRdfContent: options?.validateRdfContent ?? false,
+        rdfValidationBudgetMs: options?.rdfValidationBudgetMs ??
+            Math.min(options?.timeoutMs ?? DEFAULT_TIMEOUT_MS, DEFAULT_RDF_VALIDATION_BUDGET_MS),
     };
 }
 /**
@@ -350,30 +467,201 @@ async function probeDataDump(url, distribution, options, authHeaders, start) {
         method: 'HEAD',
         ...requestOptions,
     });
-    const contentLength = headResponse.headers.get('Content-Length');
-    const contentLengthBytes = contentLength ? parseInt(contentLength) : 0;
-    // For small or unknown-size files, do a GET to validate body content.
-    // This also handles servers that incorrectly return 0 Content-Length for HEAD.
-    if (contentLengthBytes <= 10_240) {
-        const getResponse = await fetch(url, {
-            method: 'GET',
-            ...requestOptions,
-        });
-        const body = await getResponse.text();
-        const isHttpSuccess = getResponse.status >= 200 && getResponse.status < 400;
-        const failureReason = isHttpSuccess
-            ? await validateBody(body, getResponse.headers.get('Content-Type'), url, options.timeoutMs)
-            : null;
-        const responseTimeMs = Math.round(performance.now() - start);
-        const result = new DataDumpProbeResult(url, getResponse, responseTimeMs, failureReason);
-        checkContentTypeMismatch(result, distribution);
-        return result;
+    // Validate body content only when asked to and the distribution declares an
+    // RDF media type; otherwise the probe is reachability-only and never reads a
+    // body — which keeps it from forcing a slow, generate-on-the-fly endpoint to
+    // start producing its export.
+    if (options.validateRdfContent &&
+        isDeclaredRdf(distribution) &&
+        isHttpSuccess(headResponse)) {
+        const { response, failureReason } = await validateDumpBody(url, headers, options, headResponse);
+        return finalizeDataDump(url, distribution, response, start, failureReason);
+    }
+    // Reachability only. A successful HEAD is enough; otherwise confirm with a
+    // body-less GET, which rescues servers that reject or do not implement HEAD.
+    if (isHttpSuccess(headResponse)) {
+        return finalizeDataDump(url, distribution, headResponse, start, null);
     }
+    const getResponse = await fetch(url, { method: 'GET', ...requestOptions });
+    await getResponse.body?.cancel();
+    return finalizeDataDump(url, distribution, getResponse, start, null);
+}
+/** Whether an HTTP response carries a success (2xx/3xx) status. */
+function isHttpSuccess(response) {
+    return response.status >= 200 && response.status < 400;
+}
+/** Whether the distribution declares an RDF serialization as its media type. */
+function isDeclaredRdf(distribution) {
+    const declared = distribution.mimeType?.toLowerCase();
+    return declared !== undefined && rdfContentTypes.includes(declared);
+}
+/** Build a DataDumpProbeResult and attach any Content-Type-mismatch warning. */
+function finalizeDataDump(url, distribution, response, start, failureReason) {
     const responseTimeMs = Math.round(performance.now() - start);
-    const result = new DataDumpProbeResult(url, headResponse, responseTimeMs);
+    const result = new DataDumpProbeResult(url, response, responseTimeMs, failureReason);
     checkContentTypeMismatch(result, distribution);
     return result;
 }
+/**
+ * GET the dump and validate that its body carries a triple, but only for as long
+ * as the validation budget allows. Reachability is already settled by the prior
+ * HEAD, so any shortfall — a budget that elapses before a triple, a read error,
+ * a GET that cannot start — yields a `null` failureReason (reachable,
+ * unvalidated), never a failure. Returns the response to draw metadata from
+ * (the GET, or the HEAD when the GET could not start) alongside that reason.
+ */
+async function validateDumpBody(url, headers, options, headResponse) {
+    const budgetMs = Math.min(options.rdfValidationBudgetMs, options.timeoutMs);
+    // Aborting on budget expiry stops a slow endpoint from streaming on in the
+    // background once we have given up waiting for a triple.
+    const budgetController = new AbortController();
+    let getResponse;
+    try {
+        getResponse = await fetch(url, {
+            method: 'GET',
+            headers,
+            signal: AbortSignal.any([
+                AbortSignal.timeout(options.timeoutMs),
+                budgetController.signal,
+            ]),
+        });
+    }
+    catch {
+        // The GET could not even return headers; the HEAD already proved the
+        // distribution reachable, so report it unvalidated rather than down.
+        return { response: headResponse, failureReason: null };
+    }
+    if (!isHttpSuccess(getResponse)) {
+        await getResponse.body?.cancel();
+        return { response: getResponse, failureReason: null };
+    }
+    const validation = (async () => {
+        const bounded = await readBoundedBody(getResponse, MAX_PROBE_BODY_BYTES);
+        const { text, truncated, corrupt } = await decodeProbeBody(bounded);
+        return corrupt
+            ? 'Distribution is not valid gzip'
+            : await validateBody(text, getResponse.headers.get('Content-Type'), url, budgetMs, truncated);
+    })().catch(() => null);
+    let budgetTimer;
+    const budgetExpiry = new Promise((resolve) => {
+        budgetTimer = setTimeout(() => {
+            budgetController.abort();
+            resolve(VALIDATION_TIMED_OUT);
+        }, budgetMs);
+    });
+    try {
+        const outcome = await Promise.race([validation, budgetExpiry]);
+        return {
+            response: getResponse,
+            failureReason: outcome === VALIDATION_TIMED_OUT ? null : outcome,
+        };
+    }
+    finally {
+        clearTimeout(budgetTimer);
+    }
+}
+/**
+ * Read at most `maxBytes` from a response body, then cancel the stream to free
+ * the underlying connection. Returns the bytes read and whether the body was
+ * longer than the cap (`truncated`), so the caller can tell a complete, small
+ * body — whose emptiness or parse errors are meaningful — from a deliberately
+ * cut-off prefix of a large one, where only the presence of content is
+ * conclusive. This is what keeps the probe from downloading a multi-hundred-MB
+ * streamed dump in full just to confirm it is reachable.
+ */
+async function readBoundedBody(response, maxBytes) {
+    const stream = response.body;
+    if (stream === null) {
+        return { bytes: new Uint8Array(0), truncated: false };
+    }
+    const chunks = [];
+    let total = 0;
+    let truncated = false;
+    // Breaking out of `for await` cancels the stream, which stops any further
+    // download and releases the underlying connection — so a large dump is never
+    // pulled in full once we have the prefix we need.
+    for await (const chunk of stream) {
+        chunks.push(chunk);
+        total += chunk.length;
+        if (total >= maxBytes) {
+            truncated = true;
+            break;
+        }
+    }
+    return { bytes: Buffer.concat(chunks), truncated };
+}
+/**
+ * Decode a bounded body to text for RDF validation, inflating it first when it
+ * is a gzip stream that `fetch` did not transparently decompress — e.g. a `.gz`
+ * data dump served as-is, or one labelled with a non-standard Content-Encoding
+ * (`application/gzip`) that undici does not recognise as a content coding.
+ * Detection is by the gzip magic on the delivered bytes, so a body that `fetch`
+ * already inflated (a standard `Content-Encoding: gzip`) is passed through
+ * untouched. A truncated gzip tail is expected — we only read a prefix — and
+ * inflates cleanly up to the cut, so it is never mistaken for corruption.
+ */
+async function decodeProbeBody(bounded) {
+    if (!isGzip(bounded.bytes)) {
+        return {
+            text: decodeUtf8(bounded.bytes),
+            truncated: bounded.truncated,
+            corrupt: false,
+        };
+    }
+    // The compressed body is complete only when the raw read was not itself cut
+    // off: a gzip error on a complete body is genuine corruption, on a prefix we
+    // cut it is just the dropped tail.
+    const inflated = await gunzipPrefix(bounded.bytes, MAX_PROBE_BODY_BYTES, !bounded.truncated);
+    return {
+        text: decodeUtf8(inflated.bytes),
+        truncated: bounded.truncated || inflated.truncated,
+        corrupt: inflated.corrupt,
+    };
+}
+/** Whether the bytes begin with the gzip magic number (RFC 1952 §2.3.1). */
+function isGzip(bytes) {
+    return bytes.length >= 2 && bytes[0] === 0x1f && bytes[1] === 0x8b;
+}
+/**
+ * Decode bytes as UTF-8 without throwing: an incomplete multi-byte sequence at
+ * the truncation boundary is replaced rather than fatal, since the RDF parser
+ * only needs the leading, intact portion to find the first triple.
+ */
+function decodeUtf8(bytes) {
+    return new TextDecoder('utf-8', { fatal: false }).decode(bytes);
+}
+/**
+ * Inflate up to `maxBytes` of output from a gzip prefix, stopping once the cap
+ * is reached or the input runs out. `inputComplete` says whether the caller
+ * handed us the whole compressed body (true) or a prefix it had already cut
+ * (false). An inflate error therefore means different things: on a complete body
+ * the gzip is genuinely corrupt; on a cut prefix it is just the dropped tail, so
+ * whatever inflated cleanly is reported as a (truncated) partial inflate.
+ */
+function gunzipPrefix(bytes, maxBytes, inputComplete) {
+    return new Promise((resolve) => {
+        const gunzip = createGunzip();
+        const chunks = [];
+        let total = 0;
+        // `resolve` and `destroy` are both idempotent, so the first outcome wins and
+        // any later event (e.g. a premature-close error emitted by `destroy`) is a
+        // harmless no-op — no `settled` guard needed.
+        function finish(outcome) {
+            gunzip.destroy();
+            resolve({ bytes: Buffer.concat(chunks), ...outcome });
+        }
+        gunzip.on('data', (chunk) => {
+            chunks.push(chunk);
+            total += chunk.length;
+            if (total >= maxBytes) {
+                finish({ truncated: true, corrupt: false });
+            }
+        });
+        gunzip.on('error', () => finish({ truncated: !inputComplete, corrupt: inputComplete }));
+        gunzip.on('end', () => finish({ truncated: false, corrupt: false }));
+        gunzip.end(bytes);
+    });
+}
 // The RDF serializations whose bodies we parse to confirm they carry triples. A
 // non-empty body in one of these formats that yields zero triples — an empty
 // graph such as a JSON-LD `{}`, an `<rdf:RDF/>`, or prefix-only Turtle — is a
@@ -389,9 +677,21 @@ const rdfContentTypes = [
     'application/ld+json',
     'application/rdf+xml',
 ];
-async function validateBody(body, contentType, baseIRI, timeoutMs) {
+// Serializations a streaming parser cannot validate from a truncated prefix.
+// The line/statement-oriented formats (N-Triples, N-Quads, Turtle, TriG, N3) and
+// SAX-based RDF/XML all yield their first triple from the opening chunk, but
+// JSON-LD is a single JSON value whose parser emits nothing until the whole
+// document closes — a truncated JSON-LD body parses to an ‘unclosed document’
+// error, never a triple. So a truncated body in one of these can only be
+// validated if it happened to fit the read cap in full; beyond that it is
+// inconclusive, and we must not download it in full to find out.
+const nonStreamableRdfContentTypes = ['application/ld+json'];
+async function validateBody(body, contentType, baseIRI, timeoutMs, truncated) {
     if (body.length === 0) {
-        return 'Distribution is empty';
+        // A complete, empty body is a faulty distribution; an empty *prefix* (a
+        // truncated read that yielded no bytes, e.g. a corrupt gzip header) is
+        // inconclusive — the endpoint answered, we just could not validate content.
+        return truncated ? null : 'Distribution is empty';
     }
     // Media types are case-insensitive (RFC 9110 §8.3.1), so normalise before
     // matching the lower-case allow-list — a server sending `Application/LD+JSON`
@@ -400,7 +700,13 @@ async function validateBody(body, contentType, baseIRI, timeoutMs) {
     if (!serialization || !rdfContentTypes.includes(serialization)) {
         return null;
     }
-    const outcome = await classifyRdfBody(body, serialization, baseIRI, timeoutMs);
+    if (truncated && nonStreamableRdfContentTypes.includes(serialization)) {
+        // A bounded prefix of a non-streamable serialization (JSON-LD) can never
+        // yield a triple, so skip the doomed parse and report it inconclusive — only
+        // a complete document, small enough to fit the read cap, can be validated.
+        return null;
+    }
+    const outcome = await classifyRdfBody(body, serialization, baseIRI, timeoutMs, truncated);
     switch (outcome.type) {
         case 'empty':
             return 'Distribution contains no RDF triples';
@@ -422,8 +728,13 @@ async function validateBody(body, contentType, baseIRI, timeoutMs) {
  * on expiry — and likewise when a remote `@context` is unreachable — the outcome
  * is 'inconclusive', so a valid distribution is never flagged faulty for a
  * context host's failure. `baseIRI` resolves any relative IRIs in the document.
+ *
+ * When `truncated` is true the body is only a bounded prefix of a larger one, so
+ * only finding a triple ('hasTriples') is conclusive: a parse error at the cut
+ * or a clean end with no triple yet means we did not read far enough, not that
+ * the distribution is empty or malformed, and is reported as 'inconclusive'.
  */
-function classifyRdfBody(body, contentType, baseIRI, timeoutMs) {
+function classifyRdfBody(body, contentType, baseIRI, timeoutMs, truncated) {
     return new Promise((resolve) => {
         const quads = rdfParser.parse(Readable.from([body]), {
             contentType,
@@ -441,10 +752,10 @@ function classifyRdfBody(body, contentType, baseIRI, timeoutMs) {
         }
         quads
             .on('data', () => settle({ type: 'hasTriples' }))
-            .on('error', (error) => settle(isRemoteContextError(error)
+            .on('error', (error) => settle(truncated || isRemoteContextError(error)
             ? { type: 'inconclusive' }
             : { type: 'parseError', message: error.message }))
-            .on('end', () => settle({ type: 'empty' }));
+            .on('end', () => settle(truncated ? { type: 'inconclusive' } : { type: 'empty' }));
     });
 }
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lde/distribution-probe",
-  "version": "0.1.13",
+  "version": "0.2.1",
   "repository": {
     "url": "git+https://github.com/ldelements/lde.git",
     "directory": "packages/distribution-probe"