@foss.global/codefeed 3.0.0 → 3.1.0

package/readme.md

@@ -107,13 +107,80 @@ What is measured: users with their account flags, two-factor and SSH/GPG keys; o
 
 Secrets are counted, never named: the report carries no tokens, Actions secret or variable values or names, webhook secrets or mirror remote URLs. Reasons are adapter codes or fixed text, never response bodies or URLs. Gitea has no read that lists user-scoped Actions secrets; that is listed in `limitations`, since they are reissued. User-scoped webhooks, variables and runners are read as that user through the `Sudo` header.
 
-Git-level measures come from a mirror clone of every ref Gitea advertises, `refs/pull/*` included, made with `@foss.global/gitmanager` in a fresh directory below `gitWorkDir` and removed afterwards. The reader's login and token travel as Basic credentials outside the URL, the Git command line and Git configuration; the private CA list applies to the clone as well. The mirror's distinct LFS pointers, from every reachable revision, are checked against Gitea in batches of 100: an object with a download action counts as held, one answered with error 404 as missing, and any other answer makes the LFS measures `partial`. A clone that fails or exceeds `gitTimeoutMs` makes every Git and LFS measure of that repository `partial`; only the API-reported size stays complete. A clone starts only once its scratch reservation, twice the reported size plus 64 MiB, fits the free space of the file system that holds `gitWorkDir`, counting the reservations of the clones already in flight from every reader in the process. A clone that does not fit waits for them, in arrival order; one that does not fit even alone makes every Git and LFS measure of its repository `partial` with `git-workdir-too-small`, so the inventory stays incomplete. Size `gitWorkDir` for the largest repositories measured at once.
+Git-level measures come from a mirror clone of every ref Gitea advertises, `refs/pull/*` included, made with `@foss.global/gitmanager` in a fresh directory below `gitWorkDir` and removed afterwards. The reader's login and token form Basic credentials; the credential reaches Git only through GIT_CONFIG_* entries in the child process environment; it is never in the URL, the command line or any file. The private CA list applies to the clone as well. The mirror's distinct LFS pointers, from every reachable revision, are checked against Gitea in batches of 100: an object with a download action counts as held, one answered with error 404 as missing, and any other answer makes the LFS measures `partial`. A clone that fails or exceeds `gitTimeoutMs` makes every Git and LFS measure of that repository `partial`; only the API-reported size stays complete. A clone starts only once its scratch reservation, twice the reported size plus 64 MiB, fits the free space of the file system that holds `gitWorkDir`, counting the reservations of the clones already in flight from every reader in the process. A clone that does not fit waits for them, in arrival order; one that does not fit even alone makes every Git and LFS measure of its repository `partial` with `git-workdir-too-small`, so the inventory stays incomplete. Size `gitWorkDir` for the largest repositories measured at once.
 
 Known gaps that keep an inventory incomplete:
 - A repository with its issues or pull requests unit disabled hides both through the API, so its issues, pull requests, comments, reviews, reactions and attachments are `notMeasured`. A disabled wiki hides its pages the same way. Labels, milestones, releases and Actions stay readable while their units are disabled.
 
 Listings page until an empty page or the `x-total-count` total is reached. A total that changes mid-listing or disagrees with the rows read, a repeated page, a malformed page, a failed request or the page limit makes that measure `partial` and keeps the rows already read; only a listing that does not exist, such as the wiki of a repository without one, counts as empty.
 
+### Gitea source transfer
+
+`GiteaSourceTransfer` reads one repository's Git data and LFS objects for the one-time migration, with the same reader credentials as the inventory. It never writes to Gitea: Git is read only through `git-upload-pack`, LFS only through batch `download` requests and their download actions.
+
+```ts
+import { GitManager } from '@foss.global/gitmanager';
+import { GiteaSourceTransfer } from '@foss.global/codefeed';
+
+const source = new GiteaSourceTransfer({
+  baseUrl: 'https://gitea.example.com', // HTTPS only
+  login: readerLogin,
+  token: readerToken, // read it from a file; never pass it on a command line
+  // caCertificates: [privateCaPem], gitTimeoutMs: 30 * 60 * 1000, requestTimeoutMs: 60_000, maxResponseBytes: 16 * 1024 * 1024, lfsTransferTimeoutMs: 30 * 60 * 1000,
+});
+
+// A started GitManager rooted in the caller's own scratch space; codefeed creates no scratch roots.
+const scratch = new GitManager({ dataDir: scratchDir });
+await scratch.start();
+const { repository, reportedSizeKiB } = await source.importMirror({ owner: 'team', name: 'app' }, scratch, {
+  target: { org: 'team', repo: 'app' },
+  signal,
+});
+
+const download = await source.downloadLfsObject({ owner: 'team', name: 'app' }, oid, size, { signal });
+if (download.status === 'missing') {
+  // Gitea answered error 404: the object is not held at the source.
+} else {
+  await pipeline(download.stream, destination); // errors unless the bytes are exactly the object
+}
+```
+
+`importMirror` reads the repository's size from the REST API, reserves its scratch in the process-wide `GitScratchBudget` of the file system holding the GitManager's data directory (twice the reported size plus 64 MiB, admitted in arrival order), and then clones every ref into `target` with `importMirror` of `@foss.global/gitmanager`. The clone URL is formed from `baseUrl` and the repository's owner and name; the reader's login and token reach Git as a Basic credential only through GIT_CONFIG_* entries in the child process environment, never in the URL, the command line or any file. A repository that does not fit even alone rejects with `git-workdir-too-small` without cloning. The reservation is released when the clone ends. Failures reject with a fixed `ForgeInventoryError` code (`invalid-repository`, `invalid-target`, `gitmanager-not-running`, a REST code such as `http-404`, `git-workdir-failed`, `git-workdir-too-small`, `git-mirror-failed`, `request-aborted`); GitManager error text is never passed on. An abort rejects with `request-aborted`. While the clone runs, the abort reaches the GitManager import through its own `signal` (`@foss.global/gitmanager` 2.4.0 or later): Git is ended, the call settles once it has exited, no repository is left at `target`, and the reservation is released then. The GitManager keeps serving its other work, so one engine can serve several imports and other commands at once; a `stop()` of it during the clone also rejects the import with `request-aborted`.
+
+While the import runs, GitManager holds `target`: writes to it, such as `initBareRepository`, `restoreRepositoryBundle`, `setDefaultBranch` or a Smart HTTP push, fail with a `GitManagerError` whose code is `repositoryBusy`, and a second import into the same `target` is refused at once (codefeed rejects it with `git-mirror-failed`). Reads, of `target` as of other repositories, proceed.
+
+`downloadLfsObject` sends one batch `download` request with the reader's Basic credentials, then a GET of the action's href with the action's headers only. A missing object resolves as `{ status: 'missing' }`, so a caller can list the missing oids; any other object error rejects with `lfs-object-error-<code>`, such as `lfs-object-error-422` for a size Gitea disagrees with. The returned stream never holds the whole object: it counts and hashes the bytes as they pass and holds back the last chunk until the body ended with exactly `size` bytes whose SHA-256 is `oid`. Otherwise it errors with `lfs-size-overrun`, `lfs-short-body` or `lfs-digest-mismatch`; a `content-length` other than `size` rejects with `lfs-size-mismatch` before streaming. An abort errors it with `request-aborted`, a transfer longer than `lfsTransferTimeoutMs` or idle for `requestTimeoutMs` with `lfs-transfer-timed-out`. Destroying the stream ends the download.
+
+### Git LFS batch client
+
+`LfsBatchClient` is a provider-neutral Git LFS batch API client with the `basic` transfer adapter, bound to one LFS endpoint and its credential. `GiteaSourceTransfer` downloads through it; the migration uploads to the native Git service through it.
+
+```ts
+import { LfsBatchClient } from '@foss.global/codefeed';
+
+const lfs = new LfsBatchClient({
+  endpoint: 'https://git.example.com/team/app.git/info/lfs', // https, or http only to 127.0.0.0/8 or [::1]
+  credential: { kind: 'authorization', value: `Bearer ${grantToken}` }, // or { kind: 'basic', username, password }
+  // caCertificates, requestTimeoutMs: 60_000, transferTimeoutMs: 30 * 60 * 1000, maxResponseBytes: 4 * 1024 * 1024, batchSize: 100,
+});
+const uploaded = await lfs.upload([{ oid, size, open: (signalArg) => openObjectStream(oid, signalArg) }], { signal });
+// [{ oid, size, outcome: 'uploaded' | 'present' }]
+const presence = await lfs.checkPresence([{ oid, size }], { signal }); // [{ oid, size, status: 'held' | 'missing' }]
+const download = await lfs.download({ oid, size }, { signal }); // as for downloadLfsObject
+lfs.close(); // optional: idle pooled connections close on their own
+```
+
+- `upload` sends one batch `upload` request per `batchSize` objects (at most 100). An object without an `upload` action is already present; for each other one it opens the source, PUTs exactly `size` bytes with the action's headers and `Content-Length: size`, and then calls the `verify` action when the server sent one. The bytes are hashed while they stream and the last chunk is held back, so a source that is not the object is never sent complete (`lfs-size-overrun`, `lfs-short-body`, `lfs-digest-mismatch`). Any object error rejects with `lfs-object-error-<code>`.
+- Action expiry: `expires_in` (whole seconds after the batch answer arrived) is preferred over `expires_at` (RFC 3339), as the batch API specifies; like the git-lfs client, an `expires_in` of 0 and Go's zero time mean no expiry, and any other malformed value fails with `invalid-lfs-batch`. An upload, verify or download action is used only while it stays valid for the expiry margin, 30 s or `requestTimeoutMs` when that is shorter. Objects are transferred one at a time after their batch answer, so a later object whose action would expire within the margin is asked for again in a batch of its own, once; when the fresh answer is again too short-lived the call fails with `lfs-action-expired`. A verify action that expires during its PUT is asked for the same way; a fresh answer without actions means the server holds the object.
+- `checkPresence` sends batch `download` requests and downloads nothing: a download action means `held`, error 404 `missing`. It uses no action, so expiry is validated but not enforced there.
+- The credential is sent only as the `Authorization` header of batch requests to the endpoint, never to action hrefs and never in a URL, error or result; hrefs get the action's headers only. Action headers may not set connection-level fields (`host`, `content-length`, `transfer-encoding` and similar).
+- URL policy, enforced for the endpoint and every action href: `https:`, or `http:` only to an IP literal in 127.0.0.0/8 or `[::1]`; `localhost` and every other host name over HTTP are refused (`unsafe-lfs-url`), as is user information. An href must keep the endpoint's scheme, so an HTTPS server cannot move a transfer to plain HTTP (`unsafe-lfs-href`). Redirects are never followed.
+- Every request is bounded: batch answers by `maxResponseBytes`, batch and verify requests by `requestTimeoutMs`, object transfers by `transferTimeoutMs` and an idle limit of `requestTimeoutMs`, everything by the caller's `AbortSignal`. Errors are fixed `ForgeInventoryError` codes.
+
+### Shared scratch budget
+
+`GitScratchBudget` and `mirrorScratchBytes` are exported so that every scratch GitManager in a process admits its work through the same budget as the inventory and the source transfer. `GitScratchBudget.forDirectory(directory)` returns the process-wide budget of the file system holding `directory`; `reserve(directory, bytes, signal)` resolves with a release function once the bytes fit its free space next to the reservations in flight, in arrival order, or with `null` when they do not fit even alone. `mirrorScratchBytes(reportedSizeKiB)` is the reservation of one mirror clone: twice the reported size plus 64 MiB.
+
 ## Quick Start
 
 ```ts
@@ -263,6 +330,7 @@ The canonical public result contract is exported as `ICodefeedCommitResult` from
 
 - Gitea base URL and an optional token with read access
 - For GitManager-backed feeds, a started `@foss.global/gitmanager` 2.x `GitManager` instance with imported or local repositories
+- For `GiteaSourceTransfer.importMirror`, a started `@foss.global/gitmanager` 2.4.0 or later `GitManager`, whose transfers take an abort `signal`
 - For public GitManager reads, an authoritative repository-visibility source or an explicit all-public declaration
 - Node.js 22+ (global `fetch`; dependencies such as `file-type` 22 through `@push.rocks/smartnpm` declare Node.js 22)
 
@@ -273,7 +341,8 @@ The repo contains:
 - A mocked pagination test that does not require network.
 - A GitManager provider mapping test using a local test stub.
 - A default-deny GitManager public-client mapping test using a local test stub.
-- A mocked `GiteaSourceInventory` test, a Git-level one against a local HTTPS Git, LFS batch and REST server (needs `openssl` and `git http-backend`), and an opt-in one against a Docker Gitea from `@foss.global/forgefixtures`, run with `FORGEFIXTURES=gitea pnpm test` (Docker required).
+- A mocked `GiteaSourceInventory` test, Git-level `GiteaSourceInventory` and `GiteaSourceTransfer` tests against a local HTTPS Git, LFS batch and REST server (needs `openssl` and `git http-backend`), and an opt-in one against a Docker Gitea from `@foss.global/forgefixtures`, run with `FORGEFIXTURES=gitea pnpm test` (Docker required).
+- An `LfsBatchClient` test against local loopback LFS servers over HTTP and over HTTPS with a private CA.
 
 Run tests:
 

package/ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@foss.global/codefeed',
-  version: '3.0.0',
+  version: '3.1.0',
   description: 'The @foss.global/codefeed module generates commit/activity feeds from forge providers including Gitea and GitManager.'
 }

package/ts/classes.giteasourcetransfer.ts

@@ -0,0 +1,213 @@
+import * as plugins from './plugins.js';
+import * as transport from './forgeinventory.transport.js';
+import * as validation from './forgeinventory.validation.js';
+import { ForgeRequestClient } from './forgeinventory.client.js';
+import { LfsBatchClient } from './classes.lfsbatchclient.js';
+import type { ILfsRequestOptions, TLfsDownload } from './classes.lfsbatchclient.js';
+import { giteaCloneUrl, reserveMirrorScratch } from './giteasourceinventory.git.js';
+
+export interface IGiteaSourceTransferOptions {
+  /** Instance root, including an optional deployment subpath. HTTPS is required. */
+  baseUrl: string;
+  /** The reader's login, the user name of its Git and LFS credentials. */
+  login: string;
+  /**
+   * The reader's token, retained only in this reader's memory. It is sent only
+   * to `baseUrl`, and to Git only through `GIT_CONFIG_*` entries in the child
+   * process environment; it never appears in results, reasons or errors.
+   */
+  token: string;
+  /** PEM certificates that replace the default trust store for this reader's connections, Git transfers included. */
+  caCertificates?: string[];
+  /** Wall-clock limit of one mirror clone; defaults to 30 minutes, at most six hours. */
+  gitTimeoutMs?: number;
+  /** Deadline of one REST or LFS batch request, and the idle limit of an LFS download; defaults to 60 s, at most 300 s. */
+  requestTimeoutMs?: number;
+  /** Response-size budget of one REST or LFS batch answer; defaults to 16 MiB, at most 64 MiB. */
+  maxResponseBytes?: number;
+  /** Wall-clock limit of one LFS object download; defaults to 30 minutes, at most six hours. */
+  lfsTransferTimeoutMs?: number;
+}
+
+/** A source repository, by its owner's and its own name. */
+export interface IGiteaSourceRepository {
+  owner: string;
+  name: string;
+}
+
+export interface IGiteaMirrorImportOptions {
+  /** The repository the mirror becomes in the supplied GitManager. */
+  target: plugins.fossInterfaces.data.IGitManagerRepositorySelector;
+  /**
+   * Aborts the import with `request-aborted`. An abort while the clone runs
+   * cancels it through the GitManager transfer's own signal: the call settles
+   * once Git has exited, leaves no repository at `target`, and the GitManager
+   * keeps serving its other work.
+   */
+  signal?: AbortSignal;
+}
+
+export interface IGiteaMirrorImport {
+  repository: plugins.TGitManagerRepositoryInfo;
+  /** The repository size Gitea reported, in KiB, which sized the scratch reservation. */
+  reportedSizeKiB: string;
+}
+
+/**
+ * Reads a Gitea repository's Git data and LFS objects for its one-time
+ * migration, with the reader's login and token: a mirror clone into a
+ * GitManager the caller supplies, and verified LFS object downloads. It never
+ * writes to Gitea: Git is read only through `git-upload-pack`, LFS only
+ * through batch `download` requests and their download actions.
+ */
+export class GiteaSourceTransfer {
+  readonly #options: {
+    baseUrl: string; login: string; token: string; caCertificates?: string[];
+    gitTimeoutMs: number; requestTimeoutMs: number; maxResponseBytes: number; lfsTransferTimeoutMs: number;
+  };
+  readonly #client: ForgeRequestClient;
+
+  constructor(optionsArg: IGiteaSourceTransferOptions) {
+    const baseUrl = validation.httpUrl(optionsArg.baseUrl);
+    const unsafe = (valueArg: unknown) => typeof valueArg !== 'string' || !valueArg || /[\s\u0000-\u001f\u007f]/.test(valueArg);
+    if (!baseUrl || !baseUrl.startsWith('https:') || unsafe(optionsArg.token) || unsafe(optionsArg.login) || optionsArg.login.includes(':')) {
+      throw new validation.ForgeInventoryError('invalid-options');
+    }
+    const limits = {
+      gitTimeoutMs: optionsArg.gitTimeoutMs ?? 30 * 60 * 1000,
+      requestTimeoutMs: optionsArg.requestTimeoutMs ?? 60_000,
+      maxResponseBytes: optionsArg.maxResponseBytes ?? 16 * 1024 * 1024,
+      lfsTransferTimeoutMs: optionsArg.lfsTransferTimeoutMs ?? 30 * 60 * 1000,
+    };
+    for (const [name, maximum] of [
+      ['gitTimeoutMs', 6 * 60 * 60 * 1000], ['requestTimeoutMs', 300_000], ['maxResponseBytes', 64 * 1024 * 1024], ['lfsTransferTimeoutMs', 6 * 60 * 60 * 1000],
+    ] as const) {
+      const value = limits[name];
+      if (!Number.isSafeInteger(value) || value < 1 || value > maximum) throw new validation.ForgeInventoryError('invalid-limits');
+    }
+    const caCertificates = optionsArg.caCertificates === undefined ? undefined : validation.caCertificates(optionsArg.caCertificates);
+    this.#options = { baseUrl: baseUrl.replace(/\/+$/, ''), login: optionsArg.login, token: optionsArg.token, caCertificates, ...limits };
+    this.#client = new ForgeRequestClient({
+      transport: caCertificates === undefined ? transport.globalFetchTransport : transport.createHttpsTransport(caCertificates),
+      headers: { authorization: `token ${optionsArg.token}` },
+      requestTimeoutMs: limits.requestTimeoutMs,
+      maxResponseBytes: limits.maxResponseBytes,
+    });
+  }
+
+  /** Validated, normalized instance root; contains no credentials or API suffix. */
+  public get baseUrl(): string { return this.#options.baseUrl; }
+
+  /**
+   * Clones every ref of the repository into `options.target` of `gitManager`,
+   * a started GitManager the caller roots in its own scratch space. The size
+   * Gitea reports for the repository is read first; the clone starts only once
+   * its scratch reservation fits the free space of the file system holding
+   * the GitManager's data directory, in the process-wide `GitScratchBudget`,
+   * and the reservation is released once the clone has ended. While it runs
+   * the import holds `target`: writes to it through the GitManager fail there
+   * with `repositoryBusy`. Rejects with a fixed `ForgeInventoryError` code:
+   * `invalid-repository`, `invalid-target`, `gitmanager-not-running`, a REST
+   * code such as `http-404`, `git-workdir-failed`, `git-workdir-too-small`,
+   * `git-mirror-failed` (also when GitManager refuses `target`, such as with
+   * `repositoryBusy` while another transfer holds it) or `request-aborted`
+   * (an abort, or a GitManager stop that cancelled the clone). GitManager
+   * error text is never passed on.
+   */
+  public async importMirror(
+    repositoryArg: IGiteaSourceRepository,
+    gitManagerArg: plugins.TGitManager,
+    optionsArg: IGiteaMirrorImportOptions,
+  ): Promise<IGiteaMirrorImport> {
+    const { owner, name } = this.#repository(repositoryArg);
+    const target = optionsArg?.target;
+    if (typeof target?.org !== 'string' || typeof target?.repo !== 'string') throw new validation.ForgeInventoryError('invalid-target');
+    const { signal } = optionsArg;
+    if (signal?.aborted) throw new validation.ForgeInventoryError('request-aborted');
+    if (!gitManagerArg.running) throw new validation.ForgeInventoryError('gitmanager-not-running');
+    let reportedSizeKiB: bigint;
+    try {
+      const repository = validation.record((await this.#client.request(
+        new URL(`${this.#options.baseUrl}/api/v1/repos/${encodeURIComponent(owner)}/${encodeURIComponent(name)}`), { signal },
+      )).body);
+      reportedSizeKiB = BigInt(validation.decimal(repository.size));
+    } catch (error) {
+      if (signal?.aborted) throw new validation.ForgeInventoryError('request-aborted');
+      throw error;
+    }
+    let dataDir: string;
+    try {
+      dataDir = (await gitManagerArg.getStatus()).dataDir;
+    } catch {
+      throw new validation.ForgeInventoryError('git-workdir-failed');
+    }
+    const release = await reserveMirrorScratch(dataDir, reportedSizeKiB, signal);
+    try {
+      const imported = await gitManagerArg.importMirror({
+        org: target.org,
+        repo: target.repo,
+        remoteUrl: giteaCloneUrl(this.#options.baseUrl, owner, name),
+        credential: { kind: 'basic', username: this.#options.login, password: this.#options.token },
+        caCertificates: this.#options.caCertificates,
+        timeoutMs: this.#options.gitTimeoutMs,
+      }, { signal });
+      return { repository: imported, reportedSizeKiB: reportedSizeKiB.toString() };
+    } catch (error) {
+      const { GitManagerError } = await plugins.getGitManager();
+      const aborted = signal?.aborted || (error instanceof GitManagerError && error.code === 'transferAborted');
+      // GitManager errors are redacted, but still remote text.
+      throw new validation.ForgeInventoryError(aborted ? 'request-aborted' : 'git-mirror-failed');
+    } finally {
+      // The import settles only after Git has exited, so the scratch is free again.
+      release();
+    }
+  }
+
+  /**
+   * Downloads one LFS object of the repository: a batch `download` request
+   * with the reader's Basic credentials, then a GET of the action's href with
+   * the action's headers only. Resolves with `{ status: 'missing' }` when
+   * Gitea answers error 404, otherwise with a stream of the object verified
+   * against `oid` and `size` as described for `LfsBatchClient.download`.
+   * Rejects with a fixed code, such as `lfs-object-error-422` for a size Gitea
+   * disagrees with.
+   */
+  public async downloadLfsObject(repositoryArg: IGiteaSourceRepository, oidArg: string, sizeArg: number, optionsArg: ILfsRequestOptions = {}): Promise<TLfsDownload> {
+    const { owner, name } = this.#repository(repositoryArg);
+    const client = new LfsBatchClient({
+      endpoint: `${giteaCloneUrl(this.#options.baseUrl, owner, name)}/info/lfs`,
+      // Gitea's LFS routes take the token as a Basic password, not as a `token` authorization.
+      credential: { kind: 'basic', username: this.#options.login, password: this.#options.token },
+      caCertificates: this.#options.caCertificates,
+      requestTimeoutMs: this.#options.requestTimeoutMs,
+      transferTimeoutMs: this.#options.lfsTransferTimeoutMs,
+      maxResponseBytes: this.#options.maxResponseBytes,
+      batchSize: 1,
+    });
+    try {
+      const download = await client.download({ oid: oidArg, size: sizeArg }, optionsArg);
+      if (download.status === 'available') {
+        download.stream.once('close', () => client.close());
+      } else {
+        client.close();
+      }
+      return download;
+    } catch (error) {
+      client.close();
+      throw error;
+    }
+  }
+
+  #repository(repositoryArg: IGiteaSourceRepository): IGiteaSourceRepository {
+    const single = (valueArg: unknown): string => {
+      const segments = validation.segments(valueArg);
+      if (segments.length !== 1) throw new validation.ForgeInventoryError('invalid-repository');
+      return segments[0];
+    };
+    try {
+      return { owner: single(repositoryArg?.owner), name: single(repositoryArg?.name) };
+    } catch {
+      throw new validation.ForgeInventoryError('invalid-repository');
+    }
+  }
+}