@thallylabs/migrate 0.2.24 → 0.2.26

package/README.md

@@ -171,6 +171,54 @@ from Fern MDX pages (`../../components/...`, resolved against the Fern
 project root) are migrated the same way Mintlify and Docusaurus components
 are — see "Mintlify compatibility" below.
 
+## Mintlify `sourceRef` repositories
+
+A Mintlify navigation node `{"sourceRef": "owner/repo"}` mounts the docs of another
+repository. Mintlify does not document where that mount lives, so it cannot be
+derived from `docs.json`; supply it with a repeatable flag:
+
+```bash
+create-thally-docs migrate https://github.com/OpenRouterTeam/docs my-docs \
+  --source-ref OpenRouterTeam/typescript-sdk=client-sdks/typescript
+```
+
+Without a mapping the node is skipped with a warning that names the flag.
+
+- The repository must be on github.com. Its docs root is the directory holding a
+  Mintlify `docs.json` (`docs/` first, then the repository root); without one the
+  mapping is skipped with a warning.
+- Every `.md`/`.mdx` file under that root is imported (orphans included) through the
+  same pipeline as the main site, under `<mount-path>/...`. Assets are copied under
+  `public/<mount-path>/...`. The node is replaced by the referenced `docs.json`
+  navigation with page paths prefixed by the mount path; tabs become groups, and the
+  parent group keeps its label and icon.
+- Root-absolute links inside those pages (`/models/x`, `/images/x.png`) are prefixed
+  with the mount path (sub-site semantics: Mintlify's behavior here is not
+  documented, so this is an assumption). Relative links work as-is.
+- Pages, navigation, assets and the referenced repository's custom components are
+  imported. Its components are bundled with the main site's under `src/mdx/migrated/`
+  and registered in `src/mdx/custom-components.tsx`; their content-hashed names cannot
+  clash with the main site's. Branding, colors, redirects, OpenAPI specs and
+  access-restricted pages of the referenced repository are ignored (one warning says
+  so). Access-restricted pages and the files only they use are withheld from the site
+  and are not saved anywhere (the main site saves its own under `migration-quarantine/`;
+  a referenced repository's are discarded), so recover them from that repository if
+  needed.
+- Safety limits per referenced repository, fixed constants: 5,000 files and 50 MB under
+  the docs root. Over a limit the whole repository is refused, never imported
+  partially. Symlinks and dot-directories are removed before import, and a docs
+  directory that resolves outside the clone is refused. A repository GitHub reports as
+  larger than 200 MB is refused before cloning (if the GitHub API is unreachable the
+  clone proceeds and the limits above still apply), its submodules are not fetched,
+  and a `docs.json` over 1 MB is ignored. `repo` must match `owner/repo`; the mount
+  path must be a relative lowercase slug path (no `..`, no leading slash), must not
+  start with a name the site itself uses (`api`, `admin`, `_next`, `public`,
+  `llms.txt`, `sitemap.xml`, ...), must not collide with an existing page or directory
+  of the main site, and mappings must not overlap.
+- A mapping that matches no `sourceRef` in the navigation is reported and imports
+  nothing. `migration-report.json` lists each imported repository with its page count
+  under `sourceRefs`.
+
 ## Mintlify compatibility
 
 Repository imports preserve named/default local JSX and TSX components and
@@ -214,8 +262,8 @@ same-length nested fence is widened so the inner fence no longer closes it, and
 a bare placeholder tag in prose (`<Feature> requires a plan`) that would stop
 the page compiling is escaped as text rather than losing the page.
 The Mintlify `search-bar-entry` click trigger is mapped to Thally's search
-shortcut. Markdown mixed inside interactive JSX, computed imports, and
-namespace imports are reported for manual review; unsupported source is
+shortcut. Markdown mixed inside interactive JSX and computed imports are
+reported for manual review; unsupported source is
 preserved, not replaced with empty stubs. An external package's JSX usage is
 dropped and replaced with a safe fallback (with a warning); if that package's
 binding is instead referenced outside JSX (an expression, a prop, or an
@@ -237,6 +285,30 @@ Next) and copied into any extracted client module. A source site's own
 passes a page-authored function into a client component needs the client
 wrapper described above; unsupported callback shapes retain their content
 with a warning. An unconfirmed target is warned for review.
+Other relative imports of a copied file are handled the same way: a namespace
+import (`import * as X from './comp.jsx'`) and a dynamic `import('./comp.jsx')`
+point at the copied module, and a relative side-effect import
+(`import './styles.css'`) is removed with a warning, like one from an npm
+package. A page is excluded only when the relative target is not copied (for
+example a Next.js `./metadata` module, or an `export ... from` re-export), with
+a single warning naming it. An MDX expression that names an identifier the page
+never defines (`{missing}`) is removed with a warning, since it would throw at
+render; define the name or escape the braces as `\{ \}` to keep the text.
+A value import from a local snippet (`import { FEE, fee } from
+'/snippets/c.mdx'`) is inlined when each name is a static value (string,
+number, boolean, `null`, or an object/array of those) or a self-contained
+function. Inlined functions may only use pure built-ins (`Math`, `JSON`,
+`String`, `Number`, `Boolean`, `Array`, `Date`, `Intl`, `parseInt`,
+`parseFloat`, `isNaN`, `isFinite`, `encodeURIComponent` and
+`decodeURIComponent`) and may not reach
+`Function`, constructors, `globalThis`, `fetch` or other globals; anything else
+is not inlined. A name the snippet does not export, or one that cannot be
+inlined, is bound to `undefined` (as Mintlify does for a missing export) with a
+warning, while the other names of the same import are still inlined. A snippet
+import that resolves outside the repository through a symlink is refused, and a
+snippet whose frontmatter cannot be read is not inlined (left as a comment with
+a warning). Mintlify `sourceRef` navigation nodes have no Thally equivalent and
+are reported rather than migrated.
 File paths cannot escape the repository checkout; symlinks and oversized
 graphs are rejected. This is compatibility analysis, **not a code sandbox**:
 imported JavaScript executes when the developer builds or runs the resulting
@@ -260,30 +332,43 @@ generic unknown-component fallback.
 
 ### Access-restricted pages
 
-A gated Mintlify page must never migrate as public. A page is withheld when its
-frontmatter sets a non-empty `groups` (list or string) or `public: false`
-(also `"false"`, `no`, `off`, `0`; empty `groups` restricts nothing), or when it sits
-under a navigation group or tab that sets `groups` (a list of strings) or
-`public: false`. `public: true` alone is not gating. Withheld pages are not
-written to `src/content` and are removed from navigation (emptied groups and
-tabs included), whether or not navigation lists them. The original file is
-written unchanged to `migration-quarantine/<original path>` at the project
-root, which no runtime content reader, search index, or `llms` output walks;
-`create-thally-docs` also adds it to the project's `.gitignore` (appended once,
-existing entries kept) so it is never committed or pushed. A public page that
-imports a withheld page as a component does not inline it (a `gated-page`
-warning names the importer). Invalid frontmatter fails the run rather than
-importing the page. Each page gets a
-`gated-page` warning, plus a summary; `create-thally-docs` prints these last
-under an "ACCESS-RESTRICTED CONTENT" heading and the report records them and a
-`quarantined` count. Links from other pages to a withheld page are left as
-authored and will fail `thally check`; images and other assets used only by
-gated pages are still copied to `public/`. Mintlify's site-wide authentication mode lives in its
-dashboard and cannot be read from a repository, so when any page sets
+Restricted Mintlify pages migrate as real pages with normalized access
+frontmatter that the Thally runtime enforces (see `/guides/authenticated-docs`):
+
+- Frontmatter `groups` becomes a list of trimmed group names (`groups: admin`
+  and `groups: "a, b"` become lists); `public` becomes a boolean (`"false"`,
+  `no`, `off`, `0` are `false`; `true`/`"true"` is `true`). Empty `groups`
+  restricts nothing.
+- Rules on navigation containers (a group, tab, version, dropdown, anchor,
+  language, or `menu` item) are pushed down into the frontmatter of every page
+  below them, because the runtime reads access only from pages: container
+  `groups` (a list of strings) and `public: false` restrict, and Mintlify's
+  group-level `public: true` opens pages that name no groups of their own.
+  Pages stay in navigation where they were.
+- Rules combine to the strictest: nested and page group lists intersect,
+  `public: false` anywhere wins, and a page listed under both an open and a
+  restricted container gets the restricted rule.
+
+A page whose rules cannot be carried over safely fails closed instead: a
+malformed value (`public: maybe`, `groups: [1]`), group lists that share no
+group, or frontmatter that declares `groups`/`public` but cannot be parsed or
+read. Such a page is not written to `src/content`, is removed from navigation,
+and its original is written unchanged to `migration-quarantine/<original path>`
+at the project root, which no runtime reader walks; `create-thally-docs` adds
+that folder to `.gitignore`. Each quarantined page gets a `gated-page` warning.
+
+What a restricted page uses must not leak through surfaces that are not access
+controlled: it is never inlined into another page (a `gated-page` warning names
+the importer), images and other assets used only by restricted pages are not
+copied to `public/` (they are saved under `migration-quarantine/assets/`), and
+OpenAPI operations documented only on restricted pages are withheld from the
+published spec. One `gated-page` summary counts the restricted pages and notes
+that Thally hides them from everyone until reader authentication is configured
+in `docs.json` `auth`; no `auth` block is generated, because Mintlify's
+dashboard settings cannot be read from a repository. When any page sets
 `public: true` (which implies the source site required sign-in for everything
-else) one `gated-page` warning says Thally will publish every imported page.
-Restricted-container `groups` and `public: false` are honored fail-closed even
-though Mintlify documents only `"public": true` on a group.
+else), one more `gated-page` warning says every unrestricted page will be
+public until `auth` is configured.
 
 ### Site-wide scripts, styles, fonts, and settings
 

package/dist/index.d.ts

@@ -81,6 +81,8 @@ interface MigrationDocsConfig {
     };
     /** Local site-authored CSS projected from a source docs site, served from public/. */
     stylesheets?: Array<string>;
+    /** `false` hides the breadcrumb trail; Mintlify pages show only a group eyebrow. */
+    breadcrumbs?: boolean;
     tabs: Array<MigrationNavigationTab>;
     navigation?: {
         /** How sibling documentation collections are presented to readers. */
@@ -173,6 +175,8 @@ interface MigrationDocsConfig {
         playground?: {
             display: 'interactive' | 'simple' | 'none';
         };
+        /** `false` hides the visible "OpenAPI specification" line; Mintlify does not show it. */
+        specLink?: boolean;
     };
     /** Third-party analytics in Mintlify's `integrations` shape; validated again by the renderer. */
     integrations?: {
@@ -226,6 +230,13 @@ interface MigrationPage {
     mode?: 'default' | 'wide' | 'custom' | 'center' | 'home';
     hidden?: boolean;
     noindex?: boolean;
+    /**
+     * Reader access groups (frontmatter `groups`): trimmed, non-empty names the
+     * runtime matches exactly. The page needs a signed-in reader in one of them.
+     */
+    groups?: Array<string>;
+    /** Frontmatter `public`: `true` opens the page to anyone, `false` needs any signed-in reader. */
+    public?: boolean;
     /** Social/SEO overrides from `og:*` and `twitter:*` frontmatter, keyed by the migrated frontmatter field. */
     meta?: Partial<Record<'ogTitle' | 'ogDescription' | 'ogImage' | 'twitterTitle' | 'twitterDescription' | 'twitterImage', string>>;
     /** OpenAPI operation key rendered by Thally instead of ordinary MDX. */
@@ -250,7 +261,7 @@ interface MigrationAsset {
 }
 interface MigrationWarning {
     code: 'collision' | 'invalid-page' | 'missing-page' | 'unsupported-config' | 'limit-reached' | 'fetch-failed' | 'skipped-file'
-    /** Access-restricted source page withheld from the published site (or a site-wide auth risk). */
+    /** Access-restricted source content: migrated with access rules, withheld (quarantined), or a site-wide auth risk. */
      | 'gated-page';
     message: string;
     source?: string;
@@ -277,10 +288,20 @@ interface MigrationBundle {
     remoteApiSpecs?: Array<RemoteApiSpec>;
     /** Customer-owned component source and registry; paths are repository-relative. */
     componentFiles?: Array<RenderedMigrationFile>;
-    /** Access-restricted source pages, written outside every published path. */
+    /**
+     * Restricted content that cannot be published safely (pages whose access
+     * rules are malformed or contradictory, and assets used only by restricted
+     * pages), written outside every published path.
+     */
     quarantinedFiles?: Array<RenderedMigrationFile>;
     /** Access-restricted pages the file limit dropped: neither published nor saved in quarantinedFiles. */
     droppedGatedPages?: number;
+    /** Mintlify `sourceRef` repositories imported under a mount path, with their page counts. */
+    sourceRefs?: Array<{
+        repo: string;
+        mountPath: string;
+        pages: number;
+    }>;
     docsConfig: MigrationDocsConfig;
     site?: {
         name?: string;
@@ -324,6 +345,51 @@ interface RenderedMigrationFile {
     content: string | Uint8Array;
 }
 
+/**
+ * Mintlify `sourceRef` navigation nodes mount the docs of another repository
+ * (`{"sourceRef": "owner/repo"}`). The mount path is not derivable from any
+ * docs.json, so the caller supplies it with `--source-ref owner/repo=<path>`.
+ * The referenced repository is untrusted remote content: it is validated,
+ * size-capped, stripped of symlinks and dot-directories, then run through the
+ * same repository pipeline as the main site and prefixed with its mount path.
+ */
+
+/** Per referenced repository: files and bytes under its docs root. Constants on purpose, not configurable. */
+declare const SOURCE_REF_MAX_FILES = 5000;
+declare const SOURCE_REF_MAX_BYTES = 50000000;
+/** GitHub reports repository size in KB; a larger repository is refused before it is cloned. */
+declare const SOURCE_REF_MAX_REPOSITORY_KB = 200000;
+interface SourceRefMapping {
+    /** `owner/repo`, exactly as written in the source navigation. */
+    repo: string;
+    /** Relative slug path the referenced docs are mounted at, e.g. `client-sdks/typescript`. */
+    mountPath: string;
+}
+interface SourceRefImport extends SourceRefMapping {
+    pages: Array<MigrationPage>;
+    assets: Array<MigrationAsset>;
+    /** Navigation entries with every page id already prefixed by the mount path. */
+    navigation: Array<string | MigrationNavigationGroup>;
+    /** Custom component files (and registry) the mounted pages' `<MigratedXXXX/>` tags need; merged into the main bundle. */
+    componentFiles?: Array<RenderedMigrationFile>;
+    /** Why the import is empty (or partial), plus forwarded pipeline warnings. */
+    warnings: Array<MigrationWarning>;
+}
+/** Parse repeated `owner/repo=mount/path` values; throws on anything unsafe. */
+declare function parseSourceRefFlags(values: ReadonlyArray<string>): Array<SourceRefMapping>;
+/**
+ * Why a referenced repository is too large to clone, or null when it is small
+ * enough or its size is unknown (GitHub API unreachable, rate-limited or
+ * private): an unknown size proceeds, and the post-clone caps still apply.
+ */
+declare function sourceRefOversizeReason(repo: string, fetcher?: MigrationFetcher): Promise<string | null>;
+/**
+ * Import one cloned referenced repository as a sub-site under its mount path.
+ * `repositoryDir` is a disposable clone: symlinks and dot-directories under
+ * its docs root are deleted.
+ */
+declare function importSourceRef(mapping: SourceRefMapping, repositoryDir: string): SourceRefImport;
+
 /**
  * Repository adapter for the shared migration engine. Traversal is bounded;
  * `scanFiles` follows a symbolic link only when its resolved real path stays
@@ -346,6 +412,8 @@ interface RepositoryMigrationOptions {
     sourceUrl: string;
     docsDir?: string;
     platform?: MigrationPlatform;
+    /** @internal Mount path of a sourceRef import; keeps its component files separate from any other copy of the same source. */
+    componentNamespace?: string;
     /** @internal Prefix used by additional Docusaurus docs-plugin instances. */
     docusaurusRoutePrefix?: string;
     /** @internal Prevent recursive discovery while importing one plugin root. */
@@ -360,6 +428,8 @@ interface RepositoryMigrationOptions {
     maxSourceFiles?: number;
     /** @internal Redirects are global config, read once, not per plugin instance. */
     docusaurusSkipRedirects?: boolean;
+    /** Mintlify `sourceRef` repositories already fetched and prepared by `importSourceRef`. */
+    sourceRefs?: ReadonlyArray<SourceRefImport>;
 }
 /** Parse and validate a public GitHub repository URL. */
 declare function parseGitHubRepositoryUrl(rawUrl: string): GitHubRepositorySource;
@@ -377,7 +447,11 @@ declare function gitmodulePaths(targetDir: string): Array<string>;
  * submodule-fetch-failure warning (this function otherwise returns exactly
  * as before, so existing callers are unaffected).
  */
-declare function cloneGitHubRepository(source: GitHubRepositorySource, targetDir: string, warnings?: Array<MigrationWarning>): Promise<void>;
+declare function cloneGitHubRepository(source: GitHubRepositorySource, targetDir: string, warnings?: Array<MigrationWarning>, 
+/** Untrusted referenced repositories: never follow their `.gitmodules` to further remotes. */
+options?: {
+    skipSubmodules?: boolean;
+}): Promise<void>;
 /** Detect a supported repository docs platform from unambiguous config files. */
 declare function detectRepositoryPlatform(repositoryDir: string, docsDir?: string): MigrationPlatform;
 /** Pick the first conventional content root containing Markdown. */
@@ -385,6 +459,30 @@ declare function detectRepositoryDocsDir(repositoryDir: string): string;
 /** Import an already-available repository directory into a canonical bundle. */
 declare function migrateRepository(options: RepositoryMigrationOptions): MigrationBundle;
 
+/**
+ * Mintlify-only helpers: page access rules, site-wide CSS/JS/font assets,
+ * and appearance settings. Kept apart from the generic repository walk so the
+ * rules that decide how restricted pages migrate (and what is quarantined)
+ * are auditable in one place.
+ *
+ * Access invariant: a restricted page migrates with normalized `groups` /
+ * `public` frontmatter that the Thally runtime enforces; anything ambiguous
+ * (malformed values, group lists that share no group) fails closed and the
+ * page is quarantined instead.
+ */
+
+/** Access imposed by every navigation container above one appearance of a page. */
+interface NavigationAccess {
+    /** Intersection of every container group list on the path. */
+    groups?: Array<string>;
+    /** Some container on the path sets `public: false`. */
+    hasPublicFalse: boolean;
+    /** Some container on the path sets `public: true` (Mintlify's documented group flag). */
+    hasPublicTrue: boolean;
+    /** Set when a container value is malformed or nested group lists share no group. */
+    problem?: string;
+}
+
 /**
  * Navigation projection for imported sites. Mintlify's schema is intentionally
  * more expressive than Thally's tab/group model, so complex containers are
@@ -396,21 +494,37 @@ interface MintlifyPageReference {
     navigationId: string;
     locale?: string;
 }
-/** A page listed under an access-restricted navigation container. */
-interface MintlifyGatedReference {
+/**
+ * One appearance of a page reference in navigation, with the access rules of
+ * every container above it. Thally reads access only from page frontmatter,
+ * so these are pushed down into the frontmatter of the pages they cover.
+ */
+interface MintlifyReferenceAccess {
     ref: string;
-    reason: string;
+    access: NavigationAccess;
 }
 interface MintlifyNavigationResult {
     docsConfig: MigrationDocsConfig;
     pageReferences: Array<MintlifyPageReference>;
-    /** Pages under a restricted container; withheld from `docsConfig` and `pageReferences`. */
-    gatedReferences: Array<MintlifyGatedReference>;
+    /**
+     * Access imposed by navigation containers, one entry per appearance of a
+     * page (open appearances included, so a page listed in several places can
+     * be given the strictest rule). Restricted pages stay in `docsConfig`.
+     */
+    referenceAccess: Array<MintlifyReferenceAccess>;
+    /** `sourceRef` mounts under a restricted container; never resolved, so nothing of theirs is imported. */
+    gatedSourceRefs: Array<string>;
     warnings: Array<MigrationWarning>;
 }
 interface MintlifyProjectionOptions {
     /** Public URL migrations may expose page hrefs including the docs mount path. */
     pathPrefix?: string;
+    /**
+     * Resolves a `sourceRef` repository (`owner/repo`) to the navigation entries
+     * of its imported docs, already prefixed with their mount path. Returns null
+     * when the repository was not mapped with `--source-ref`.
+     */
+    resolveSourceRef?: (repo: string) => Array<string | MigrationNavigationGroup> | null;
 }
 /** Preserve Mintlify's implicit `/section` route for section landing pages. */
 declare function addMintlifyDirectoryRedirects(config: MigrationDocsConfig, pages: Array<MigrationPage>): MigrationDocsConfig;
@@ -589,4 +703,4 @@ declare function migrateUrl(options: UrlMigrationOptions): Promise<MigrationBund
  */
 declare function hydrateRemoteApiSpecs(bundle: MigrationBundle, fetcher?: MigrationFetcher): Promise<MigrationBundle>;
 
-export { type GitHubRepositorySource, type MigrationAsset, type MigrationBundle, type MigrationDocsConfig, type MigrationFetchRequest, type MigrationFetchResponse, type MigrationFetcher, type MigrationNavigationGroup, type MigrationNavigationTab, type MigrationPage, type MigrationPlatform, type MigrationWarning, type RenderedMigrationFile, type RepositoryMigrationOptions, type UrlMigrationOptions, addMintlifyDirectoryRedirects, buildNavigationFromPages, cloneGitHubRepository, defaultMigrationFetcher, detectRepositoryDocsDir, detectRepositoryPlatform, gitmodulePaths, hydrateRemoteApiSpecs, mergeMigrationConfig, migrateRepository, migrateUrl, normalizeMdx, parseGitHubRepositoryUrl, parseMarkdownPage, projectFernNavigation, projectMintlifyIntegrations, projectMintlifyNavigation, pruneMissingNavigationPages, readFernConfig, readMintlifyConfig, renderMigrationFiles, validateIntegrations, validateMigrationUrl };
+export { type GitHubRepositorySource, type MigrationAsset, type MigrationBundle, type MigrationDocsConfig, type MigrationFetchRequest, type MigrationFetchResponse, type MigrationFetcher, type MigrationNavigationGroup, type MigrationNavigationTab, type MigrationPage, type MigrationPlatform, type MigrationWarning, type RenderedMigrationFile, type RepositoryMigrationOptions, SOURCE_REF_MAX_BYTES, SOURCE_REF_MAX_FILES, SOURCE_REF_MAX_REPOSITORY_KB, type SourceRefImport, type SourceRefMapping, type UrlMigrationOptions, addMintlifyDirectoryRedirects, buildNavigationFromPages, cloneGitHubRepository, defaultMigrationFetcher, detectRepositoryDocsDir, detectRepositoryPlatform, gitmodulePaths, hydrateRemoteApiSpecs, importSourceRef, mergeMigrationConfig, migrateRepository, migrateUrl, normalizeMdx, parseGitHubRepositoryUrl, parseMarkdownPage, parseSourceRefFlags, projectFernNavigation, projectMintlifyIntegrations, projectMintlifyNavigation, pruneMissingNavigationPages, readFernConfig, readMintlifyConfig, renderMigrationFiles, sourceRefOversizeReason, validateIntegrations, validateMigrationUrl };