@thallylabs/migrate 0.2.23 → 0.2.25

package/README.md

@@ -171,6 +171,54 @@ from Fern MDX pages (`../../components/...`, resolved against the Fern
 project root) are migrated the same way Mintlify and Docusaurus components
 are — see "Mintlify compatibility" below.
 
+## Mintlify `sourceRef` repositories
+
+A Mintlify navigation node `{"sourceRef": "owner/repo"}` mounts the docs of another
+repository. Mintlify does not document where that mount lives, so it cannot be
+derived from `docs.json`; supply it with a repeatable flag:
+
+```bash
+create-thally-docs migrate https://github.com/OpenRouterTeam/docs my-docs \
+  --source-ref OpenRouterTeam/typescript-sdk=client-sdks/typescript
+```
+
+Without a mapping the node is skipped with a warning that names the flag.
+
+- The repository must be on github.com. Its docs root is the directory holding a
+  Mintlify `docs.json` (`docs/` first, then the repository root); without one the
+  mapping is skipped with a warning.
+- Every `.md`/`.mdx` file under that root is imported (orphans included) through the
+  same pipeline as the main site, under `<mount-path>/...`. Assets are copied under
+  `public/<mount-path>/...`. The node is replaced by the referenced `docs.json`
+  navigation with page paths prefixed by the mount path; tabs become groups, and the
+  parent group keeps its label and icon.
+- Root-absolute links inside those pages (`/models/x`, `/images/x.png`) are prefixed
+  with the mount path (sub-site semantics: Mintlify's behavior here is not
+  documented, so this is an assumption). Relative links work as-is.
+- Pages, navigation, assets and the referenced repository's custom components are
+  imported. Its components are bundled with the main site's under `src/mdx/migrated/`
+  and registered in `src/mdx/custom-components.tsx`; their content-hashed names cannot
+  clash with the main site's. Branding, colors, redirects, OpenAPI specs and
+  access-restricted pages of the referenced repository are ignored (one warning says
+  so). Access-restricted pages and the files only they use are withheld from the site
+  and are not saved anywhere (the main site saves its own under `migration-quarantine/`;
+  a referenced repository's are discarded), so recover them from that repository if
+  needed.
+- Safety limits per referenced repository, fixed constants: 5,000 files and 50 MB under
+  the docs root. Over a limit the whole repository is refused, never imported
+  partially. Symlinks and dot-directories are removed before import, and a docs
+  directory that resolves outside the clone is refused. A repository GitHub reports as
+  larger than 200 MB is refused before cloning (if the GitHub API is unreachable the
+  clone proceeds and the limits above still apply), its submodules are not fetched,
+  and a `docs.json` over 1 MB is ignored. `repo` must match `owner/repo`; the mount
+  path must be a relative lowercase slug path (no `..`, no leading slash), must not
+  start with a name the site itself uses (`api`, `admin`, `_next`, `public`,
+  `llms.txt`, `sitemap.xml`, ...), must not collide with an existing page or directory
+  of the main site, and mappings must not overlap.
+- A mapping that matches no `sourceRef` in the navigation is reported and imports
+  nothing. `migration-report.json` lists each imported repository with its page count
+  under `sourceRefs`.
+
 ## Mintlify compatibility
 
 Repository imports preserve named/default local JSX and TSX components and
@@ -214,8 +262,8 @@ same-length nested fence is widened so the inner fence no longer closes it, and
 a bare placeholder tag in prose (`<Feature> requires a plan`) that would stop
 the page compiling is escaped as text rather than losing the page.
 The Mintlify `search-bar-entry` click trigger is mapped to Thally's search
-shortcut. Markdown mixed inside interactive JSX, computed imports, and
-namespace imports are reported for manual review; unsupported source is
+shortcut. Markdown mixed inside interactive JSX and computed imports are
+reported for manual review; unsupported source is
 preserved, not replaced with empty stubs. An external package's JSX usage is
 dropped and replaced with a safe fallback (with a warning); if that package's
 binding is instead referenced outside JSX (an expression, a prop, or an
@@ -237,6 +285,30 @@ Next) and copied into any extracted client module. A source site's own
 passes a page-authored function into a client component needs the client
 wrapper described above; unsupported callback shapes retain their content
 with a warning. An unconfirmed target is warned for review.
+Other relative imports of a copied file are handled the same way: a namespace
+import (`import * as X from './comp.jsx'`) and a dynamic `import('./comp.jsx')`
+point at the copied module, and a relative side-effect import
+(`import './styles.css'`) is removed with a warning, like one from an npm
+package. A page is excluded only when the relative target is not copied (for
+example a Next.js `./metadata` module, or an `export ... from` re-export), with
+a single warning naming it. An MDX expression that names an identifier the page
+never defines (`{missing}`) is removed with a warning, since it would throw at
+render; define the name or escape the braces as `\{ \}` to keep the text.
+A value import from a local snippet (`import { FEE, fee } from
+'/snippets/c.mdx'`) is inlined when each name is a static value (string,
+number, boolean, `null`, or an object/array of those) or a self-contained
+function. Inlined functions may only use pure built-ins (`Math`, `JSON`,
+`String`, `Number`, `Boolean`, `Array`, `Date`, `Intl`, `parseInt`,
+`parseFloat`, `isNaN`, `isFinite`, `encodeURIComponent` and
+`decodeURIComponent`) and may not reach
+`Function`, constructors, `globalThis`, `fetch` or other globals; anything else
+is not inlined. A name the snippet does not export, or one that cannot be
+inlined, is bound to `undefined` (as Mintlify does for a missing export) with a
+warning, while the other names of the same import are still inlined. A snippet
+import that resolves outside the repository through a symlink is refused, and a
+snippet whose frontmatter cannot be read is not inlined (left as a comment with
+a warning). Mintlify `sourceRef` navigation nodes have no Thally equivalent and
+are reported rather than migrated.
 File paths cannot escape the repository checkout; symlinks and oversized
 graphs are rejected. This is compatibility analysis, **not a code sandbox**:
 imported JavaScript executes when the developer builds or runs the resulting

package/dist/index.d.ts

@@ -81,6 +81,8 @@ interface MigrationDocsConfig {
     };
     /** Local site-authored CSS projected from a source docs site, served from public/. */
     stylesheets?: Array<string>;
+    /** `false` hides the breadcrumb trail; Mintlify pages show only a group eyebrow. */
+    breadcrumbs?: boolean;
     tabs: Array<MigrationNavigationTab>;
     navigation?: {
         /** How sibling documentation collections are presented to readers. */
@@ -173,6 +175,8 @@ interface MigrationDocsConfig {
         playground?: {
             display: 'interactive' | 'simple' | 'none';
         };
+        /** `false` hides the visible "OpenAPI specification" line; Mintlify does not show it. */
+        specLink?: boolean;
     };
     /** Third-party analytics in Mintlify's `integrations` shape; validated again by the renderer. */
     integrations?: {
@@ -255,6 +259,18 @@ interface MigrationWarning {
     message: string;
     source?: string;
 }
+/** A repository-configured remote OpenAPI spec, downloaded after the bundle is built. */
+interface RemoteApiSpec {
+    url: string;
+    tabLabel?: string;
+    parentTab?: string;
+    icon?: string;
+    hidden?: boolean;
+    /** Operation keys (`get /x`) only access-restricted pages document; marked `x-excluded` after download. */
+    withheldOperations?: Array<string>;
+    /** Operation keys published pages or docs.json also name; keeps a shared `$ref` path item public. */
+    keptOperations?: Array<string>;
+}
 interface MigrationBundle {
     sourceUrl: string;
     sourceKind: 'repository' | 'url';
@@ -262,19 +278,19 @@ interface MigrationBundle {
     pages: Array<MigrationPage>;
     assets: Array<MigrationAsset>;
     /** Repository-configured remote specs awaiting a bounded network fetch by the host. */
-    remoteApiSpecs?: Array<{
-        url: string;
-        tabLabel?: string;
-        parentTab?: string;
-        icon?: string;
-        hidden?: boolean;
-    }>;
+    remoteApiSpecs?: Array<RemoteApiSpec>;
     /** Customer-owned component source and registry; paths are repository-relative. */
     componentFiles?: Array<RenderedMigrationFile>;
     /** Access-restricted source pages, written outside every published path. */
     quarantinedFiles?: Array<RenderedMigrationFile>;
     /** Access-restricted pages the file limit dropped: neither published nor saved in quarantinedFiles. */
     droppedGatedPages?: number;
+    /** Mintlify `sourceRef` repositories imported under a mount path, with their page counts. */
+    sourceRefs?: Array<{
+        repo: string;
+        mountPath: string;
+        pages: number;
+    }>;
     docsConfig: MigrationDocsConfig;
     site?: {
         name?: string;
@@ -318,6 +334,51 @@ interface RenderedMigrationFile {
     content: string | Uint8Array;
 }
 
+/**
+ * Mintlify `sourceRef` navigation nodes mount the docs of another repository
+ * (`{"sourceRef": "owner/repo"}`). The mount path is not derivable from any
+ * docs.json, so the caller supplies it with `--source-ref owner/repo=<path>`.
+ * The referenced repository is untrusted remote content: it is validated,
+ * size-capped, stripped of symlinks and dot-directories, then run through the
+ * same repository pipeline as the main site and prefixed with its mount path.
+ */
+
+/** Per referenced repository: files and bytes under its docs root. Constants on purpose, not configurable. */
+declare const SOURCE_REF_MAX_FILES = 5000;
+declare const SOURCE_REF_MAX_BYTES = 50000000;
+/** GitHub reports repository size in KB; a larger repository is refused before it is cloned. */
+declare const SOURCE_REF_MAX_REPOSITORY_KB = 200000;
+interface SourceRefMapping {
+    /** `owner/repo`, exactly as written in the source navigation. */
+    repo: string;
+    /** Relative slug path the referenced docs are mounted at, e.g. `client-sdks/typescript`. */
+    mountPath: string;
+}
+interface SourceRefImport extends SourceRefMapping {
+    pages: Array<MigrationPage>;
+    assets: Array<MigrationAsset>;
+    /** Navigation entries with every page id already prefixed by the mount path. */
+    navigation: Array<string | MigrationNavigationGroup>;
+    /** Custom component files (and registry) the mounted pages' `<MigratedXXXX/>` tags need; merged into the main bundle. */
+    componentFiles?: Array<RenderedMigrationFile>;
+    /** Why the import is empty (or partial), plus forwarded pipeline warnings. */
+    warnings: Array<MigrationWarning>;
+}
+/** Parse repeated `owner/repo=mount/path` values; throws on anything unsafe. */
+declare function parseSourceRefFlags(values: ReadonlyArray<string>): Array<SourceRefMapping>;
+/**
+ * Why a referenced repository is too large to clone, or null when it is small
+ * enough or its size is unknown (GitHub API unreachable, rate-limited or
+ * private): an unknown size proceeds, and the post-clone caps still apply.
+ */
+declare function sourceRefOversizeReason(repo: string, fetcher?: MigrationFetcher): Promise<string | null>;
+/**
+ * Import one cloned referenced repository as a sub-site under its mount path.
+ * `repositoryDir` is a disposable clone: symlinks and dot-directories under
+ * its docs root are deleted.
+ */
+declare function importSourceRef(mapping: SourceRefMapping, repositoryDir: string): SourceRefImport;
+
 /**
  * Repository adapter for the shared migration engine. Traversal is bounded;
  * `scanFiles` follows a symbolic link only when its resolved real path stays
@@ -340,6 +401,8 @@ interface RepositoryMigrationOptions {
     sourceUrl: string;
     docsDir?: string;
     platform?: MigrationPlatform;
+    /** @internal Mount path of a sourceRef import; keeps its component files separate from any other copy of the same source. */
+    componentNamespace?: string;
     /** @internal Prefix used by additional Docusaurus docs-plugin instances. */
     docusaurusRoutePrefix?: string;
     /** @internal Prevent recursive discovery while importing one plugin root. */
@@ -354,6 +417,8 @@ interface RepositoryMigrationOptions {
     maxSourceFiles?: number;
     /** @internal Redirects are global config, read once, not per plugin instance. */
     docusaurusSkipRedirects?: boolean;
+    /** Mintlify `sourceRef` repositories already fetched and prepared by `importSourceRef`. */
+    sourceRefs?: ReadonlyArray<SourceRefImport>;
 }
 /** Parse and validate a public GitHub repository URL. */
 declare function parseGitHubRepositoryUrl(rawUrl: string): GitHubRepositorySource;
@@ -371,7 +436,11 @@ declare function gitmodulePaths(targetDir: string): Array<string>;
  * submodule-fetch-failure warning (this function otherwise returns exactly
  * as before, so existing callers are unaffected).
  */
-declare function cloneGitHubRepository(source: GitHubRepositorySource, targetDir: string, warnings?: Array<MigrationWarning>): Promise<void>;
+declare function cloneGitHubRepository(source: GitHubRepositorySource, targetDir: string, warnings?: Array<MigrationWarning>, 
+/** Untrusted referenced repositories: never follow their `.gitmodules` to further remotes. */
+options?: {
+    skipSubmodules?: boolean;
+}): Promise<void>;
 /** Detect a supported repository docs platform from unambiguous config files. */
 declare function detectRepositoryPlatform(repositoryDir: string, docsDir?: string): MigrationPlatform;
 /** Pick the first conventional content root containing Markdown. */
@@ -400,11 +469,19 @@ interface MintlifyNavigationResult {
     pageReferences: Array<MintlifyPageReference>;
     /** Pages under a restricted container; withheld from `docsConfig` and `pageReferences`. */
     gatedReferences: Array<MintlifyGatedReference>;
+    /** `sourceRef` mounts under a restricted container; never resolved, so nothing of theirs is imported. */
+    gatedSourceRefs: Array<string>;
     warnings: Array<MigrationWarning>;
 }
 interface MintlifyProjectionOptions {
     /** Public URL migrations may expose page hrefs including the docs mount path. */
     pathPrefix?: string;
+    /**
+     * Resolves a `sourceRef` repository (`owner/repo`) to the navigation entries
+     * of its imported docs, already prefixed with their mount path. Returns null
+     * when the repository was not mapped with `--source-ref`.
+     */
+    resolveSourceRef?: (repo: string) => Array<string | MigrationNavigationGroup> | null;
 }
 /** Preserve Mintlify's implicit `/section` route for section landing pages. */
 declare function addMintlifyDirectoryRedirects(config: MigrationDocsConfig, pages: Array<MigrationPage>): MigrationDocsConfig;
@@ -583,4 +660,4 @@ declare function migrateUrl(options: UrlMigrationOptions): Promise<MigrationBund
  */
 declare function hydrateRemoteApiSpecs(bundle: MigrationBundle, fetcher?: MigrationFetcher): Promise<MigrationBundle>;
 
-export { type GitHubRepositorySource, type MigrationAsset, type MigrationBundle, type MigrationDocsConfig, type MigrationFetchRequest, type MigrationFetchResponse, type MigrationFetcher, type MigrationNavigationGroup, type MigrationNavigationTab, type MigrationPage, type MigrationPlatform, type MigrationWarning, type RenderedMigrationFile, type RepositoryMigrationOptions, type UrlMigrationOptions, addMintlifyDirectoryRedirects, buildNavigationFromPages, cloneGitHubRepository, defaultMigrationFetcher, detectRepositoryDocsDir, detectRepositoryPlatform, gitmodulePaths, hydrateRemoteApiSpecs, mergeMigrationConfig, migrateRepository, migrateUrl, normalizeMdx, parseGitHubRepositoryUrl, parseMarkdownPage, projectFernNavigation, projectMintlifyIntegrations, projectMintlifyNavigation, pruneMissingNavigationPages, readFernConfig, readMintlifyConfig, renderMigrationFiles, validateIntegrations, validateMigrationUrl };
+export { type GitHubRepositorySource, type MigrationAsset, type MigrationBundle, type MigrationDocsConfig, type MigrationFetchRequest, type MigrationFetchResponse, type MigrationFetcher, type MigrationNavigationGroup, type MigrationNavigationTab, type MigrationPage, type MigrationPlatform, type MigrationWarning, type RenderedMigrationFile, type RepositoryMigrationOptions, SOURCE_REF_MAX_BYTES, SOURCE_REF_MAX_FILES, SOURCE_REF_MAX_REPOSITORY_KB, type SourceRefImport, type SourceRefMapping, type UrlMigrationOptions, addMintlifyDirectoryRedirects, buildNavigationFromPages, cloneGitHubRepository, defaultMigrationFetcher, detectRepositoryDocsDir, detectRepositoryPlatform, gitmodulePaths, hydrateRemoteApiSpecs, importSourceRef, mergeMigrationConfig, migrateRepository, migrateUrl, normalizeMdx, parseGitHubRepositoryUrl, parseMarkdownPage, parseSourceRefFlags, projectFernNavigation, projectMintlifyIntegrations, projectMintlifyNavigation, pruneMissingNavigationPages, readFernConfig, readMintlifyConfig, renderMigrationFiles, sourceRefOversizeReason, validateIntegrations, validateMigrationUrl };