@vendure-io/docs-provider 0.9.0 → 0.10.0

package/docs/mdx-testing.md

@@ -1,10 +1,10 @@
-# MDX Compilation Testing
+# Documentation Testing
 
-`@vendure-io/docs-provider` includes a testing module that allows you to validate MDX files **before publishing** your documentation package. This catches compilation errors early, reducing the feedback loop between publishing and discovering issues.
+`@vendure-io/docs-provider` includes a comprehensive testing module that validates your documentation **before publishing**. It catches errors early, reducing the feedback loop between publishing and discovering issues.
 
-## Why Test MDX Files?
+## Why Test Documentation?
 
-MDX compilation errors only surface when the documentation website attempts to render your pages. Without pre-publish testing, this creates a slow feedback loop:
+Documentation errors only surface when the website attempts to render your pages. Without pre-publish testing, this creates a slow feedback loop:
 
 1. Write MDX content
 2. Publish package to npm
@@ -15,9 +15,18 @@ MDX compilation errors only surface when the documentation website attempts to r
 
 The testing module lets you catch errors at step 1, before any publishing occurs.
 
+## What Gets Validated?
+
+The testing module runs four validators in sequence:
+
+1. **Frontmatter Validation** - Validates frontmatter against the DocPageMeta schema
+2. **Admonition Validation** - Checks for correct admonition syntax (bracket notation)
+3. **Code Language Validation** - Ensures code blocks use valid Shiki language identifiers
+4. **MDX Compilation** - Full AST parsing to catch syntax errors
+
 ## Installation
 
-The testing module is included in `@vendure-io/docs-provider`. No additional installation is required, but ensure you have the package installed:
+The testing module is included in `@vendure-io/docs-provider`. No additional installation is required:
 
 ```bash
 npm install @vendure-io/docs-provider
@@ -27,30 +36,30 @@ bun add @vendure-io/docs-provider
 
 ## Quick Start
 
-### Testing Your Manifest
+### Using runAllTests (Recommended)
 
-The simplest way to test all MDX files in your documentation package:
+The simplest way to validate all documentation in your package:
 
 ```typescript
-import { manifest } from './src/manifest'
-import { testManifestMdx, formatTestReport } from '@vendure-io/docs-provider/testing'
+// scripts/test-docs.ts
+import { manifest } from '../src/manifest'
+import { runAllTests } from '@vendure-io/docs-provider/testing'
 
-async function main() {
-  const report = await testManifestMdx(manifest)
+runAllTests(manifest)
+```
 
-  console.log(formatTestReport(report))
-  process.exit(report.failed > 0 ? 1 : 0)
-}
+That's it! This single function call:
 
-main()
-```
+- Runs all 4 validators in sequence
+- Prints a unified report to the console
+- Exits with code 1 if any errors are found
 
-Save this as `scripts/test-mdx.ts` and run it:
+Save this as `scripts/test-docs.ts` and run it:
 
 ```bash
-npx tsx scripts/test-mdx.ts
+npx tsx scripts/test-docs.ts
 # or with bun
-bun run scripts/test-mdx.ts
+bun run scripts/test-docs.ts
 ```
 
 ### Adding to package.json
@@ -60,25 +69,25 @@ Add a script to your `package.json` for easy execution:
 ```json
 {
   "scripts": {
-    "test:mdx": "tsx scripts/test-mdx.ts",
-    "prepublishOnly": "npm run test:mdx"
+    "test:docs": "tsx scripts/test-docs.ts",
+    "prepublishOnly": "npm run test:docs"
   }
 }
 ```
 
-The `prepublishOnly` script ensures MDX validation runs before every npm publish.
+The `prepublishOnly` script ensures validation runs before every npm publish.
 
-## API Reference
+## Unified Test Runner
 
-### testManifestMdx
+### runAllTests
 
-Tests all MDX files referenced in a manifest's navigation tree.
+Runs all documentation validators on a manifest.
 
 ```typescript
-async function testManifestMdx(
+async function runAllTests(
   manifest: DocsPackageManifest,
-  options?: TestManifestOptions,
-): Promise<MdxTestReport>
+  options?: TestRunnerOptions,
+): Promise<TestRunnerReport>
 ```
 
 **Parameters:**
@@ -88,6 +97,175 @@ async function testManifestMdx(
 
 **Options:**
 
+| Option          | Type                                      | Default | Description                                          |
+| --------------- | ----------------------------------------- | ------- | ---------------------------------------------------- |
+| `verbose`       | `boolean`                                 | `false` | Show details for all files, not just failures        |
+| `failFast`      | `boolean`                                 | `false` | Stop on first error in any validator                 |
+| `exit`          | `boolean`                                 | `true`  | Auto-exit process with appropriate code              |
+| `silent`        | `boolean`                                 | `false` | Suppress all console output                          |
+| `frontmatter`   | `boolean \| ValidateFrontmatterOptions`   | `true`  | Frontmatter validation options, or `false` to skip   |
+| `admonitions`   | `boolean \| ValidateAdmonitionsOptions`   | `true`  | Admonition validation options, or `false` to skip    |
+| `codeLanguages` | `boolean \| ValidateCodeLanguagesOptions` | `true`  | Code language validation options, or `false` to skip |
+| `mdx`           | `boolean \| TestManifestOptions`          | `true`  | MDX compilation options, or `false` to skip          |
+
+**Example with options:**
+
+```typescript
+import { manifest } from '../src/manifest'
+import { runAllTests } from '@vendure-io/docs-provider/testing'
+
+// Skip frontmatter validation, enable verbose output
+runAllTests(manifest, {
+  verbose: true,
+  frontmatter: false,
+})
+```
+
+**Example for programmatic use:**
+
+```typescript
+import { manifest } from '../src/manifest'
+import { runAllTests } from '@vendure-io/docs-provider/testing'
+
+async function main() {
+  const report = await runAllTests(manifest, {
+    exit: false, // Don't auto-exit, we want the report
+    silent: true, // Suppress default output
+  })
+
+  // Custom handling
+  if (!report.success) {
+    console.error(`Found ${report.totalErrors} errors`)
+    // Do something with report.frontmatter, report.admonitions, etc.
+  }
+}
+
+main()
+```
+
+**Example output:**
+
+```
+Documentation Test Report: my-plugin
+============================================================
+
+Summary
+------------------------------------------------------------
+  ✓ Frontmatter       OK           (25 files)
+  ✓ Admonitions       OK           (25 files)
+  ✗ Code Languages    2 errors     (142 blocks)
+  ✓ MDX Compilation   25/25 passed (25 files)
+
+Total errors: 2
+Total time: 156.32ms
+
+Details
+------------------------------------------------------------
+
+Code Language Validation:
+  Language 'env' - Use 'dotenv' instead of 'env'
+    - docs/configuration.mdx:15
+    - docs/deployment.mdx:42
+
+✗ 2 errors found.
+```
+
+### TestRunnerReport
+
+```typescript
+interface TestRunnerReport {
+  /** Package ID from the manifest */
+  packageId: string
+  /** Frontmatter validation report (if run) */
+  frontmatter?: FrontmatterValidationReport
+  /** Admonition validation report (if run) */
+  admonitions?: AdmonitionValidationReport
+  /** Code language validation report (if run) */
+  codeLanguages?: CodeLanguageValidationReport
+  /** MDX compilation report (if run) */
+  mdx?: MdxTestReport
+  /** Total number of errors across all validators */
+  totalErrors: number
+  /** Total time taken in milliseconds */
+  totalTime: number
+  /** Whether all validations passed */
+  success: boolean
+}
+```
+
+---
+
+## Individual Validators
+
+For fine-grained control, you can run validators individually.
+
+### Frontmatter Validation
+
+Validates that all MDX files have valid frontmatter according to the DocPageMeta schema.
+
+```typescript
+import { manifest } from './src/manifest'
+import { validateFrontmatter, formatFrontmatterReport } from '@vendure-io/docs-provider/testing'
+
+const report = validateFrontmatter(manifest)
+
+if (report.errors.length > 0) {
+  console.error(formatFrontmatterReport(report))
+  process.exit(1)
+}
+```
+
+**Options:**
+
+| Option     | Type       | Default     | Description                   |
+| ---------- | ---------- | ----------- | ----------------------------- |
+| `failFast` | `boolean`  | `false`     | Stop on first error           |
+| `onError`  | `Function` | `undefined` | Callback for each error found |
+
+**Example output:**
+
+```
+Frontmatter Validation Report: my-plugin
+==================================================
+
+Files scanned: 25
+Errors found: 2
+Total time: 12.45ms
+
+Errors:
+--------------------------------------------------
+
+  docs/getting-started.mdx
+    - title: Page title is required
+
+  docs/advanced.mdx
+    - lastModified: Invalid datetime
+
+Frontmatter requirements:
+  - title: Required, non-empty string
+  - description: Optional string
+  - keywords: Optional array of strings
+  - sidebarLabel: Optional string
+  - hidden: Optional boolean
+  - order: Optional integer
+  - lastModified: Optional ISO 8601 datetime
+```
+
+### MDX Compilation Testing
+
+#### testManifestMdx
+
+Tests all MDX files referenced in a manifest's navigation tree.
+
+```typescript
+async function testManifestMdx(
+  manifest: DocsPackageManifest,
+  options?: TestManifestOptions,
+): Promise<MdxTestReport>
+```
+
+**Options:**
+
 | Option          | Type            | Default         | Description           |
 | --------------- | --------------- | --------------- | --------------------- |
 | `failFast`      | `boolean`       | `false`         | Stop on first failure |
@@ -98,15 +276,19 @@ async function testManifestMdx(
 **Example with progress reporting:**
 
 ```typescript
+import { testManifestMdx, formatTestReport } from '@vendure-io/docs-provider/testing'
+
 const report = await testManifestMdx(manifest, {
   onProgress: (current, total, result) => {
     const status = result.success ? '✓' : '✗'
     console.log(`[${current}/${total}] ${status} ${result.filePath}`)
   },
 })
+
+console.log(formatTestReport(report))
 ```
 
-### compileMdx
+#### compileMdx
 
 Compiles a single MDX string and returns the result.
 
@@ -142,106 +324,136 @@ if (result.success) {
 }
 ```
 
-### formatTestReport
+#### createProgressReporter
 
-Formats a test report for console output.
+Creates a simple progress reporter for console output.
 
 ```typescript
-function formatTestReport(report: MdxTestReport, verbose?: boolean): string
+import { testManifestMdx, createProgressReporter } from '@vendure-io/docs-provider/testing'
+
+const report = await testManifestMdx(manifest, {
+  onProgress: createProgressReporter(),
+})
 ```
 
-**Parameters:**
+#### getDefaultRemarkPlugins / getDefaultRehypePlugins
 
-- `report` - The test report to format
-- `verbose` - If `true`, shows all results. If `false`, only shows failures.
+Get the default plugins used for MDX compilation. Useful for extending the plugin list.
 
-**Example output (verbose):**
+```typescript
+import {
+  testManifestMdx,
+  getDefaultRemarkPlugins,
+  getDefaultRehypePlugins,
+} from '@vendure-io/docs-provider/testing'
+import myCustomPlugin from './my-plugin'
 
+const report = await testManifestMdx(manifest, {
+  remarkPlugins: [...getDefaultRemarkPlugins(), myCustomPlugin],
+  rehypePlugins: getDefaultRehypePlugins(),
+})
 ```
-MDX Compilation Report: my-plugin
-==================================================
 
-Total files: 5
-Passed: 4 (80.0%)
-Failed: 1
-Total time: 156.32ms
+---
 
-Results:
---------------------------------------------------
-[PASS] docs/getting-started.mdx (45.21ms)
-[PASS] docs/installation.mdx (32.11ms)
-[PASS] docs/configuration.mdx (28.54ms)
-[PASS] docs/api-reference.mdx (35.67ms)
-[FAIL] docs/advanced.mdx (14.79ms)
-       Error (line 42:15): Unexpected token
+## Types Reference
 
-1 file(s) failed to compile.
-```
+### TestRunnerReport
 
-### createProgressReporter
+```typescript
+interface TestRunnerReport {
+  packageId: string
+  frontmatter?: FrontmatterValidationReport
+  admonitions?: AdmonitionValidationReport
+  codeLanguages?: CodeLanguageValidationReport
+  mdx?: MdxTestReport
+  totalErrors: number
+  totalTime: number
+  success: boolean
+}
+```
 
-Creates a simple progress reporter for console output.
+### FrontmatterError
 
 ```typescript
-function createProgressReporter(): (
-  current: number,
-  total: number,
-  result: MdxCompilationResult,
-) => void
+interface FrontmatterError {
+  file: string
+  absolutePath: string
+  issues: string[] // Zod validation issues
+}
 ```
 
-**Example:**
+### FrontmatterValidationReport
 
 ```typescript
-import { testManifestMdx, createProgressReporter } from '@vendure-io/docs-provider/testing'
-
-const report = await testManifestMdx(manifest, {
-  onProgress: createProgressReporter(),
-})
+interface FrontmatterValidationReport {
+  packageId: string
+  filesScanned: number
+  errors: FrontmatterError[]
+  totalTime: number
+}
 ```
 
-### getDefaultRemarkPlugins / getDefaultRehypePlugins
-
-Get the default plugins used for MDX compilation. Useful for extending the plugin list.
+### AdmonitionError
 
 ```typescript
-function getDefaultRemarkPlugins(): PluggableList
-function getDefaultRehypePlugins(): PluggableList
+interface AdmonitionError {
+  file: string
+  absolutePath: string
+  line: number
+  column: number
+  content: string
+  admonitionType: string
+  label: string
+  suggestion: string
+}
 ```
 
-**Example (extending plugins):**
+### AdmonitionValidationReport
 
 ```typescript
-import {
-  testManifestMdx,
-  getDefaultRemarkPlugins,
-  getDefaultRehypePlugins,
-} from '@vendure-io/docs-provider/testing'
-import myCustomPlugin from './my-plugin'
+interface AdmonitionValidationReport {
+  packageId: string
+  filesScanned: number
+  errors: AdmonitionError[]
+  totalTime: number
+}
+```
 
-const report = await testManifestMdx(manifest, {
-  remarkPlugins: [...getDefaultRemarkPlugins(), myCustomPlugin],
-  rehypePlugins: getDefaultRehypePlugins(),
-})
+### CodeLanguageError
+
+```typescript
+interface CodeLanguageError {
+  file: string
+  absolutePath: string
+  line: number
+  language: string
+  content: string
+  suggestion?: string
+}
 ```
 
-## Types
+### CodeLanguageValidationReport
+
+```typescript
+interface CodeLanguageValidationReport {
+  packageId: string
+  filesScanned: number
+  codeBlocksFound: number
+  errors: CodeLanguageError[]
+  totalTime: number
+}
+```
 
 ### MdxCompilationResult
 
 ```typescript
 interface MdxCompilationResult {
-  /** Path to the MDX file that was compiled */
   filePath: string
-  /** Whether compilation was successful */
   success: boolean
-  /** Error message if compilation failed */
   error?: string
-  /** Line number where the error occurred */
   line?: number
-  /** Column number where the error occurred */
   column?: number
-  /** Time taken to compile in milliseconds */
   compilationTime: number
 }
 ```
@@ -250,40 +462,74 @@ interface MdxCompilationResult {
 
 ```typescript
 interface MdxTestReport {
-  /** Package ID from the manifest */
   packageId: string
-  /** Total number of MDX files tested */
   totalFiles: number
-  /** Number of files that compiled successfully */
   passed: number
-  /** Number of files that failed to compile */
   failed: number
-  /** Individual results for each file */
   results: MdxCompilationResult[]
-  /** Total time taken in milliseconds */
   totalTime: number
 }
 ```
 
+---
+
 ## Integration with Vitest
 
-You can integrate MDX testing into your existing test suite:
+You can integrate documentation testing into your existing test suite:
 
 ```typescript
-// tests/mdx-compilation.test.ts
+// tests/docs-validation.test.ts
 import { describe, it, expect } from 'vitest'
 import { manifest } from '../src/manifest'
-import { testManifestMdx } from '@vendure-io/docs-provider/testing'
+import { runAllTests } from '@vendure-io/docs-provider/testing'
 
-describe('MDX Compilation', () => {
-  it('should compile all MDX files without errors', async () => {
-    const report = await testManifestMdx(manifest)
+describe('Documentation Validation', () => {
+  it('should pass all documentation tests', async () => {
+    const report = await runAllTests(manifest, {
+      exit: false,
+      silent: true,
+    })
 
-    expect(report.failed).toBe(0)
+    expect(report.success).toBe(true)
   }, 30000) // Increase timeout for large doc sets
 })
 ```
 
+Or test individual validators:
+
+```typescript
+import { describe, it, expect } from 'vitest'
+import { manifest } from '../src/manifest'
+import {
+  validateFrontmatter,
+  validateAdmonitions,
+  validateCodeLanguages,
+  testManifestMdx,
+} from '@vendure-io/docs-provider/testing'
+
+describe('Documentation Validation', () => {
+  it('should have valid frontmatter', () => {
+    const report = validateFrontmatter(manifest)
+    expect(report.errors).toHaveLength(0)
+  })
+
+  it('should use correct admonition syntax', () => {
+    const report = validateAdmonitions(manifest)
+    expect(report.errors).toHaveLength(0)
+  })
+
+  it('should use valid code languages', () => {
+    const report = validateCodeLanguages(manifest)
+    expect(report.errors).toHaveLength(0)
+  })
+
+  it('should compile all MDX files', async () => {
+    const report = await testManifestMdx(manifest)
+    expect(report.failed).toBe(0)
+  }, 30000)
+})
+```
+
 ## Supported MDX Features
 
 The testing module uses the same remark/rehype plugins as the Vendure documentation website:
@@ -295,7 +541,7 @@ The testing module uses the same remark/rehype plugins as the Vendure documentat
 This is an info callout.
 :::
 
-:::warning Custom Title
+:::warning[Custom Title]
 This is a warning with a custom title.
 :::
 ```
@@ -329,8 +575,148 @@ export function example() {
 </Callout>
 ````
 
+### Admonition Validation
+
+Validates that admonitions use the correct bracket syntax required by remark-directive v4+.
+
+```typescript
+import { manifest } from './src/manifest'
+import { validateAdmonitions, formatAdmonitionReport } from '@vendure-io/docs-provider/testing'
+
+const report = validateAdmonitions(manifest)
+
+if (report.errors.length > 0) {
+  console.error(formatAdmonitionReport(report))
+  process.exit(1)
+}
+```
+
+**Valid syntax:**
+
+```mdx
+:::warning[Deprecated]
+This feature is deprecated.
+:::
+```
+
+**Invalid syntax (caught by validator):**
+
+```mdx
+:::warning Deprecated
+This feature is deprecated.
+:::
+```
+
+### Code Language Validation
+
+Validates that code blocks use supported Shiki language identifiers.
+
+```typescript
+import { manifest } from './src/manifest'
+import { validateCodeLanguages, formatCodeLanguageReport } from '@vendure-io/docs-provider/testing'
+
+const report = validateCodeLanguages(manifest)
+
+if (report.errors.length > 0) {
+  console.error(formatCodeLanguageReport(report))
+  process.exit(1)
+}
+```
+
+**Options:**
+
+| Option                | Type                     | Default     | Description                       |
+| --------------------- | ------------------------ | ----------- | --------------------------------- |
+| `failFast`            | `boolean`                | `false`     | Stop on first error               |
+| `onError`             | `Function`               | `undefined` | Callback for each error found     |
+| `additionalAliases`   | `Record<string, string>` | `{}`        | Extra aliases to consider valid   |
+| `additionalLanguages` | `string[]`               | `[]`        | Extra languages to consider valid |
+
+#### Utility Functions
+
+**isValidCodeLanguage** - Check if a language identifier is valid:
+
+```typescript
+import { isValidCodeLanguage } from '@vendure-io/docs-provider/testing'
+
+isValidCodeLanguage('typescript') // true
+isValidCodeLanguage('ts') // true (alias)
+isValidCodeLanguage('env') // true (maps to 'dotenv')
+isValidCodeLanguage('foobar') // false
+```
+
+**getCanonicalLanguage** - Get the canonical Shiki language name:
+
+```typescript
+import { getCanonicalLanguage } from '@vendure-io/docs-provider/testing'
+
+getCanonicalLanguage('env') // 'dotenv'
+getCanonicalLanguage('text') // 'ini'
+getCanonicalLanguage('terminal') // 'shellscript'
+```
+
+#### Common Language Issues
+
+| Invalid     | Use Instead | Notes                        |
+| ----------- | ----------- | ---------------------------- |
+| `env`       | `dotenv`    | For `.env` files             |
+| `text`      | (none)      | Remove language or use `ini` |
+| `plaintext` | (none)      | Remove language or use `ini` |
+| `terminal`  | `bash`      | Or `shell`, `sh`, `zsh`      |
+
+#### Supported Languages
+
+The validator includes all 328 languages bundled with Shiki v3.22+. Common languages include:
+
+- **Web**: `html`, `css`, `javascript`, `typescript`, `jsx`, `tsx`, `json`, `yaml`
+- **Backend**: `python`, `ruby`, `php`, `java`, `go`, `rust`, `csharp`
+- **Shell**: `bash`, `shell`, `sh`, `zsh`, `powershell`
+- **Config**: `ini`, `toml`, `dotenv`, `nginx`, `docker`
+- **Data**: `sql`, `graphql`, `xml`, `csv`
+- **Markup**: `markdown`, `mdx`, `latex`
+
+See the full list at [shiki.style/languages](https://shiki.style/languages)
+
+---
+
 ## Common Errors
 
+### Missing or Invalid Frontmatter
+
+The frontmatter validator catches:
+
+- Missing `title` field (required)
+- Invalid `lastModified` format (must be ISO 8601)
+- Wrong types (e.g., `keywords` as string instead of array)
+
+```mdx
+---
+title: My Page Title
+description: Optional description
+keywords:
+  - keyword1
+  - keyword2
+---
+
+# Content
+```
+
+### Invalid Admonition Syntax
+
+```
+:::warning Deprecated
+```
+
+Must use bracket syntax:
+
+```
+:::warning[Deprecated]
+```
+
+### Unsupported Code Language
+
+Using `env` instead of `dotenv`, or `terminal` instead of `bash`.
+
 ### Unclosed JSX Tags
 
 ```
@@ -362,56 +748,46 @@ Check for:
 - Unescaped special characters (`<`, `>`, `{`, `}`)
 - JavaScript expressions not wrapped in curly braces
 
-### Missing Frontmatter
-
-While the MDX compiler doesn't require frontmatter, the docs website does. Ensure all files have at least a `title`:
-
-```mdx
----
-title: My Page Title
 ---
 
-# Content
-```
-
 ## Best Practices
 
-1. **Run tests before commits**: Add MDX testing to your pre-commit hooks
-2. **Use CI/CD**: Include MDX testing in your continuous integration pipeline
-3. **Test incrementally**: Use `failFast: true` during development for faster feedback
-4. **Verbose output for debugging**: Use `formatTestReport(report, true)` when troubleshooting
+1. **Use `runAllTests()` as your default** - It runs all validators and handles exit codes automatically
+2. **Run tests before commits** - Add documentation testing to your pre-commit hooks
+3. **Use CI/CD** - Include documentation testing in your continuous integration pipeline
+4. **Test incrementally** - Use `failFast: true` during development for faster feedback
+5. **Verbose output for debugging** - Use `verbose: true` when troubleshooting
 
-## Example: Complete Test Script
+---
 
-Here's a complete test script with all features:
+## Complete Examples
+
+### Minimal Test Script (Recommended)
 
 ```typescript
-// scripts/test-mdx.ts
+// scripts/test-docs.ts
 import { manifest } from '../src/manifest'
-import {
-  testManifestMdx,
-  formatTestReport,
-  createProgressReporter,
-} from '@vendure-io/docs-provider/testing'
+import { runAllTests } from '@vendure-io/docs-provider/testing'
 
-async function main() {
-  console.log('Testing MDX files...\n')
+runAllTests(manifest)
+```
 
-  const report = await testManifestMdx(manifest, {
-    onProgress: createProgressReporter(),
-  })
+Run with:
 
-  console.log('\n')
-  console.log(formatTestReport(report, process.argv.includes('--verbose')))
+```bash
+bun run scripts/test-docs.ts
+```
 
-  if (report.failed > 0) {
-    process.exit(1)
-  }
-}
+### With Options
 
-main().catch((error) => {
-  console.error('Test runner error:', error)
-  process.exit(1)
+```typescript
+// scripts/test-docs.ts
+import { manifest } from '../src/manifest'
+import { runAllTests } from '@vendure-io/docs-provider/testing'
+
+runAllTests(manifest, {
+  verbose: process.argv.includes('--verbose'),
+  failFast: process.argv.includes('--fail-fast'),
 })
 ```
 
@@ -419,8 +795,45 @@ Run with:
 
 ```bash
 # Standard output
-bun run scripts/test-mdx.ts
+bun run scripts/test-docs.ts
 
 # Verbose output showing all results
-bun run scripts/test-mdx.ts --verbose
+bun run scripts/test-docs.ts --verbose
+
+# Stop on first error
+bun run scripts/test-docs.ts --fail-fast
+```
+
+### Programmatic Usage
+
+```typescript
+// scripts/test-docs.ts
+import { manifest } from '../src/manifest'
+import { runAllTests, formatTestRunnerReport } from '@vendure-io/docs-provider/testing'
+
+async function main() {
+  const report = await runAllTests(manifest, {
+    exit: false,
+    silent: true,
+  })
+
+  // Custom output
+  console.log(formatTestRunnerReport(report, true))
+
+  // Custom error handling
+  if (!report.success) {
+    console.error(`\nFailed with ${report.totalErrors} errors`)
+
+    if (report.frontmatter?.errors.length) {
+      console.error(`- ${report.frontmatter.errors.length} frontmatter errors`)
+    }
+    if (report.mdx?.failed) {
+      console.error(`- ${report.mdx.failed} MDX compilation errors`)
+    }
+
+    process.exit(1)
+  }
+}
+
+main()
 ```