@entropicwarrior/sdoc 0.1.14 → 0.1.17

package/README.md

@@ -15,7 +15,7 @@ Markdown has no formal structure — section boundaries are ambiguous, extractio
 
 ## What's in the Box
 
-**The format** — a formal specification (`lexica/specification.sdoc`) with EBNF grammar, plus a comprehensive authoring guide written as an AI agent skill document. Drop `lexica/sdoc-authoring.sdoc` into any AI agent's context and it can read and write SDOC immediately.
+**The format** — a formal specification (`lexica/specification.sdoc`) with EBNF grammar, plus a comprehensive authoring guide written as an AI agent skill document. Drop `docs/reference/sdoc-authoring.sdoc` into any AI agent's context and it can read and write SDOC immediately.
 
 **A zero-dependency JavaScript parser** — `src/sdoc.js` parses SDOC into a format-neutral AST. No runtime dependencies, works anywhere Node runs. Parsing and rendering are cleanly separated — build your own renderers on top.
 
@@ -44,7 +44,7 @@ Open any `.sdoc` file and click the preview icon in the editor title bar, or run
 node tools/build-slides.js deck.sdoc -o slides.html
 ```
 
-Each top-level scope becomes a slide. Set `type: slides` in `@meta`. See `lexica/slide-authoring.sdoc` for the full authoring guide.
+Each top-level scope becomes a slide. Set `type: slides` in `@meta`. See `docs/reference/slide-authoring.sdoc` for the full authoring guide.
 
 ### Export to PDF or HTML
 
@@ -63,6 +63,19 @@ python3 tools/serve_docs.py docs/
 
 Or use the **SDOC: Browse Documents** command from the VS Code Command Palette.
 
+## For AI Agents
+
+This repo provides an [`llms.txt`](https://raw.githubusercontent.com/entropicwarrior/sdoc/main/llms.txt) file at the repo root for AI agent discovery.
+
+Key resources for agents:
+
+| Resource | What it gives you |
+|---|---|
+| [`docs/reference/sdoc-authoring.sdoc`](https://raw.githubusercontent.com/entropicwarrior/sdoc/main/docs/reference/sdoc-authoring.sdoc) | Skill document — drop into context to read/write SDOC immediately |
+| [`lexica/specification.sdoc`](https://raw.githubusercontent.com/entropicwarrior/sdoc/main/lexica/specification.sdoc) | Formal spec with EBNF grammar |
+
+All `.sdoc` files are designed for progressive disclosure — read the `@about` scope first (~50 tokens), then scan headings, then load only the section you need.
+
 ## Format at a Glance
 
 ```sdoc
@@ -161,9 +174,10 @@ Per-folder `sdoc.config.json` or per-file `@meta` scope for custom CSS, headers,
 ## Learning the Format
 
 - `docs/guide/intro.sdoc` — what SDOC is and why
+- `docs/guide/why-sdoc.sdoc` — the case for SDOC over Markdown
 - `docs/tutorials/first-steps.sdoc` — write your first document
+- `docs/reference/sdoc-authoring.sdoc` — authoring guide with quick reference and common mistakes
 - `docs/reference/syntax.sdoc` — full syntax reference
-- `SDOC_GUIDE.md` — quick reference in Markdown (also used by AI tools)
 
 ## Contributing
 

package/docs/guide/intro.sdoc

@@ -0,0 +1,112 @@
+# Introduction to SDOC
+{
+    @meta {
+        sdoc-version: 0.1
+    }
+
+    # What is SDOC?
+    {
+        SDOC is a plain text format for writing structured documentation. It uses explicit scoping to define document structure, with optional braces for unambiguous nesting.
+
+        The simplest SDOC document is just a heading followed by content:
+
+        ```
+# My Document
+
+This is a paragraph.
+        ```
+
+        For richer structure, use braces to scope content explicitly:
+
+        ```
+# My Section
+{
+    Content goes here.
+}
+        ```
+
+        Scopes nest naturally:
+
+        ```
+# Outer
+{
+    # Inner
+    {
+        Nested content.
+    }
+}
+        ```
+
+        Or use braceless scopes for simpler documents:
+
+        ```
+# Outer
+{
+    # Section A
+    Content for section A.
+
+    # Section B
+    Content for section B.
+}
+        ```
+
+        Heading depth is determined by nesting, not by the number of `#` characters. A single `#` is all you need.
+    }
+
+    # Why SDOC?
+    {
+        # For Humans
+        {
+            SDOC reads like plain text. The braces are there, but they fade into the background the same way indentation does. You get:
+
+            {[.]
+                - **Unlimited nesting** with no ambiguity. Scope boundaries are explicit, not guessed from heading levels or indentation. A ten-level-deep legal document parses the same way as a three-section readme.
+                - **Familiar inline formatting.** Bold, italic, code, links, and lists work the way you expect from Markdown. There is almost nothing new to learn.
+                - **No rendering required.** The raw file is the document. You can read and edit it in any text editor, on any device, with no tooling at all.
+                - **Stable cross-references.** Tag any scope with `@id` and reference it with `@id` in text. Rename the heading, the reference still works.
+            }
+        }
+
+        # For AI Agents and Tools
+        {
+            This is where SDOC's design pays off most. Every property of the format was chosen to make programmatic consumption reliable and token-efficient.
+
+            **Deterministic parsing.** Two parsers, same input, same tree. Always. Markdown cannot guarantee this because section boundaries are inferred from heading levels, and different parsers make different choices about where a section ends. In SDOC, a scope starts with `{` and ends with the matching `}`. There is no ambiguity to resolve. This is also a security property: when every parser produces the same tree, there is no interpretation gap that could be exploited to inject or hide content in automated processing pipelines.
+
+            **Exact section extraction.** An agent that needs "the error handling section" calls `get_section("error-handling")` and gets exactly that scope back, with its children, every time. In Markdown, extracting a section means guessing: does it end at the next heading of equal depth? Lesser depth? What if heading levels are inconsistent? SDOC makes this a solved problem.
+
+            **Token-efficient navigation.** Agents do not need to load entire files. With progressive disclosure, the cost of a targeted lookup is roughly:
+
+            {[.]
+                - **Discovery:** list all files with summaries \~200 tokens for 50 files
+                - **Orientation:** list section headings of one file \~50-100 tokens
+                - **Detail:** load one section \~200-1000 tokens
+            }
+
+            Total: roughly 750 tokens for a precise answer. Loading the whole file in Markdown would cost 5,000+ tokens and pollute the context window with irrelevant material.
+
+            **AI agents can write it reliably.** Brace matching is unambiguous to produce. A template that says "fill in the content between the braces" is straightforward. In Markdown, agents must track heading levels carefully, and a single `##` where `###` was needed silently restructures the entire document.
+
+            **Structured metadata in the same format.** The `@meta` scope uses the same SDOC syntax as everything else. No YAML frontmatter with different escaping rules parsed by different tools. Metadata is just another scope.
+        }
+    }
+
+    # Core Concepts
+    {
+        # Scopes
+        {
+            A scope is the fundamental building block. It has a heading line starting with `#`, an optional `@id` tag, and a body block in braces:
+
+            ```
+# Section Title @section-id
+{
+    Body text here.
+}
+            ```
+
+            Scopes nest to any depth. Inline formatting (`*bold*`, `` `code` ``, `[links](url)`, `@references`, `$math$`) works the same way as Markdown inside any scope.
+
+            See `reference/syntax.sdoc` for the full list of features, or follow `tutorials/first-steps.sdoc` to try it hands-on.
+        }
+    }
+}

package/docs/guide/notion-sync.sdoc

@@ -0,0 +1,177 @@
+# Notion Sync
+{
+    @meta {
+        sdoc-version: 0.1
+    }
+
+    SDOC files can be synced to Notion as native blocks -- headings, paragraphs, lists, tables, code, and images all render as real Notion content. The repo is the source of truth; Notion is a read-only mirror that stays up to date automatically via CI.
+
+    # How It Works
+    {
+        {[#]
+            - An SDOC file declares its Notion page ID in its `@meta` scope
+            - The sync tool parses the file and converts the AST to Notion API block objects
+            - It clears the target page and replaces the content with the rendered blocks
+            - A callout banner is added at the top: "Auto-synced from path/file.sdoc — do not edit in Notion"
+        }
+
+        Top-level SDOC scopes render as **toggle headings** in Notion, matching the collapsible behaviour of the HTML preview. Deeper scopes render as flat headings due to Notion API nesting limits.
+    }
+
+    # Setup
+    {
+        # 1. Create a Notion Integration
+        {
+            {[#]
+                - Go to [https://www.notion.so/profile/integrations](https://www.notion.so/profile/integrations)
+                - Click **New integration**
+                - Give it a name (e.g. "SDOC Sync") and select your workspace
+                - Copy the **Internal Integration Secret** (starts with `secret_`)
+            }
+        }
+
+        # 2. Share Pages with the Integration
+        {
+            For each Notion page you want to sync to:
+
+            {[#]
+                - Open the page in Notion
+                - Click the **...** menu (top right) and select **Connect to**
+                - Find and select your integration
+            }
+
+            Copy the page ID from the URL. It is the 32-character hex string at the end:
+
+            ```
+https://www.notion.so/My-Page-3135217a0d6380acbcecfed64316ec91
+                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+                                  this is the page ID
+            ```
+        }
+
+        # 3. Add the Page ID to Your SDOC File
+        {
+            Add a `notion-page` property to the file's `@meta` scope:
+
+            ```sdoc
+# My Document @meta
+{
+    type: doc
+    notion-page: 3135217a0d6380acbcecfed64316ec91
+}
+            ```
+
+            Only files with `notion-page` in their meta will be synced. Files without it are ignored.
+        }
+
+        # 4. Set the Notion Token
+        {
+            The token is **never committed to the repo**. Set it as an environment variable:
+
+            ```bash
+export NOTION_TOKEN=secret_abc123...
+            ```
+
+            For CI, add it as a repository secret (see @ci-setup below).
+        }
+    }
+
+    # Running Manually
+    {
+        Sync a single file:
+
+        ```bash
+node tools/sync-notion.js path/to/file.sdoc --verbose
+        ```
+
+        Scan a directory and sync all files that have `notion-page` in their meta:
+
+        ```bash
+node tools/sync-notion.js docs/ --verbose
+        ```
+
+        Dry run (renders blocks to stdout without calling the Notion API):
+
+        ```bash
+node tools/sync-notion.js docs/ --dry-run
+        ```
+
+        Override the page ID from the command line (useful for testing):
+
+        ```bash
+node tools/sync-notion.js file.sdoc --page-id abc123def456 --verbose
+        ```
+    }
+
+    # CI Setup @ci-setup
+    {
+        This section describes a workflow to add to **your own repository** that uses SDOC (the sdoc repo itself does not include this workflow). A GitHub Actions workflow can sync automatically on every push. Add the Notion token as a repository secret:
+
+        {[#]
+            - Go to your repo's **Settings > Secrets and variables > Actions**
+            - Click **New repository secret**
+            - Name: `NOTION_TOKEN`, Value: your integration secret
+        }
+
+        Then add a workflow file at `.github/workflows/sync-notion.yml`:
+
+        ```yaml
+name: Sync SDOC to Notion
+
+on:
+  push:
+    branches: [main, develop]
+    paths:
+      - '**.sdoc'
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+
+      - name: Sync to Notion
+        env:
+          NOTION_TOKEN: ${{ secrets.NOTION_TOKEN }}
+        run: node tools/sync-notion.js . --verbose
+        ```
+
+        This triggers only when `.sdoc` files change on push to `main` or `develop`.
+    }
+
+    # What Gets Rendered
+    {
+        {[table]
+            SDOC Element | Notion Block
+            Scopes (depth 1) | Toggle heading (collapsible)
+            Scopes (depth 2+) | Flat headings with content as siblings
+            Paragraphs | Paragraph blocks
+            Bullet lists | Bulleted list items
+            Numbered lists | Numbered list items
+            Task lists | To-do items (checked/unchecked)
+            Tables | Table blocks with rows
+            Code blocks | Code blocks (with language highlighting)
+            Blockquotes | Quote blocks
+            Horizontal rules | Divider blocks
+            Images (absolute URLs) | Image blocks
+            Images (relative paths) | Skipped (not supported by Notion)
+            Links | Rich text with link
+            Bold, italic, strikethrough, code | Rich text annotations
+            Refs (`@id`) | Literal text
+        }
+    }
+
+    # Limitations
+    {
+        {[.]
+            - **One-way sync only.** The repo is the source of truth. Edits made in Notion are overwritten on the next sync.
+            - **Images require absolute URLs.** Notion cannot fetch images from relative file paths. Use full `https://` URLs for images that should appear in Notion.
+            - **Nesting depth.** Notion's API limits block nesting to 2 levels. Only the top-level scope renders as a toggle heading; deeper scopes render as flat headings.
+            - **Refs are literal.** SDOC `@references` render as plain text in Notion since there is no equivalent for internal document cross-references.
+        }
+    }
+}

package/docs/guide/setup.sdoc

@@ -0,0 +1,112 @@
+# Setup
+{
+    @meta {
+        sdoc-version: 0.1
+    }
+
+    # Requirements
+    {
+        {[.]
+            - [Visual Studio Code](https://code.visualstudio.com/) (v1.95 or later)
+            - The SDOC extension (`.vsix` file)
+        }
+    }
+
+    # Installing the Extension
+    {
+        Build the extension from source:
+
+        ```bash
+npm install
+npm run package
+        ```
+
+        This produces a `.vsix` file in the `dist/` folder. Install it:
+
+        ```bash
+code --install-extension dist/sdoc-*.vsix
+        ```
+
+        Or install from within VS Code: open the Command Palette, run "Extensions: Install from VSIX...", and select the `.vsix` file.
+    }
+
+    # Previewing Documents
+    {
+        Once installed, open any `.sdoc` file and use one of:
+
+        {[.]
+            - **SDOC: Open Preview** to preview in the current pane
+            - **SDOC: Open Preview to the Side** for a side-by-side view
+        }
+
+        The preview updates live as you edit.
+    }
+
+    # Interactive Preview
+    {
+        The preview panel supports interactive features:
+
+        {[.]
+            - **Click-to-navigate**: click any element in the preview to jump to its source line in the editor
+            - **Collapsible scopes**: hover over a heading with children to reveal a toggle triangle. Click it to collapse or expand the section. Collapse state persists across refreshes.
+            - **Links**: clicking a hyperlink in the preview opens it in your default browser
+        }
+    }
+
+    # Formatting Documents
+    {
+        Use **Format Document** (Shift+Option+F on macOS, Shift+Alt+F on Windows/Linux) to auto-indent the current `.sdoc` file based on brace depth. The formatter respects your VS Code tab size and spaces/tabs preference.
+
+        Formatting is purely cosmetic -- it adjusts indentation without changing document structure. Code blocks are left untouched.
+    }
+
+    # Export and Print
+    {
+        Three export commands are available from the Command Palette or the editor title bar when an `.sdoc` file is open:
+
+        {[.]
+            - **SDOC: Export HTML** -- saves a standalone `.html` file with all styles embedded
+            - **SDOC: Export PDF** -- exports the document as an A4 PDF via headless Chrome
+            - **SDOC: Open in Browser** -- opens the document in the system browser
+        }
+
+        HTML and browser outputs support collapsible scope toggles. PDF export requires Chrome or Chromium installed on the system. For command-line export, see `tools/build-doc.js` in the [CLI reference](../reference/cli.sdoc).
+    }
+
+    # Configuration
+    {
+        Place a `sdoc.config.json` in your project root to set defaults for all documents:
+
+        ```json
+{
+    "style": "path/to/custom.css",
+    "header": "My Project",
+    "footer": "Footer text"
+}
+        ```
+
+        {[.]
+            - `style` replaces the default stylesheet (path relative to the config file)
+            - `styleAppend` adds CSS after the base stylesheet
+            - `header` and `footer` set page-level text with inline formatting
+        }
+
+        Configs are hierarchical: place `sdoc.config.json` in subfolders to override parent settings for documents in that folder.
+
+        See `examples/sdoc.config.json` for a sample config and `examples/sdoc.template.css` for the default theme.
+    }
+
+    # Document Server
+    {
+        Browse all `.sdoc` files in the project with a local HTTP server:
+
+        ```bash
+python3 tools/serve_docs.py                    # serve current dir on :4070
+python3 tools/serve_docs.py docs/ -p 8080      # serve docs/ on :8080
+        ```
+
+        Or use the **SDOC: Browse Documents** command from the VSCode Command Palette. It prompts you to serve from the entire workspace or choose a specific subfolder.
+
+        The viewer opens in your browser with a sidebar, search, and split-pane comparison. Edits to `.sdoc` files are reflected on browser refresh.
+    }
+}

package/docs/index.sdoc

@@ -0,0 +1,39 @@
+# SDOC Documentation
+{
+    @meta {
+        sdoc-version: 0.1
+    }
+
+    # Overview
+    {
+        SDOC ("Simple/Smart Documentation") is a plain text documentation format with explicit scoping. Unlike Markdown, structure is never inferred — braces make scope boundaries explicit, which means tools and AI agents can extract exactly the section they need, every time, with no heuristics. Inline formatting is borrowed from Markdown, so there is almost nothing new to learn.
+
+        Use the sidebar to navigate. Shift+Click two docs to compare them side by side.
+    }
+
+    # Quick Links
+    {
+        {[.]
+            - Guide: Introduction — `guide/intro.sdoc`
+            - Guide: Why SDOC — `guide/why-sdoc.sdoc`
+            - Guide: Setup — `guide/setup.sdoc`
+            - Guide: Notion Sync — `guide/notion-sync.sdoc`
+            - Tutorial: First Steps — `tutorials/first-steps.sdoc`
+            - Reference: SDOC Authoring — `reference/sdoc-authoring.sdoc`
+            - Reference: Slide Authoring — `reference/slide-authoring.sdoc`
+            - Reference: Syntax — `reference/syntax.sdoc`
+            - Reference: API — `reference/api.sdoc`
+            - Reference: CLI — `reference/cli.sdoc`
+        }
+    }
+
+    # Sections
+    {[.]
+        - Guide
+        { Getting started with SDOC: what it is, why it exists, and how to set it up. }
+        - Reference
+        { Authoring guides, syntax details, the JavaScript API, and CLI tools. }
+        - Tutorials
+        { Step-by-step walkthroughs for common tasks. }
+    }
+}