noblog 0.2.2 → 0.3.0-beta.1

package/README.md

@@ -1,208 +1,364 @@
 # 📝 Noblog
 
-> Transform your Notion content into beautiful static blogs with Astro
+> Generate Astro-compatible Markdown from your Notion database.
 
 [![npm version](https://badge.fury.io/js/noblog.svg)](https://badge.fury.io/js/noblog)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Noblog generates static content from Notion databases and provides a pluggable component/theme marketplace for bloggers. Inspired by [notablog](https://github.com/dragonman225/notablog) but more flexible thanks to the power of Astro.
+Noblog fetches published Notion pages, converts their blocks to Markdown, and writes a complete blog structure with Astro frontmatter. Each page lives once in a canonical store. Database posts also receive public symlinks, while child pages are referenced directly in the store. Custom output, store, and layout paths are supported.
 
-## 🆚 Why Noblog vs notion-to-md?
+The renderer includes logic derived from [notion-to-md](https://github.com/souvikinator/notion-to-md), with additional handling for nested formatting and blog generation.
 
-While Noblog uses [notion-to-md](https://github.com/souvikinator/notion-to-md) under the hood, it provides significant improvements for blog generation:
+## Requirements
 
-### 🔧 **Enhanced Markdown Generation**
+- Node.js 24 or newer
+- A Notion integration with read access to your database
+- A Notion database with `publish` and `date` properties
 
-**Problem with notion-to-md:**
-When dealing with complex formatting like nested styles (e.g., `{BLOCK1}` and `{BLOCK2}` both bold, but only `BLOCK1` also italic), notion-to-md generates malformed markdown:
+## Installation
 
-```markdown
-_**BLOCK1**_**BLOCK2** # Broken: unmerged tags
+```bash
+npm install -g noblog
 ```
 
-**Noblog's Solution:**
-
-- **Merged Tag Handling**: Properly merges nested formatting tags
-- **Clean Markdown Output**: Generates valid, renderer-friendly markdown
-- **Astro-Optimized**: Specifically designed for static site generation
-
-### 🏗️ **Complete Blog Structure**
+For use as a library in another package:
 
-**notion-to-md:** Just converts pages to markdown
+```bash
+npm install noblog
+```
 
-**Noblog:**
+## Configuration
 
-- ✅ Automatic file structure generation
-- ✅ Parent/child page organization
-- ✅ Astro-compatible frontmatter
-- ✅ Customizable save paths
-- ✅ Layout path customization
-- ✅ Theme marketplace ready
+Set the following environment variables in your shell or in a `.env` file in the directory where you run `noblog`:
 
-### ⚡ **One-Command Solution**
+```bash
+NOTION_API_SECRET="your-notion-api-secret"
+NOBLOG_DATABASE_ID="your-notion-database-id"
+```
 
-Instead of manually handling notion-to-md output and building your own file structure:
+The CLI loads `.env` automatically. Credentials are validated only when generating content, so `--help`, `--version`, and `config check` do not require them. Keep credentials in environment variables or `.env`; YAML configuration has no credential fields.
 
-```bash
-# With Noblog - everything handled automatically
-noblog --layout ../../layouts/BlogLayout.astro
+## Usage
 
-# With notion-to-md - you'd need to build everything yourself
+```text
+noblog [generate] [options] [SAVE_PATH] [SUB_PATH]
 ```
 
-> **Note:** Noblog includes improvements to notion-to-md's core parsing logic, fixing edge cases like unmerged formatting tags that can break markdown renderers.
+Running `noblog` without arguments immediately generates content with the default settings. It does not print help.
 
-## 🚀 Quick Start
+### Commands
 
-### Installation
+| Command | Description |
+| --- | --- |
+| `noblog` / `noblog generate` | Generate blog content |
+| `noblog config check --config PATH` | Validate YAML without credentials or network access |
+| `noblog help` | Show general help |
+| `noblog version` | Show the installed version |
 
-```bash
-npm install -g noblog
-```
+### Options
 
-### Basic Usage
+| Option | Description |
+| --- | --- |
+| `-v, --verbose` | Print diagnostics and error stack traces |
+| `--config PATH` | Load YAML configuration for generation |
+| `--layout PATH` | Set the layout path relative to `SAVE_PATH` |
+| `--store PATH` | Set the canonical Markdown store (default: `.noblog/pages/`) |
+| `-h, --help` | Show general or command-specific help |
+| `--version` | Show the installed version |
+| `--` | Treat all remaining inputs as positional paths |
 
-```bash
-noblog
-```
+### Positional arguments
 
-That's it! Noblog will fetch your Notion content and generate static markdown files.
+| Argument | Default | Description |
+| --- | --- | --- |
+| `SAVE_PATH` | `./src/pages/posts/` | Public view directory for database posts |
+| `SUB_PATH` | — | Deprecated compatibility argument; ignored with a warning |
 
-### 🛠️ Command-Line Options
-
-Noblog supports various command-line options to customize your blog generation:
+### Examples
 
 ```bash
-# Basic usage with default settings
+# Generate with all defaults
 noblog
 
-# Custom layout path
+# Equivalent explicit command
+noblog generate
+
+# Use a custom layout
 noblog --layout ../../layouts/BlogLayout.astro
 
-# Custom save and sub paths with layout
-noblog ./content/blog/ ./content/blog/subpages/ --layout ../../../layouts/CustomLayout.astro
+# Write database-post symlinks to ./content/blog/
+noblog ./content/blog/ --layout ../../layouts/CustomLayout.astro
 
-# Verbose output for debugging
-noblog -v
-# or
-noblog --verbose
-```
+# Store the real Markdown files in a custom canonical store
+noblog --store content/.noblog-pages
 
-**Available Options:**
+# Generate using YAML configuration
+noblog --config ./noblog.yaml
 
-- `--layout PATH` - Custom layout path relative to the posts directory (default: `../../layouts/MarkdownPostLayout.astro`)
-- `-v, --verbose` - Print more messages for debugging
-- `-h, --help` - Show help message
-- `--version` - Show version information
+# Validate configuration without credentials or Notion access
+noblog config check --config ./noblog.yaml
 
-**Positional Arguments:**
+# Print diagnostics
+noblog generate -v
 
-- `SAVE_PATH` - Directory to save main posts (default: `./src/pages/posts/`)
-- `SUB_PATH` - Subdirectory for nested content (default: `nob_children/`)
+# Show help for the generate command and its arguments
+noblog generate --help
+```
+
+## YAML configuration
 
-## 📋 Setup Guide
+Pass `--config PATH` to load a versioned YAML file. Configuration is explicit; Noblog does not search for a YAML file automatically. Generation resolves each output setting with this precedence:
 
-### 1. Create Your Notion Database
+```text
+built-in defaults < YAML configuration < explicit CLI flags and SAVE_PATH
+```
 
-Duplicate this [template database](https://www.notion.so/humbornjo/fa0faae85c504934a4a86cfa70302850?v=2abd1079ae134fbd8df2604765baa1df) and start writing your blog posts in Notion.
+For example, this config adapts output and frontmatter to a theme with a native date, custom Notion properties, and typed boolean/number fields:
+
+```yaml
+version: 1
+notion:
+  properties:
+    title: Name
+    publishedAt: Published
+    series: Series
+    sensitive: Sensitive
+    top: Top
+output:
+  savePath: src/pages/posts/
+  storePath: .noblog/pages/
+  layoutPath: ../../layouts/MarkdownPostLayout.astro
+  recursive: true
+frontmatter:
+  fields:
+    title:
+      from: title
+      required: true
+    timestamp:
+      from: publishedAt
+      as: date
+      required: true
+    series:
+      from: series
+      omitEmpty: true
+    sensitive:
+      from: sensitive
+      default: false
+    top:
+      from: top
+      default: 0
+    toc:
+      value: true
+```
 
-### 2. Configure Environment Variables
+The keys under `notion.properties` are logical source names; their values must exactly match Notion property names. Custom sources such as `series` must be declared there before frontmatter can reference them. Built-in sources need no declaration.
 
-Set up your Notion API credentials:
+Validate a file without Notion credentials, a Notion client, or network access:
 
 ```bash
-export NOBLOG_DATABASE_ID="your-notion-database-id"
-export NOTION_API_SECRET="your-notion-api-secret"
+noblog config check --config ./noblog.yaml
 ```
 
-### 3. Choose a Template
+The machine-readable JSON Schema is generated from the runtime Zod schema and shipped as `schema/config.json` (package export: `noblog/config.schema.json`). For a local installation, enable editor validation with:
 
-Pick an Astro template that supports Noblog. Here's a starter template: **[nobloger](https://github.com/humbornjo/nobloger)**
+```yaml
+# yaml-language-server: $schema=./node_modules/noblog/schema/config.json
+```
 
-### 4. Deploy Your Site
+The schema checks structure and native types. `config check` additionally enforces cross-field rules such as declared custom sources. Invalid YAML, unsupported versions, unknown keys, and invalid field definitions exit with status 2.
 
-Run `noblog` to generate your content, then deploy your Astro site to your preferred hosting platform.
+A standard Agent Skill is included at [`skills/noblog-config/SKILL.md`](skills/noblog-config/SKILL.md), with detailed references and Nobloger/ThoughtLite examples. Copy that directory into your agent's skills location or point the agent at it. The skill inspects the theme's actual collection schema, frontmatter consumers, and routes before generating YAML; it does not need or request Notion credentials.
 
-## 📁 File Structure
+## Output structure
 
-By default, Noblog organizes your content like this:
+With the default paths, Noblog names files by Notion page ID. All real Markdown files are flat in `.noblog/pages/`; database posts also have relative symlinks in `SAVE_PATH`:
 
-```
+```text
 project-root/
+├── .noblog/
+│   └── pages/
+│       ├── <database-page-id>.md  # real file
+│       └── <child-page-id>.md     # real file
 └── src/
     └── pages/
         └── posts/
-            ├── blog-post-1.md          # Direct database entries
-            ├── blog-post-2.md
-            └── nob_children/
-                ├── sub-post-1.md       # Referenced/child pages
-                ├── sub-post-2.md
-                └── sub-post-3.md
+            └── <database-page-id>.md  # relative symlink
 ```
 
-### Custom Save Paths
+Only database posts receive public views. Child pages live exclusively in the canonical store, and child links use `./<child-page-id>.md` so they resolve as siblings after Vite/Astro resolves symlinks to real paths. Child pages therefore stay out of `src/pages/posts/*.md` listings. Use `--store PATH` to change the canonical store independently of the public path.
 
-You can customize the output directories:
+**Routing migration:** Noblog no longer creates `/nob_children/` routes. Configure the theme to load and route child pages from `.noblog/pages/` (or the custom store). Existing nested view files are not deleted; remove or redirect them separately after updating the theme. Existing `SUB_PATH` arguments are ignored with a warning.
 
-```bash
-noblog <SAVE_PATH> <SUB_PATH>
+Generation is repeatable. Existing regular database-post view files with matching page IDs are replaced by links, and all canonical files are updated. Unrelated files are left unchanged; directories and other non-file outputs are not replaced.
 
-# Example:
-noblog ./content/blog/ ./content/blog/subpages/
-```
+The store and views must remain together when building or deploying. If generated content is committed, include both; otherwise regenerate before building. Creating the views requires filesystem symlink support. On Windows, this may require Developer Mode or elevated privileges.
 
-## 🎯 Markdown Output
+## Frontmatter
 
-Generated markdown files include Astro-compatible frontmatter:
+Without custom Notion or frontmatter mappings, generated files retain the legacy Astro-compatible frontmatter:
 
-```markdown
+```yaml
 ---
-layout: ../../layouts/MarkdownPostLayout.astro
+layout: ../../src/layouts/MarkdownPostLayout.astro
 title: "Your Blog Post Title"
-tags: ["tag1", "tag2", "tag3"]
-pubDate: 2024-01-24
-archived: false
 description: "Your post description here"
+tags: ["tag1", "tag2", "tag3"]
+pubDate: "2024-01-24"
+archived: "false"
 ---
+```
 
-# Your content starts here...
+When `frontmatter.fields` is present, it defines the complete output field set; legacy fields are not implicitly added. When only `notion.properties` is present, Noblog uses the declarative equivalent of the legacy fields (`layout`, `title`, `description`, `tags`, `pubDate`, and `archived`) with the configured property names.
+
+Every custom field has exactly one of:
+
+- `from`: read a built-in source or a key declared under `notion.properties`.
+- `value`: emit a static native string, finite number, boolean, null, array, or nested object.
+
+Built-in sources are `title`, `description`, `tags`, `publishedAt`, `archived`, `layoutPath`, `pageId`, and `isDatabasePage`. Without explicit property mappings, the conventional Notion names are `description`, `tags`, `date`, and `archived`; title comes from the last Notion title property.
+
+Field behavior can be refined with:
+
+| Directive | Behavior |
+| --- | --- |
+| `default` | Use this value when the source is empty |
+| `required` | Fail generation if the value remains empty after defaults; errors include the field and page ID |
+| `omitEmpty` | Omit an empty value after default/required processing |
+| `as: string` | Convert a string, number, or boolean to a string; date-like strings remain quoted |
+| `as: date` | Convert a valid date string or finite number to an unquoted YAML timestamp |
+
+Empty means `undefined`, `null`, `""`, or `[]`; `false` and `0` are preserved. Processing order is source/static value, default, required check, omission, then conversion.
+
+Notion values are normalized according to their actual property types. Text, select/status, dates, URLs, times, users, and unique IDs become strings; checkboxes and numeric properties remain native; multi-selects, people, relations, and files become string arrays. Formula results follow their own value type. Unsupported or missing properties become `undefined`, so defaults and omission rules apply. See the [property-type reference](skills/noblog-config/references/notion-property-types.md) for the complete table.
+
+This makes native theme schemas possible without stringly-typed frontmatter: `as: string` can produce `pubDate: "2024-01-24"`, while `as: date` produces `timestamp: 2024-01-24`, and booleans/numbers remain unquoted unless conversion is requested.
+
+Vite and Astro resolve Markdown symlinks to their real paths, so layout imports must be relative to the canonical store. The CLI interprets `--layout` relative to `SAVE_PATH`, then writes the equivalent store-relative path for every page.
+
+## Programmatic API
+
+The package exposes a reusable renderer and filesystem writer. The optional Notion adapter is configured explicitly; importing `noblog` has no environment or network side effects.
+
+```ts
+import {
+  createNotionSource,
+  renderPages,
+  resolveStoreLayoutPath,
+  writePages,
+} from "noblog";
+
+const notionApiSecret = process.env["NOTION_API_SECRET"];
+const databaseId = process.env["NOBLOG_DATABASE_ID"];
+if (!notionApiSecret || !databaseId) {
+  throw new Error("Notion credentials are required");
+}
+
+const notion = createNotionSource({ notionApiSecret, databaseId });
+
+const savePath = "./content/blog/";
+const storePath = ".noblog/pages/";
+const layoutPath = resolveStoreLayoutPath({
+  savePath,
+  storePath,
+  layoutPath: "../../layouts/MarkdownPostLayout.astro",
+});
+
+const posts = await notion.listPages();
+const pages = await renderPages(posts, notion, { layoutPath });
+
+await writePages(pages, { savePath, storePath });
 ```
 
-## 🔧 Requirements
+`renderPages` recursively collects child pages by default and returns complete Markdown documents with frontmatter. Set `recursive: false` to render only the supplied database pages. `writePages` writes every canonical file but returns only the public database-post view paths. When rendering for a canonical store, use `resolveStoreLayoutPath` as above so layout imports remain valid after symlink resolution.
+
+Use `renderPage` for a single page, or supply any implementation of the `NotionReader` interface instead of the built-in adapter. Lower-level helpers such as `renderRichText` and `renderFrontmatter` are exported as well. To customize a block, pass a `transformers` map in `RenderOptions`; return a string to override the default renderer or `undefined` to fall back to it. Each key narrows its handler's block payload to the corresponding Notion block type, so a `paragraph` transformer can access `block.paragraph` directly. The programmatic API does not load `.env`; load environment variables yourself or pass explicit configuration.
+
+The same declarative mapping is available programmatically, or you can provide a `FrontmatterAdapter` function for full control:
+
+```ts
+import { createFrontmatterAdapter, renderPages } from "noblog";
+
+const frontmatter = createFrontmatterAdapter({
+  title: { from: "title", required: true },
+  timestamp: { from: "publishedAt", as: "date" },
+  draft: { from: "archived", default: false },
+});
+
+const pages = await renderPages(posts, notion, {
+  layoutPath,
+  propertyMap: { publishedAt: "Published" },
+  frontmatter,
+});
+```
 
-**Any Astro template can be used with Noblog!** The only requirement is that your template includes a layout component.
+An adapter receives `{ page, metadata, layoutPath, isDatabasePage }` and returns a plain object. It can use the raw Notion page or normalized metadata, and root `undefined` values are omitted. Values follow the exported `FrontmatterValue` contract: strings, numbers, booleans, `null`, `Date`, and readonly arrays/objects containing those values. Arrays, null, dates, and class instances are rejected as adapter roots.
 
-This means you can use:
+## Exit codes
 
-- ✅ Any existing Astro blog template
-- ✅ Your custom Astro theme
-- ✅ Popular Astro themes from the community
-- ✅ The starter template: **[nobloger](https://github.com/humbornjo/nobloger)**
+| Code | Meaning |
+| --- | --- |
+| `0` | Help, version, successful validation, or successful generation |
+| `1` | Generation or unexpected runtime failure |
+| `2` | Invalid arguments, YAML, or configuration |
 
-### Layout Customization
+## Development
 
-By default, Noblog uses `../../layouts/MarkdownPostLayout.astro` as the layout path. You can customize this using the `--layout` option:
+Noblog uses [Stricli](https://bloomberg.github.io/stricli/) for typed command routing, [tsdown](https://tsdown.dev/) for CLI/library builds, and [Vitest](https://vitest.dev/) for tests.
 
 ```bash
-# Use a custom layout
-noblog --layout ../../layouts/BlogLayout.astro
+pnpm install
 
-# Use with custom save paths
-noblog ./content/blog/ ./content/blog/subpages/ --layout ../../../layouts/CustomLayout.astro
-```
+# Run the TypeScript CLI directly
+pnpm dev --help
+
+# Run unit tests and type checking
+pnpm test
+pnpm typecheck
 
-**Important:** The layout path is relative to the posts directory where your markdown files are saved. For example:
-- If your save path is `./src/pages/posts/`, the default layout path `../../layouts/MarkdownPostLayout.astro` points to `./src/layouts/MarkdownPostLayout.astro`
-- If you change the save path to `./content/blog/`, you might need `--layout ../../layouts/CustomLayout.astro` to point to `./content/layouts/CustomLayout.astro`
+# Regenerate the published JSON Schema (also runs during build)
+pnpm schema:generate
 
-Adjust the layout path based on where your Astro layout files are located relative to your posts directory.
+# Build and smoke-test the actual dist artifacts
+pnpm test:built
 
-## 🙏 Acknowledgments
+# Run the complete verification suite
+pnpm check
+```
+
+### Project structure
+
+```text
+src/
+├── cli.ts                    # Process entry
+├── main.ts                   # Dotenv loading and dependency composition
+├── app.ts                    # Routes, help/version, and exit handling
+├── config-file.ts            # YAML loading and validation
+├── commands/
+│   ├── config*.ts            # Offline configuration validation
+│   ├── generate.ts           # Declarative arguments and documentation
+│   └── generate.impl.ts      # Lazy-loaded command action
+├── workflows/
+│   └── generate.ts           # CLI-oriented generation orchestration
+└── lib/
+    ├── noblog/               # Reusable rendering and file-writing package
+    │   ├── config.ts         # Runtime YAML schema and types
+    │   ├── metadata.ts       # Notion property normalization
+    │   ├── frontmatter-fields.ts # Declarative frontmatter adapter
+    │   ├── pages.ts          # Page collection, caching, and recursion
+    │   ├── blocks.ts         # Notion block-tree loading and rendering
+    │   ├── rich-text.ts      # Rich-text annotations and links
+    │   ├── markdown.ts       # Markdown formatting primitives
+    │   ├── frontmatter.ts    # Legacy/custom frontmatter serialization
+    │   └── writer.ts         # Canonical store and database-post symlinks
+    └── notion/client.ts      # Explicit Notion SDK adapter
+schema/config.json            # Generated JSON Schema
+skills/noblog-config/         # Agent Skill, references, and examples
+```
 
-Built upon the excellent work of these projects:
+## Acknowledgments
 
-- [Notion API](https://developers.notion.com/reference/intro) - Official Notion API documentation
-- [Notion SDK](https://developers.notion.com/docs/getting-started) - Notion's official SDK
-- [notion-to-md](https://github.com/souvikinator/notion-to-md) - Notion to Markdown converter
-- [notablog](https://github.com/dragonman225/notablog) - Original inspiration
-- [astro-notion-blog](https://github.com/otoyo/astro-notion-blog) - Astro + Notion integration
+- [Notion API](https://developers.notion.com/reference/intro) and [Notion SDK](https://github.com/makenotion/notion-sdk-js)
+- [notion-to-md](https://github.com/souvikinator/notion-to-md)
+- [notablog](https://github.com/dragonman225/notablog)
+- [astro-notion-blog](https://github.com/otoyo/astro-notion-blog)
+- [nobloger](https://github.com/humbornjo/nobloger), a starter Astro template