flex-md 1.1.0 → 2.0.0

package/README.md

@@ -1,6 +1,10 @@
 # flex-md
 
-Parse and stringify **FlexMD Frames**: semi-structured Markdown that is easy for humans + deterministic for machines.
+Parse and stringify **FlexMD**: semi-structured Markdown with three powerful layers:
+
+- **Layer A**: FlexMD Frames - reliable round-trip Markdown ↔ JSON
+- **Layer B**: Output Format Spec (OFS) - LLM-friendly Markdown contracts
+- **Layer C**: Detection & Extraction - find FlexMD in arbitrary text
 
 ## Install
 
@@ -8,7 +12,11 @@ Parse and stringify **FlexMD Frames**: semi-structured Markdown that is easy for
 npm i flex-md
 ```
 
-## Usage
+## Quick Start
+
+### Layer A: FlexMD Frames
+
+Parse and stringify semi-structured Markdown with frames, metadata, and payloads:
 
 ```javascript
 import { parseFlexMd, stringifyFlexMd } from "flex-md";
@@ -25,20 +33,310 @@ Hello
 
 const doc = parseFlexMd(md);
 console.log(doc.frames[0]?.type); // "message"
+console.log(doc.frames[0]?.meta?.tags); // ["auth", "login"]
 
 const back = stringifyFlexMd(doc, { skipEmpty: true });
 console.log(back);
+```
 
-// New in v1.1.0: Validation
+**Validation:**
+
+```javascript
 import { validateFlexMd } from "flex-md";
+
 const result = validateFlexMd(md);
 if (!result.valid) {
   console.log("Validation errors:", result.errors);
 }
 ```
 
-### Notes
-- Frames start with: `[[type key=value ...]]`
-- Metadata lines: `@key: value`
-- Payload binding: `@payload:name: X` then the next fenced block is payload X
-- **Validation**: `validateFlexMd` checks for unterminated frames, dangling payload declarations, and unterminated code fences.
+**Datatype Handling:**
+
+```javascript
+// Infer types automatically
+const doc = parseFlexMd(md, { metaTypeMode: "infer" });
+// @priority: 5 → number
+// @enabled: true → boolean
+// @value: null → null
+
+// Or use a schema
+const doc2 = parseFlexMd(md, {
+  metaTypeMode: "schema",
+  metaSchema: {
+    priority: "number",
+    enabled: "boolean"
+  }
+});
+```
+
+---
+
+### Layer B: Output Format Spec (OFS)
+
+Create LLM-friendly Markdown contracts and extract structured data:
+
+```javascript
+import {
+  parseOutputFormatSpec,
+  stringifyOutputFormatSpec,
+  enrichInstructions,
+  validateOutput,
+  extractOutput
+} from "flex-md";
+
+// Define an output format
+const spec = {
+  descriptorType: "output_format_spec",
+  format: "markdown",
+  sectionOrderMatters: false,
+  sections: [
+    { name: "Short answer", kind: "prose" },
+    { name: "Reasoning", kind: "ordered_list" },
+    { name: "Assumptions", kind: "list" }
+  ],
+  tablesOptional: true,
+  tables: [],
+  emptySectionValue: "None"
+};
+
+// Generate minimal LLM instructions
+const instructions = enrichInstructions(spec);
+console.log(instructions);
+// Output:
+// Rules:
+// - If a section is empty, write `None`.
+// - Ordered-list sections must use numbered items (nested allowed).
+// - List sections must use '-' bullets (nested allowed).
+
+// Stringify for LLM prompt
+const ofsMarkdown = stringifyOutputFormatSpec(spec);
+
+// Later, validate LLM response
+const llmResponse = `
+## Short answer
+The answer is 42.
+
+## Reasoning
+1. First, we analyze the question
+2. Then we compute the result
+
+## Assumptions
+- The universe is deterministic
+- We have sufficient compute
+`;
+
+const validation = validateOutput(llmResponse, spec);
+console.log(validation.ok); // true
+
+// Extract structured data
+const extracted = extractOutput(llmResponse, spec);
+console.log(extracted.sectionsByName["Short answer"].md);
+// "The answer is 42."
+
+console.log(extracted.sectionsByName["Reasoning"].list);
+// { kind: "list", ordered: true, items: [...] }
+```
+
+---
+
+### Outline & Tree Building
+
+Build nested heading trees from any Markdown:
+
+```javascript
+import { buildOutline, renderOutline } from "flex-md";
+
+const md = `
+# Introduction
+Some intro text
+
+## Background
+More details
+
+### History
+Even more details
+
+## Methodology
+Our approach
+`;
+
+const outline = buildOutline(md);
+console.log(outline.nodes[0].title); // "Introduction"
+console.log(outline.nodes[0].children[0].title); // "Background"
+console.log(outline.nodes[0].children[0].children[0].title); // "History"
+
+// Render back to Markdown
+const rendered = renderOutline(outline);
+```
+
+---
+
+### List & Table Parsing
+
+Parse nested lists and tables:
+
+```javascript
+import { parseList, parsePipeTable, extractAllTables } from "flex-md";
+
+// Parse nested lists
+const listMd = `
+- Item 1
+  - Nested 1.1
+  - Nested 1.2
+- Item 2
+`;
+
+const list = parseList(listMd);
+console.log(list.items[0].children.length); // 2
+
+// Parse ordered lists
+const orderedMd = `
+1. First step
+   1. Sub-step 1.1
+   2. Sub-step 1.2
+2. Second step
+`;
+
+const orderedList = parseList(orderedMd);
+console.log(orderedList.ordered); // true
+
+// Parse tables
+const tableMd = `
+| Name | Age |
+|------|-----|
+| Alice | 30 |
+| Bob | 25 |
+`;
+
+const table = parsePipeTable(tableMd);
+console.log(table.columns); // ["Name", "Age"]
+console.log(table.rows); // [["Alice", "30"], ["Bob", "25"]]
+
+// Extract all tables from a document
+const allTables = extractAllTables(documentMd);
+```
+
+---
+
+### Layer C: Detection & Extraction
+
+Find and parse FlexMD from arbitrary text:
+
+```javascript
+import { detectObjects, parseAny } from "flex-md";
+
+const mixedText = `
+Some random text here.
+
+\`\`\`flexmd
+[[message role=user]]
+Hello from FlexMD!
+\`\`\`
+
+More text.
+
+\`\`\`json
+{
+  "frames": [
+    {"type": "message", "body_md": "JSON FlexDocument"}
+  ]
+}
+\`\`\`
+`;
+
+// Detect all objects
+const detected = detectObjects(mixedText);
+console.log(detected[0].kind); // "flexmd_fence"
+console.log(detected[0].confidence); // 1.0
+console.log(detected[1].kind); // "flexdoc_json_fence"
+console.log(detected[1].confidence); // 0.9
+
+// Parse everything
+const result = parseAny(mixedText);
+console.log(result.flexDocs.length); // 2
+console.log(result.remainder); // "Some random text here.\n\nMore text."
+```
+
+---
+
+## Features
+
+- ✅ **Layer A**: FlexMD Frames with metadata and payloads
+- ✅ **Layer B**: Output Format Spec for LLM contracts
+- ✅ **Layer C**: Multi-tier detection from arbitrary text
+- ✅ **Nested heading trees**: Build and render outline structures
+- ✅ **List parsing**: Support for nested ordered/unordered lists
+- ✅ **Table parsing**: GFM pipe tables with ordered table support
+- ✅ **Datatype handling**: Infer or schema-based type conversion
+- ✅ **Validation**: Validate FlexMD syntax and OFS compliance
+- ✅ **TypeScript**: Full type definitions included
+
+## Documentation
+
+- [SPEC.md](./SPEC.md) - Complete specification with all details
+- [Examples](./examples/) - More usage examples (coming soon)
+
+## Use Cases
+
+### LLM Prompting & Response Parsing
+
+```javascript
+// 1. Define what you want from the LLM
+const spec = {
+  descriptorType: "output_format_spec",
+  format: "markdown",
+  sectionOrderMatters: false,
+  sections: [
+    { name: "Analysis", kind: "prose" },
+    { name: "Steps", kind: "ordered_list" },
+    { name: "Risks", kind: "list" }
+  ],
+  tablesOptional: true,
+  tables: [],
+  emptySectionValue: "None"
+};
+
+// 2. Generate instructions
+const instructions = enrichInstructions(spec);
+
+// 3. Send to LLM with your prompt + instructions
+
+// 4. Validate and extract response
+const validation = validateOutput(llmResponse, spec);
+const data = extractOutput(llmResponse, spec);
+```
+
+### Structured Conversation Storage
+
+```javascript
+const conversation = parseFlexMd(`
+[[message role=user id=msg1]]
+What is the capital of France?
+
+[[message role=assistant id=msg2]]
+The capital of France is Paris.
+`);
+
+// Store as JSON, query, manipulate, then stringify back
+const md = stringifyFlexMd(conversation);
+```
+
+### Markdown Documentation Processing
+
+```javascript
+// Build outline from docs
+const outline = buildOutline(documentationMd);
+
+// Extract specific sections
+const extracted = extractOutput(documentationMd, {
+  sections: [
+    { name: "Installation", kind: "prose" },
+    { name: "API Reference", kind: "list" }
+  ],
+  // ... rest of spec
+});
+```
+
+## License
+
+MIT