flex-md 4.1.0 → 4.2.0

package/README.md

@@ -7,6 +7,7 @@ Flex-MD is a TypeScript library for building and enforcing **Markdown Output Con
 - 📏 **System Parts Protocol**: Standardized size hints that guide LLMs AND enable token prediction
 - 🧠 **Smart Toolbox**: Cognitive cost analysis, confidence scoring, and improvement detection
 - 🔧 **Auto-Fix**: Automatically improve specs with one command
+- 🔄 **Markdown to JSON**: Convert sectioned MD to structured objects with camel-casing (v4.1)
 
 ## Key Features
 
@@ -25,6 +26,7 @@ Flex-MD is a TypeScript library for building and enforcing **Markdown Output Con
 - **Confidence Scoring**: Know how accurate your token estimates will be
 - **Improvement Detection**: Find issues and get actionable suggestions
 - **Auto-Fix**: Apply improvements automatically
+- **Markdown to JSON**: Transform sectioned Markdown into structured JSON with camel-cased keys
 
 ## Installation
 
@@ -432,6 +434,171 @@ const v3 = `
 - `formatImprovementReport(analysis)` - Format improvements
 - `formatSmartReport(analysis)` - Format complete analysis
 
+## Markdown to JSON Transformation (v4.1)
+
+Flex-MD includes a robust utility to convert sectioned Markdown (headings and their respective bodies) into a standard JavaScript object/JSON. This is particularly useful for extracting structured data from LLM responses without needing a complex schema.
+
+### Features:
+- **Automatic Camel-Casing**: Headings like `### Short Answer` or `===Next Steps` are automatically converted to valid camelCase keys (`shortAnswer`, `nextSteps`).
+- **Robust Newline Handling**: Automatically handles both actual newlines and literal `\\n` escape sequences often found in LLM outputs.
+- **Support for All Heading Types**: Works with standard `###` headings and `===key` alternative delimiters.
+
+### Usage:
+
+```typescript
+import { markdownToJson } from 'flex-md';
+
+const md = `
+### Short Answer
+The asset is a server named server1 with private IP 192.168.1.1.
+
+### Next Steps
+1. Document in CMDB
+2. Perform security check
+`;
+
+const data = markdownToJson(md);
+
+console.log(data.shortAnswer); 
+// "The asset is a server named server1 with private IP 192.168.1.1."
+
+console.log(data.nextSteps); 
+// "1. Document in CMDB\\n2. Perform security check"
+```
+
+## NX Flex-MD: Schema-Driven Extraction (Advanced)
+
+For more complex scenarios where you need to enforce a specific JSON schema, handle fuzzy matching of headings, or apply automatic fixes to the data, Flex-MD exposes the **NX Flex-MD** toolset (powered by `nx-md-parser`).
+
+### Feature Highlights:
+- **Schema Validation**: Define the exact structure and types (`string`, `number`, `boolean`, `array`, `object`) you expect.
+- **Intelligent Heading Matching**: Matches headings even if they aren't an exact match (e.g., "Summary" vs "Exec Summary").
+- **Automatic Type Conversion**: Converts Markdown strings to numbers or booleans based on your schema.
+- **Auto-Fixing**: Automatically corrects common formatting issues to match the schema.
+
+### Usage:
+
+```typescript
+import { JSONTransformer, Schema } from 'flex-md';
+
+// 1. Define a schema
+const schema = Schema.object({
+  status: Schema.string(),
+  score: Schema.number(),
+  isVerified: Schema.boolean(),
+  tags: Schema.array(Schema.string())
+});
+
+// 2. Create the transformer
+const transformer = new JSONTransformer(schema);
+
+// 3. Transform complex Markdown
+const md = `
+### Status
+Active and operational
+
+### Score
+95.5
+
+### Verified
+Yes
+
+### Tags
+- production
+- critical
+`;
+
+const { result, status, errors } = transformer.transformMarkdown(md);
+
+if (status === 'validated' || status === 'fixed') {
+  console.log(result);
+  /*
+  {
+    status: "Active and operational",
+    score: 95.5,
+    isVerified: true,
+    tags: ["production", "critical"]
+  }
+  */
+}
+```
+
+## Integrated OFS Transformation (The "Best of Both Worlds")
+
+Flex-MD allows you to use its native **Output Format Spec (OFS)** (the markdown-style description) as the source of truth for **NX-MD-Parser's** structured extraction. This combines the simplicity of describing output with markdown and the power of schema-driven JSON transformation.
+
+### Usage:
+
+```typescript
+import { parseOutputFormatSpec, transformWithOfs } from 'flex-md';
+
+// 1. Describe your output naturally
+const spec = parseOutputFormatSpec(`
+## Output format
+- Executive Summary — text (required)
+- Key Findings — ordered list (required)
+- Technical Specs — table (optional)
+  Columns: Component, Version, Notes
+`);
+
+// 2. Transfrom Markdown output using the spec
+const md = `
+### Summary
+The system is fully operational with 99.9% uptime.
+
+### Findings
+1. Scalability improved by 40%
+2. Memory leak in cache module fixed
+`;
+
+const { result, status } = transformWithOfs(md, spec);
+
+if (status === 'validated' || status === 'fixed') {
+  console.log(result['Executive Summary']); // "The system is fully operational..."
+  console.log(result['Key Findings']);      // ["Scalability improved...", "Memory leak..."]
+}
+```
+
+### Why use this?
+1. **Fuzzy Matching**: Even if the LLM slightly changes the heading (e.g., "Summary" instead of "Executive Summary"), the adapter will correctly map it based on the spec.
+2. **Type Enforcement**: Lists and Tables are automatically converted to JSON arrays and objects.
+3. **Single Source of Truth**: Use the same spec to guide the LLM AND parse its response.
+
+## Spec Memory: Remember & Recall
+
+Flex-MD includes an in-memory storage feature that allows you to "remember" an Output Format Spec and later reuse it by a unique `recallId`. This is especially useful for maintaining state within a single execution environment.
+
+### Usage:
+
+```typescript
+import { remember, transformWithOfs } from 'flex-md';
+
+// 1. Remember a spec extracted from instructions
+const instructions = `
+## Output format
+- Confidence — number (required)
+- Reason — text (required)
+`;
+const recallId = remember(instructions); // Returns "unique-uuid-..."
+
+// 2. Later, use the recallId instead of the spec object
+const md = `
+### Confidence
+0.95
+
+### Reason
+Everything looks correctly formatted based on initial evidence.
+`;
+
+const { result } = transformWithOfs(md, recallId);
+console.log(result.Confidence); // 0.95
+```
+
+### Why use this?
+- **Cleaner Code**: Passes simple strings instead of complex spec objects.
+- **Reference-based workflow**: Useful when multiple parts of your system need to agree on the same specification.
+- **Efficiency**: The spec is parsed once and reused.
+
 ## Documentation
 
 Detailed guides can be found in the [docs](./docs) folder:

package/SPEC.md

@@ -471,6 +471,14 @@ Nested occurrences remain as children in outline.
 * `detectObjects(text) -> DetectedObject[]`
 * `parseAny(text) -> { flexDocs: FlexDocument[]; markdownSnippets: string[]; remainder: string }`
 
+### Shared Utilities
+
+* `markdownToJson(md) -> Record<string, any>`: Convert headings to camel-cased keys and bodies to values. Robustly handles escaped newlines.
+* `ofsToSchema(spec) -> SchemaType`: Convert a Flex-MD `OutputFormatSpec` to an `nx-md-parser` `SchemaType`.
+* `transformWithOfs(md, specOrRecallId) -> TransformResult`: High-level wrapper to transform Markdown using an OFS definition or a stored recall ID.
+* `remember(instructions: string) -> string`: Parse and store an OFS from instructions, returning a unique recall ID.
+* `recall(id: string) -> OutputFormatSpec | undefined`: Retrieve a stored OFS.
+
 ---
 
 ## Appendix A — A canonical OFS generator (example)

package/dist/index.cjs

@@ -14,7 +14,7 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
     for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.enforceFlexMd = exports.repairToMarkdownLevel = exports.detectResponseKind = exports.buildIssuesEnvelopeAuto = exports.buildIssuesEnvelope = exports.parseIssuesEnvelope = exports.processResponseMarkdown = exports.extractFromMarkdown = exports.checkConnection = exports.hasFlexMdContract = exports.checkCompliance = exports.validateMarkdownAgainstOfs = exports.enrichInstructionsWithFlexMd = exports.enrichInstructions = exports.buildMarkdownGuidance = exports.stringifyOutputFormatSpec = exports.validateFormat = exports.parseOutputFormatSpec = exports.buildOutline = void 0;
+exports.jsonToMarkdown = exports.Schema = exports.MarkdownParser = exports.JSONTransformer = exports.enforceFlexMd = exports.repairToMarkdownLevel = exports.detectResponseKind = exports.buildIssuesEnvelopeAuto = exports.buildIssuesEnvelope = exports.parseIssuesEnvelope = exports.processResponseMarkdown = exports.extractFromMarkdown = exports.checkConnection = exports.hasFlexMdContract = exports.checkCompliance = exports.validateMarkdownAgainstOfs = exports.enrichInstructionsWithFlexMd = exports.enrichInstructions = exports.buildMarkdownGuidance = exports.stringifyOutputFormatSpec = exports.recall = exports.remember = exports.transformWithOfs = exports.ofsToSchema = exports.validateFormat = exports.parseOutputFormatSpec = exports.buildOutline = void 0;
 // Core SFMD Types
 __exportStar(require("./types.js"), exports);
 __exportStar(require("./strictness/types.js"), exports);
@@ -26,6 +26,12 @@ Object.defineProperty(exports, "buildOutline", { enumerable: true, get: function
 var parser_js_1 = require("./ofs/parser.js");
 Object.defineProperty(exports, "parseOutputFormatSpec", { enumerable: true, get: function () { return parser_js_1.parseOutputFormatSpec; } });
 Object.defineProperty(exports, "validateFormat", { enumerable: true, get: function () { return parser_js_1.validateFormat; } });
+var adapter_js_1 = require("./ofs/adapter.js");
+Object.defineProperty(exports, "ofsToSchema", { enumerable: true, get: function () { return adapter_js_1.ofsToSchema; } });
+Object.defineProperty(exports, "transformWithOfs", { enumerable: true, get: function () { return adapter_js_1.transformWithOfs; } });
+var memory_js_1 = require("./ofs/memory.js");
+Object.defineProperty(exports, "remember", { enumerable: true, get: function () { return memory_js_1.remember; } });
+Object.defineProperty(exports, "recall", { enumerable: true, get: function () { return memory_js_1.recall; } });
 var stringify_js_1 = require("./ofs/stringify.js");
 Object.defineProperty(exports, "stringifyOutputFormatSpec", { enumerable: true, get: function () { return stringify_js_1.stringifyOutputFormatSpec; } });
 var enricher_js_1 = require("./ofs/enricher.js");
@@ -60,3 +66,9 @@ Object.defineProperty(exports, "enforceFlexMd", { enumerable: true, get: functio
 __exportStar(require("./detect/json/index.js"), exports);
 // Token Estimation
 __exportStar(require("./tokens/index.js"), exports);
+// NX-MD-Parser Integration (Structured MD -> JSON)
+var nx_md_parser_1 = require("nx-md-parser");
+Object.defineProperty(exports, "JSONTransformer", { enumerable: true, get: function () { return nx_md_parser_1.JSONTransformer; } });
+Object.defineProperty(exports, "MarkdownParser", { enumerable: true, get: function () { return nx_md_parser_1.MarkdownParser; } });
+Object.defineProperty(exports, "Schema", { enumerable: true, get: function () { return nx_md_parser_1.Schema; } });
+Object.defineProperty(exports, "jsonToMarkdown", { enumerable: true, get: function () { return nx_md_parser_1.jsonToMarkdown; } });

package/dist/index.d.ts

@@ -3,6 +3,8 @@ export * from "./strictness/types.js";
 export * from "./md/parse.js";
 export { buildOutline } from "./md/outline.js";
 export { parseOutputFormatSpec, validateFormat } from "./ofs/parser.js";
+export { ofsToSchema, transformWithOfs } from "./ofs/adapter.js";
+export { remember, recall } from "./ofs/memory.js";
 export { stringifyOutputFormatSpec } from "./ofs/stringify.js";
 export { buildMarkdownGuidance, enrichInstructions, enrichInstructionsWithFlexMd } from "./ofs/enricher.js";
 export { validateMarkdownAgainstOfs } from "./validate/validate.js";
@@ -16,3 +18,5 @@ export { repairToMarkdownLevel } from "./pipeline/repair.js";
 export { enforceFlexMd } from "./pipeline/enforce.js";
 export * from "./detect/json/index.js";
 export * from "./tokens/index.js";
+export { JSONTransformer, MarkdownParser, Schema, jsonToMarkdown } from "nx-md-parser";
+export type { SchemaType, ValidationStatus, TransformResult, MarkdownSection } from "nx-md-parser";

package/dist/index.js

@@ -6,6 +6,8 @@ export * from "./md/parse.js";
 export { buildOutline } from "./md/outline.js";
 // Output Format Spec (OFS)
 export { parseOutputFormatSpec, validateFormat } from "./ofs/parser.js";
+export { ofsToSchema, transformWithOfs } from "./ofs/adapter.js";
+export { remember, recall } from "./ofs/memory.js";
 export { stringifyOutputFormatSpec } from "./ofs/stringify.js";
 export { buildMarkdownGuidance, enrichInstructions, enrichInstructionsWithFlexMd } from "./ofs/enricher.js";
 // Validation & Extraction
@@ -24,3 +26,5 @@ export { enforceFlexMd } from "./pipeline/enforce.js";
 export * from "./detect/json/index.js";
 // Token Estimation
 export * from "./tokens/index.js";
+// NX-MD-Parser Integration (Structured MD -> JSON)
+export { JSONTransformer, MarkdownParser, Schema, jsonToMarkdown } from "nx-md-parser";

package/dist/md/parse.d.ts

@@ -26,5 +26,6 @@ export interface ParsedSection {
 }
 export declare function parseHeadingsAndSections(md: string): ParsedSection[];
 export declare function extractBullets(body: string): string[];
+export declare function parseMarkdownTable(body: string, columns: string[]): Record<string, string>[];
 export declare function isIssuesEnvelopeCheck(md: string): IssuesEnvelope;
 export declare function markdownToJson(md: string): Record<string, any>;

package/dist/md/parse.js

@@ -90,12 +90,35 @@ export function extractBullets(body) {
     const lines = body.split(/\r?\n/);
     const out = [];
     for (const line of lines) {
-        const m = line.match(/^\s*[-*]\s+(.*)\s*$/);
+        const m = line.match(/^\s*[-*•]\s+(.*)\s*$/) || line.match(/^\s*\d+\.\s+(.*)\s*$/);
         if (m)
-            out.push(m[1] ?? "");
+            out.push(m[1].trim());
     }
     return out;
 }
+export function parseMarkdownTable(body, columns) {
+    const lines = body.split(/\r?\n/).map(l => l.trim()).filter(l => l.startsWith("|"));
+    if (lines.length < 2)
+        return [];
+    // Identification of header vs separator
+    const headerLine = lines[0];
+    const separatorLine = lines[1];
+    // Check if second line is a separator (e.g. |---|---|)
+    const isSeparator = separatorLine && /^[|\s-:]+$/.test(separatorLine);
+    const dataStartIndex = isSeparator ? 2 : 1;
+    const dataLines = lines.slice(dataStartIndex);
+    const results = [];
+    for (const line of dataLines) {
+        const cells = line.split("|").map(c => c.trim()).filter((_, i, arr) => i > 0 && i < arr.length - 1);
+        const row = {};
+        // Match cells to provided columns by index
+        for (let i = 0; i < columns.length; i++) {
+            row[columns[i]] = cells[i] || "";
+        }
+        results.push(row);
+    }
+    return results;
+}
 export function isIssuesEnvelopeCheck(md) {
     const parsed = parseHeadingsAndSections(md);
     const want = ["status", "issues", "expected", "found", "how to fix"].map(normalizeName);

package/dist/ofs/adapter.d.ts

@@ -0,0 +1,10 @@
+import { type SchemaType, type TransformResult } from "nx-md-parser";
+import { type OutputFormatSpec } from "../types.js";
+/**
+ * Converts a Flex-MD OutputFormatSpec to an nx-md-parser Schema.
+ */
+export declare function ofsToSchema(spec: OutputFormatSpec): SchemaType;
+/**
+ * Transforms markdown text using a Flex-MD OutputFormatSpec or a recallId.
+ */
+export declare function transformWithOfs<T = any>(md: string, specOrRecallId: OutputFormatSpec | string): TransformResult<T>;

package/dist/ofs/adapter.js

@@ -0,0 +1,101 @@
+import { JSONTransformer, Schema } from "nx-md-parser";
+import { recall } from "./memory.js";
+import { parseHeadingsAndSections, extractBullets, parseMarkdownTable, normalizeName } from "../md/parse.js";
+/**
+ * Converts a Flex-MD OutputFormatSpec to an nx-md-parser Schema.
+ */
+export function ofsToSchema(spec) {
+    const properties = {};
+    for (const section of spec.sections) {
+        let type;
+        switch (section.kind) {
+            case "list":
+            case "ordered_list":
+                type = Schema.array(Schema.string());
+                break;
+            case "table":
+            case "ordered_table":
+                if (section.columns && section.columns.length > 0) {
+                    const rowProps = {};
+                    for (const col of section.columns) {
+                        rowProps[col] = Schema.string();
+                    }
+                    type = Schema.array(Schema.object(rowProps));
+                }
+                else {
+                    type = Schema.string();
+                }
+                break;
+            case "text":
+            default:
+                type = Schema.string();
+                break;
+        }
+        properties[section.name] = type;
+    }
+    if (spec.tables && spec.tables.length > 0) {
+        for (const table of spec.tables) {
+            const tableKey = table.columns.join("_");
+            const rowProps = {};
+            for (const col of table.columns) {
+                rowProps[col] = Schema.string();
+            }
+            properties[tableKey] = Schema.array(Schema.object(rowProps));
+        }
+    }
+    return Schema.object(properties);
+}
+/**
+ * Transforms markdown text using a Flex-MD OutputFormatSpec or a recallId.
+ */
+export function transformWithOfs(md, specOrRecallId) {
+    let spec;
+    if (typeof specOrRecallId === "string") {
+        spec = recall(specOrRecallId);
+        if (!spec) {
+            return {
+                status: "failed",
+                result: null,
+                errors: [`Recall ID "${specOrRecallId}" not found in memory.`]
+            };
+        }
+    }
+    else {
+        spec = specOrRecallId;
+    }
+    // 1. Parse sections using Flex-MD parser
+    const parsedSections = parseHeadingsAndSections(md);
+    const parsedObj = {};
+    // 2. Map sections to OFS and apply complex parsing (tables/lists)
+    for (const sectionSpec of spec.sections) {
+        const normName = normalizeName(sectionSpec.name);
+        // Find section with similar name
+        const found = parsedSections.find(s => normalizeName(s.heading.name) === normName);
+        if (found) {
+            let value;
+            switch (sectionSpec.kind) {
+                case "list":
+                case "ordered_list":
+                    value = extractBullets(found.body);
+                    break;
+                case "table":
+                case "ordered_table":
+                    if (sectionSpec.columns) {
+                        value = parseMarkdownTable(found.body, sectionSpec.columns);
+                    }
+                    else {
+                        value = found.body.trim();
+                    }
+                    break;
+                default:
+                    value = found.body.trim();
+                    break;
+            }
+            parsedObj[sectionSpec.name] = value;
+        }
+    }
+    // 3. Transform using nx-md-parser for schema validation and auto-fixing
+    const schema = ofsToSchema(spec);
+    const transformer = new JSONTransformer(schema);
+    return transformer.transform(parsedObj);
+}

package/dist/ofs/memory.d.ts

@@ -0,0 +1,10 @@
+import { type OutputFormatSpec } from "../types.js";
+/**
+ * Parses Flex-MD instructions and "remembers" the Output Format Spec.
+ * Returns a unique recallId that can be used later.
+ */
+export declare function remember(instructions: string): string;
+/**
+ * Recalls a previously remembered Output Format Spec.
+ */
+export declare function recall(id: string): OutputFormatSpec | undefined;

package/dist/ofs/memory.js

@@ -0,0 +1,22 @@
+import { parseOutputFormatSpec } from "./parser.js";
+import { randomUUID } from "node:crypto";
+const specMemory = new Map();
+/**
+ * Parses Flex-MD instructions and "remembers" the Output Format Spec.
+ * Returns a unique recallId that can be used later.
+ */
+export function remember(instructions) {
+    const spec = parseOutputFormatSpec(instructions);
+    if (!spec) {
+        throw new Error("No valid Output Format Spec found in instructions.");
+    }
+    const id = randomUUID();
+    specMemory.set(id, spec);
+    return id;
+}
+/**
+ * Recalls a previously remembered Output Format Spec.
+ */
+export function recall(id) {
+    return specMemory.get(id);
+}

package/dist/ofs/parser.js

@@ -140,8 +140,15 @@ export function parseOutputFormatSpec(md, opts = {}) {
         }
         // If not a bullet and we have a current section, it's an instruction
         if (currentSection && line.length > 0) {
-            const existing = currentSection.instruction || "";
-            currentSection.instruction = existing ? `${existing} ${line}` : line;
+            // Support "Columns: A, B, C" in instructions for tables
+            const colMatch = line.match(/^Columns:\s*(.+)$/i);
+            if (colMatch && (currentSection.kind === "table" || currentSection.kind === "ordered_table")) {
+                currentSection.columns = colMatch[1].split(",").map(c => c.trim()).filter(Boolean);
+            }
+            else {
+                const existing = currentSection.instruction || "";
+                currentSection.instruction = existing ? `${existing} ${line}` : line;
+            }
         }
     }
     if (!sections.length)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "flex-md",
-  "version": "4.1.0",
+  "version": "4.2.0",
   "description": "Parse and stringify FlexMD: semi-structured Markdown with three powerful layers - Frames, Output Format Spec (OFS), and Detection/Extraction.",
   "license": "MIT",
   "author": "",
@@ -48,6 +48,8 @@
     "vitest": "^4.0.16"
   },
   "dependencies": {
-    "nx-helpers": "^1.5.0"
+    "nd": "^1.2.0",
+    "nx-helpers": "^1.5.0",
+    "nx-md-parser": "^1.4.0"
   }
 }