mdsel 0.1.2 → 0.1.3

package/README.md

@@ -10,38 +10,30 @@ mdsel parses Markdown documents into semantic trees and exposes machine-addressa
 
 ```bash
 $ mdsel README.md
-h1 mdsel
- h2 Demo
- h2 Installation
- h2 Quick Start
- h2 Commands
-  h3 index
-  h3 select
- h2 Selectors
-  h3 Syntax
-  h3 Node Types
-  h3 Index Syntax
-  h3 Examples
-  h3 Index Semantics
- h2 Output Format
- h2 Error Handling
- h2 Development
- h2 License
+h1.0 mdsel
+ h2.0 Demo
+ h2.1 Installation
+ h2.2 Quick Start
+ h2.3 Usage
+  h3.0 Index (files only)
+  h3.1 Select (files + selectors)
+  h3.2 Search (fuzzy matching)
+ h2.4 Selectors
+ h2.5 Output Format
+ h2.6 Error Handling
+ h2.7 Development
+ h2.8 License
 ---
-code:19 para:23 list:5 table:3
+code:29 para:29 list:5 table:4
 ```
 
 **2. Select specific content by selector:**
 
 ```bash
-$ mdsel README.md h2.1
-```
-```
+$ mdsel h2.1 README.md
 ## Installation
 
-​```bash
 npm install -g mdsel
-​```
 
 **Requirements**: Node.js >=18.0.0
 ```
@@ -49,9 +41,7 @@ npm install -g mdsel
 **3. Drill into nested content:**
 
 ```bash
-$ mdsel README.md h2.1/code.0
-```
-```
+$ mdsel "h2.1/code.0" README.md
 npm install -g mdsel
 ```
 
@@ -66,143 +56,134 @@ npm install -g mdsel
 ## Quick Start
 
 ```bash
-# Index a document to discover available selectors
+# Index a document to see its structure
 mdsel README.md
 
-# Select a specific heading (shorthand)
-mdsel README.md h1.0
+# Select a specific section by index
+mdsel h2.1 README.md
+
+# Select the entire document
+mdsel '*' README.md
 
-# Select the first code block under a heading
-mdsel README.md "h2.0/code.0"
+# Select a nested element (first code block under second h2)
+mdsel "h2.1/code.0" README.md
 
-# Select multiple headings with comma syntax
-mdsel README.md h2.0,2
+# Select multiple sections at once
+mdsel h2.0 h2.1 README.md
 
-# Select multiple selectors at once
-mdsel README.md h2.0 h2.1 code.0
+# Select a range of sections
+mdsel h2.0-2 README.md
 
-# Limit output to first 10 lines
-mdsel README.md "h2.0?head=10"
+# Fuzzy search when you don't know the exact selector
+mdsel "installation" README.md
 
-# Limit output to last 5 lines
-mdsel README.md "h2.0?tail=5"
+# Limit output to first N lines
+mdsel "h2.0?head=10" README.md
 
-# Use JSON output for programmatic consumption
+# JSON output for programmatic use
 mdsel --json README.md
 ```
 
 ## Usage
 
-mdsel automatically detects whether arguments are files or selectors:
-
 ```bash
-mdsel <files...> [selectors...]
 mdsel [options] <files...> [selectors...]
 ```
 
+Arguments are auto-detected: `.md` files and existing paths are files, everything else is a selector.
+
 **Options**:
 - `--json` - Output JSON instead of text
+- `--help` - Show help
 
 ### Index (files only)
 
-When only files are provided, mdsel outputs the document structure:
+When only files are provided, outputs the document structure:
 
 ```bash
 mdsel README.md
-mdsel README.md docs/API.md
-mdsel --json README.md
 ```
 
-**Text Output** (default):
 ```
 h1.0 mdsel
- h2.0 Installation
- h2.1 Quick Start
- h2.2 Usage
-  h3.0 Index
-  h3.1 Select
+ h2.0 Demo
+ h2.1 Installation
+ h2.2 Quick Start
+ h2.3 Usage
+  h3.0 Index (files only)
+  h3.1 Select (files + selectors)
+  h3.2 Search (fuzzy matching)
 ---
-code:19 para:23 list:5 table:3
+code:29 para:29 list:5 table:4
 ```
 
-**JSON Output** (`--json` flag):
-```json
-{
-  "success": true,
-  "command": "index",
-  "timestamp": "2025-01-15T10:30:00.000Z",
-  "data": {
-    "documents": [
-      {
-        "namespace": "readme",
-        "file_path": "README.md",
-        "headings": [...],
-        "blocks": {
-          "paragraphs": 5,
-          "code_blocks": 2,
-          "lists": 1,
-          "tables": 0,
-          "blockquotes": 0
-        }
-      }
-    ],
-    "summary": {
-      "total_documents": 1,
-      "total_nodes": 8,
-      "total_selectors": 8
-    }
-  }
-}
-```
+The index shows:
+- Heading hierarchy with selectors (e.g., `h1.0`, `h2.0`)
+- Indentation reflecting document structure
+- Block counts for code, paragraphs, lists, tables
 
 ### Select (files + selectors)
 
-When both files and selectors are provided, mdsel retrieves matching content:
+When selectors are provided, retrieves matching content:
 
 ```bash
-# Single selector
-mdsel README.md h2.0
-
-# Multiple selectors
-mdsel README.md h2.0 h2.1 code.0
-
-# Cross-document selection
-mdsel README.md GUIDE.md h1.0
-
-# With query parameters
-mdsel README.md "h2.0?head=10"
-
-# Range selection
-mdsel README.md h2.1-3
+# Single result - content only
+mdsel h2.1 README.md
+```
 
-# Comma list selection
-mdsel README.md h2.0,2,4
 ```
+## Installation
 
-**Text Output** (default):
+npm install -g mdsel
+
+**Requirements**: Node.js >=18.0.0
 ```
-## Quick Start
 
-To get started...
+```bash
+# Multiple results - prefixed with selector
+mdsel h2.0 h2.1 README.md
 ```
 
-**Multiple Results** (selector prefix):
 ```
 heading:h2.0:
-## Installation
-content...
+## Demo
+...
 heading:h2.1:
-## Quick Start
-content...
+## Installation
+...
+```
+
+```bash
+# Errors show suggestions
+mdsel h2.99 README.md
 ```
 
-**Error Output**:
 ```
 !h2.99
-Index out of range: document has 3 h2 headings
+Index out of range: document has 9 h2 headings
 ~h2.0 ~h2.1 ~h2.2
 ```
 
+### Search (fuzzy matching)
+
+When input doesn't look like a selector, mdsel performs fuzzy search:
+
+```bash
+mdsel "installation" README.md
+```
+
+```
+Search results for "installation":
+
+readme::h2.1 (100% match)
+  Installation
+
+readme::code.9 (74% match)
+  ## Installation npm install -g mdsel ...
+```
+
+Search returns selectors you can use directly to fetch the content.
+
 ## Selectors
 
 Selectors are path-based, ordinal, stateless, and deterministic. They resemble CSS/XPath conceptually but are purpose-built for Markdown.
@@ -223,6 +204,7 @@ Selectors are path-based, ordinal, stateless, and deterministic. They resemble C
 
 | Category | Full Form | Shorthand |
 |----------|-----------|-----------|
+| Wildcard | `*` | `*` |
 | Root | `root` | - |
 | Headings | `heading:h1` ... `heading:h6` | `h1` ... `h6` |
 | Sections | `section` | - |
@@ -248,6 +230,7 @@ Two equivalent notations are supported:
 
 **Basic selection**:
 ```bash
+*                   # Entire document (wildcard)
 root                # Document root
 h1.0                # First h1 heading
 h2.1                # Second h2 heading

package/dist/cli.mjs

@@ -855,6 +855,10 @@ function tokenize(input) {
         advance();
         addToken("EQUALS" /* EQUALS */, "=", pos);
         continue;
+      case "*":
+        advance();
+        addToken("STAR" /* STAR */, "*", pos);
+        continue;
       case '"':
       case "'": {
         const strPos = position();
@@ -976,7 +980,9 @@ var Parser = class {
     let nodeType;
     let subtype;
     let index;
-    if (this.match("ROOT" /* ROOT */)) {
+    if (this.match("STAR" /* STAR */)) {
+      nodeType = "all";
+    } else if (this.match("ROOT" /* ROOT */)) {
       nodeType = "root";
     } else if (this.match("HEADING" /* HEADING */)) {
       nodeType = "heading";
@@ -1420,6 +1426,10 @@ function resolvePathSegments(context, segments) {
       currentNodes = currentNodes.map(({ node, path }) => ({ node, path: [...path, node] }));
       continue;
     }
+    if (segment.nodeType === "all") {
+      currentNodes = currentNodes.map(({ node, path }) => ({ node, path: [...path, node] }));
+      continue;
+    }
     const currentNode = currentNodes[0]?.node;
     const currentPath = currentNodes[0]?.path ?? [];
     const matches = findMatchingChildren(currentNode, segment);
@@ -1558,6 +1568,9 @@ function segmentToStringWithIndex(segment, index) {
   return segStr;
 }
 function segmentToString(segment) {
+  if (segment.nodeType === "all") {
+    return "*";
+  }
   let segStr = segment.nodeType;
   if (segment.subtype) {
     segStr += `:${segment.subtype}`;
@@ -1684,6 +1697,193 @@ function selectorToString2(selector) {
   return result;
 }
 
+// src/resolver/search.ts
+import { toString as toString2 } from "mdast-util-to-string";
+function searchDocument(tree, namespace, query, options = {}) {
+  const { maxResults = 10, minScore = 0.3, includeCode = true } = options;
+  const results = [];
+  const normalizedQuery = query.toLowerCase().trim();
+  if (!normalizedQuery) {
+    return [];
+  }
+  const headingIndices = { 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0 };
+  const blockIndices = {
+    paragraph: 0,
+    code: 0,
+    list: 0,
+    table: 0,
+    blockquote: 0
+  };
+  for (let i = 0; i < tree.children.length; i++) {
+    const node = tree.children[i];
+    if (!node) continue;
+    const nodeType = node.type;
+    const text = toString2(node);
+    const normalizedText = text.toLowerCase();
+    const matchResult = calculateMatchScore(normalizedQuery, normalizedText, text);
+    if (matchResult.score >= minScore) {
+      let selector;
+      let type;
+      if (nodeType === "heading" && "depth" in node) {
+        const depth = node.depth;
+        const idx = headingIndices[depth] ?? 0;
+        selector = `${namespace}::h${depth}.${idx}`;
+        type = `heading:h${depth}`;
+        headingIndices[depth] = idx + 1;
+      } else if (nodeType in blockIndices) {
+        const idx = blockIndices[nodeType] ?? 0;
+        const shorthand = getBlockShorthand(nodeType);
+        selector = `${namespace}::${shorthand}.${idx}`;
+        type = nodeType;
+        blockIndices[nodeType] = idx + 1;
+        if (nodeType === "code" && !includeCode) {
+          continue;
+        }
+      } else {
+        continue;
+      }
+      results.push({
+        selector,
+        type,
+        preview: createPreview(text, normalizedQuery, 100),
+        content: text,
+        score: matchResult.score,
+        matchType: matchResult.matchType
+      });
+    } else {
+      if (nodeType === "heading" && "depth" in node) {
+        const depth = node.depth;
+        headingIndices[depth] = (headingIndices[depth] ?? 0) + 1;
+      } else if (nodeType in blockIndices) {
+        blockIndices[nodeType] = (blockIndices[nodeType] ?? 0) + 1;
+      }
+    }
+  }
+  const sectionResults = searchHeadingSections(tree, namespace, normalizedQuery, minScore);
+  results.push(...sectionResults);
+  return results.sort((a, b) => b.score - a.score).slice(0, maxResults);
+}
+function searchHeadingSections(tree, namespace, normalizedQuery, minScore) {
+  const results = [];
+  const headingIndices = { 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0 };
+  for (let i = 0; i < tree.children.length; i++) {
+    const node = tree.children[i];
+    if (!node || node.type !== "heading" || !("depth" in node)) {
+      continue;
+    }
+    const depth = node.depth;
+    const idx = headingIndices[depth] ?? 0;
+    headingIndices[depth] = idx + 1;
+    const sectionParts = [toString2(node)];
+    for (let j = i + 1; j < tree.children.length; j++) {
+      const sibling = tree.children[j];
+      if (!sibling) break;
+      if (sibling.type === "heading" && "depth" in sibling && sibling.depth <= depth) {
+        break;
+      }
+      sectionParts.push(toString2(sibling));
+    }
+    const sectionText = sectionParts.join("\n");
+    const normalizedSection = sectionText.toLowerCase();
+    const matchResult = calculateMatchScore(normalizedQuery, normalizedSection, sectionText);
+    if (matchResult.score >= minScore) {
+      const selector = `${namespace}::h${depth}.${idx}`;
+      const existing = results.find((r) => r.selector === selector);
+      if (!existing) {
+        results.push({
+          selector,
+          type: `heading:h${depth}`,
+          preview: createPreview(sectionText, normalizedQuery, 100),
+          content: sectionText,
+          score: matchResult.score * 0.9,
+          // Slightly lower score for section matches
+          matchType: matchResult.matchType
+        });
+      }
+    }
+  }
+  return results;
+}
+function calculateMatchScore(normalizedQuery, normalizedText, originalText) {
+  if (normalizedText === normalizedQuery) {
+    return { score: 1, matchType: "exact" };
+  }
+  if (normalizedText.includes(normalizedQuery)) {
+    const coverage = normalizedQuery.length / normalizedText.length;
+    const score = 0.7 + coverage * 0.25;
+    return { score, matchType: "substring" };
+  }
+  const queryWords = normalizedQuery.split(/\s+/).filter((w) => w.length > 0);
+  const textWords = normalizedText.split(/\s+/).filter((w) => w.length > 0);
+  if (queryWords.length > 0) {
+    let matchedWords = 0;
+    for (const queryWord of queryWords) {
+      for (const textWord of textWords) {
+        if (textWord === queryWord) {
+          matchedWords++;
+          break;
+        }
+        if (textWord.includes(queryWord) || queryWord.includes(textWord)) {
+          matchedWords += 0.7;
+          break;
+        }
+        const distance = levenshteinDistance(queryWord, textWord);
+        const maxLen = Math.max(queryWord.length, textWord.length);
+        const similarity = (maxLen - distance) / maxLen;
+        if (similarity > 0.7) {
+          matchedWords += similarity * 0.5;
+          break;
+        }
+      }
+    }
+    const wordMatchRatio = matchedWords / queryWords.length;
+    if (wordMatchRatio > 0.3) {
+      return { score: wordMatchRatio * 0.6, matchType: "fuzzy" };
+    }
+  }
+  return { score: 0, matchType: "fuzzy" };
+}
+function createPreview(text, query, maxLength) {
+  const normalizedText = text.toLowerCase();
+  const index = normalizedText.indexOf(query);
+  if (index === -1) {
+    const cleaned = text.replace(/\s+/g, " ").trim();
+    if (cleaned.length <= maxLength) {
+      return cleaned;
+    }
+    return cleaned.slice(0, maxLength - 3) + "...";
+  }
+  const start = Math.max(0, index - Math.floor((maxLength - query.length) / 2));
+  const end = Math.min(text.length, start + maxLength);
+  let preview = text.slice(start, end).replace(/\s+/g, " ");
+  if (start > 0) {
+    preview = "..." + preview;
+  }
+  if (end < text.length) {
+    preview = preview + "...";
+  }
+  return preview.trim();
+}
+function getBlockShorthand(type) {
+  switch (type) {
+    case "paragraph":
+      return "para";
+    case "blockquote":
+      return "quote";
+    default:
+      return type;
+  }
+}
+function searchMultipleDocuments(documents, query, options = {}) {
+  const allResults = [];
+  for (const doc of documents) {
+    const docResults = searchDocument(doc.tree, doc.namespace, query, options);
+    allResults.push(...docResults);
+  }
+  const maxResults = options.maxResults ?? 10;
+  return allResults.sort((a, b) => b.score - a.score).slice(0, maxResults);
+}
+
 // src/cli/commands/select-command.ts
 async function selectCommand(selector, files, options = {}) {
   const useJson = options.json === true;
@@ -1698,22 +1898,32 @@ async function selectCommand(selector, files, options = {}) {
     return;
   }
   let selectorAst;
+  let parseError = null;
   try {
     selectorAst = parseSelector(selector);
   } catch (error) {
     if (error instanceof SelectorParseError) {
-      const errorEntry = createErrorEntry(
-        "INVALID_SELECTOR",
-        error.code,
-        error.message,
-        void 0,
-        selector
-      );
-      outputError2([errorEntry], useJson);
-      exitWithCode(ExitCode.ERROR);
+      parseError = error;
+    } else {
+      throw error;
+    }
+  }
+  if (parseError) {
+    const looksLikeSearch = isLikelySearchQuery(selector);
+    if (looksLikeSearch) {
+      await performSearchFallback(selector, files, options);
       return;
     }
-    throw error;
+    const errorEntry = createErrorEntry(
+      "INVALID_SELECTOR",
+      parseError.code,
+      parseError.message,
+      void 0,
+      selector
+    );
+    outputError2([errorEntry], useJson);
+    exitWithCode(ExitCode.ERROR);
+    return;
   }
   const truncateOptions = {};
   if (selectorAst.queryParams) {
@@ -1945,6 +2155,121 @@ async function selectMultiCommand(selectors, files, options = {}) {
   }
   exitWithCode(allUnresolved.length > 0 ? ExitCode.ERROR : ExitCode.SUCCESS);
 }
+function isLikelySearchQuery(input) {
+  if (!input.trim()) {
+    return false;
+  }
+  if (input.includes("::")) {
+    return false;
+  }
+  if (input.includes("[") && !input.includes("]") || !input.includes("[") && input.includes("]")) {
+    return false;
+  }
+  if (/^(h[1-6]|code|para|paragraph|list|table|quote|blockquote|root|section|heading|block)[\.\[]/.test(input)) {
+    return false;
+  }
+  if (/^(heading|block):/.test(input)) {
+    return false;
+  }
+  return true;
+}
+async function performSearchFallback(query, files, options) {
+  const useJson = options.json === true;
+  const documents = [];
+  const parseErrors = [];
+  for (const file of files) {
+    try {
+      const result = await parseFile(file);
+      const namespace = deriveNamespace(file);
+      documents.push({
+        namespace,
+        tree: result.ast
+      });
+    } catch (error) {
+      if (error instanceof ParserError) {
+        parseErrors.push(
+          createErrorEntry(
+            error.code,
+            error.code,
+            error.message,
+            error.filePath
+          )
+        );
+      } else if (error instanceof Error) {
+        parseErrors.push(createErrorEntry("PROCESSING_ERROR", "UNKNOWN", error.message, file));
+      }
+    }
+  }
+  if (documents.length === 0) {
+    outputError2(parseErrors, useJson);
+    exitWithCode(ExitCode.ERROR);
+    return;
+  }
+  const searchResults = searchMultipleDocuments(documents, query, {
+    maxResults: 10,
+    minScore: 0.3
+  });
+  if (searchResults.length === 0) {
+    const unresolved = [
+      {
+        selector: query,
+        reason: `No matches found for search query "${query}"`,
+        suggestions: []
+      }
+    ];
+    if (useJson) {
+      const response = formatSelectResponse([], unresolved);
+      console.log(JSON.stringify(response));
+    } else {
+      console.log(formatSearchText(query, []));
+    }
+    exitWithCode(ExitCode.ERROR);
+    return;
+  }
+  if (useJson) {
+    const matches = formatSearchResultsAsMatches(searchResults);
+    const response = {
+      success: true,
+      command: "search",
+      query,
+      timestamp: (/* @__PURE__ */ new Date()).toISOString(),
+      data: {
+        matches,
+        unresolved: []
+      }
+    };
+    console.log(JSON.stringify(response));
+  } else {
+    console.log(formatSearchText(query, searchResults));
+  }
+  exitWithCode(ExitCode.SUCCESS);
+}
+function formatSearchResultsAsMatches(results) {
+  return results.map((result) => ({
+    selector: result.selector,
+    type: result.type,
+    content: result.content,
+    truncated: false,
+    children_available: [],
+    search_score: result.score,
+    match_type: result.matchType
+  }));
+}
+function formatSearchText(query, results) {
+  if (results.length === 0) {
+    return `No matches found for: ${query}`;
+  }
+  const parts = [];
+  parts.push(`Search results for "${query}":`);
+  parts.push("");
+  for (const result of results) {
+    const score = Math.round(result.score * 100);
+    parts.push(`${result.selector} (${score}% match)`);
+    parts.push(`  ${result.preview}`);
+    parts.push("");
+  }
+  return parts.join("\n").trimEnd();
+}
 
 // src/cli/commands/format-command.ts
 function formatCommand(command, options = {}) {
@@ -2023,7 +2348,17 @@ function partitionArgs(args) {
   return { files, selectors };
 }
 var program = new Command();
-program.name("mdsel").description(pkg.description).version(pkg.version).option("--json", "Output JSON instead of minimal text").argument("[args...]", "Files and/or selectors").action(async (args) => {
+program.name("mdsel").description(
+  `${pkg.description}
+
+Examples:
+  mdsel README.md                 Index document structure
+  mdsel h2.1 README.md            Select second h2 section
+  mdsel '*' README.md             Select entire document
+  mdsel "h2.1/code.0" README.md   Select nested content
+  mdsel "installation" README.md  Fuzzy search
+  mdsel --json README.md          Output as JSON`
+).version(pkg.version).option("--json", "Output JSON instead of text").argument("[args...]", "Markdown files and selectors (auto-detected)").action(async (args) => {
   try {
     const globalOpts = program.opts();
     if (args.length === 0) {
@@ -2057,7 +2392,7 @@ program.name("mdsel").description(pkg.description).version(pkg.version).option("
 program.command("format").description("Output format specification for tool descriptions").argument("[command]", "Command to describe (index, select, or omit for all)").option("--example", "Show example output instead of terse spec").action((command, options) => {
   formatCommand(command, options);
 });
-program.command("index").description("Index markdown files (optional - same as just providing files)").argument("<files...>", "Markdown files to index").action(async (files) => {
+program.command("index").description("Index markdown files (alternative to: mdsel <files>)").argument("<files...>", "Markdown files to index").action(async (files) => {
   try {
     const globalOpts = program.opts();
     await indexCommand(files, { json: globalOpts.json });
@@ -2066,7 +2401,7 @@ program.command("index").description("Index markdown files (optional - same as j
     process.exit(ExitCode.ERROR);
   }
 });
-program.command("select").description("Select content from markdown files (optional - same as providing files + selectors)").argument("<selector>", "Selector to match").argument("<files...>", "Markdown files to search").action(async (selector, files) => {
+program.command("select").description("Select content (alternative to: mdsel <selector> <files>)").argument("<selector>", "Selector to match").argument("<files...>", "Markdown files to search").action(async (selector, files) => {
   try {
     const globalOpts = program.opts();
     const selectors = splitSelectorList(selector);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mdsel",
-  "version": "0.1.2",
+  "version": "0.1.3",
   "description": "Declarative Markdown semantic selection CLI for LLM agents",
   "type": "module",
   "bin": {