@udx/mq 0.1.1

package/package.json

@@ -0,0 +1,67 @@
+{
+  "name": "@udx/mq",
+  "version": "0.1.1",
+  "description": "Markdown Query - jq for Markdown documents",
+  "main": "mq.js",
+  "type": "module",
+  "bin": {
+    "mq": "./mq.js"
+  },
+  "scripts": {
+    "test": "NODE_OPTIONS=--experimental-vm-modules npx mocha",
+    "test:gist": "NODE_OPTIONS=--experimental-vm-modules npx mocha test/gist-integration.test.mjs",
+    "test:integration": "NODE_OPTIONS=--experimental-vm-modules npx mocha test/integration.test.mjs", 
+    "test:all": "npm test",
+    "start": "node mq.js",
+    "prepublishOnly": "npm test"
+  },
+  "keywords": [
+    "markdown",
+    "query",
+    "jq",
+    "transform",
+    "documentation",
+    "cli",
+    "toc",
+    "headings",
+    "code-blocks",
+    "links",
+    "analysis"
+  ],
+  "author": "UDX",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/andypotanin/udx.dev.git",
+    "directory": "tools/mq"
+  },
+  "bugs": {
+    "url": "https://github.com/andypotanin/udx.dev/issues"
+  },
+  "homepage": "https://github.com/andypotanin/udx.dev/tree/main/tools/mq#readme",
+  "files": [
+    "mq.js",
+    "README.md",
+    "lib/**/*.js",
+    "examples"
+  ],
+  "engines": {
+    "node": ">=14.16"
+  },
+  "dependencies": {
+    "commander": "^11.1.0",
+    "js-yaml": "^4.1.0",
+    "mdast-util-from-markdown": "^1.3.0",
+    "mdast-util-gfm": "^3.1.0",
+    "mdast-util-to-markdown": "^1.5.0",
+    "mdast-util-to-string": "^3.2.0",
+    "micromark-extension-gfm": "^3.0.0",
+    "unist-util-visit": "^4.1.1"
+  },
+  "peerDependencies": {
+    "@udx/mcurl": "^0.3.0"
+  },
+  "devDependencies": {
+    "mocha": "^11.1.0"
+  }
+}

package/readme.md

@@ -0,0 +1,242 @@
+# mq - Markdown Query
+
+A powerful tool for querying, transforming, and analyzing markdown documents, designed as an extension for [@udx/mcurl](../mcurl).
+
+## Quick Start
+
+Installing mq
+
+```bash
+npm install -g @udx/mq
+```
+
+Test installation:
+
+```bash
+mq --version
+```
+
+### Basic Usage
+
+```bash
+# Extract all headings
+mq '.headings[]' -i ../../README.md
+
+# Get the table of contents
+mq '.toc' -i ../../README.md
+
+# Extract all code blocks with their language
+mq '.codeBlocks[] | {language, content}' -i ../../README.md
+
+# Make all code blocks collapsible
+mq -t '.codeBlocks[] |= makeCollapsible' -i ../../README.md
+
+# Analyze document structure
+mq --analyze -i ../../README.md
+
+# Show document structure (headings hierarchy)
+mq --structure -i ../../README.md
+
+# Count document elements
+mq --count -i ../../README.md
+```
+
+### Using Piped Input
+
+You can also use mq with piped input from the tools/mq directory:
+
+```bash
+cat ../../README.md | mq '.headings[]'
+```
+
+## Concept
+
+Just as `jq` allows you to query and transform JSON data, `mq` provides a similar experience for markdown documents. It parses markdown into a structured object that you can query, transform, and analyze using a familiar syntax.
+
+## Features
+
+- Query markdown documents with a jq-like syntax
+- Transform markdown content with various operations
+- Extract specific elements like headings, code blocks, links
+- Generate table of contents
+- Analyze document structure and content
+- Count document elements
+- Integration with mcurl
+
+## Query Syntax
+
+mq uses a simplified jq-like syntax for querying markdown elements:
+
+```bash
+# Basic element selection
+.headings[]                # All headings
+.headings[0]               # First heading
+.headings[] | select(.level == 2)  # All h2 headings
+.links[]                   # All links
+.codeBlocks[]              # All code blocks
+.toc                       # Table of contents
+
+# Filtering and transformations
+.headings[] | select(.text | contains("Introduction"))  # Headings containing "Introduction"
+.codeBlocks[] | select(.language == "javascript")       # JavaScript code blocks
+.links[] | select(.href | startswith("http"))           # External links
+```
+
+## Transform Operations
+
+Transform operations modify the markdown structure:
+
+```bash
+# Make code blocks collapsible
+mq -t '.codeBlocks[] |= makeCollapsible' -i test/fixtures/content-website.md
+
+# Add a descriptive TOC
+mq -t '.toc |= makeDescriptive' -i test/fixtures/content-website.md
+
+# Add cross-links section after TOC
+mq -t '.toc |= addCrossLinks(["worker.md", "platform.md"])' -i test/fixtures/content-website.md
+
+# Fix heading hierarchy
+mq -t '.headings |= fixHierarchy' -i test/fixtures/content-website.md
+```
+
+## Analysis Features
+
+mq provides powerful analysis capabilities:
+
+```bash
+# Analyze document structure and content
+mq --analyze -i test/fixtures/content-website.md
+
+# Show document structure (headings hierarchy)
+mq --structure -i test/fixtures/content-website.md
+
+# Count document elements
+mq --count -i test/fixtures/content-website.md
+```
+
+## Piping with Other Tools
+
+mq works seamlessly with Unix pipes and other tools:
+
+```bash
+# Find markdown files with inconsistent heading hierarchy
+find ../../content/ -name "*.md" | xargs -I{} sh -c 'cat {} | mq ".headings | validateHierarchy" || echo "Issue in {}"'
+
+# Extract all external links from documentation
+find ../../content/ -name "*.md" | xargs cat | mq '.links[] | select(.href | startswith("http")) | .href' | sort | uniq
+
+# Generate a combined TOC from multiple files
+find ../../content/architecture/ -name "*.md" | xargs cat | mq '.headings[] | select(.level <= 2) | {file: input_filename, heading: .text}'
+```
+
+## Integration with mCurl
+
+Use mq as a content handler extension for mCurl:
+
+```javascript
+import { registerContentHandler } from '@udx/mcurl';
+import { markdownHandler } from '@udx/mq';
+
+// Register the markdown handler with mcurl
+registerContentHandler('text/markdown', markdownHandler);
+
+// Fetch and transform markdown content
+const result = await mcurl('https://udx.io', {
+  mqQuery: '.headings[] | select(.level == 2) | .text'
+});
+```
+
+## Common Use Cases
+
+### 1. Documentation Analysis
+
+```bash
+# Find documents missing a proper TOC
+find test/fixtures/ -name "*.md" | xargs -I{} sh -c 'cat {} | mq ".toc | length" | grep -q "^0$" && echo "{} missing TOC"'
+
+# List all unique tags used in documentation
+find test/fixtures/ -name "*.md" | xargs cat | mq '.tags[]?' | sort | uniq -c | sort -nr
+
+# Analyze content distribution
+find test/fixtures/ -name "*.md" | xargs -I{} sh -c 'cat {} | mq --analyze > {}.analysis.md'
+```
+
+### 2. Batch Transformations
+
+```bash
+# Add cross-links between related documents
+for file in test/fixtures/*.md; do
+  cat $file | mq -t '.toc |= addCrossLinks(["worker.md", "platform.md"])' > $file.new
+  mv $file.new $file
+done
+
+# Make all code blocks collapsible
+find test/fixtures/ -name "*.md" | xargs -I{} sh -c 'cat {} | mq -t ".codeBlocks[] |= makeCollapsible" > {}.new && mv {}.new {}'
+```
+
+### 3. Documentation Validation
+
+```bash
+# Validate all links are working
+find test/fixtures/ -name "*.md" | xargs cat | mq '.links[] | .href' | xargs -I{} curl -s -o /dev/null -w "%{http_code} {}\n" {} | grep -v "^200"
+
+# Check for consistent heading structure
+find test/fixtures/ -name "*.md" | xargs -I{} sh -c 'cat {} | mq ".headings | validateHierarchy" || echo "Issue in {}"'
+```
+
+## Examples
+
+See the [examples](./examples) directory for more usage examples.
+
+## Document Comparison Examples
+
+Use `mq` to compare and analyze differences between architecture documents. Run these commands from the tools/mq directory:
+
+```bash
+
+# Compare document statistics between files
+mq --count -i test/fixtures/stateless.md | grep 'Headings\|Paragraphs\|Links\|CodeBlocks'
+
+# Find unique sections in a specific document (e.g., mobile credential-related sections)
+mq '.headings[] | select(.text | contains("Mobile Credential"))' -i test/fixtures/stateless.md
+
+# Compare content distribution to identify most detailed sections
+mq --analyze -i test/fixtures/stateless.md | grep -A20 'Content Distribution'
+
+# Extract links to see different external references
+mq '.links[]' -i test/fixtures/stateless.md
+```
+
+For inline comparison without creating temporary files, use command substitution:
+
+```bash
+# Make sure to run these from the tools/mq directory
+
+# Directly compare heading counts
+echo "Stateless: $(mq --count -i test/fixtures/stateless.md | grep 'Headings' | awk '{print $2}')"
+
+# Find terms that appear in one document but not the other
+comm -23 <(mq '.headings[].text' -i test/fixtures/stateless.md | sort) \
+         <(mq '.headings[].text' -i test/fixtures/cloud-automation-group.md | sort)
+```
+
+### Document Comparison Script
+
+Create a simple shell script to help compare architecture documents:
+
+```bash
+#!/bin/bash
+# Save as tools/mq/compare-architectures.sh and make executable with chmod +x
+
+echo "=== Comparing Document Statistics ==="
+echo "Stateless:"
+mq --count -i test/fixtures/stateless.md | grep 'Headings\|Paragraphs\|Links\|CodeBlocks'
+
+echo "\n=== Finding Mobile Credential Sections ==="
+mq '.headings[] | select(.text | contains("Mobile") or .text | contains("Credential"))' -i test/fixtures/stateless.md | grep "text"
+```
+
+## License
+
+MIT