@monoes/monomindcli 2.15.2 → 2.15.4

package/org-skills/markitdown-document-conversion/references/security.md

@@ -0,0 +1,246 @@
+# Security and Privacy
+
+MarkItDown is a converter, not a sandbox. It performs file and network I/O with the privileges of the current process and loads parser dependencies for complex, attacker-controlled formats.
+
+## Threat Model
+
+Treat all of these as untrusted unless provenance is established:
+
+- Paths, filenames, and URIs supplied by users or agents
+- Uploaded PDFs, Office files, archives, notebooks, images, audio, and EPUBs
+- HTTP response headers and redirects
+- Installed plugins
+- Markdown produced from external documents
+- LLM/OCR responses
+
+Potential impacts include:
+
+- Reading arbitrary local files
+- Server-side request forgery (SSRF)
+- Access to loopback services or cloud metadata
+- Archive/decompression bombs and memory exhaustion
+- Parser vulnerabilities
+- Credential or document exfiltration to cloud/LLM providers
+- Prompt injection carried through converted content
+- Arbitrary Python execution through plugins
+
+## API Risk Levels
+
+| API | Input capability | Recommended use |
+|---|---|---|
+| `convert_local()` | Local path only | Trusted, allowlisted local files |
+| `convert_stream()` | Caller-controlled bytes | Preferred for validated uploads |
+| `convert_response()` | Existing HTTP response | After caller-enforced network policy |
+| `convert_uri()` | `file:`, `data:`, `http:`, `https:` | Trusted and validated URI only |
+| `convert()` | Dispatches across all of the above | Trusted polymorphic input only |
+
+Use the narrowest API that satisfies the task.
+
+## Local File Controls
+
+Before `convert_local()`:
+
+1. Resolve the candidate path and the approved root.
+2. Confirm the resolved candidate remains under that root.
+3. Reject symlinks when their semantics are not explicitly required.
+4. Require a regular file; reject devices, FIFOs, sockets, and directories.
+5. Enforce an extension/MIME allowlist.
+6. Apply a maximum byte size before opening.
+7. Open with the least-privileged process account.
+8. Keep sensitive directories outside the process's readable namespace.
+
+Do not build a path by concatenating an untrusted filename with a directory string.
+
+The bundled batch and literature scripts use `convert_local()` and skip symlink inputs by default.
+
+## HTTP(S) and SSRF Controls
+
+Do not pass a user-controlled URL directly to `convert()` or `convert_uri()`.
+
+An application-controlled downloader should enforce all of the following:
+
+- Permit only required schemes, preferably HTTPS.
+- Reject embedded usernames/passwords and malformed authorities.
+- Canonicalize internationalized hostnames.
+- Resolve DNS and reject every loopback, private, link-local, multicast, reserved, unspecified, and metadata-service address in IPv4 and IPv6.
+- Revalidate the destination on every redirect.
+- Defend against DNS rebinding; validation before connection alone is insufficient.
+- Disable ambient proxy/environment behavior unless it is an explicit part of policy.
+- Use connect and read timeouts.
+- Limit redirect count.
+- Enforce `Content-Length` when present and a streaming byte limit regardless of the header.
+- Limit decompressed size.
+- Allowlist response MIME types.
+- Verify TLS normally; never disable certificate verification.
+- Avoid forwarding credentials across origins.
+- Pass the validated `requests.Response` to `convert_response()`.
+
+MarkItDown's own HTTP path calls a Requests session and buffers the complete response before parsing. It does not impose these application-level controls.
+
+## `file:` and `data:` URIs
+
+`file:` URIs can read anything the process user can read. MarkItDown permits an empty authority or `localhost`; this is not a path allowlist.
+
+`data:` URIs can contain arbitrarily large base64 payloads. Decode size grows beyond encoded size, and conversion can create additional copies.
+
+Prefer `convert_local()` or a bounded `convert_stream()` path instead of accepting arbitrary URI strings.
+
+## Streams and Memory
+
+- `convert_stream()` copies a non-seekable stream completely into memory.
+- `convert_response()` buffers the complete HTTP response.
+- ZIP, large tables, base64 images, and generated Markdown can substantially expand data.
+- `--keep-data-uris` preserves full embedded payloads and can expose hidden/sensitive data.
+
+Set input, output, member-count, page-count, time, and memory limits outside MarkItDown.
+
+## Archives
+
+MarkItDown's ZIP converter iterates members and applies nested conversion. Before accepting an untrusted archive, enforce:
+
+- Maximum compressed bytes
+- Maximum total uncompressed bytes
+- Maximum member count
+- Maximum single-member bytes
+- Maximum nested archive depth
+- Compression-ratio threshold
+- File-type allowlist
+- Conversion timeout
+
+MarkItDown does not turn an archive into a safe bundle merely because it processes members in memory.
+
+## Parser Supply Chain
+
+Use a pinned MarkItDown release:
+
+```bash
+uv pip install "markitdown[all]==0.1.6"
+```
+
+Relevant release history:
+
+- 0.1.2 moved XML/OMML parsing to `defusedxml`.
+- 0.1.3 required safe ExifTool usage at version 12.24 or later.
+- 0.1.4 updated Mammoth and `pdfminer.six` for published vulnerabilities.
+- 0.1.6 clarified the project's I/O security posture.
+
+For production:
+
+- Lock transitive dependencies and retain hashes.
+- Scan the environment and container image.
+- Rebuild when parser security advisories are published.
+- Test representative documents after upgrades.
+
+## Plugins
+
+Plugins are arbitrary Python code loaded through package entry points. They run in the same process and can access its files, network, environment, and credentials.
+
+- Keep `enable_plugins=False` by default.
+- Pin and review every plugin.
+- Inspect install/build hooks and transitive dependencies.
+- Run risky plugins in a sandbox with synthetic files.
+- Do not install by hashtag or name similarity alone.
+
+The official `markitdown-ocr` plugin still introduces external LLM calls and expands parser dependencies.
+
+## External Processing Map
+
+| Feature | What leaves the process | Destination |
+|---|---|---|
+| HTTP(S), RSS, Wikipedia, Bing | Request metadata; downloaded response returns | Requested host |
+| YouTube conversion | Page and transcript requests | YouTube/transcript service |
+| Built-in audio transcription | Recorded audio | Google Web Speech via `SpeechRecognition` |
+| LLM image description | Prompt and base64 image | Configured OpenAI-compatible provider |
+| `markitdown-ocr` | Prompt plus embedded/full-page images | Configured vision provider |
+| Document Intelligence | Complete selected file | Configured Azure resource |
+| Content Understanding | Complete selected document/image/audio/video | Configured Azure resource |
+
+Do not describe `[all]` as fully offline. It installs capabilities whose use can make network calls.
+
+## Credentials
+
+- Use only the named credential required by the selected service.
+- Prefer workload identity or managed identity for Azure.
+- Never enumerate, copy, print, or forward the process environment.
+- Never place a key in a command-line argument, source file, Markdown output, manifest, or log.
+- Keep provider base URLs fixed to a reviewed endpoint; do not combine a credential with a user-controlled endpoint.
+- Scope keys to the minimum service/project and rotate them.
+
+MarkItDown's Azure converters look for the named `AZURE_API_KEY`; otherwise they use `DefaultAzureCredential`. OpenAI-compatible clients use their own provider-specific configuration.
+
+## Sensitive Scientific and Clinical Data
+
+Before cloud, audio, or LLM processing:
+
+- Confirm consent and data-processing agreements.
+- Check PHI/PII, genomic-identifiability, unpublished results, export controls, and sponsor restrictions.
+- Use an approved region and tenant.
+- Minimize pages/modalities sent.
+- Understand provider retention and training policies.
+- Record the processing method in provenance without recording secrets.
+
+When approval is absent, use a local parser/OCR workflow.
+
+## MCP Server
+
+`markitdown-mcp` has no authentication and exposes broad URI conversion. Even localhost mode can be reached by other local processes/users.
+
+- Prefer STDIO.
+- Never bind directly to non-local interfaces.
+- Isolate filesystem and network access.
+- Mount only the required directory read-only.
+- Keep plugins disabled.
+
+See `mcp_and_plugins.md`.
+
+## Prompt Injection in Converted Content
+
+Documents can contain text such as "ignore previous instructions," links to malicious resources, or commands disguised as analysis steps. Conversion preserves that content.
+
+When Markdown is used with an agent or RAG system:
+
+1. Label it as untrusted source material.
+2. Separate it from system/developer instructions.
+3. Do not grant tools based on instructions found in the document.
+4. Require independent authorization for file, shell, network, or credential actions.
+5. Preserve source/page provenance for claims.
+6. Sanitize active HTML and dangerous links before rendering.
+
+## Output and Logging
+
+Avoid logging:
+
+- Full source paths when they reveal study/patient information
+- Document contents
+- Base64 data URIs
+- Provider request/response bodies
+- API keys, authorization headers, or signed URLs
+
+Useful safe provenance:
+
+- Content hash
+- Redacted source identifier
+- MarkItDown/plugin version
+- Converter mode
+- Timestamp
+- Approved provider/analyzer identifier
+- Success/failure category
+
+## Preflight Checklist
+
+- [ ] Source and requester are authorized
+- [ ] Narrow conversion API selected
+- [ ] Local path or network destination validated
+- [ ] Input/output/resource limits applied
+- [ ] Plugins disabled or reviewed
+- [ ] External transmission disclosed and approved
+- [ ] Credentials scoped and not logged
+- [ ] Converted Markdown treated as untrusted data
+- [ ] Result checked against the source
+- [ ] Original retained as authority
+
+## Sources
+
+- MarkItDown security guidance: https://github.com/microsoft/markitdown/blob/v0.1.6/README.md#security-considerations
+- MCP security guidance: https://github.com/microsoft/markitdown/blob/v0.1.6/packages/markitdown-mcp/README.md#security-considerations
+- Release history: https://github.com/microsoft/markitdown/releases

package/org-skills/markitdown-document-conversion/references/workflows.md

@@ -0,0 +1,309 @@
+# Practical Workflows
+
+All workflows target MarkItDown 0.1.6 and use `.markdown`, not the legacy `.text_content` alias.
+
+## 1. One Trusted Local Document
+
+```python
+from hashlib import sha256
+from pathlib import Path
+
+from markitdown import MarkItDown
+
+source = Path("protocol.docx").resolve(strict=True)
+output = Path("protocol.md")
+
+result = MarkItDown().convert_local(source)
+output.write_text(result.markdown, encoding="utf-8")
+
+provenance = {
+    "source": source.name,
+    "sha256": sha256(source.read_bytes()).hexdigest(),
+    "title": result.title,
+    "output": str(output),
+}
+print(provenance)
+```
+
+For very large files, compute the hash incrementally rather than loading the source again.
+
+## 2. Binary Upload
+
+Validate size and type before conversion. Use `BytesIO` plus explicit hints:
+
+```python
+from io import BytesIO
+
+from markitdown import MarkItDown, StreamInfo
+
+
+def convert_pdf_upload(payload: bytes, filename: str) -> str:
+    max_bytes = 25 * 1024 * 1024
+    if len(payload) > max_bytes:
+        raise ValueError("PDF exceeds the 25 MiB conversion limit")
+    if not payload.startswith(b"%PDF-"):
+        raise ValueError("Payload does not have a PDF signature")
+
+    result = MarkItDown().convert_stream(
+        BytesIO(payload),
+        stream_info=StreamInfo(
+            extension=".pdf",
+            mimetype="application/pdf",
+            filename=filename,
+        ),
+    )
+    return result.markdown
+```
+
+Signature checks are only one validation layer; they do not make the parser sandboxed.
+
+## 3. Batch Directory Conversion
+
+```bash
+python scripts/batch_convert.py inputs/ outputs/ \
+  --recursive \
+  --extensions .pdf .docx .pptx .xlsx .csv .html \
+  --manifest outputs/manifest.json
+```
+
+Properties:
+
+- Local paths only
+- Symlinks skipped
+- Deterministic path ordering
+- Relative directories preserved
+- Output name retains the source suffix, e.g. `paper.pdf.md`
+- Existing output skipped unless `--overwrite`
+- Plugins off unless `--plugins`
+- Nonzero exit when a conversion fails
+
+Retry failed files after installing the missing format extra:
+
+```bash
+uv pip install "markitdown[pdf,docx,pptx,xlsx]==0.1.6"
+python scripts/batch_convert.py inputs/ outputs/ --recursive --overwrite
+```
+
+Use a fresh output directory when comparing converter versions.
+
+## 4. Literature Collection
+
+Input naming convention:
+
+```text
+Author_Year_Title.pdf
+```
+
+Convert and build indexes:
+
+```bash
+python scripts/convert_literature.py papers/ literature/ \
+  --recursive \
+  --create-index
+```
+
+Organize by inferred year:
+
+```bash
+python scripts/convert_literature.py papers/ literature/ \
+  --recursive \
+  --organize-by-year \
+  --create-index
+```
+
+Each output contains provenance front matter:
+
+```yaml
+---
+title: "Example title"
+author: "Smith"
+year: "2025"
+source: "incoming/Smith_2025_Example_Title.pdf"
+converted_at: "2026-07-23T17:00:00+00:00"
+markitdown_version: "0.1.6"
+---
+```
+
+The helper also writes `catalog.json` and `INDEX.md` when `--create-index` is requested. Metadata inferred from a filename is a convenience, not authoritative bibliographic metadata.
+
+## 5. RAG Ingestion
+
+Recommended sequence:
+
+1. Validate and hash the source.
+2. Convert with a pinned package/plugin version.
+3. Keep source metadata separate from converted text.
+4. Check non-empty output and expected section markers.
+5. Sanitize active HTML and unsafe links if Markdown will be rendered.
+6. Chunk by semantic headings with overlap.
+7. Store source/hash/converter/page-or-section provenance with every chunk.
+8. Mark chunks as untrusted source content in the retrieval prompt.
+9. Retain the original source for verification.
+
+Example conversion envelope:
+
+```python
+from dataclasses import asdict, dataclass
+from hashlib import sha256
+from importlib.metadata import version
+from pathlib import Path
+
+from markitdown import MarkItDown
+
+
+@dataclass(frozen=True)
+class ConvertedDocument:
+    source_name: str
+    source_sha256: str
+    converter_version: str
+    title: str | None
+    markdown: str
+
+
+def convert_for_ingestion(path: Path) -> ConvertedDocument:
+    path = path.resolve(strict=True)
+    digest = sha256(path.read_bytes()).hexdigest()
+    result = MarkItDown().convert_local(path)
+    if not result.markdown.strip():
+        raise ValueError(f"Conversion produced no text: {path.name}")
+
+    return ConvertedDocument(
+        source_name=path.name,
+        source_sha256=digest,
+        converter_version=version("markitdown"),
+        title=result.title,
+        markdown=result.markdown,
+    )
+```
+
+Do not treat Markdown line numbers as stable page citations. MarkItDown does not expose PDF page coordinates.
+
+## 6. Mixed-Format Study Folder
+
+Use a two-pass workflow:
+
+### Pass A: inventory
+
+- Count files by extension and byte size.
+- Detect duplicates by content hash.
+- Exclude temporary, lock, and hidden files.
+- Identify encrypted/unsupported inputs.
+- Classify which files require external OCR/cloud processing.
+
+### Pass B: convert
+
+- Use local built-ins first.
+- Route only failed scanned/complex files to an approved OCR/cloud path.
+- Record the selected mode per file.
+- Compare file count, failure count, and output count.
+
+This minimizes cost and external data transfer.
+
+## 7. Spreadsheet Orientation, Then Structured Analysis
+
+Use MarkItDown to understand workbook organization:
+
+```python
+from markitdown import MarkItDown
+
+preview = MarkItDown().convert_local("assay-results.xlsx").markdown
+print(preview)
+```
+
+Then use a spreadsheet/dataframe library for calculations:
+
+- Inspect formulas and cached values explicitly.
+- Select the authoritative sheet/range.
+- Preserve data types.
+- Validate merged cells and hidden rows.
+
+Do not parse a scientific numeric dataset back out of Markdown when the original workbook is available.
+
+## 8. Presentation Review
+
+```python
+from markitdown import MarkItDown
+
+result = MarkItDown().convert_local("lab-meeting.pptx")
+```
+
+Review:
+
+- Slide boundaries
+- Title hierarchy
+- Speaker notes
+- Text in charts/images
+- Claims encoded only by color/position
+
+If images carry essential meaning, add an approved vision description or OCR pass and label generated text as model-derived.
+
+## 9. Scanned-PDF Escalation
+
+1. Try built-in PDF extraction.
+2. Detect unexpectedly short/empty output relative to page/file size.
+3. Confirm the source is scanned by visual inspection.
+4. Choose:
+   - Local OCR/layout parser for restricted data
+   - `markitdown-ocr` for approved vision-provider processing
+   - Document Intelligence for Azure layout/OCR
+   - Content Understanding for structured fields/multimodal processing
+5. Validate a page sample and record the selected service/model/analyzer.
+
+Do not automatically send failed local conversions to a cloud service.
+
+## 10. Application-Controlled Remote Resource
+
+The application—not MarkItDown—should:
+
+- Validate destination and every redirect
+- Enforce time and byte limits
+- Restrict MIME types
+- Apply authentication safely
+- Produce a bounded `requests.Response`
+
+Then:
+
+```python
+result = MarkItDown().convert_response(validated_response)
+```
+
+See `security.md` for the complete SSRF policy. A simple scheme check is not sufficient.
+
+## 11. Deterministic Regression Corpus
+
+Keep a small, redistributable corpus covering:
+
+- Text-native and scanned PDFs
+- DOCX headings/tables/equations
+- PPTX notes/grouped shapes
+- XLSX multiple sheets/merged cells
+- CSV quoted fields and Unicode
+- HTML links/lists/tables
+- EPUB chapters
+- JPEG/PNG metadata
+
+For each fixture, assert:
+
+- Conversion succeeds or fails with the expected category.
+- Required headings/sentinel text are present.
+- No source is unexpectedly routed to network/cloud.
+- Output is UTF-8 and below a reasonable size.
+- Package/plugin versions are recorded.
+
+Avoid asserting a full byte-for-byte Markdown snapshot unless exact formatting stability is required; semantic assertions are less brittle.
+
+## 12. Conversion Quality Report
+
+Track per file:
+
+- Source identifier and hash
+- Bytes and extension
+- Converter mode
+- MarkItDown/plugin version
+- Output characters/lines
+- Optional title
+- Warning/failure category
+- Manual validation status
+- External provider/analyzer, if any
+
+The bundled batch manifest provides a baseline for these records.

package/{dist/src/orgrt/role-skills/mcp-builder.md → org-skills/mcp-builder/SKILL.md}

@@ -1,3 +1,11 @@
+---
+name: mcp-builder
+description: "MCP Builder role best practices. Designs and builds Model Context Protocol servers — custom tools, resources, and prompts that extend what an AI agent can actually do."
+tags: ["engineering"]
+tools: ["monograph_query","monograph_context","monograph_impact"]
+license: Apache-2.0
+source: https://github.com/monoes/monomind
+---
 # MCP Builder — Best Practices
 
 ## Focus