rbtr-lang-markdown 2026.9.0.dev1__tar.gz

rbtr_lang_markdown-2026.9.0.dev1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alejandro Giacometti
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rbtr_lang_markdown-2026.9.0.dev1/PKG-INFO

@@ -0,0 +1,79 @@
+Metadata-Version: 2.4
+Name: rbtr-lang-markdown
+Version: 2026.9.0.dev1
+Summary: rbtr — Markdown language plugin
+Keywords: code-search,code-index,tree-sitter,static-analysis,semantic-search,developer-tools,markdown
+Author: Alejandro Giacometti
+Author-email: Alejandro Giacometti <alejandro.giacometti@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Typing :: Typed
+Requires-Dist: rbtr==2026.9.0.dev1
+Requires-Dist: tree-sitter-markdown
+Requires-Python: >=3.13
+Project-URL: Homepage, https://github.com/janrito/rbtr
+Project-URL: Repository, https://github.com/janrito/rbtr
+Project-URL: Documentation, https://github.com/janrito/rbtr/tree/main/packages/rbtr-lang-markdown#readme
+Project-URL: Issues, https://github.com/janrito/rbtr/issues
+Project-URL: Changelog, https://github.com/janrito/rbtr/releases
+Description-Content-Type: text/markdown
+
+# rbtr-lang-markdown
+
+Markdown support for [rbtr]. A **default** plugin — installed with
+rbtr itself (`pip install rbtr`).
+
+[rbtr]: https://github.com/janrito/rbtr/tree/main/packages/rbtr#readme
+
+## What it ingests
+
+A custom chunker (not a tree-sitter query): it splits a document by its
+**heading hierarchy** — each section is a chunk, scoped by its ancestor
+headings. Links and fenced code blocks are extracted on top.
+
+- **Sections** — headings → doc sections; `scope` is the `::` chain of
+  enclosing headings. A headingless document falls back to plaintext chunks.
+- **Links** — local `[text](path.md)` and `[text](path.md#fragment)` →
+  imports (the fragment becomes `names`). External and fragment-only links
+  are skipped.
+- **Fenced code** — delegated to the block's language (see below).
+
+## Chunks produced
+
+```markdown
+# Guide                    <!-- doc_section "Guide"                       -->
+## Setup                   <!-- doc_section "Setup", scope "Guide"         -->
+See [the API](api.md#run). <!-- import, metadata {module: api.md, names: run} -->
+```
+
+## Embedded / injected chunks
+
+A fenced block delegates to its language, extracted at real line numbers —
+and delegation **recurses**, so a block that itself embeds another language
+resolves all the way down (markdown → html → js):
+
+````markdown
+```yaml
+service: greeter       <!-- doc_section "service" (yaml)               -->
+```
+```html
+<main>                 <!-- doc_section "main"    (html)               -->
+  <script>
+    function boot() {} <!-- function "boot"      (javascript, nested)  -->
+  </script>
+</main>
+```
+````
+
+## Grammar & dependencies
+
+Uses the `tree-sitter-markdown` grammar. No runtime dependency on other
+language plugins; the test suite dev-depends on the plugins its sample's
+fenced blocks delegate to (bash, css, html, javascript, python, scss — plus
+svelte/toml/yaml via core until those are packaged).

rbtr_lang_markdown-2026.9.0.dev1/README.md

@@ -0,0 +1,53 @@
+# rbtr-lang-markdown
+
+Markdown support for [rbtr]. A **default** plugin — installed with
+rbtr itself (`pip install rbtr`).
+
+[rbtr]: https://github.com/janrito/rbtr/tree/main/packages/rbtr#readme
+
+## What it ingests
+
+A custom chunker (not a tree-sitter query): it splits a document by its
+**heading hierarchy** — each section is a chunk, scoped by its ancestor
+headings. Links and fenced code blocks are extracted on top.
+
+- **Sections** — headings → doc sections; `scope` is the `::` chain of
+  enclosing headings. A headingless document falls back to plaintext chunks.
+- **Links** — local `[text](path.md)` and `[text](path.md#fragment)` →
+  imports (the fragment becomes `names`). External and fragment-only links
+  are skipped.
+- **Fenced code** — delegated to the block's language (see below).
+
+## Chunks produced
+
+```markdown
+# Guide                    <!-- doc_section "Guide"                       -->
+## Setup                   <!-- doc_section "Setup", scope "Guide"         -->
+See [the API](api.md#run). <!-- import, metadata {module: api.md, names: run} -->
+```
+
+## Embedded / injected chunks
+
+A fenced block delegates to its language, extracted at real line numbers —
+and delegation **recurses**, so a block that itself embeds another language
+resolves all the way down (markdown → html → js):
+
+````markdown
+```yaml
+service: greeter       <!-- doc_section "service" (yaml)               -->
+```
+```html
+<main>                 <!-- doc_section "main"    (html)               -->
+  <script>
+    function boot() {} <!-- function "boot"      (javascript, nested)  -->
+  </script>
+</main>
+```
+````
+
+## Grammar & dependencies
+
+Uses the `tree-sitter-markdown` grammar. No runtime dependency on other
+language plugins; the test suite dev-depends on the plugins its sample's
+fenced blocks delegate to (bash, css, html, javascript, python, scss — plus
+svelte/toml/yaml via core until those are packaged).

rbtr_lang_markdown-2026.9.0.dev1/pyproject.toml

@@ -0,0 +1,95 @@
+[build-system]
+requires = ["uv_build>=0.11.26,<1"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-name = "rbtr_lang_markdown"
+
+[tool.uv.sources.rbtr]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-bash]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-css]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-html]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-javascript]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-python]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-scss]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-svelte]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-toml]
+workspace = true
+
+[tool.uv.sources.rbtr-lang-yaml]
+workspace = true
+
+[project]
+name = "rbtr-lang-markdown"
+version = "2026.9.0-dev1"
+description = "rbtr — Markdown language plugin"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = [
+    "code-search",
+    "code-index",
+    "tree-sitter",
+    "static-analysis",
+    "semantic-search",
+    "developer-tools",
+    "markdown",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Software Development :: Libraries",
+    "Topic :: Text Processing :: Indexing",
+    "Typing :: Typed",
+]
+requires-python = ">=3.13"
+dependencies = [
+    "rbtr==2026.9.0-dev1",
+    "tree-sitter-markdown",
+]
+
+[[project.authors]]
+name = "Alejandro Giacometti"
+email = "alejandro.giacometti@gmail.com"
+
+[project.urls]
+Homepage = "https://github.com/janrito/rbtr"
+Repository = "https://github.com/janrito/rbtr"
+Documentation = "https://github.com/janrito/rbtr/tree/main/packages/rbtr-lang-markdown#readme"
+Issues = "https://github.com/janrito/rbtr/issues"
+Changelog = "https://github.com/janrito/rbtr/releases"
+
+[project.entry-points."rbtr.languages"]
+markdown = "rbtr_lang_markdown.plugin:markdown"
+
+[dependency-groups]
+dev = [
+    "rbtr[test]",
+    "rbtr-lang-bash",
+    "rbtr-lang-css",
+    "rbtr-lang-html",
+    "rbtr-lang-javascript",
+    "rbtr-lang-python",
+    "rbtr-lang-scss",
+    "rbtr-lang-svelte",
+    "rbtr-lang-toml",
+    "rbtr-lang-yaml",
+]

rbtr_lang_markdown-2026.9.0.dev1/pyproject.toml.orig

@@ -0,0 +1,84 @@
+[build-system]
+requires = ["uv_build>=0.11.26,<1"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-name = "rbtr_lang_markdown"
+# No source-exclude: the plugin's tests, samples, and snapshots ship with
+# the wheel (they are the language's golden record), and the .scm queries
+# ship as package data.
+
+[project]
+name = "rbtr-lang-markdown"
+version = "2026.9.0-dev1"
+description = "rbtr — Markdown language plugin"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+  { name = "Alejandro Giacometti", email = "alejandro.giacometti@gmail.com" },
+]
+keywords = [
+  "code-search",
+  "code-index",
+  "tree-sitter",
+  "static-analysis",
+  "semantic-search",
+  "developer-tools",
+  "markdown",
+]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Software Development :: Libraries",
+  "Topic :: Text Processing :: Indexing",
+  "Typing :: Typed",
+]
+requires-python = ">=3.13"
+dependencies = [
+  "rbtr==2026.9.0-dev1",
+  "tree-sitter-markdown",
+]
+
+[project.urls]
+Homepage = "https://github.com/janrito/rbtr"
+Repository = "https://github.com/janrito/rbtr"
+Documentation = "https://github.com/janrito/rbtr/tree/main/packages/rbtr-lang-markdown#readme"
+Issues = "https://github.com/janrito/rbtr/issues"
+Changelog = "https://github.com/janrito/rbtr/releases"
+
+[project.entry-points."rbtr.languages"]
+# The value resolves to the language's `LanguageRegistration` (named by its
+# id) — see the rbtr ARCHITECTURE "External plugins" note.
+markdown = "rbtr_lang_markdown.plugin:markdown"
+
+[tool.uv.sources]
+rbtr = { workspace = true }
+# The sample's fenced code blocks delegate to these (plus svelte/toml/yaml,
+# which are still in core and come via `rbtr` until they are packaged — add
+# them here when they are).
+rbtr-lang-bash = { workspace = true }
+rbtr-lang-css = { workspace = true }
+rbtr-lang-html = { workspace = true }
+rbtr-lang-javascript = { workspace = true }
+rbtr-lang-python = { workspace = true }
+rbtr-lang-scss = { workspace = true }
+rbtr-lang-svelte = { workspace = true }
+rbtr-lang-toml = { workspace = true }
+rbtr-lang-yaml = { workspace = true }
+
+[dependency-groups]
+dev = [
+  "rbtr[test]",
+  "rbtr-lang-bash",
+  "rbtr-lang-css",
+  "rbtr-lang-html",
+  "rbtr-lang-javascript",
+  "rbtr-lang-python",
+  "rbtr-lang-scss",
+  "rbtr-lang-svelte",
+  "rbtr-lang-toml",
+  "rbtr-lang-yaml",
+]

rbtr_lang_markdown-2026.9.0.dev1/src/rbtr_lang_markdown/__init__.py

@@ -0,0 +1 @@
+"""Markdown language plugin package."""

rbtr_lang_markdown-2026.9.0.dev1/src/rbtr_lang_markdown/injections.scm

@@ -0,0 +1,4 @@
+; Fenced code blocks: the info string names the embedded language.
+(fenced_code_block
+  (info_string (language) @injection.language)
+  (code_fence_content) @injection.content)

rbtr_lang_markdown-2026.9.0.dev1/src/rbtr_lang_markdown/links.scm

@@ -0,0 +1 @@
+(inline_link (link_destination) @dest)

rbtr_lang_markdown-2026.9.0.dev1/src/rbtr_lang_markdown/plugin.py

@@ -0,0 +1,258 @@
+"""Markdown language plugin.
+
+Splits Markdown by heading hierarchy using tree-sitter.  Headed
+sections produce one `doc_section` chunk per heading, containing
+the heading and its direct content (excluding nested subsections).
+Headingless documents fall back to `chunk_plaintext`.
+
+Extracted chunks::
+
+    # Title                         → doc_section "Title", scope ""
+    Intro text.                       (content: heading + intro)
+
+    ## Section A                    → doc_section "Section A", scope "Title"
+    Body A.                           (content: heading + body, excludes
+                                       child sections)
+
+    First paragraph.                → raw_chunk (plaintext fallback)
+    Second paragraph.               → raw_chunk
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from typing import TYPE_CHECKING
+
+from tree_sitter import Language, Parser, Query, QueryCursor
+
+from rbtr.domain.models import Chunk, ChunkKind, ImportMeta
+from rbtr.languages.chunks import chunk_plaintext, last_line
+from rbtr.languages.registration import LanguageRegistration, load_query
+
+if TYPE_CHECKING:
+    from tree_sitter import Node, Range
+
+# URL schemes that indicate external links (not cross-references).
+_EXTERNAL_SCHEMES = ("http://", "https://", "mailto:", "ftp://")
+
+# Block-level query: captures headed sections.
+_SECTION_QUERY = load_query(__package__, "sections")
+
+# Injection query: a fenced code block delegates its content to the language
+# named in the fence's info string (```python, ```sh). The info-string name
+# is captured dynamically and resolved to a language id by the runner.
+
+
+# ── Chunking ─────────────────────────────────────────────────────────
+
+
+def _has_headings(root: Node) -> bool:
+    """Return True if any section in the tree has an atx_heading."""
+    for child in root.children:
+        if child.type == "section":
+            if any(gc.type == "atx_heading" for gc in child.children):
+                return True
+            if _has_headings(child):
+                return True
+    return False
+
+
+def _section_depth(node: Node) -> int:
+    """Count ancestor nodes of the same type.
+
+    In the tree-sitter markdown grammar, sections nest:
+    a `## Sub` section is a child `section` node of the
+    `# Top` section.  The depth (number of same-typed
+    ancestors) maps directly to the heading level minus
+    one: depth 0 = `#`, depth 1 = `##`, etc.
+    """
+    depth = 0
+    ancestor = node.parent
+    while ancestor is not None:
+        if ancestor.type == node.type:
+            depth += 1
+        ancestor = ancestor.parent
+    return depth
+
+
+def _section_own_content(node: Node, content_bytes: bytes) -> str:
+    """Return the section's own text, excluding nested subsections.
+
+    A parent section's byte span covers its children.  Trimming
+    at the first child of the same type gives us only the
+    heading and the prose that belongs directly to this section.
+    """
+    end_byte = node.end_byte
+    for child in node.children:
+        if child.type == node.type:
+            end_byte = child.start_byte
+            break
+    return content_bytes[node.start_byte : end_byte].decode("utf-8", errors="replace").strip()
+
+
+def _extract_sections(
+    content_bytes: bytes,
+    grammar: Language,
+    file_path: str,
+    blob_sha: str,
+    ranges: list[Range] | None,
+) -> Iterator[Chunk]:
+    """Run the section query and yield one chunk per headed section.
+
+    The tree-sitter markdown grammar represents headings as
+    nested `section` nodes.  This function queries for all
+    sections that contain an `atx_heading` and produces one
+    `doc_section` chunk per match.
+
+    **Scope** is the parent heading chain, built by maintaining
+    a stack of heading names indexed by depth.  When a section
+    at depth *d* is encountered, the stack is truncated to *d*
+    entries (popping deeper headings from a previous branch),
+    the current scope is read from the remaining stack, and
+    the new heading is pushed.
+
+    **Content** is trimmed to exclude nested subsections — see
+    `_section_own_content`.
+    """
+    query = Query(grammar, _SECTION_QUERY)
+    parser = Parser(grammar)
+    if ranges is not None:
+        parser.included_ranges = ranges
+    tree = parser.parse(content_bytes)
+    matches = QueryCursor(query).matches(tree.root_node)
+
+    # Heading names indexed by depth.  Tracks the current
+    # branch of the heading tree so we can reconstruct the
+    # scope ("Top::Mid") for each section.
+    scope_stack: list[str] = []
+
+    for _pattern_idx, capture_dict in matches:
+        section_nodes = capture_dict.get("doc_section", [])
+        name_nodes = capture_dict.get("_section_name", [])
+        if not section_nodes or not name_nodes:
+            continue
+
+        node = section_nodes[0]
+        name = name_nodes[0].text.decode("utf-8", errors="replace") if name_nodes[0].text else ""
+
+        # Maintain the scope stack: pop back to this section's
+        # depth, read scope, then push the current heading.
+        depth = _section_depth(node)
+        while len(scope_stack) > depth:
+            scope_stack.pop()
+        scope_segments = list(scope_stack)
+        scope_stack.append(name)
+
+        text = _section_own_content(node, content_bytes)
+        if not text:
+            continue
+
+        line_start = node.start_point[0] + 1
+        yield Chunk.model_validate(
+            {
+                "blob_sha": blob_sha,
+                "file_path": file_path,
+                "kind": ChunkKind.DOC_SECTION,
+                "name": name,
+                "scope": scope_segments,
+                "language": "markdown",
+                "content": text,
+                "line_start": line_start,
+                "line_end": last_line(node),
+            }
+        )
+
+
+def chunk_markdown(
+    file_path: str,
+    blob_sha: str,
+    content: str,
+    grammar: Language,
+    ranges: list[Range] | None = None,
+) -> Iterator[Chunk]:
+    """Split Markdown by heading hierarchy using tree-sitter."""
+    if not content.strip():
+        return
+
+    content_bytes = content.encode("utf-8")
+    parser = Parser(grammar)
+    if ranges is not None:
+        parser.included_ranges = ranges
+    tree = parser.parse(content_bytes)
+
+    if _has_headings(tree.root_node):
+        yield from _extract_sections(content_bytes, grammar, file_path, blob_sha, ranges)
+    else:
+        yield from chunk_plaintext(file_path, blob_sha, content)
+
+    # Extract local links as IMPORT chunks using the inline parser.
+    yield from _extract_links(content_bytes, file_path, blob_sha, ranges)
+
+
+# ── Link extraction ──────────────────────────────────────────────────
+
+_LINK_QUERY = load_query(__package__, "links")
+
+
+def _extract_links(
+    content_bytes: bytes,
+    file_path: str,
+    blob_sha: str,
+    ranges: list[Range] | None,
+) -> Iterator[Chunk]:
+    """Extract local links as IMPORT chunks using the inline parser.
+
+    Parses the full content with `tree_sitter_markdown.inline_language()`
+    and queries for `inline_link` nodes.  External URLs and same-file
+    fragment-only links are skipped.
+    """
+    import tree_sitter_markdown  # deferred: heavy native lib
+
+    inline_lang = Language(tree_sitter_markdown.inline_language())
+    inline_parser = Parser(inline_lang)
+    if ranges is not None:
+        inline_parser.included_ranges = ranges
+    inline_tree = inline_parser.parse(content_bytes)
+
+    query = Query(inline_lang, _LINK_QUERY)
+    for _pattern_idx, captures in QueryCursor(query).matches(inline_tree.root_node):
+        for dest_node in captures.get("dest", []):
+            if dest_node.text is None:
+                continue
+            dest = dest_node.text.decode("utf-8", errors="replace")
+
+            # Skip external URLs and fragment-only anchors.
+            if any(dest.startswith(s) for s in _EXTERNAL_SCHEMES) or dest.startswith("#"):
+                continue
+
+            # Split path#fragment.
+            module = dest
+            names = ""
+            if "#" in dest:
+                module, names = dest.rsplit("#", 1)
+
+            yield Chunk(
+                blob_sha=blob_sha,
+                file_path=file_path,
+                kind=ChunkKind.IMPORT,
+                name=dest,
+                scope="",
+                content=dest,
+                metadata=ImportMeta(module=module, names=names),
+                line_start=dest_node.start_point[0] + 1,
+                line_end=last_line(dest_node),
+            )
+
+
+# ── Plugin ───────────────────────────────────────────────────────────
+
+
+markdown = LanguageRegistration(
+    id="markdown",
+    extensions=frozenset({".md"}),
+    grammar_module="tree_sitter_markdown",
+    injection_query=load_query(__package__, "injections"),
+    extraction_serial=5,
+)
+
+markdown.chunker(chunk_markdown)

rbtr_lang_markdown-2026.9.0.dev1/src/rbtr_lang_markdown/sections.scm

@@ -0,0 +1,3 @@
+(section
+  (atx_heading
+    (inline) @_section_name)) @doc_section

rbtr_lang_markdown-2026.9.0.dev1/src/rbtr_lang_markdown/tests/__snapshots__/test_samples/test_edges_match_snapshot.json

@@ -0,0 +1,8 @@
+[
+  "guides/setup.md::troubleshooting.md -> guides/troubleshooting.md::A step failed [documents]",
+  "guides/setup.md::troubleshooting.md -> guides/troubleshooting.md::Troubleshooting [documents]",
+  "markdown.md::api.md#format-greeting -> api.md::format-greeting [documents]",
+  "markdown.md::config.md -> config.md::Configuration [documents]",
+  "markdown.md::guides/setup.md -> guides/setup.md::Setup [documents]",
+  "markdown.md::locales.md -> locales.md::Locales [documents]"
+]