asciisourcerer 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: abee321c7941006227f85f6a006a9b6ea795f8bcfd0e5c3a5837b11400e43a23
-  data.tar.gz: 92dc5d0bfb4435f1d1a42a558ed1aa975d297aff3201a4cd40266da21e86f816
+  metadata.gz: 758bb9f8f554228a7d67b894471d79d966b0e6495b7b2da2854dcdfb18544e40
+  data.tar.gz: c82bb9ba5cf1391d78feb56682c470c6ded9d49f0d469678eb27801e1e30e9af
 SHA512:
-  metadata.gz: 70c932c2a7875d27cec949952bc413dce9c240fe19f8fabbcd756bfa1c94d970d0d9d217538d68c2fb7325b4040dc9cae4ad71fa4c595ab89a8ffa6b650aab1d
-  data.tar.gz: 39b7c3e368ed5149012f4e07d6c33809d6b11e5e82e75f8562ab5edae9fde98b32928407801cdd6941951d375ac488c008b2857d3378a2115c4767b398194f1a
+  metadata.gz: d64e160a0e2ddc95e8114781c88e8f1a25cfaad689c43fd121e4bcae2c68b3e6bd8d5bc332464e496938b5d1e8cd769ce90ea74943c3d29edea593c560793d9b
+  data.tar.gz: 3a3a656576871fe4fdb3b4927d8cd3ed5b094f8f959ecbd2ccaf2ab4db16551ec6a136948526d137ff43b66ca043b808d935c8a22babd4bbd39db38811c2565e

data/README.adoc

@@ -37,11 +37,11 @@ endif::[]
 :this_prod_name: {this_proj_name}
 // end::universal-settings[]
 :this_prod_vrsn_major: 0
-:this_prod_vrsn_minor: 2
+:this_prod_vrsn_minor: 3
 :this_prod_vrsn_majmin: {this_prod_vrsn_major}.{this_prod_vrsn_minor}
-:this_prod_vrsn_patch: 1
+:this_prod_vrsn_patch: 0
 :this_prod_vrsn: {this_prod_vrsn_majmin}.{this_prod_vrsn_patch}
-:next_prod_vrsn: 0.3.0
+:next_prod_vrsn: 0.4.0
 // end::global-settings[]
 :toc: macro
 :toclevels: 4
@@ -50,6 +50,7 @@ endif::[]
 
 AsciiSourcerer is a Ruby library for radical single-sourcing of documentation and product data, primarily from AsciiDoc, YAML, and Liquid templating operations.
 
+
 [[intro]]
 == Introduction
 
@@ -98,7 +99,7 @@ AsciiDoc-to-Markdown conversion::
 Convert AsciiDoc documents to Markdown for agentic consumption, with a focus on preserving semantic structure and document frontmatter.
 See <<markdowngrade>>.
 
-AsciiDoc source inspection::
+AsciiDoc and Markdown source inspection::
 Skim AsciiDoc documents to produce machine-oriented outlines of sections, code blocks, tables, and other semantic elements for tooling and agent consumption.
 See <<source-skim>>.
 
@@ -299,20 +300,21 @@ Override it per call with the `canonical_prefix:` keyword argument.
 Canonical blocks use AsciiDoc-style `tag::`/`end::` markers embedded inside comments.
 Three comment styles are recognized so the same prime can manage files of any type:
 
-[cols="3m,5m",options="header"]
+[cols="3,5m",options="header"]
 |===
 | Style | Example
-| HTML / Markdown comment
-| <!-- tag::universal-intro[] -->
-| AsciiDoc line comment
+| HTML / Markdown
+| <!-- tag::universal-intro[] +++-->+++
+| AsciiDoc / JavaScript
 | // tag::universal-intro[]
-| Shell / Ruby / YAML comment
+| Shell / Ruby / YAML / INI
 | # tag::universal-intro[]
 |===
 
-The trailing `[]` is optional in all three styles.
+A block with tag names beginning with the configured prefix (default `universal-`) is treated as canonical and managed by Sync/Cast.
+
+Any block wrapped in `tag::_skip[]` and `end::_skip[]` will _not_ be passed from prime to target even during the init operation.
 
-A tag whose name begins with the configured prefix (default `universal-`) is treated as canonical and managed by Sync/Cast.
 All other tagged regions in a target file are left entirely alone, and their presence beside a canonical block never triggers a warning.
 
 [[sync-cast-operations]]
@@ -321,6 +323,8 @@ All other tagged regions in a target file are left entirely alone, and their pre
 init::
 One-time operation that renders the prime template (the whole file, not just canonical blocks) through Liquid and writes the result as a new target file.
 Use this to bootstrap a repo-local copy of a project template.
++
+NOTE: Only blocks marked `_skip` are not carried over.
 
 sync::
 Ongoing operation that scans for canonical blocks, replaces their content with the prime version (after optional Liquid rendering), and leaves everything else verbatim.
@@ -332,7 +336,8 @@ The `result.diff` file contains the rendered content (`init`) or a unified diff
 [NOTE]
 ====
 If a canonical block is absent from a target file, `sync` emits a warning.
-That warning is suppressed when the target contains a non-canonical tag sharing the same suffix — for example, `local-intro` alongside canonical `universal-intro` — indicating a deliberate project-local override.
+That warning is suppressed when the target contains a non-canonical tag sharing the same suffix indicating a deliberate project-local override.
+Example: `local-intro` alongside canonical `universal-intro.
 ====
 
 [[sync-cast-liquid]]
@@ -352,6 +357,18 @@ This is {{ data.variables.name }} version {{ data.variables.version }}.
 <!-- end::universal-intro[] -->
 ----
 
+Since each block is rendered independently, variables cannot reference content from other blocks.
+Use a segment wrapped in `_liquid` tags to set variables inline that will be used across all subsequent blocks.
+This `_liquid` segment only has access to variables passed into the main render call, not to any content blocks.
+
+[source,markdown]
+----
+<!-- tag::_liquid -->
+{%- assign name = data.variables.this_proj_name | default: 'Unknown Project' %}
+{%- assign version = data.variables.this_prod_vrsn | default: '0.0.0' %}
+<!-- end::_liquid -->
+----
+
 [[sync-cast-bootstrap]]
 ==== Bootstrap example
 
@@ -360,7 +377,7 @@ This is {{ data.variables.name }} version {{ data.variables.version }}.
 Sourcerer::Sync.init(
   'templates/AGENTS.markdown',
   'AGENTS.md',
-  data: { 'project' => 'my-gem', 'org' => 'DocOps' }
+  data: { 'project' => 'my-gem', 'org' => 'ACME Co' }
 )
 ----
 
@@ -370,7 +387,7 @@ Pass `dry_run: true` to return the rendered content in `result.diff` without wri
 [[templating-liquid-runtime]]
 === Templating and Liquid Runtime
 
-Sourcerer supports Liquid and ERB, but its Liquid support is intentionally aligned with Jekyll’s runtime.
+Sourcerer supports Liquid and ERB, but its Liquid support is intentionally aligned with Jekyll's runtime.
 This keeps template behavior consistent with Jekyll projects while allowing DocOps Lab to register filters and tags.
 
 If you are building a Jekyll-compatible templating pipeline, prefer Liquid.
@@ -379,7 +396,7 @@ If you want a low-friction Ruby template for internal tooling, ERB is available.
 [[pipelines]]
 === Prebuild and Rendering Pipelines
 
-Sourcerer’s rendering pipeline is optimized for build tooling and prebuild steps.
+Sourcerer's rendering pipeline is optimized for build tooling and prebuild steps.
 A typical prebuild might load attributes from `README.adoc`, extract tagged snippets into `build/snippets/`, and render YAML plus Liquid templates into `build/docs/`.
 
 The API is intentionally small.
@@ -419,10 +436,14 @@ Schema-aware filters (including SGYML-specific classification filters) should be
 [[source-skim]]
 === SourceSkim
 
-`Sourcerer::SourceSkim` generates machine-oriented _skims_ of AsciiDoc source documents.
+`Sourcerer::SourceSkim` generates machine-oriented _skims_ of AsciiDoc and Markdown source documents.
 
-A skim is a structured, JSON/YAML-ready outline of selected source elements:
+A skim is a structured, JSON/YAML-ready outline of selected source elements.
+AsciiDoc skims include:
 sections, code blocks, definition lists, tables, images, and more.
+
+Markdown skims include only sections and frontmatter, since other semantic elements are not reliably parseable in Markdown's freeform syntax.
+
 Skims are intended to help tooling and agents inspect documentation source without ingesting full file contents.
 
 .Example: Skim a file for sections and code blocks
@@ -439,6 +460,9 @@ skim = Sourcerer::SourceSkim.skim_file('docs/install.adoc', categories: [:code_b
 
 # Skim inline content
 skim = Sourcerer::SourceSkim.skim_string(raw_adoc_content)
+
+# Skim Markdown file with flat sections only
+skim = Sourcerer::SourceSkim.skim_file('README.md', forms: [:flat])
 ----
 
 Section shapes::
@@ -446,6 +470,7 @@ Pass `forms: [:tree]` (default), `forms: [:flat]`, or both.
 Tree shape preserves nesting; flat shape adds `parent_id` and expresses child section IDs as an array.
 
 Categories::
+(AsciiDoc skims only.)
 Default output includes `attributes_custom`, `definition_lists`, `code_blocks`, `literal_blocks`, `examples`, `sidebars`, `tables`, and `images`.
 Opt-in only (excluded by default): `attributes_builtin`, `admonitions`, `quotes`.
 Pass `categories:` with an explicit array of symbols to restrict or expand output.
@@ -455,6 +480,7 @@ The skim output schema is at `specs/data/asciidoc-source-skim.schema.json` and a
 
 [[skim-asciidoctor-extension]]
 ==== Asciidoctor extension
+
 `Sourcerer::SourceSkim::TreeProcessorExtension` integrates SourceSkim into Asciidoctor parsing pipelines.
 It stores the result as the `source-skim-result` document attribute and reads `source-skim-forms` and `source-skim-categories` document attributes for per-document configuration.
 
@@ -584,7 +610,7 @@ r.enum.to_a  # => []
 == Integrations (Implemented and Planned)
 
 These notes describe how AsciiSourcerer relates to existing and future downstream tools.
-For most users, these tools are the recommended way to access AsciiSourcerer’s capabilities, since they provide a richer context for configuration, input/output management, and workflow orchestration.
+For most users, these tools are the recommended way to access AsciiSourcerer's capabilities, since they provide a richer context for configuration, input/output management, and workflow orchestration.
 
 https://github.com/DocOps/lab/tree/main/gems/docopslab-dev[docopslab-dev]::
 A harness for DocOps Lab developer tasks and operations.
@@ -608,24 +634,61 @@ All the benefits of extended YAML ingest and Liquid processing are available via
 === Alpha Scripts/CLIs
 
 The AsciiSourcerer repo does host some CLI utilities and scripts that make direct usage of APIs hosted in the `asciisourcerer` gem.
-These are largely prototype utilities that will eventually make it into official DocOps Lab applications with proper I/O surfaces.
+
+These are largely prototype utilities that may eventually make it into official DocOps Lab applications with proper I/O surfaces.
 
 [WARNING]
 These scripts are *neither* rigorously tested *nor* officially supported.
 They may be altered, deprecated, or dropped in backward-incompatible ways.
-Use them as examples but do not rely on them in production.
+Use them as examples or utilities but do not rely on them in production.
 
-To use such a script, copy it from the repo's `scripts/` directory into your project, then run it with `ruby` or `bundle exec ruby` as appropriate.
+Current scripts include: ::
 
-Current scripts include::
-`skim_asciidoc.rb`:::
-For AsciiDoc source inspection.
-Generates YAML or JSON versions of `Sourcerer::SourceSkim` output for a given AsciiDoc document or collection of documents.
+link:https://github.com/DocOps/asciisourcerer/blob/main/scripts/skim_markup.rb[`skim_markup.rb`]:::
+(Formerly `skim_asciidoc.rb`.)
+For AsciiDoc/Markdown source inspection.
+Generates YAML or JSON versions of `Sourcerer::SourceSkim` output for a given AsciiDoc document, Markdown file, or collection of such.
 
-`mark_down_grade.rb`:::
+link:https://github.com/DocOps/asciisourcerer/blob/main/scripts/mark_down_grade.rb[`mark_down_grade.rb`]:::
 For AsciiDoc-to-Markdown conversion.
 This script orchestrates `Sourcerer::AsciiDoc.mark_down_grade` and is a recommended starting point for users needing this functionality without building their own Ruby integration.
 
+To use one of these ad hock CLIs, copy it from the repo's `scripts/` directory into your project, then run it with `ruby` or `bundle exec ruby` as appropriate.
+
+[[usage-example]]
+==== Setup / Usage Example
+
+For these scripts to work, the `asciisourcerer` gem must be available.
+
+. Add `asciisourcerer` to (or create) `Gemfile` in the root of a project repo.
++
+[source,ruby]
+----
+source 'https://rubygems.org'
+gem 'asciisourcerer'
+----
+
+. Install the library and dependencies.
++
+[.prompt]
+ bundle install
+
+For each script you wish to use:
+
+[start=3]
+. Download the script you want.
++
+.Example: writes current GH copy to a local `scripts/` path.
+[.prompt]
+ curl -o scripts/skim_markup.rb https://raw.githubusercontent.com/DocOps/asciisourcerer/refs/heads/main/scripts/skim_markup.rb
++
+Change `-o scripts/` to wherever you wish to save the script locally.
+
+. Run the script.
++
+[.prompt]
+ bundle exec ruby scripts/skim_markup.rb --help
+
 
 [[development]]
 == Development
@@ -729,14 +792,15 @@ Standard release flow:
 
 . Run `./scripts/build.sh` to validate the environment, run tests, and build the gem file to `pkg/`.
 
-. Publish to RubyGems. 
-+
- RUBYGEMS_API_KEY=<rubygems.org key> ./scripts/publish.sh
-
 . Tag the release in Git:
 +
 [.prompt,subs=+attributes]
- git tag v{this_prod_vrsn} && git push origin v{this_prod_vrsn}`.
+ git tag v{this_prod_vrsn}
+ git push origin v{this_prod_vrsn}
+
+. Publish to RubyGems. 
++
+ RUBYGEMS_API_KEY=<rubygems.org key> ./scripts/publish.sh
 
 [NOTE]
 AsciiSourcerer does not yet publish a release history document.

data/lib/sourcerer/asciidoc.rb

@@ -4,6 +4,7 @@ require 'asciidoctor'
 require 'fileutils'
 require 'yaml'
 require 'cgi'
+require_relative 'yaml_frontmatter'
 
 module Sourcerer
   # AsciiDoc-focused primitives for attribute loading, region extraction,
@@ -35,7 +36,7 @@ module Sourcerer
   #
   # @see https://asciidoctor.org/ Asciidoctor Documentation
   module AsciiDoc
-    YAML_FRONTMATTER_REGEXP = /\A(---\s*\n.*?\n)(---\s*\n?)/m
+    YAML_FRONTMATTER_REGEXP = Sourcerer::YamlFrontmatter::REGEXP
     YAML_FRONT_MATTER_REGEXP = YAML_FRONTMATTER_REGEXP
     PAGE_ATTRIBUTE_PREFIX = 'page-'
 
@@ -327,14 +328,7 @@ module Sourcerer
     # @param source_text [String]
     # @return [Hash]
     def self.extract_yaml_frontmatter source_text
-      match = source_text.match(YAML_FRONTMATTER_REGEXP)
-      return {} unless match
-
-      frontmatter_payload = match[1].sub(/\A---\s*\n/, '')
-      parsed = YAML.safe_load(frontmatter_payload, aliases: true)
-      parsed.is_a?(Hash) ? parsed : {}
-    rescue Psych::SyntaxError
-      {}
+      Sourcerer::YamlFrontmatter.extract(source_text)
     end
 
     # Remove leading YAML front matter fence block from AsciiDoc source.
@@ -342,7 +336,7 @@ module Sourcerer
     # @param source_text [String]
     # @return [String]
     def self.strip_yaml_frontmatter source_text
-      source_text.sub(YAML_FRONTMATTER_REGEXP, '')
+      Sourcerer::YamlFrontmatter.strip(source_text)
     end
 
     # Compatibility alias.
@@ -483,5 +477,65 @@ module Sourcerer
                          :normalize_extract_tags,
                          :collect_tagged_content,
                          :normalize_mark_down_grade_options
+
+    # Utilities for filtering and partitioning Asciidoctor document attributes.
+    #
+    # Separates user-defined ("custom") attributes from those injected by
+    # Asciidoctor at parse time ("built-in").
+    #
+    # @example
+    #   custom  = Sourcerer::AsciiDoc::AttributesFilter.user_attributes(doc)
+    #   builtin = Sourcerer::AsciiDoc::AttributesFilter.builtin_attributes(doc)
+    module AttributesFilter
+      # Attribute keys injected by Asciidoctor at parse time.
+      BUILTIN_ATTR_KEYS = (Asciidoctor::DEFAULT_ATTRIBUTES.keys + %w[
+        asciidoctor asciidoctor-version
+        attribute-missing attribute-undefined
+        authorcount
+        docdate docdatetime docdir docfile docfilesuffix docname doctime doctitle doctype docyear
+        embedded
+        htmlsyntax
+        iconsdir
+        localdate localdatetime localtime localyear
+        max-include-depth
+        notitle
+        outfilesuffix
+        stylesdir
+        toc-position
+        user-home
+      ]).freeze
+
+      BUILTIN_ATTR_PATTERNS = [
+        /^backend(-|$)/,
+        /^basebackend(-|$)/,
+        /^doctype-/,
+        /^filetype(-|$)/,
+        /^safe-mode-/
+      ].freeze
+
+      module_function
+
+      # Returns user-defined attributes, excluding Asciidoctor built-ins.
+      #
+      # @param doc [Asciidoctor::Document]
+      # @return [Hash{String => String}]
+      def user_attributes doc
+        doc.attributes.reject do |k, _|
+          BUILTIN_ATTR_KEYS.include?(k) ||
+            BUILTIN_ATTR_PATTERNS.any? { |pat| pat.match?(k) }
+        end
+      end
+
+      # Returns built-in Asciidoctor attributes injected at parse time.
+      #
+      # @param doc [Asciidoctor::Document]
+      # @return [Hash{String => String}]
+      def builtin_attributes doc
+        doc.attributes.select do |k, _|
+          BUILTIN_ATTR_KEYS.include?(k) ||
+            BUILTIN_ATTR_PATTERNS.any? { |pat| pat.match?(k) }
+        end
+      end
+    end
   end
 end

data/lib/sourcerer/source_skim/markdown_skimmer.rb

@@ -0,0 +1,108 @@
+# frozen_string_literal: true
+
+module Sourcerer
+  module SourceSkim
+    # Parses Markdown content and produces a JSON-ready skim hash.
+    #
+    # Heading levels are mapped to mirror AsciiDoc document structure:
+    # a single +#+ heading becomes the document title (level 0); all subsequent
+    # +#++#+ headings become sections starting at level 1. This keeps Markdown
+    # and AsciiDoc skim output shapes consistent.
+    #
+    # A new instance should be created per-document call. External callers should
+    # use {Sourcerer::SourceSkim.skim_file} or {Sourcerer::SourceSkim.skim_string}
+    # with a Markdown file or +format: :markdown+ rather than instantiating this
+    # class directly.
+    # @api private
+    class MarkdownSkimmer
+      # Matches ATX-style Markdown headings: one to six leading # characters.
+      MD_HEADING_RE = /^(\#{1,6})\s+(.+?)\s*$/
+
+      # @param content [String] raw Markdown text
+      # @param config [Config]
+      # @return [Hash] JSON-ready skim
+      def process content, config: Config.new(forms: [:flat])
+        @config = config
+
+        fm               = Sourcerer::YamlFrontmatter.extract(content)
+        body             = Sourcerer::YamlFrontmatter.strip(content)
+        offset           = content.lines.length - body.lines.length
+        title, sections  = extract_title_and_sections(body, offset)
+
+        result = {
+          title:       title || fm['title'].to_s,
+          frontmatter: fm
+        }
+        result[:sections_flat] = sections if @config.flat?
+        result[:sections_tree] = build_tree(sections) if @config.tree?
+        result
+      end
+
+      private
+
+      # Scan body content for ATX headings.
+      #
+      # The first +#+ heading is treated as the document title (level 0) and
+      # returned separately. All remaining headings are mapped to section level
+      # +hashes - 1+ so that +##+ becomes level 1, +###+ becomes level 2, etc.
+      #
+      # Lines inside fenced code blocks (delimited by +```+ or +~~~+) are skipped
+      # so that comment lines such as +# rubocop comment+ are not mistaken for headings.
+      def extract_title_and_sections content, offset
+        title    = nil
+        sections = []
+        in_fence = nil
+
+        content.each_line.with_index(1) do |line, lineno|
+          stripped  = line.chomp
+          in_fence, fence_line = update_fence(stripped, in_fence)
+          next if fence_line || in_fence
+
+          m = stripped.match(MD_HEADING_RE)
+          next unless m
+
+          hashes = m[1].length
+          if hashes == 1 && title.nil?
+            title = m[2]
+          else
+            sections << { text: m[2], level: hashes - 1, starts_at: lineno + offset }
+          end
+        end
+
+        [title, sections]
+      end
+
+      # Returns +[new_fence_state, is_fence_line]+ for the given stripped line.
+      #
+      # A fence line (the opening or closing +```+/+~~~+ marker) should always
+      # be skipped by the caller regardless of the new fence state.
+      def update_fence stripped, in_fence
+        m = stripped.match(/\A(`{3,}|~{3,})/)
+        return [in_fence, false] unless m
+        return [m[1], true] if in_fence.nil?
+        return [nil, true] if stripped.start_with?(in_fence)
+
+        [in_fence, false]
+      end
+
+      # Build a nested section tree (Array) from a flat section list.
+      #
+      # Returns an Array of top-level (level 1) section nodes, each with a
+      # +:sections+ array of children, mirroring the shape produced by
+      # {Skimmer} for AsciiDoc documents.
+      def build_tree sections
+        roots = []
+        stack = [{ level: 0, sections: roots }]
+
+        sections.each do |h|
+          node = h.merge(sections: [])
+          stack.pop while stack.size > 1 && stack.last[:level] >= h[:level]
+          stack.last[:sections] << node
+          stack << node
+        end
+
+        roots
+      end
+    end
+  end
+end

data/lib/sourcerer/source_skim/skimmer.rb

@@ -42,11 +42,11 @@ module Sourcerer
 
         if @config.include?(:attributes_custom)
           result[:attributes_custom] =
-            Sourcerer::AttributesFilter.user_attributes(document)
+            Sourcerer::AsciiDoc::AttributesFilter.user_attributes(document)
         end
         if @config.include?(:attributes_builtin)
           result[:attributes_builtin] =
-            Sourcerer::AttributesFilter.builtin_attributes(document)
+            Sourcerer::AsciiDoc::AttributesFilter.builtin_attributes(document)
         end
 
         result[:sections_tree] = tree if @config.tree?

data/lib/sourcerer/source_skim.rb

@@ -2,61 +2,96 @@
 
 require 'asciidoctor'
 require 'logger'
-require_relative 'attributes_filter'
+require_relative 'yaml_frontmatter'
 require_relative 'source_skim/config'
 require_relative 'source_skim/skimmer'
+require_relative 'source_skim/markdown_skimmer'
 
 module Sourcerer
-  # SourceSkim produces machine-oriented skims of AsciiDoc source documents.
+  # SourceSkim produces machine-oriented skims of markup source documents.
   #
   # A skim is a structured, JSON-ready representation of selected source elements
   # intended to help automated tooling inspect documentation source and identify
   # likely areas of interest when related product code changes.
   #
-  # @example Skim a file with default tree output
+  # AsciiDoc files are fully parsed by Asciidoctor and yield rich semantic
+  # output (sections, attributes, code blocks, tables, etc.). Markdown files
+  # yield frontmatter and section headings only, since Markdown has no
+  # standardised semantic block model.
+  #
+  # The format is auto-detected from the file extension when using +skim_file+.
+  # Pass +format: :markdown+ or +format: :asciidoc+ to +skim_string+ to
+  # disambiguate when there is no path to inspect.
+  #
+  # @example Skim an AsciiDoc file (auto-detected)
   #   skim = Sourcerer::SourceSkim.skim_file('docs/install.adoc')
   #
+  # @example Skim a Markdown file (auto-detected)
+  #   skim = Sourcerer::SourceSkim.skim_file('docs/guide.md')
+  #
   # @example Skim with both tree and flat section shapes
   #   skim = Sourcerer::SourceSkim.skim_file('docs/install.adoc', forms: [:tree, :flat])
   #
-  # @example Skim a content string
-  #   skim = Sourcerer::SourceSkim.skim_string(adoc_content, forms: [:flat])
+  # @example Skim a Markdown string explicitly
+  #   skim = Sourcerer::SourceSkim.skim_string(content, format: :markdown)
   #
-  # @example Skim with caller-supplied attribute overrides
+  # @example Skim with caller-supplied Asciidoctor attribute overrides
   #   skim = Sourcerer::SourceSkim.skim_file('docs/ref.adoc', attributes: { 'env' => 'prod' })
   module SourceSkim
-    NULL_LOGGER = Logger.new(IO::NULL)
-    LOAD_OPTS   = { safe: :safe, sourcemap: true, logger: NULL_LOGGER }.freeze
+    NULL_LOGGER       = Logger.new(IO::NULL)
+    LOAD_OPTS         = { safe: :safe, sourcemap: true, logger: NULL_LOGGER,
+                          attributes: { 'skip-front-matter' => '' } }.freeze
 
-    # Skim the AsciiDoc file at +file_path+.
+    # Skim the markup file at +file_path+.
     #
-    # @param file_path [String] path to the .adoc source file
-    # @param forms [Array<Symbol>] section shape(s) to emit: +:tree+, +:flat+, or both
-    # @param categories [Array<Symbol>, nil] element categories to include;
-    #   nil uses {DEFAULT_CATEGORIES} (everything except +attributes_builtin+)
-    # @param attributes [Hash{String => String}] arbitrary Asciidoctor attribute
-    #   overrides applied at parse time, e.g. <tt>'env' => 'test'</tt>.
-    #   Useful for toggling conditionals or injecting values that affect which
-    #   blocks are visible to the parser.
+    # Format is auto-detected from the file extension (.adoc → AsciiDoc;
+    # .md / .markdown → Markdown). Override with +format: :asciidoc+ or
+    # +format: :markdown+.
+    #
+    # @param file_path [String] path to the source file
+    # @param forms [Array<Symbol>, nil] section shape(s) to emit: +:tree+, +:flat+,
+    #   or both. Defaults to +[:tree]+ for AsciiDoc and +[:flat]+ for Markdown.
+    # @param format [Symbol, nil] +:asciidoc+ or +:markdown+; nil auto-detects
+    # @param categories [Array<Symbol>, nil] AsciiDoc only. Element categories to
+    #   include; nil uses {DEFAULT_CATEGORIES}. Silently ignored for Markdown.
+    # @param attributes [Hash{String => String}] AsciiDoc only. Asciidoctor
+    #   attribute overrides. Silently ignored for Markdown.
     # @return [Hash] JSON-ready skim
-    def self.skim_file file_path, forms: [:tree], categories: nil, attributes: {}
-      opts = LOAD_OPTS.merge(attributes: attributes)
-      doc  = Asciidoctor.load_file(file_path, **opts)
-      skim_doc(doc, forms: forms, categories: categories)
+    def self.skim_file file_path, forms: nil, format: nil, categories: nil, attributes: {}
+      fmt = format || detect_format(file_path)
+      if fmt == :markdown
+        config = Config.new(forms: forms || [:flat])
+        MarkdownSkimmer.new.process(File.read(file_path), config: config)
+      else
+        attrs = LOAD_OPTS[:attributes].merge(attributes)
+        opts  = LOAD_OPTS.merge(attributes: attrs)
+        doc   = Asciidoctor.load_file(file_path, **opts)
+        skim_doc(doc, forms: forms || [:tree], categories: categories)
+      end
     end
 
-    # Skim AsciiDoc source from a +content+ string.
+    # Skim markup source from a +content+ string.
     #
-    # @param content [String] raw AsciiDoc markup
-    # @param forms [Array<Symbol>] section shape(s) to emit
-    # @param categories [Array<Symbol>, nil] element categories to include
-    # @param attributes [Hash{String => String}] arbitrary Asciidoctor attribute
-    #   overrides applied at parse time
+    # +format:+ must be provided when the content is Markdown, since there is
+    # no file extension to inspect. Defaults to +:asciidoc+ for backward
+    # compatibility.
+    #
+    # @param content [String] raw markup text
+    # @param format [Symbol] +:asciidoc+ (default) or +:markdown+
+    # @param forms [Array<Symbol>, nil] section shape(s) to emit
+    # @param categories [Array<Symbol>, nil] AsciiDoc only
+    # @param attributes [Hash{String => String}] AsciiDoc only
     # @return [Hash] JSON-ready skim
-    def self.skim_string content, forms: [:tree], categories: nil, attributes: {}
-      opts = LOAD_OPTS.merge(attributes: attributes)
-      doc  = Asciidoctor.load(content, **opts)
-      skim_doc(doc, forms: forms, categories: categories)
+    def self.skim_string content, format: :asciidoc, forms: nil, categories: nil, attributes: {}
+      if format == :markdown
+        config = Config.new(forms: forms || [:flat])
+        MarkdownSkimmer.new.process(content, config: config)
+      else
+        attrs = LOAD_OPTS[:attributes].merge(attributes)
+        opts  = LOAD_OPTS.merge(attributes: attrs)
+        doc   = Asciidoctor.load(content, **opts)
+        skim_doc(doc, forms: forms || [:tree], categories: categories)
+      end
     end
 
     # Skim an already-parsed Asciidoctor +document+.
@@ -72,5 +107,16 @@ module Sourcerer
       config = Config.new(forms: forms, categories: categories)
       Skimmer.new.process(doc, config: config)
     end
+
+    # @api private
+    def self.detect_format file_path
+      ext = File.extname(file_path).downcase
+      if Sourcerer::MARKDOWN_EXTS.include?(ext)
+        :markdown
+      else
+        :asciidoc
+      end
+    end
+    private_class_method :detect_format
   end
 end

data/lib/sourcerer/sync/cast.rb

@@ -72,7 +72,8 @@ module Sourcerer
       # @return [CastResult]
       def self.init prime_path, target_path, data: {}, dry_run: false
         prime_text = File.read(prime_path)
-        rendered = data.empty? ? prime_text : render_liquid_string(prime_text, data)
+        clean_text = strip_meta_blocks(prime_text)
+        rendered   = data.empty? ? clean_text : render_liquid_string(clean_text, data)
 
         unless dry_run
           FileUtils.mkdir_p(File.dirname(File.expand_path(target_path)))
@@ -107,15 +108,23 @@ module Sourcerer
         prime_text = File.read(@prime_path)
         target_text = File.read(@target_path)
 
+        # Parse with canonical_prefix: '' so that ALL tagged regions -- including
+        # the non-canonical _liquid preamble block -- surface as Block objects
+        # rather than being swallowed into TextSegments.
         prime_segments  = BlockParser.parse(
           prime_text,
-          canonical_prefix: @canonical_prefix,
+          canonical_prefix: '',
           tag_patterns: @tag_patterns)
         target_segments = BlockParser.parse(
           target_text,
-          canonical_prefix: @canonical_prefix,
+          canonical_prefix: '',
           tag_patterns: @tag_patterns)
 
+        # Extract the _liquid preamble from the prime (non-canonical; not synced as a
+        # canonical block but used to carry Liquid variable context to all rendered content).
+        prime_liquid_block = prime_segments.find { |s| s.is_a?(BlockParser::Block) && s.tag == '_liquid' }
+        liquid_preamble = prime_liquid_block&.content.to_s
+
         prime_blocks = BlockParser.extract_canonical(prime_segments, canonical_prefix: @canonical_prefix)
         target_blocks, errors = validate_target_canonical(target_segments)
 
@@ -129,7 +138,10 @@ module Sourcerer
         end
 
         warnings = collect_warnings(prime_blocks, target_blocks, target_text)
-        new_segments, applied_changes = apply_prime_blocks(target_segments, prime_blocks)
+        new_segments, applied_changes = apply_prime_blocks(
+          target_segments, prime_blocks,
+          prime_liquid_block: prime_liquid_block,
+          liquid_preamble: liquid_preamble)
 
         new_text = reconstruct(new_segments)
         diff = generate_diff(target_text, new_text) if applied_changes.any? || @dry_run
@@ -156,6 +168,23 @@ module Sourcerer
         template.render(data.transform_keys(&:to_s))
       end
 
+      # Remove every underscore-prefixed meta block (+_skip+, +_liquid+, etc.) from
+      # a prime text before it is written to a target during {.init}.
+      # These blocks carry template instructions or Liquid context that are only
+      # meaningful during the prime→target rendering pass, not in the output file.
+      # @api private
+      def self.strip_meta_blocks text
+        tag_patterns = BlockParser.build_tag_patterns(
+          BlockParser::DEFAULT_TAG_SYNTAX_START,
+          BlockParser::DEFAULT_TAG_SYNTAX_END,
+          BlockParser::DEFAULT_COMMENT_SYNTAX_PATTERNS)
+        segments = BlockParser.parse(text, canonical_prefix: '', tag_patterns: tag_patterns)
+        segments
+          .reject { |s| s.is_a?(BlockParser::Block) && s.tag.start_with?('_') }
+          .map { |s| s.is_a?(BlockParser::Block) ? "#{s.open_line}#{s.content}#{s.close_line}" : s.content }
+          .join
+      end
+
       private
 
       # Collect canonical blocks from target, raising errors for duplicates.
@@ -192,29 +221,65 @@ module Sourcerer
         warnings
       end
 
-      def apply_prime_blocks target_segments, prime_blocks
+      def apply_prime_blocks target_segments, prime_blocks,
+        prime_liquid_block: nil, liquid_preamble: ''
         applied_changes = []
+        has_preamble = !liquid_preamble.empty?
+        liquid_seen = false
 
         new_segments = target_segments.map do |segment|
-          next segment unless segment.is_a?(BlockParser::Block) && canonical?(segment.tag)
-          next segment unless prime_blocks.key?(segment.tag)
+          if segment.is_a?(BlockParser::Block)
+            if segment.tag == '_liquid'
+              # Sync the _liquid block content from prime to target
+              liquid_seen = true
+              next segment unless prime_liquid_block
+              next segment if prime_liquid_block.content == segment.content
+
+              applied_changes << '_liquid'
+              BlockParser::Block.new(
+                tag: '_liquid',
+                open_line: segment.open_line,
+                content: prime_liquid_block.content,
+                close_line: segment.close_line)
+
+            elsif canonical?(segment.tag)
+              next segment unless prime_blocks.key?(segment.tag)
+
+              prime_content = prime_blocks[segment.tag].content
+              rendered_content = render_content(prime_content, preamble: liquid_preamble)
+
+              if rendered_content == segment.content
+                segment
+              else
+                applied_changes << segment.tag
+                BlockParser::Block.new(
+                  tag: segment.tag,
+                  open_line: segment.open_line,
+                  content: rendered_content,
+                  close_line: segment.close_line)
+              end
+
+            else
+              segment
+            end
+
+          elsif segment.is_a?(BlockParser::TextSegment) && has_preamble && liquid_seen
+            # Render in-between text with the preamble context, but only after the
+            # _liquid block has been encountered so all variables are in scope.
+            rendered_text = render_content(segment.content, preamble: liquid_preamble)
+            if rendered_text == segment.content
+              segment
+            else
+              applied_changes << 'document-text'
+              BlockParser::TextSegment.new(content: rendered_text)
+            end
 
-          prime_content = prime_blocks[segment.tag].content
-          rendered_content = render_content(prime_content)
-
-          if rendered_content == segment.content
-            segment
           else
-            applied_changes << segment.tag
-            BlockParser::Block.new(
-              tag: segment.tag,
-              open_line: segment.open_line,
-              content: rendered_content,
-              close_line: segment.close_line)
+            segment
           end
         end
 
-        [new_segments, applied_changes]
+        [new_segments, applied_changes.uniq]
       end
 
       def reconstruct segments
@@ -244,10 +309,11 @@ module Sourcerer
         end
       end
 
-      def render_content content
-        return content if @data.empty?
+      def render_content content, preamble: ''
+        return content if @data.empty? && preamble.empty?
 
-        self.class.render_liquid_string(content, @data)
+        full = preamble.empty? ? content : "#{preamble}#{content}"
+        self.class.render_liquid_string(full, @data)
       end
 
       def generate_diff old_text, new_text

data/lib/sourcerer/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Sourcerer
-  VERSION = '0.2.1'
+  VERSION = '0.3.0'
 end

data/lib/sourcerer/yaml_frontmatter.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+require 'yaml'
+
+module Sourcerer
+  # Single owner of YAML frontmatter parsing across Sourcerer.
+  #
+  # Both AsciiDoc pages (the Jekyll convention of embedding +---+-fenced YAML
+  # at the top of a +.adoc+ file) and Markdown pages use the same syntax.
+  # All Sourcerer code that needs to detect, extract, or remove a frontmatter
+  # block delegates here instead of duplicating logic or constants.
+  module YamlFrontmatter
+    # Matches a leading +---+-fenced YAML block at the start of a file.
+    # Content between the fences must be non-empty (+.+?+, lazy).
+    # The closing fence must be followed by a newline.
+    REGEXP = /\A(---\s*\n.+?\n)(---\s*\n)/m
+
+    module_function
+
+    # Parse the YAML frontmatter from +source_text+ and return it as a Hash.
+    #
+    # Returns an empty Hash when no frontmatter is present or when the YAML
+    # is malformed.
+    #
+    # @param source_text [String]
+    # @return [Hash]
+    def extract source_text
+      match = source_text.match(REGEXP)
+      return {} unless match
+
+      frontmatter_payload = match[1].sub(/\A---\s*\n/, '')
+      parsed = YAML.safe_load(frontmatter_payload, aliases: true)
+      parsed.is_a?(Hash) ? parsed : {}
+    rescue Psych::SyntaxError
+      {}
+    end
+
+    # Return +source_text+ with the leading YAML frontmatter block removed.
+    #
+    # @param source_text [String]
+    # @return [String]
+    def strip source_text
+      source_text.sub(REGEXP, '')
+    end
+  end
+end

data/lib/sourcerer.rb

@@ -17,11 +17,18 @@ require_relative 'sourcerer/yaml'
 # Requiring `sourcerer` also makes adjacent public constants (for example,
 # `Sourcerer::Builder`) available to downstream callers.
 module Sourcerer
+  # File extensions recognised as Markdown source files.
+  MARKDOWN_EXTS  = %w[.md .markdown].freeze
+
+  # File extensions recognised as AsciiDoc source files.
+  ASCIIDOC_EXTS  = %w[.adoc .asciidoc .asc .ad].freeze
+
   autoload :AttributesFilter, 'sourcerer/attributes_filter'
-  autoload :Jekyll, 'sourcerer/jekyll'
-  autoload :MarkDownGrade, 'sourcerer/mark_down_grade'
-  autoload :SourceSkim, 'sourcerer/source_skim'
-  autoload :Sync, 'sourcerer/sync'
+  autoload :YamlFrontmatter,     'sourcerer/yaml_frontmatter'
+  autoload :Jekyll,              'sourcerer/jekyll'
+  autoload :MarkDownGrade,       'sourcerer/mark_down_grade'
+  autoload :SourceSkim,          'sourcerer/source_skim'
+  autoload :Sync,                'sourcerer/sync'
 
   DEPRECATED_FACADE_METHODS = {
     # DO NOT add new public methods to this surface

data/specs/docs/frontmatter-reader_prd.adoc

@@ -0,0 +1,47 @@
+= FrontmatterReader: PRD
+:status: implemented
+:version: 0.3.0
+
+== Overview
+
+`Sourcerer::SourceSkim` needs to incorporate a simpler Skim object that derives data/content from frontmatter YAML and section headings.
+
+`SourceSkim` currently generates structured, semantic metadata and content outline about AsciiDoc source files.
+But many use cases (Jekyll site pipelines, documentation indexers, cross-repo search tools) need to work with Markdown files, as well:
+
+* Frontmatter data (layout, title, navigation hints, etc.)
+* Section headings and their levels:
+** a flat list of section headings and their line numbers, and/or
+** a nested hierarchy of sections and subsections.
+
+== Output Shape
+
+The core operation yields a skim in one of the following configurations:
+
+[source,ruby]
+----
+{
+  title:         'Introduction',                                    # String — from first # heading
+  frontmatter:   { 'layout' => 'docs', 'title' => 'Guide', ... },  # Hash
+  sections_flat: [                                                 # Array
+    { text: 'Getting Started', level: 1, starts_at: 13 },
+    { text: 'Prerequisites',   level: 2, starts_at: 17 },
+    ...
+  ]
+}
+----
+
+[source,ruby]
+----
+{
+  title:       'Introduction',                                    # String
+  frontmatter: { 'layout' => 'docs', 'title' => 'Guide', ... },  # Hash
+  sections_tree: [                                               # Array (same shape as AsciiDoc)
+    { text: 'Getting Started', level: 1, starts_at: 13, sections: [
+      { text: 'Prerequisites', level: 2, starts_at: 17, sections: [] }
+    ] },
+    { text: 'Advanced Usage', level: 1, starts_at: 22, sections: [] },
+    ...
+  ]
+}
+----

metadata

@@ -1,13 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: asciisourcerer
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - DocOps Lab
+autorequire:
 bindir: bin
 cert_chain: []
-date: 1980-01-02 00:00:00.000000000 Z
+date: 2026-04-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: asciidoctor
@@ -122,7 +123,6 @@ files:
 - lib/asciisourcerer.rb
 - lib/sourcerer.rb
 - lib/sourcerer/asciidoc.rb
-- lib/sourcerer/attributes_filter.rb
 - lib/sourcerer/builder.rb
 - lib/sourcerer/jekyll.rb
 - lib/sourcerer/jekyll/bootstrapper.rb
@@ -135,6 +135,7 @@ files:
 - lib/sourcerer/rendering.rb
 - lib/sourcerer/source_skim.rb
 - lib/sourcerer/source_skim/config.rb
+- lib/sourcerer/source_skim/markdown_skimmer.rb
 - lib/sourcerer/source_skim/skimmer.rb
 - lib/sourcerer/sync.rb
 - lib/sourcerer/sync/block_parser.rb
@@ -144,12 +145,15 @@ files:
 - lib/sourcerer/util/pathifier.rb
 - lib/sourcerer/version.rb
 - lib/sourcerer/yaml.rb
+- lib/sourcerer/yaml_frontmatter.rb
+- specs/docs/frontmatter-reader_prd.adoc
 homepage: https://github.com/DocOps/asciisourcerer
 licenses:
 - MIT
 metadata:
   allowed_push_host: https://rubygems.org
   rubygems_mfa_required: 'true'
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -164,7 +168,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.7.2
+rubygems_version: 3.4.19
+signing_key:
 specification_version: 4
 summary: APIs for specialized handling of AsciiDoc, YAML, and Liquid documents.
 test_files: []

data/lib/sourcerer/attributes_filter.rb

@@ -1,72 +0,0 @@
-# frozen_string_literal: true
-
-require 'asciidoctor'
-
-module Sourcerer
-  # Utilities for filtering and partitioning Asciidoctor document attributes.
-  #
-  # The primary use case is separating user-defined ("custom") attributes from
-  # those injected by Asciidoctor at parse time ("built-in"). This distinction
-  # matters when a skim consumer needs to inspect only the attributes an author
-  # explicitly set in their source.
-  #
-  # Additional attribute manipulation helpers may be added here over time.
-  #
-  # @example
-  #   custom = Sourcerer::AttributesFilter.user_attributes(doc)
-  #   builtin = Sourcerer::AttributesFilter.builtin_attributes(doc)
-  module AttributesFilter
-    # Attribute keys injected by Asciidoctor at parse time rather than defined
-    # by the document author.
-    BUILTIN_ATTR_KEYS = (Asciidoctor::DEFAULT_ATTRIBUTES.keys + %w[
-      asciidoctor asciidoctor-version
-      attribute-missing attribute-undefined
-      authorcount
-      docdate docdatetime docdir docfile docfilesuffix docname doctime doctitle doctype docyear
-      embedded
-      htmlsyntax
-      iconsdir
-      localdate localdatetime localtime localyear
-      max-include-depth
-      notitle
-      outfilesuffix
-      stylesdir
-      toc-position
-      user-home
-    ]).freeze
-
-    BUILTIN_ATTR_PATTERNS = [
-      /^backend(-|$)/,
-      /^basebackend(-|$)/,
-      /^doctype-/,
-      /^filetype(-|$)/,
-      /^safe-mode-/
-    ].freeze
-
-    module_function
-
-    # Returns a hash of user-defined attributes, excluding any key that belongs
-    # to Asciidoctor's built-in set.
-    #
-    # @param doc [Asciidoctor::Document]
-    # @return [Hash{String => String}]
-    def user_attributes doc
-      doc.attributes.reject do |k, _|
-        BUILTIN_ATTR_KEYS.include?(k) ||
-          BUILTIN_ATTR_PATTERNS.any? { |pat| pat.match?(k) }
-      end
-    end
-
-    # Returns a hash of built-in Asciidoctor attributes, i.e., those injected at
-    # parse time rather than authored in the document.
-    #
-    # @param doc [Asciidoctor::Document]
-    # @return [Hash{String => String}]
-    def builtin_attributes doc
-      doc.attributes.select do |k, _|
-        BUILTIN_ATTR_KEYS.include?(k) ||
-          BUILTIN_ATTR_PATTERNS.any? { |pat| pat.match?(k) }
-      end
-    end
-  end
-end