markly 0.15.0 → 0.15.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d9b96430051a815282d6f83d487fc0c0cdc636b174a1f1f775f0574950e8bab3
-  data.tar.gz: d16b1c808df1030e6bcc9c67c99f88a0eeae8d8e604b68745e0d8f4c16ac171d
+  metadata.gz: 78c9ac76ac7421696ae270097bcf7e960f78b3ae8850232fa9d7c1d6a5e5d893
+  data.tar.gz: eb0e5d5ebcfb8cf4f128dc61ac3989eb789a31ae1f9ec7e99ecd6c9482859edc
 SHA512:
-  metadata.gz: 9e7e24631dbd95cb90ee889919be9bb6a267c6e2cf31e7355dd21f5e0c903a83a288178bffcf4430eed4ed58b3d2c014e8e3617629a9c703f063069a03cfeaf7
-  data.tar.gz: 2c51b26c4ce3d2467079ce968f0d2bae307090b49b59b0acfb7396ad00c140e276f76d9291ac4bc65e24e368005b6929990521de4749fecd678f1b6b0ed6d77d
+  metadata.gz: 2c68b3899e33d96fcfeb9ad0c2458f67dffd9b9f8d5a1e2f1b2cd8a158faf74d5b4e1246f59b7257606b8b2218175a3563ad048824878ae6b04f6a1f5b7bfc28
+  data.tar.gz: 27d4228f0c7f158d1397117ce744f4c00ed7a59fff8253756281dc3f87d01aa642759a71187e9ee137d051d91a9d059a739addaa8b9f8f3b23b544f7ebb3f18b

data/context/abstract-syntax-tree.md

@@ -0,0 +1,94 @@
+# Abstract Syntax Tree
+
+This guide explains how to use Markly's abstract syntax tree (AST) to parse and manipulate Markdown documents.
+
+## Parsing
+
+You can parse Markdown to a `Document` node using `Markly.parse`:
+
+~~~ ruby
+require 'markly'
+
+document = Markly.parse('*Hello* world')
+
+pp document 
+~~~
+
+This will print out the following:
+
+~~~
+#<Markly::Node(document):
+	source_position={:start_line=>1, :start_column=>1, :end_line=>1, :end_column=>13}
+	children=[#<Markly::Node(paragraph):
+			 source_position={:start_line=>1, :start_column=>1, :end_line=>1, :end_column=>13}
+			 children=[#<Markly::Node(emph):
+						source_position={:start_line=>1, :start_column=>1, :end_line=>1, :end_column=>7}
+						children=[#<Markly::Node(text): source_position={:start_line=>1, :start_column=>2, :end_line=>1, :end_column=>6}, string_content="Hello">]>,
+					#<Markly::Node(text): source_position={:start_line=>1, :start_column=>8, :end_line=>1, :end_column=>13}, string_content=" world">]>]>
+~~~
+
+As you can see, a document consists of a root node, which contains several children, they themselves containing children, and so on. We refer to this as the abstract syntax tree (AST).
+
+## Example: Walking the AST
+
+You can use `walk` or `each` to iterate over nodes:
+
+	- `walk` will iterate on a node and recursively iterate on a node's children.
+	- `each` will iterate on a node and its children, but no further.
+
+<!-- end list -->
+
+``` ruby
+require "markly"
+
+document = Markly.parse("# The site\n\n [GitHub](https://www.github.com)")
+
+# Walk tree and print out URLs for links:
+document.walk do |node|
+	if node.type == :link
+		puts "URL = #{node.url}"
+	end
+end
+
+# Capitalize all regular text in headers:
+document.walk do |node|
+	if node.type == :header
+		node.each do |subnode|
+			if subnode.type == :text
+				subnode.string_content = subnode.string_content.upcase
+			end
+		end
+	end
+end
+
+# Transform links to regular text:
+document.walk do |node|
+	if node.type == :link
+		node.insert_before(node.first_child)
+		node.delete
+	end
+end
+```
+
+### Creating a Custom Renderer
+
+You can also derive a class from {ruby Markly::Renderer::HTML} class. Using a pure Ruby renderer is slower, but allows you to customize the output. For example:
+
+``` ruby
+class MyHtmlRenderer < Markly::Renderer::HTML
+	def initialize
+		super
+		@header_id = 1
+	end
+	
+	def header(node)
+		block do
+			out("<h", node.header_level, " id=\"", @header_id, "\">", :children, "</h", node.header_level, ">")
+			@header_id += 1
+		end
+	end
+end
+
+my_renderer = MyHtmlRenderer.new
+puts my_renderer.render(document)
+```

data/context/getting-started.md

@@ -0,0 +1,101 @@
+# Getting Started
+
+This guide explains now to install and use Markly.
+
+## Installation 
+
+Add the gem to your project:
+
+	$ bundle add markly
+
+## Usage
+
+Markly's most basic usage is to convert Markdown to HTML. You can do this in a few ways:
+
+~~~ ruby
+require 'markly'
+
+Markly.render_html('Hi *there*')
+# <p>Hi <em>there</em></p>\n
+~~~
+
+You can also parse a string to receive a `Document` node. You can then print that node to HTML, iterate over the children, and other fun node stuff. For example:
+
+~~~ ruby
+require 'markly'
+
+document = Markly.parse('*Hello* world')
+puts(document.to_html) # <p>Hi <em>there</em></p>\n
+
+document.walk do |node|
+	puts node.type # [:document, :paragraph, :text, :emph, :text]
+end
+~~~
+
+## Options
+
+Markly accepts integer flags which control how the Markdown is parsed and rendered.
+
+### Parse Options
+
+| Name                                 | Description
+| ------------------------------------ | -----------
+| `Markly::DEFAULT`                    | The default parsing system.
+| `Markly::UNSAFE`                     | Allow raw/custom HTML and unsafe links.
+| `Markly::FOOTNOTES`                  | Parse footnotes.
+| `Markly::LIBERAL_HTML_TAG`           | Support liberal parsing of inline HTML tags.
+| `Markly::SMART`                      | Use smart punctuation (curly quotes, etc.).
+| `Markly::STRIKETHROUGH_DOUBLE_TILDE` | Parse strikethroughs by double tildes (compatibility with [redcarpet](https://github.com/vmg/redcarpet))
+| `Markly::VALIDATE_UTF8`              | Replace illegal sequences with the replacement character `U+FFFD`.
+
+### Render Options
+
+| Name                                    | Description                                                     |
+| --------------------------------------- | --------------------------------------------------------------- |
+| `Markly::DEFAULT`                       | The default rendering system.                                   |
+| `Markly::UNSAFE`                        | Allow raw/custom HTML and unsafe links.                         |
+| `Markly::GITHUB_PRE_LANG`               | Use GitHub-style `<pre lang>` for fenced code blocks.           |
+| `Markly::HARD_BREAKS`                   | Treat `\n` as hardbreaks (by adding `<br/>`).                   |
+| `Markly::NO_BREAKS`                     | Translate `\n` in the source to a single whitespace.            |
+| `Markly::SOURCE_POSITION`               | Include source position in rendered HTML.                       |
+| `Markly::TABLE_PREFER_STYLE_ATTRIBUTES` | Use `style` insted of `align` for table cells.                  |
+| `Markly::FULL_INFO_STRING`              | Include full info strings of code blocks in separate attribute. |
+
+### Passing Options
+
+To apply a single option, pass it in as a flags option:
+
+``` ruby
+Markly.parse("\"Hello,\" said the spider.", flags: Markly::SMART)
+# <p>“Hello,” said the spider.</p>\n
+```
+
+To have multiple options applied, `|` (or) the flags together:
+
+``` ruby
+Markly.render_html("\"'Shelob' is my name.\"", flags: Markly::HARD_BREAKS|Markly::SOURCE_POSITION)
+```
+
+## Extensions
+
+Both `render_html` and `parse` take an optional `extensions:` argument defining the extensions you want enabled as your CommonMark document is being processed:
+
+``` ruby
+Markly.render_html("<script>hi</script>", flags: Markly::UNSAFE, extensions: [:tagfilter])
+```
+
+The documentation for these extensions are [defined in this spec](https://github.github.com/gfm/), and the rationale is provided [in this blog post](https://githubengineering.com/a-formal-spec-for-github-markdown/).
+
+The available extensions are:
+
+  - `:table` - This provides support for tables.
+  - `:tasklist` - This provides support for task list items.
+  - `:strikethrough` - This provides support for strikethroughs.
+  - `:autolink` - This provides support for automatically converting URLs to anchor tags.
+  - `:tagfilter` - This escapes [several "unsafe" HTML tags](https://github.github.com/gfm/#disallowed-raw-html-extension-), causing them to not have any effect.
+
+## Developing Locally
+
+After cloning the repo:
+
+	$ bake build test

data/context/headings.md

@@ -0,0 +1,115 @@
+# Headings
+
+This guide explains how to work with headings in Markly, including extracting them for navigation and handling duplicate heading text.
+
+## Unique ID Generation
+
+When rendering HTML with `ids: true`, duplicate heading text automatically gets unique IDs to avoid collisions. This is particularly useful when multiple sections have the same title (e.g., multiple "Deployment" sections under different parent headings).
+
+``` ruby
+markdown = <<~MARKDOWN
+  ## Kubernetes
+  
+  ### Deployment
+  
+  ## Systemd
+  
+  ### Deployment
+MARKDOWN
+
+renderer = Markly::Renderer::HTML.new(ids: true)
+html = renderer.render(Markly.parse(markdown))
+
+# Generates:
+# <section id="kubernetes">...</section>
+# <section id="deployment">...</section>
+# <section id="systemd">...</section>
+# <section id="deployment-2">...</section>
+```
+
+The first occurrence gets the clean ID, subsequent duplicates get numbered suffixes (`-2`, `-3`, etc.).
+
+## Extracting Headings for Table of Contents
+
+The `Headings` class can extract headings for building navigation or table of contents:
+
+``` ruby
+document = Markly.parse(markdown)
+headings = Markly::Renderer::Headings.extract(document, min_level: 2, max_level: 3)
+
+headings.each do |heading|
+	puts "#{heading.level}: #{heading.text} (#{heading.anchor})"
+end
+
+# Output:
+# 2: Kubernetes (kubernetes)
+# 3: Deployment (deployment)
+# 2: Systemd (systemd)
+# 3: Deployment (deployment-2)
+```
+
+Each `Heading` object has:
+- `level` - The heading level (1-6)
+- `text` - The plain text content
+- `anchor` - The unique ID/anchor
+- `node` - The original Markly AST node
+
+### Level Filtering
+
+Use `min_level` and `max_level` to filter which heading levels to extract:
+
+``` ruby
+# Only extract h2 and h3 headings
+headings = Markly::Renderer::Headings.extract(document, min_level: 2, max_level: 3)
+
+# Only h1 headings
+headings = Markly::Renderer::Headings.extract(document, min_level: 1, max_level: 1)
+```
+
+## Custom Heading Strategies
+
+For advanced use cases, you can provide a custom `Headings` instance to the HTML renderer:
+
+### Sharing State Across Documents
+
+To ensure IDs remain unique across multiple documents:
+
+``` ruby
+# Share heading state across multiple documents
+headings = Markly::Renderer::Headings.new
+renderer = Markly::Renderer::HTML.new(headings: headings)
+
+doc1_html = renderer.render(Markly.parse(doc1_markdown))
+doc2_html = renderer.render(Markly.parse(doc2_markdown))
+# IDs remain unique across both documents
+```
+
+### Custom ID Generation
+
+Subclass `Headings` to implement alternative ID generation strategies:
+
+``` ruby
+class HierarchicalHeadings < Markly::Renderer::Headings
+	def initialize
+		super
+		@parent_context = []
+	end
+	
+	def anchor_for(node)
+		base = base_anchor_for(node)
+		
+		# Custom logic: could incorporate parent heading context to generate IDs like "kubernetes-deployment" instead of "deployment-2"
+		
+		if @ids.key?(base)
+			@ids[base] += 1
+			"#{base}-#{@ids[base]}"
+		else
+			@ids[base] = 1
+			base
+		end
+	end
+end
+
+renderer = Markly::Renderer::HTML.new(headings: HierarchicalHeadings.new)
+```
+

data/context/index.yaml

@@ -0,0 +1,20 @@
+# Automatically generated context index for Utopia::Project guides.
+# Do not edit then files in this directory directly, instead edit the guides and then run `bake utopia:project:agent:context:update`.
+---
+description: CommonMark parser and renderer. Written in C, wrapped in Ruby.
+metadata:
+  documentation_uri: https://ioquatix.github.io/markly/
+  funding_uri: https://github.com/sponsors/ioquatix/
+  source_code_uri: https://github.com/ioquatix/markly.git
+files:
+- path: getting-started.md
+  title: Getting Started
+  description: This guide explains now to install and use Markly.
+- path: abstract-syntax-tree.md
+  title: Abstract Syntax Tree
+  description: This guide explains how to use Markly's abstract syntax tree (AST)
+    to parse and manipulate Markdown documents.
+- path: headings.md
+  title: Headings
+  description: This guide explains how to work with headings in Markly, including
+    extracting them for navigation and handling duplicate heading text.

data/ext/markly/blocks.c

@@ -72,6 +72,20 @@ static CMARK_INLINE bool S_is_space_or_tab(char c) {
   return (c == ' ' || c == '\t');
 }
 
+// Returns true if block is being finalized on the same line it ends.
+// This happens for:
+// - Document node (special case)
+// - Fenced code blocks (end on the closing fence line)
+// - Setext headings (end on the underline)
+// - Any block finalized on the same line it started (e.g., single-line HTML blocks)
+static CMARK_INLINE bool S_ends_on_current_line(cmark_parser *parser, cmark_node *b) {
+  return S_type(b) == CMARK_NODE_DOCUMENT ||
+         (S_type(b) == CMARK_NODE_CODE_BLOCK && b->as.code.fenced) ||
+         (S_type(b) == CMARK_NODE_HEADING && b->as.heading.setext) ||
+         // Single-line blocks: finalized on same line they started
+         b->start_line == parser->line_number;
+}
+
 static void S_parser_feed(cmark_parser *parser, const unsigned char *buffer,
                           size_t len, bool eof);
 
@@ -288,17 +302,15 @@ static cmark_node *finalize(cmark_parser *parser, cmark_node *b) {
   bool has_content;
 
   parent = b->parent;
-  assert(b->flags &
-         CMARK_NODE__OPEN); // shouldn't call finalize on closed blocks
+  assert(b->flags & CMARK_NODE__OPEN); // shouldn't call finalize on closed blocks
   b->flags &= ~CMARK_NODE__OPEN;
 
   if (parser->curline.size == 0) {
-    // end of input - line number has not been incremented
+    // end of input - line number has not been incremented:
     b->end_line = parser->line_number;
     b->end_column = parser->last_line_length;
-  } else if (S_type(b) == CMARK_NODE_DOCUMENT ||
-             (S_type(b) == CMARK_NODE_CODE_BLOCK && b->as.code.fenced) ||
-             (S_type(b) == CMARK_NODE_HEADING && b->as.heading.setext)) {
+  } else if (S_ends_on_current_line(parser, b)) {
+    // Block ends on current line (line_number already incremented):
     b->end_line = parser->line_number;
     b->end_column = parser->curline.size;
     if (b->end_column && parser->curline.ptr[b->end_column - 1] == '\n')
@@ -306,6 +318,7 @@ static cmark_node *finalize(cmark_parser *parser, cmark_node *b) {
     if (b->end_column && parser->curline.ptr[b->end_column - 1] == '\r')
       b->end_column -= 1;
   } else {
+    // Block ended on a previous line:
     b->end_line = parser->line_number - 1;
     b->end_column = parser->last_line_length;
   }

data/lib/markly/node.rb

@@ -6,6 +6,7 @@
 # Copyright, 2017, by Goro Fuji.
 # Copyright, 2018, by Jerry van Leeuwen.
 # Copyright, 2020-2025, by Samuel Williams.
+# Copyright, 2025, by Olle Jonsson.
 
 require_relative "node/inspect"
 

data/lib/markly/renderer/generic.rb

@@ -26,9 +26,9 @@ module Markly
 			def out(*args)
 				args.each do |arg|
 					if arg == :children
-						@node.each {|child| out(child)}
+						@node.each{|child| out(child)}
 					elsif arg.is_a?(Array)
-						arg.each {|x| render(x)}
+						arg.each{|x| render(x)}
 					elsif arg.is_a?(Node)
 						render(arg)
 					else
@@ -43,7 +43,7 @@ module Markly
 					document(node)
 					@stream.string
 				elsif @in_plain && node.type != :text && node.type != :softbreak
-					node.each {|child| render(child)}
+					node.each{|child| render(child)}
 				else
 					send(node.type, node)
 				end

data/lib/markly/renderer/html.rb

@@ -10,7 +10,13 @@
 
 require_relative "generic"
 require_relative "headings"
-require "cgi"
+
+require "cgi/escape"
+
+# Compatibility for older Ruby versions where escape_html alias doesn't exist:
+unless CGI.respond_to?(:escape_html)
+	require "cgi"
+end
 
 module Markly
 	module Renderer
@@ -256,10 +262,10 @@ module Markly
 			end
 			
 			TABLE_CELL_ALIGNMENT = {
-						left: ' align="left"',
-						right: ' align="right"',
-						center: ' align="center"'
-					}.freeze
+				left: ' align="left"',
+				right: ' align="right"',
+				center: ' align="center"'
+			}.freeze
 			
 			def table_cell(node)
 				align = TABLE_CELL_ALIGNMENT.fetch(@alignments[@column_index], "")

data/lib/markly/version.rb

@@ -7,5 +7,5 @@
 # Copyright, 2020-2025, by Samuel Williams.
 
 module Markly
-	VERSION = "0.15.0"
+	VERSION = "0.15.2"
 end

data/license.md

@@ -17,9 +17,11 @@ Copyright, 2018, by Akira Matsuda.
 Copyright, 2018, by Danny Iachini.  
 Copyright, 2019-2020, by Tomoya Chiba.  
 Copyright, 2019, by Brett Walker.  
-Copyright, 2020, by Olle Jonsson.  
+Copyright, 2020-2025, by Olle Jonsson.  
 Copyright, 2020-2025, by Samuel Williams.  
 Copyright, 2024, by Ross Kaffenberger.  
+Copyright, 2025, by Henrik Nyh.  
+Copyright, 2025, by Peter H. Boling.  
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/readme.md

@@ -1,18 +1,14 @@
 # Markly
 
-A parser and abstract syntax tree for Markdown documents (CommonMark compatible) in Ruby. Originally forked from
-[CommonMarker](https://github.com/gjtorikian/commonmarker). It also includes extensions to the CommonMark spec as
-documented in the [GitHub Flavored Markdown spec](http://github.github.com/gfm/), such as support for tables,
-strikethroughs, and autolinking.
+A parser and abstract syntax tree for Markdown documents (CommonMark compatible) in Ruby. Originally forked from [CommonMarker](https://github.com/gjtorikian/commonmarker). It also includes extensions to the CommonMark spec as documented in the [GitHub Flavored Markdown spec](http://github.github.com/gfm/), such as support for tables, strikethroughs, and autolinking.
 
 [![Development Status](https://github.com/ioquatix/markly/workflows/Test/badge.svg)](https://github.com/ioquatix/markly/actions?workflow=Test)
 
 ## Motivation
 
-This code base was originally forked from [Commonmarker](https://github.com/gjtorikian/commonmarker) before they
-switched from `cmark-gfm` (C) to `comrak` (Rust). The original implementation provided access to the abstract syntax
-tree (AST), which is useful for building tools on top of Markdown. The Rust implementation does not provide this
-functionality, and so this fork was created to continue to provide these (and more) features.
+This code base was originally forked from [Commonmarker](https://github.com/gjtorikian/commonmarker) before theyswitched from `cmark-gfm` (C) to `comrak` (Rust). The original implementation provided access to the abstract syntaxtree (AST), which is useful for building tools on top of Markdown. The Rust implementation did not provide thisfunctionality, and so this fork was created to continue to provide these (and more) features.
+
+It should be noted that `commonmarker` re-introduced AST access, but the original C implementation in this fork is [3-4x faster at processing Markdown into HTML](https://github.com/gjtorikian/commonmarker?tab=readme-ov-file#benchmarks) and has a more advanced HTML generation and AST processing features.
 
 ## Usage
 
@@ -28,6 +24,10 @@ Please see the [project documentation](https://ioquatix.github.io/markly/) for m
 
 Please see the [project releases](https://ioquatix.github.io/markly/releases/index) for all releases.
 
+### v0.15.1
+
+  - Add agent context.
+
 ### v0.15.0
 
   - Introduced `Markly::Renderer::Headings` class for extracting headings from markdown documents with automatic duplicate ID resolution. When rendering HTML with `ids: true`, duplicate heading text now automatically gets unique IDs (`deployment`, `deployment-2`, `deployment-3`). The `Headings` class can also be used to extract headings for building navigation or table of contents.

data/releases.md

@@ -1,5 +1,9 @@
 # Releases
 
+## v0.15.1
+
+  - Add agent context.
+
 ## v0.15.0
 
   - Introduced `Markly::Renderer::Headings` class for extracting headings from markdown documents with automatic duplicate ID resolution. When rendering HTML with `ids: true`, duplicate heading text now automatically gets unique IDs (`deployment`, `deployment-2`, `deployment-3`). The `Headings` class can also be used to extract headings for building navigation or table of contents.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: markly
 version: !ruby/object:Gem::Version
-  version: 0.15.0
+  version: 0.15.2
 platform: ruby
 authors:
 - Garen Torikian
@@ -14,13 +14,15 @@ authors:
 - Andrew Anderson
 - Ben Woosley
 - Goro Fuji
+- Olle Jonsson
 - Tomoya Chiba
 - Akira Matsuda
 - Danny Iachini
+- Henrik Nyh
 - Jerry van Leeuwen
 - Michael Camilleri
 - Mu-An Chiou
-- Olle Jonsson
+- Peter H. Boling
 - Roberto Hidalgo
 - Ross Kaffenberger
 - Vitaliy Klachkov
@@ -62,6 +64,10 @@ extensions:
 - ext/markly/extconf.rb
 extra_rdoc_files: []
 files:
+- context/abstract-syntax-tree.md
+- context/getting-started.md
+- context/headings.md
+- context/index.yaml
 - ext/markly/arena.c
 - ext/markly/autolink.c
 - ext/markly/autolink.h