maquina_remend 0.1.0

data/docs/repairs.md

@@ -0,0 +1,391 @@
+# Repairs
+
+Every repair `maquina_remend` makes, and every place it refuses to make one.
+One section per repair: the option that controls it, the broken input, the
+repaired output.
+
+All outputs below are real. Each is the return value of `MaquinaRemend.call`
+on the input shown, at the stated options.
+
+* [Emphasis](#emphasis) — `bold:`, `italic:`, `bold_italic:`
+* [Inline code](#inline-code) — `inline_code:`
+* [Strikethrough](#strikethrough) — `strikethrough:`
+* [Links and images](#links-and-images) — `links:`, `images:`, `link_mode:`
+* [Math](#math) — `block_math:`, `inline_math:`
+* [Setext headings](#setext-headings) — `setext_headings:`
+* [Truncated HTML tags](#truncated-html-tags) — `html_tags:`
+* [Application tags](#application-tags) — `app_tags:`
+* [Comparison operators](#comparison-operators) — `comparison_operators:`
+* [Single tildes](#single-tildes) — `single_tilde:`
+* [Dangling escapes](#dangling-escapes) — `dangling_escape:`
+* [Guards](#guards) — what is never repaired
+
+---
+
+## Emphasis
+
+`bold:`, `italic:` and `bold_italic:`, all on by default.
+
+An emphasis run opened and not yet closed is closed at the end of the buffer.
+The closing run matches the opening one: three asterisks are closed with three,
+not one.
+
+```ruby
+MaquinaRemend.call("**unclosed bold")  # => "**unclosed bold**"
+MaquinaRemend.call("*unclosed")        # => "*unclosed*"
+MaquinaRemend.call("_unclosed")        # => "_unclosed_"
+MaquinaRemend.call("***both")          # => "***both***"
+```
+
+Switch one off and its delimiter is left alone:
+
+```ruby
+MaquinaRemend.call("**unclosed bold", bold: false)  # => "**unclosed bold"
+```
+
+The underscore form follows CommonMark's flanking rules, so an underscore
+inside a word is an identifier and not an opener — see
+[Guards](#guards).
+
+## Inline code
+
+`inline_code:`, on by default.
+
+```ruby
+MaquinaRemend.call("`code")  # => "`code`"
+```
+
+An opening backtick with no partner is given one. The repair does not fire
+inside a fenced code block, where a backtick is content.
+
+## Strikethrough
+
+`strikethrough:`, on by default.
+
+```ruby
+MaquinaRemend.call("~~struck")  # => "~~struck~~"
+```
+
+A *single* tilde is a different problem with a different answer; see
+[Single tildes](#single-tildes).
+
+## Links and images
+
+`links:` for `[label](url)`, `images:` for `![alt](url)`, both on by default.
+`link_mode:` decides what a half-finished one becomes.
+
+A label that is still being typed is closed with an empty destination, so
+nothing links anywhere until the real URL arrives:
+
+```ruby
+MaquinaRemend.call("[label")  # => "[label]()"
+MaquinaRemend.call("![alt")   # => "![alt]()"
+```
+
+A destination that is still being typed is replaced by a placeholder:
+
+```ruby
+MaquinaRemend.call("[label](http")           # => "[label](#)"
+MaquinaRemend.call("[label](https://exa")    # => "[label](#)"
+```
+
+**A half-streamed destination is never kept.** `https://exa` is a perfectly
+valid URL to the wrong host, and model output is untrusted; the destination is
+a placeholder until the whole of it has arrived.
+
+### `link_mode:`
+
+| Value | `[label` | `[label](http` |
+|---|---|---|
+| `:protocol` (default) | `[label]()` | `[label](#)` |
+| `:text_only` | `label` | `label` |
+
+`:text_only` unwraps the markup and keeps the label, which is the right choice
+for a host that would rather show no link than a dead one:
+
+```ruby
+MaquinaRemend.call("[label", link_mode: :text_only)        # => "label"
+MaquinaRemend.call("[label](http", link_mode: :text_only)  # => "label"
+```
+
+Any other value raises:
+
+```ruby
+MaquinaRemend.call("x", link_mode: :guess)
+# MaquinaRemend::UnknownOption: link_mode must be one of protocol, text_only
+```
+
+Each flag guards its own kind, so images can be left broken while links are
+repaired:
+
+```ruby
+MaquinaRemend.call("![alt", images: false)  # => "![alt"
+```
+
+## Math
+
+`block_math:` on by default, `inline_math:` **off** by default.
+
+```ruby
+MaquinaRemend.call("$$x = 1")  # => "$$x = 1$$"
+```
+
+`$$` is unambiguous, so block math is repaired. A single `$` is not:
+
+```ruby
+MaquinaRemend.call("$x = 1")                     # => "$x = 1"
+MaquinaRemend.call("$x = 1", inline_math: true)  # => "$x = 1$"
+```
+
+**`inline_math:` is off because a bare dollar sign is currency far more often
+than it is mathematics.** Turn it on only if you know your corpus. Even with it
+on, a paired pair of dollar signs in prose is read as two prices rather than one
+equation:
+
+```ruby
+MaquinaRemend.call("costs $5 and $10", inline_math: true)
+# => "costs $5 and $10"
+```
+
+## Setext headings
+
+`setext_headings:`, on by default.
+
+A setext underline arrives one character at a time, and a lone `=` under a line
+of text is not yet a heading. It is padded out to the width of the heading text:
+
+```ruby
+MaquinaRemend.call("Title\n=")  # => "Title\n====="
+```
+
+Padding to the text's own width is what keeps the repair idempotent: repairing
+the result changes nothing, because the underline is already the right length.
+
+## Truncated HTML tags
+
+`html_tags:`, on by default.
+
+A tag whose `>` has not arrived is dropped, along with the whitespace in front
+of it. Left in place, it makes the renderer swallow everything that streams in
+afterwards as attribute soup:
+
+```ruby
+MaquinaRemend.call("text <div cla")  # => "text"
+```
+
+The tag comes back on the frame where its `>` finally arrives. This handler is
+also what makes [Application tags](#application-tags) simple: by the time that
+handler runs, every tag it can see is a whole one.
+
+## Application tags
+
+`app_tags:`, whose value is **a list of tag names** — `thinking`, `answer`,
+`tool_call`, `citation` and `scratchpad` by default, or `false` to switch the
+repair off.
+
+Models emit XML-ish tags that carry application meaning rather than
+presentation. An opening one with no closer is a broken document, and not
+subtly:
+
+```ruby
+MaquinaRemend.call("<thinking>\nstill reasoning")
+# => "<thinking>\nstill reasoning</thinking>"
+```
+
+### Why an open tag is broken
+
+CommonMark ends an HTML block at the first blank line. The moment the model
+types one, the rest of the answer leaves the tag:
+
+```
+<thinking>          parses as   <thinking>
+reasoning                       reasoning
+                                <p>the answer</p>
+the answer
+```
+
+In the parsed document those paragraphs are siblings of the tag. In the
+browser's DOM they are its children, because an unknown element is never
+implicitly closed — so the whole answer renders as reasoning. Appending the
+closer makes the two agree, and it costs nothing once the model's own closer
+arrives, because the buffer is then balanced and the repair does not fire.
+
+### Why it is a list and not a boolean
+
+**Closing every unbalanced tag would close the ones that must not be closed.**
+`<br>` and `<img>` have no closer at all. A `<div>` wrapper is routinely opened
+in one streamed block and closed several blocks later, so a speculative
+`</div>` would land in the middle of its own content. An application tag is
+different in kind: the model emits it whole, it wraps one region of the answer,
+and the host knows its name in advance. That is why the option takes names —
+the host is the only party that knows which tags are its own.
+
+So supply your own registry:
+
+```ruby
+MaquinaRemend.call("<plan>\nstep one", app_tags: %w[plan])
+# => "<plan>\nstep one</plan>"
+```
+
+A name not in the list is left open:
+
+```ruby
+MaquinaRemend.call("<thinking>\nreasoning", app_tags: %w[answer])
+# => "<thinking>\nreasoning"
+```
+
+And `false` switches the repair off entirely:
+
+```ruby
+MaquinaRemend.call("<thinking>\nstill reasoning", app_tags: false)
+# => "<thinking>\nstill reasoning"
+```
+
+Anything that is neither a list of names nor `false` is refused:
+
+```ruby
+MaquinaRemend.call("x", app_tags: 3)
+# MaquinaRemend::UnknownOption: app_tags must be false or a list of tag names
+```
+
+### Nesting, placement and the cases it declines
+
+Nested tags are closed innermost first, so the nesting the model opened is the
+nesting the renderer sees:
+
+```ruby
+MaquinaRemend.call("<answer>\n<citation>\nsrc")
+# => "<answer>\n<citation>\nsrc</citation></answer>"
+```
+
+Where the closer goes is not cosmetic. When the open tag is in the same block
+as the tail, the closer goes flush against it. When the tail has moved on to a
+later block, the closers go after a blank line, one per line — a line holding
+two closing tags is not an HTML block, and would be wrapped in a paragraph:
+
+```ruby
+MaquinaRemend.call("<thinking>\n\n```ruby\nputs \"<answer>\"\n```\n\nstill thinking")
+# => "<thinking>\n\n```ruby\nputs \"<answer>\"\n```\n\nstill thinking\n\n</thinking>"
+```
+
+That example also shows the fence winning: the `<answer>` inside the code block
+is content, not markup, and opens nothing.
+
+A self-closing tag opened nothing, so nothing is appended:
+
+```ruby
+MaquinaRemend.call("<citation id=\"1\"/>\n\nThe next paragraph.")
+# => "<citation id=\"1\"/>\n\nThe next paragraph."
+```
+
+A tag whose attributes are still being typed has no `>` yet, so it is dropped
+before this repair runs and there is nothing to close. The enclosing tag, which
+did arrive whole, is closed:
+
+```ruby
+MaquinaRemend.call("<answer>\nUsing <tool_call name=\"weather\" ci")
+# => "<answer>\nUsing</answer>"
+```
+
+The closer itself streams in one character at a time, and the document stays
+well formed at every frame in between:
+
+```ruby
+MaquinaRemend.call("<thinking>\nreasoning\n</thinkin")
+# => "<thinking>\nreasoning</thinking>"
+
+MaquinaRemend.call("<thinking>\nreasoning\n</thinking>")
+# => "<thinking>\nreasoning\n</thinking>"
+```
+
+On the first frame the truncated closer is dropped and a whole one is put back.
+On the next, the model's own closer has landed and nothing is appended — the
+speculative closer is *replaced* by the real one, never added to it.
+
+## Comparison operators
+
+`comparison_operators:`, on by default.
+
+A `>` inside a list item is escaped, so that a reflow or a continuation line
+cannot turn it into a blockquote nested inside the item:
+
+```ruby
+MaquinaRemend.call("- a > b")  # => "- a \\> b"
+```
+
+(That is Ruby's `inspect` form of a single backslash: the buffer becomes
+`- a \> b`.)
+
+## Single tildes
+
+`single_tilde:`, on by default.
+
+A lone tilde must not be read as a half-typed `~~`. It is escaped so it renders
+as a literal tilde instead of opening a strikethrough that never closes:
+
+```ruby
+MaquinaRemend.call("20~25°C")  # => "20\\~25°C"
+```
+
+## Dangling escapes
+
+`dangling_escape:`, on by default.
+
+A buffer cut in the middle of a LaTeX command, or after any lone backslash,
+ends in a backslash that would escape whatever the next repair appends. It is
+dropped first:
+
+```ruby
+MaquinaRemend.call("$$x = \\frac{-b \\")  # => "$$x = \\frac{-b $$"
+MaquinaRemend.call("media ecuación \\")   # => "media ecuación "
+```
+
+Without it the math repair produces `\$$` — an escaped dollar that leaves the
+math open, so the next pass appends another `$$`, and the pass after that
+another. This is why the handler runs first.
+
+## Guards
+
+A repair that cannot be made safe is not made. A false repair corrupts a
+message that was never broken, and no later frame undoes it.
+
+Nothing is ever completed inside a fenced code block, an inline code span, or
+math, because in those places broken-looking markup is the content:
+
+```ruby
+MaquinaRemend.call("```\n**not bold")   # => "```\n**not bold"
+MaquinaRemend.call("`a * b`")           # => "`a * b`"
+MaquinaRemend.call("$$a_1 + b_2$$")     # => "$$a_1 + b_2$$"
+MaquinaRemend.call("\\(a_1\\)")         # => "\\(a_1\\)"
+```
+
+The last one matters: math context is not only `$`. LaTeX `\( ... \)` is math
+too, and the underscore in it is a subscript.
+
+Underscores inside a word are an identifier, not emphasis:
+
+```ruby
+MaquinaRemend.call("some_var_name")  # => "some_var_name"
+```
+
+Two dollar signs in a sentence are two prices:
+
+```ruby
+MaquinaRemend.call("costs $5 and $10")  # => "costs $5 and $10"
+```
+
+And a complete document is returned byte for byte, whatever is in it. That is
+the property every repair above is written to preserve; see
+[streaming.md](streaming.md#the-three-properties).
+
+| Input | Result | Why |
+|---|---|---|
+| `` ```\n**not bold `` | unchanged | never complete anything inside a fence |
+| `` `a * b` `` | unchanged | the asterisk is literal inside code |
+| `` `<thinking>` `` | unchanged | a tag inside code is content, not markup |
+| `<citation id="1"/>` | unchanged | a self-closing tag opened nothing |
+| `$$a_1 + b_2$$` | unchanged | underscores are subscripts |
+| `\(a_1\)` | unchanged | math context is not only `$` |
+| `some_var_name` | unchanged | underscores inside a word are an identifier |
+| `costs $5 and $10` | unchanged | currency, not inline math |
+| any complete document | byte-identical | the no-op property |

data/docs/streaming.md

@@ -0,0 +1,162 @@
+# Using it in a stream
+
+`maquina_remend` is one function with no memory. The whole of using it in a
+stream is: on every frame, call it on the whole buffer so far, and render what
+comes back.
+
+```ruby
+require "maquina_remend"
+
+buffer = +""
+
+stream.each do |chunk|
+  buffer << chunk
+  render MaquinaRemend.call(buffer)
+end
+```
+
+The repaired string is for the renderer. **Keep appending chunks to `buffer`,
+never to the repaired output** — the repair is a view of the buffer, not a
+replacement for it. Appending the next chunk to `"**bold**"` gives you a
+document the model never wrote.
+
+## What a frame looks like
+
+The output of consecutive frames is not a growing prefix of one string; the
+completion moves along as the tail does, and it disappears the moment the model
+supplies its own closer. Every line below is a real frame:
+
+```
+""                             -> ""
+"Reading"                      -> "Reading"
+"Reading *"                    -> "Reading *"
+"Reading **"                   -> "Reading **"
+"Reading **t"                  -> "Reading **t**"
+"Reading **the"                -> "Reading **the**"
+"Reading **the manua"          -> "Reading **the manua**"
+"Reading **the manual"         -> "Reading **the manual**"
+"Reading **the manual*"        -> "Reading **the manual*"
+"Reading **the manual**"       -> "Reading **the manual**"
+"Reading **the manual** now."  -> "Reading **the manual** now."
+```
+
+Two things to read out of that. The reader sees bold text from the frame after
+the delimiter completes, instead of two literal asterisks for the length of the
+phrase. And the last frame is byte-identical to the input: once the document is
+finished, the gem is not in the picture at all.
+
+Application tags behave the same way across frames:
+
+```
+""                                     -> ""
+"<thi"                                 -> ""
+"<thinkin"                             -> ""
+"<thinking>\ns"                        -> "<thinking>\ns</thinking>"
+"<thinking>\nshort"                    -> "<thinking>\nshort</thinking>"
+"<thinking>\nshort\n</t"               -> "<thinking>\nshort</thinking>"
+"<thinking>\nshort\n</think"           -> "<thinking>\nshort</thinking>"
+"<thinking>\nshort\n</thinking>"       -> "<thinking>\nshort\n</thinking>"
+"<thinking>\nshort\n</thinking>\n\nDo" -> "<thinking>\nshort\n</thinking>\n\nDo"
+```
+
+The half-typed opening tag is dropped rather than shown. The speculative closer
+appears as soon as the opener is whole, and is *replaced* by the model's own
+closer rather than added to it.
+
+## The three properties
+
+Three properties hold over a corpus of whole documents at every truncation
+point, and they are what a caller is allowed to rely on.
+
+### Idempotence
+
+`call(call(x)) == call(x)`. Repairing a repaired buffer changes nothing.
+
+This is what makes the function safe to put anywhere in a pipeline. You do not
+have to know whether some earlier layer already called it, and a retried or
+replayed frame cannot accumulate closers.
+
+### No-op on well-formed input
+
+A document with nothing broken in it comes back byte for byte — including this
+page, which the test suite runs through `MaquinaRemend.call` and compares to
+itself.
+
+So the finished message is never the gem's output in any meaningful sense: it
+is the model's own bytes. Nothing needs to be undone at the end of a stream.
+
+### Prefix safety
+
+Every truncation point of every document is repairable without raising. The
+buffer can be cut anywhere — between the bytes of a multibyte character's
+markup neighbourhood, inside a table row, inside a fence's info string — and
+the call returns a String.
+
+For a caller this means no rescue around the call, and no minimum chunk size.
+Cut wherever the transport cuts.
+
+A fourth property is measured rather than proved: an 8KB buffer is repaired in
+under a millisecond, so calling it on every frame of a stream is not a cost
+worth engineering around.
+
+## Deciding how to render the tail
+
+A renderer often needs to know where the tail sits before it decides what to
+do with it — whether to run a syntax highlighter over a code block whose closing
+fence has not arrived, for one. That question has a public answer that does not
+involve repairing anything:
+
+```ruby
+context = MaquinaRemend.context(buffer)
+context.in_code_fence?  # => true
+context.open_fence_info # => "ruby"
+```
+
+`Context` is documented in [handlers.md](handlers.md#the-context-object).
+
+## Reusing a pipeline across frames
+
+Options are validated once, in the constructor. When the same options are used
+for every frame, build the pipeline once:
+
+```ruby
+pipeline = MaquinaRemend::Pipeline.new(inline_math: true)
+
+stream.each do |chunk|
+  buffer << chunk
+  render pipeline.call(buffer)
+end
+```
+
+It holds no state between buffers; this only avoids re-validating the option
+hash, and turns a bad option into a startup error rather than a surprise
+mid-stream.
+
+## What it deliberately does not do
+
+**It does not parse markdown.** It repairs a tail and hands back a String. What
+renders it is your business — CommonMark, a client-side renderer, anything.
+
+**It does not render HTML.** Nothing here escapes or sanitizes anything. Model
+output is untrusted, and it is still untrusted after this gem has seen it;
+sanitize downstream exactly as you would have without it.
+
+**It does not diff frames or compute deltas.** Every call is over the whole
+buffer, from scratch. A caller that wants deltas builds them on top, over the
+repaired output.
+
+**It does not hold a session.** No object accumulates chunks, no state carries
+from one frame to the next. Two frames repaired in either order, on two
+different machines, give the same answers.
+
+**It does not guess at a destination.** A half-arrived URL becomes a
+placeholder rather than a link somewhere unintended — see
+[repairs.md](repairs.md#links-and-images).
+
+**It does not repair the middle of a document.** Everything is anchored at the
+tail, because the tail is the only part of a streaming buffer that is
+provisional. A construct broken in the middle was broken by the model, and
+inventing a fix for it would silently rewrite a finished message.
+
+**It does not require Rails, or anything else.** Zero runtime dependencies is a
+property of the gem, asserted by the test suite in a bare Ruby process.

data/lib/maquina_remend/context.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+module MaquinaRemend
+  # What a handler is allowed to know about where the cursor sits. A handler that
+  # needs more than this is a sign the scanner is missing a state, not a sign
+  # this object should grow.
+  #
+  # Every handler — built-in or custom — receives one of these as its second
+  # argument, and MaquinaRemend.context returns one for callers that want the
+  # reading without the repair.
+  #
+  # ```ruby
+  # context = MaquinaRemend.context(buffer)
+  # renderer.highlight! unless context.in_code_fence?
+  # ```
+  #
+  # The three predicates are the three places where broken-looking markup is the
+  # content rather than a defect, which is why they are the whole of what a
+  # handler may ask:
+  #
+  # | Predicate | True when the tail sits inside |
+  # |---|---|
+  # | #in_code_fence? | an unclosed fenced code block |
+  # | #in_inline_code? | an unclosed backtick span in the current paragraph |
+  # | #in_math? | an unclosed `$$` block-math span |
+  #
+  # Instances are built by MaquinaRemend::Scanner and are read-only; there is no
+  # reason for a host to construct one directly, but nothing stops it.
+  class Context
+    # The info string of the open fence, or `nil` when no fence is open or the
+    # fence carried no info string.
+    #
+    # ```ruby
+    # MaquinaRemend.context("~~~ruby\ndef call").open_fence_info # => "ruby"
+    # MaquinaRemend.context("~~~\nplain").open_fence_info        # => nil
+    # ```
+    #
+    # This is what lets a renderer pick a highlighter for a code block whose
+    # closing fence has not streamed in yet.
+    attr_reader :open_fence_info
+
+    # The options hash the pipeline was built with — MaquinaRemend::DEFAULTS
+    # merged with whatever the caller passed. A custom handler that wants to
+    # read a host-specific setting can find it here.
+    #
+    # Empty when the context came from MaquinaRemend.context, which does no
+    # repair and therefore has no options.
+    attr_reader :options
+
+    # Builds a context. Called by MaquinaRemend::Scanner#context; a host has no
+    # reason to call it, but the arguments are the four states plus the options.
+    #
+    # * `in_code_fence:` — whether an unclosed fence is open
+    # * `in_inline_code:` — whether an unclosed backtick span is open
+    # * `in_math:` — whether an unclosed `$$` span is open
+    # * `open_fence_info:` — the open fence's info string, or `nil`
+    # * `options:` — the resolved option hash
+    def initialize(in_code_fence:, in_inline_code:, in_math:, open_fence_info: nil, options: {})
+      @in_code_fence = in_code_fence
+      @in_inline_code = in_inline_code
+      @in_math = in_math
+      @open_fence_info = open_fence_info
+      @options = options
+    end
+
+    # True when the buffer's tail sits inside a fenced code block whose closing
+    # fence has not arrived.
+    #
+    # Every built-in handler returns early on this: a fence is the one place
+    # where broken-looking markup is the content. A custom handler is *not*
+    # short-circuited, so it has to ask for itself.
+    def in_code_fence? = @in_code_fence
+
+    # True when the current paragraph has an odd number of backticks once the
+    # balanced spans are removed — an inline code span is still open.
+    def in_inline_code? = @in_inline_code
+
+    # True when an unclosed `$$` block-math span is open.
+    #
+    # Asked of the whole document rather than the current paragraph, because
+    # block math spans paragraphs — but only of the parts of the document that
+    # are prose, so a buffer that merely *writes about* `$$x = 1` inside a code
+    # span has no open math.
+    def in_math? = @in_math
+  end
+end