herbst 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b1a11bff9f5b5f00be6b23b297bc01faf785e1f661124b90129e9a509ecb10b4
-  data.tar.gz: 1d36830e13ca7faf498a8e30dd9997d58ffbc8b4027d832780969c3c9ab400c4
+  metadata.gz: 97c0f8a85d40558521510f33882058615c8530334598498b8db1407d799655d2
+  data.tar.gz: 917a913efabe6050570792a30fb49a12e6e9086b8e7ccf3fac90c2c35563f533
 SHA512:
-  metadata.gz: f1334e96f1c98dfa0a1a40d79955c45fcba7ef9ebf7ee0d943545c21c2288bcadb3114783226f3f63b2fc0199908b716de2f35ca95bc2949839b870bf320f37e
-  data.tar.gz: f93966c0949452056e56629a501640141d51ff2f2a72cedd3209379a9e650ad9fada8e6d91ece8e55cc87f82197d0b85adbcdf2b92b94f805d38fd67d91e1aec
+  metadata.gz: aa0044094d6f1fac7a519a105bcf848bc40ed2568039eef3c6288e4d23bb66b39c12c0d8f6006c6b284c7afb6a062a69aa0d612323c17d87eceef08c43fbe6ec
+  data.tar.gz: 56799703b87cfdef92316360560be4939c0f0d6cbcd9f25ff08c7f5d6d3dbfefe6a467fe6b18457eecc25f5fec02b94fbcda9ac9e5fe17ccfa691da6a94ae4fb

data/README.md

@@ -1,4 +1,4 @@
-<h2 align="center">Herbst 🍂</h2>
+<h2 align="center">🍂 Herbst</h2>
 
 <h4 align="center">Herb Syntax Tree</h4>
 
@@ -10,13 +10,13 @@ Herbst (German for "autumn") provides tooling to translate various HTML templati
 
 ## Why Herbst?
 
-Herbst bridges the gap between different HTML templating languages and the Herb ecosystem by:
+Herb's tooling (the linter, formatter, language server and `Herb::Engine`) works on Herb's syntax tree, which Herb builds from HTML+ERB. Herbst builds the same tree from other templating languages, so you can:
 
-- **Translating** Haml, Slim, and other HTML templating languages to Herb's unified syntax tree
-- **Enabling** Herb tooling (linter, formatter, language server) to work with any supported templating language
-- **Unifying** the developer experience across different HTML templating languages
+- **Convert** Haml and Slim templates to ERB, one file or a whole `app/views` at a time
+- **Compile** them with `Herb::Engine`, by converting them to ERB first
+- **Use** Herb's HTML-aware tooling on templates that aren't written in ERB
 
-All these HTML templating languages eventually produce HTML, and Herbst ensures they can all benefit from Herb's HTML-aware tooling.
+All of these languages render HTML in the end, so they can all share the same syntax tree.
 
 ## Installation
 
@@ -34,7 +34,41 @@ gem install herbst
 
 ## Usage
 
-Herbst is currently in early development. Usage instructions will be added as the gem matures.
+Herbst is currently in early development and supports converting Haml and Slim to ERB. Add the `haml` or `slim` gem for the languages you convert, since Herbst doesn't depend on either.
+
+Converting and printing ERB is pure Ruby. Formatting the output (`format: true`, and the default for the CLI) runs the Herb formatter through `npx`, so it requires Node.js.
+
+```ruby
+require "herbst"
+
+Herbst::Haml.parse("%p= greeting")   # => Herb::AST::DocumentNode
+Herbst::Haml.to_erb("%p= greeting")  # => "<p><%= greeting %></p>\n"
+Herbst::Haml.to_erb(source, format: true)
+
+Herbst::Slim.parse("p = greeting")     # => Herb::AST::DocumentNode
+Herbst::Slim.to_erb("p = greeting")    # => "<p><%= greeting %></p>"
+
+Herbst::IdentityPrinter.print(Herb.parse(erb, track_whitespace: true).value)
+```
+
+Herbst doesn't compile templates. To compile Haml or Slim, compile its ERB with `Herb::Engine`:
+
+```ruby
+Herb::Engine.new(Herbst::Haml.to_erb(source), escape: true).src
+```
+
+From the command line:
+
+```bash
+herbst convert app/views/posts/show.html.haml   # prints the ERB
+herbst convert app/views                        # writes show.html.erb next to show.html.haml or .slim
+```
+
+A single file is printed unless you pass `--write`. Directories, and several files at once, are written next to their sources with the `.haml` or `.slim` extension replaced by `.erb`, so `show.html.haml` becomes `show.html.erb`. A file without a format gets `html`, so `show.haml` becomes `show.html.erb`. Pass `--default-format FORMAT` for another format, or `--no-default-format` to write `show.erb`. The Haml and Slim files are kept, and existing ERB files are skipped unless you pass `--force`. Pass `--no-format` to skip the Herb formatter.
+
+Constructs that cannot be converted exactly (for example splatted attribute hashes, which become `tag.attributes(...)`) are reported as warnings. Files that can't be converted, for example because they use an unsupported filter such as `:markdown`, are reported and the remaining files are still converted.
+
+The CLI also checks that the ERB parses back into exactly the tree Herbst built, and reports any difference. Pass `--no-verify` to skip the check. The same check is available as `Herbst::RoundTrip.difference(document, erb)`. The CLI exits with status 1 when a file fails to convert or the check finds a difference.
 
 ## Development
 

data/Rakefile

@@ -7,4 +7,14 @@ require "rubocop/rake_task"
 Minitest::TestTask.create
 RuboCop::RakeTask.new
 
-task default: %i[test rubocop]
+desc "Generate RBS signatures from the inline annotations"
+task :rbs do
+  sh "bundle exec rbs-inline --opt-out --output=sig/ lib/"
+end
+
+desc "Type check with Steep"
+task :steep do
+  sh "bundle exec steep check"
+end
+
+task default: [:test, :rubocop, :steep]

data/exe/herbst

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative "../lib/herbst"
+
+exit Herbst::CLI.new(ARGV).run

data/herbst.gemspec

@@ -11,11 +11,12 @@ Gem::Specification.new do |spec|
   spec.summary = "Herb Syntax Tree - Unified syntax tree for HTML templating languages"
   spec.description = "Herbst provides tooling to translate Haml, Slim, and other HTML templating languages into the unified syntax tree used by Herb, enabling HTML-aware tooling across different HTML templating languages."
   spec.homepage = "https://github.com/marcoroth/herbst"
+  spec.license = "MIT"
   spec.required_ruby_version = ">= 3.2.0"
 
   spec.metadata["homepage_uri"] = spec.homepage
   spec.metadata["source_code_uri"] = "https://github.com/marcoroth/herbst"
-  spec.metadata["changelog_uri"] = "https://github.com/marcoroth/herbst/blob/main/CHANGELOG.md"
+  spec.metadata["changelog_uri"] = "https://github.com/marcoroth/herbst/releases"
   spec.metadata["rubygems_mfa_required"] = "true"
 
   spec.files = Dir[
@@ -24,7 +25,8 @@ Gem::Specification.new do |spec|
     "Rakefile",
     "README.md",
     "lib/**/*.rb",
-    "sig/**/*.rbs",
+    "sig/herbst.rbs",
+    "sig/herbst/**/*.rbs",
     "exe/*"
   ]
 
@@ -32,5 +34,5 @@ Gem::Specification.new do |spec|
   spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.add_dependency "herb"
+  spec.add_dependency "herb", ">= 0.11.1"
 end

data/lib/herbst/builder.rb

@@ -0,0 +1,358 @@
+# frozen_string_literal: true
+
+require "herb"
+require "herb/html/util"
+require "prism"
+
+module Herbst
+  module Builder
+    #: (String, String) -> Herb::Token
+    def token(type, value)
+      Herb::Token.from(type, value)
+    end
+
+    #: (Array[Herb::AST::Node]) -> Herb::AST::DocumentNode
+    def document(children)
+      Herb::AST::DocumentNode.build(children: children)
+    end
+
+    #: (String) -> Herb::AST::HTMLTextNode
+    def text(content)
+      Herb::AST::HTMLTextNode.build(content: +content)
+    end
+
+    #: (String) -> Herb::AST::LiteralNode
+    def literal(content)
+      Herb::AST::LiteralNode.build(content: +content)
+    end
+
+    #: (?String) -> Herb::AST::WhitespaceNode
+    def whitespace(value = " ")
+      Herb::AST::WhitespaceNode.build(value: token("TOKEN_WHITESPACE", value))
+    end
+
+    #: (String, ?opening: String) -> (Herb::AST::ERBContentNode | Herb::AST::ERBYieldNode)
+    def erb(code, opening: "<%=")
+      return Herb::AST::ERBYieldNode.build(**erb_tag(opening, code)) if yield?(code)
+
+      Herb::AST::ERBContentNode.build(**erb_tag(opening, code), parsed: true, valid: Prism.parse(code).errors.empty?)
+    end
+
+    #: (String) -> bool
+    def yield?(code)
+      code.include?("yield") && !Prism.parse(code).value.breadth_first_search { |node| node.is_a?(Prism::YieldNode) }.nil?
+    end
+
+    #: (String) -> Herb::AST::ERBCommentNode
+    def erb_comment(content)
+      Herb::AST::ERBCommentNode.build(**erb_tag("<%#", content))
+    end
+
+    #: () -> Herb::AST::ERBEndNode
+    def erb_end
+      Herb::AST::ERBEndNode.build(**erb_tag("<%", "end"))
+    end
+
+    #: (String, Array[Herb::AST::Node], ?subsequent: (Herb::AST::ERBIfNode | Herb::AST::ERBElseNode)?, ?end_node: Herb::AST::ERBEndNode?) -> Herb::AST::ERBIfNode
+    def erb_if(code, statements, subsequent: nil, end_node: erb_end)
+      Herb::AST::ERBIfNode.build(**erb_tag("<%", code), statements: statements, subsequent: subsequent, end_node: end_node)
+    end
+
+    #: (String, Array[Herb::AST::Node], ?else_clause: Herb::AST::ERBElseNode?) -> Herb::AST::ERBUnlessNode
+    def erb_unless(code, statements, else_clause: nil)
+      Herb::AST::ERBUnlessNode.build(**erb_tag("<%", code), statements: statements, else_clause: else_clause, end_node: erb_end)
+    end
+
+    #: (Array[Herb::AST::Node]) -> Herb::AST::ERBElseNode
+    def erb_else(statements)
+      Herb::AST::ERBElseNode.build(**erb_tag("<%", "else"), statements: statements)
+    end
+
+    #: (String, Array[Herb::AST::ERBWhenNode], ?else_clause: Herb::AST::ERBElseNode?) -> Herb::AST::ERBCaseNode
+    def erb_case(code, conditions, else_clause: nil)
+      Herb::AST::ERBCaseNode.build(**erb_tag("<%", code), conditions: conditions, else_clause: else_clause, end_node: erb_end)
+    end
+
+    #: (String, Array[Herb::AST::Node]) -> Herb::AST::ERBWhenNode
+    def erb_when(code, statements)
+      Herb::AST::ERBWhenNode.build(**erb_tag("<%", code), statements: statements)
+    end
+
+    #: (String, Array[Herb::AST::Node], ?rescue_clause: Herb::AST::ERBRescueNode?, ?else_clause: Herb::AST::ERBElseNode?, ?ensure_clause: Herb::AST::ERBEnsureNode?) -> Herb::AST::ERBBeginNode
+    def erb_begin(code, statements, rescue_clause: nil, else_clause: nil, ensure_clause: nil)
+      Herb::AST::ERBBeginNode.build(
+        **erb_tag("<%", code),
+        statements: statements,
+        rescue_clause: rescue_clause,
+        else_clause: else_clause,
+        ensure_clause: ensure_clause,
+        end_node: erb_end
+      )
+    end
+
+    #: (String, Array[Herb::AST::Node], ?subsequent: Herb::AST::ERBRescueNode?) -> Herb::AST::ERBRescueNode
+    def erb_rescue(code, statements, subsequent: nil)
+      Herb::AST::ERBRescueNode.build(**erb_tag("<%", code), statements: statements, subsequent: subsequent)
+    end
+
+    #: (Array[Herb::AST::Node]) -> Herb::AST::ERBEnsureNode
+    def erb_ensure(statements)
+      Herb::AST::ERBEnsureNode.build(**erb_tag("<%", "ensure"), statements: statements)
+    end
+
+    #: (String, Array[Herb::AST::Node]) -> Herb::AST::ERBWhileNode
+    def erb_while(code, statements)
+      Herb::AST::ERBWhileNode.build(**erb_tag("<%", code), statements: statements, end_node: erb_end)
+    end
+
+    #: (String, Array[Herb::AST::Node]) -> Herb::AST::ERBUntilNode
+    def erb_until(code, statements)
+      Herb::AST::ERBUntilNode.build(**erb_tag("<%", code), statements: statements, end_node: erb_end)
+    end
+
+    #: (String, Array[Herb::AST::Node]) -> Herb::AST::ERBForNode
+    def erb_for(code, statements)
+      Herb::AST::ERBForNode.build(**erb_tag("<%", code), statements: statements, end_node: erb_end)
+    end
+
+    #: (String, Array[Herb::AST::Node], ?opening: String) -> Herb::AST::ERBBlockNode
+    def erb_block(code, body, opening: "<%")
+      Herb::AST::ERBBlockNode.build(**erb_tag(opening, code), body: body, block_arguments: block_arguments(code), end_node: erb_end)
+    end
+
+    #: (String) -> Array[Herb::AST::RubyParameterNode]
+    def block_arguments(code)
+      # Like Herb, only a block passed to the first statement's method call has parameters, so
+      # `items.each do |item|` has them and `list = items.map do |item|` doesn't.
+      call = Prism.parse("#{code}\nend").value.statements.body.first
+      return [] unless call.is_a?(Prism::CallNode)
+
+      block = call.block
+      return [] unless block.is_a?(Prism::BlockNode)
+
+      block_parameters = block.parameters
+      return [] unless block_parameters.is_a?(Prism::BlockParametersNode)
+
+      parameters = block_parameters.parameters
+      return [] unless parameters
+
+      all = [*parameters.requireds, *parameters.optionals, parameters.rest, *parameters.posts, *parameters.keywords, parameters.keyword_rest, parameters.block].compact
+      all = all.flat_map { |parameter| destructured(parameter) }
+
+      all.filter_map { |parameter| ruby_parameter(parameter) }
+    end
+
+    #: (Prism::node) -> Herb::AST::RubyParameterNode?
+    def ruby_parameter(parameter)
+      case parameter
+      when Prism::RequiredParameterNode
+        ruby_parameter_node(parameter.name, "positional", true)
+      when Prism::OptionalParameterNode
+        ruby_parameter_node(parameter.name, "positional", false, parameter.value)
+      when Prism::RestParameterNode
+        ruby_parameter_node(parameter.name, "rest", false)
+      when Prism::RequiredKeywordParameterNode
+        ruby_parameter_node(parameter.name, "keyword", true)
+      when Prism::OptionalKeywordParameterNode
+        ruby_parameter_node(parameter.name, "keyword", false, parameter.value)
+      when Prism::KeywordRestParameterNode
+        ruby_parameter_node(parameter.name, "keyword_rest", false)
+      when Prism::BlockParameterNode
+        ruby_parameter_node(parameter.name, "block", false)
+      end
+    end
+
+    #: (Symbol?, String, bool, ?Prism::node?) -> Herb::AST::RubyParameterNode?
+    def ruby_parameter_node(name, kind, required, value = nil)
+      return unless name
+
+      default = Herb::AST::RubyLiteralNode.build(content: +value.slice) if value
+
+      Herb::AST::RubyParameterNode.build(
+        name: token("TOKEN_IDENTIFIER", name.to_s), default_value: default, kind: +kind, required: required
+      )
+    end
+
+    #: (String, ?attributes: Array[Herb::AST::Node], ?body: Array[Herb::AST::Node], ?self_closing: bool) -> Herb::AST::HTMLElementNode
+    def element(name, attributes: [], body: [], self_closing: false)
+      void = Herb::HTML::Util.void_element?(name)
+
+      self_closing &&= !void
+
+      children = open_tag_children(attributes)
+      children << whitespace if self_closing
+
+      open_tag = Herb::AST::HTMLOpenTagNode.build(
+        tag_opening: token("TOKEN_HTML_TAG_START", "<"),
+        tag_name: token("TOKEN_IDENTIFIER", name),
+        tag_closing: self_closing ? token("TOKEN_HTML_TAG_SELF_CLOSE", "/>") : token("TOKEN_HTML_TAG_END", ">"),
+        children: children,
+        is_void: self_closing
+      )
+
+      if Herb::HTML::Util.raw_text_element?(name)
+        body = merge_literals(body.map do |node|
+          node.is_a?(Herb::AST::HTMLTextNode) ? literal(node.content.to_s) : node
+        end)
+      end
+
+      # Herb reads the content of elements like `<script>` and `<textarea>` as text and ERB, so Herb
+      # builds one that contains anything else, like the templates in `<script type="text/ng-template">`.
+      if Herb::HTML::Util.raw_text_element?(name) || Herb::HTML::Util.rcdata_element?(name)
+        text_only = body.all? do |node|
+          node.is_a?(Herb::AST::LiteralNode) || node.is_a?(Herb::AST::HTMLTextNode) || node.is_a?(Herb::AST::ERBContentNode)
+        end
+
+        unless text_only
+          source = IdentityPrinter.print(element(name, attributes: attributes, body: []).tap { |node| node.body.replace(body) })
+
+          return Herb.parse(source, track_whitespace: true).value.children.fetch(0) #: Herb::AST::HTMLElementNode
+        end
+      end
+
+      close_tag = unless void || self_closing
+                    Herb::AST::HTMLCloseTagNode.build(
+                      tag_opening: token("TOKEN_HTML_TAG_START_CLOSE", "</"),
+                      tag_name: token("TOKEN_IDENTIFIER", name),
+                      tag_closing: token("TOKEN_HTML_TAG_END", ">")
+                    )
+                  end
+
+      Herb::AST::HTMLElementNode.build(
+        open_tag: open_tag,
+        tag_name: token("TOKEN_IDENTIFIER", name),
+        body: body,
+        close_tag: close_tag,
+        is_void: void || self_closing,
+        element_source: "HTML"
+      )
+    end
+
+    #: (String | Herb::AST::ERBContentNode, ?Array[Herb::AST::Node]?) -> Herb::AST::HTMLAttributeNode
+    def attribute(name, value = nil)
+      name_node = Herb::AST::HTMLAttributeNameNode.build(children: [name.is_a?(String) ? literal(name) : name])
+      return Herb::AST::HTMLAttributeNode.build(name: name_node) if value.nil?
+
+      Herb::AST::HTMLAttributeNode.build(
+        name: name_node,
+        equals: token("TOKEN_EQUALS", "="),
+        value: Herb::AST::HTMLAttributeValueNode.build(
+          open_quote: token("TOKEN_QUOTE", '"'),
+          children: merge_literals(value),
+          close_quote: token("TOKEN_QUOTE", '"'),
+          quoted: true
+        )
+      )
+    end
+
+    #: (Array[Herb::AST::Node]) -> Herb::AST::HTMLCommentNode
+    def comment(children)
+      children = children.map { |node| node.is_a?(Herb::AST::HTMLTextNode) ? literal(node.content.to_s) : node }
+
+      # Herb doesn't parse elements inside a comment, so Herb builds a comment that contains any.
+      unless children.all? { |node| node.is_a?(Herb::AST::LiteralNode) || node.is_a?(Herb::AST::ERBContentNode) }
+        source = "<!--#{children.map { |node| IdentityPrinter.print(node) }.join}-->"
+
+        return Herb.parse(source, track_whitespace: true).value.children.fetch(0) #: Herb::AST::HTMLCommentNode
+      end
+
+      Herb::AST::HTMLCommentNode.build(
+        comment_start: token("TOKEN_HTML_COMMENT_START", "<!--"),
+        children: merge_literals(children),
+        comment_end: token("TOKEN_HTML_COMMENT_END", "-->")
+      )
+    end
+
+    #: (String) -> Herb::AST::HTMLDoctypeNode
+    def doctype(content)
+      Herb::AST::HTMLDoctypeNode.build(
+        tag_opening: token("TOKEN_HTML_DOCTYPE", "<!DOCTYPE"),
+        children: [literal(content)],
+        tag_closing: token("TOKEN_HTML_TAG_END", ">")
+      )
+    end
+
+    #: (String) -> Herb::AST::XMLDeclarationNode
+    def xml_declaration(content)
+      Herb::AST::XMLDeclarationNode.build(
+        tag_opening: token("TOKEN_XML_DECLARATION", "<?xml"),
+        children: [literal(content)],
+        tag_closing: token("TOKEN_XML_DECLARATION_END", "?>")
+      )
+    end
+
+    #: (Prism::node) -> Array[Prism::node]
+    def destructured(parameter)
+      return [parameter] unless parameter.is_a?(Prism::MultiTargetNode)
+
+      [*parameter.lefts, *parameter.rights].flat_map { |target| destructured(target) }
+    end
+
+    #: (Array[Herb::AST::Node]) -> Array[Herb::AST::Node]
+    def open_tag_children(attributes)
+      attributes.each_with_index.flat_map do |node, index|
+        # Herb reads ERB in an open tag as an attribute name when an attribute with a value follows it.
+        valued = attributes.drop(index + 1).any? { |following| following.is_a?(Herb::AST::HTMLAttributeNode) && following.equals }
+
+        if node.is_a?(Herb::AST::ERBContentNode) && valued
+          [whitespace, attribute(node)]
+        else
+          [whitespace, node]
+        end
+      end
+    end
+
+    #: [T < Herb::AST::Node] (T) -> T
+    def merge_texts!(node)
+      node.instance_variables.each do |name|
+        value = node.instance_variable_get(name)
+
+        if value.is_a?(Array)
+          value.each { |child| merge_texts!(child) if child.is_a?(Herb::AST::Node) }
+          value.replace(merge_adjacent_texts(value))
+        elsif value.is_a?(Herb::AST::Node)
+          merge_texts!(value)
+        end
+      end
+
+      node
+    end
+
+    #: (Array[Herb::AST::Node]) -> Array[Herb::AST::Node]
+    def merge_adjacent_texts(nodes)
+      nodes.each_with_object([]) do |node, merged|
+        last = merged.last
+
+        if node.is_a?(Herb::AST::HTMLTextNode) && last.is_a?(Herb::AST::HTMLTextNode)
+          merged[-1] = text(last.content.to_s + node.content.to_s)
+        else
+          merged << node
+        end
+      end
+    end
+
+    #: (Array[Herb::AST::Node]) -> Array[Herb::AST::Node]
+    def merge_literals(nodes)
+      nodes.each_with_object([]) do |node, merged|
+        next if node.is_a?(Herb::AST::LiteralNode) && node.content.to_s.empty?
+
+        last = merged.last
+
+        if node.is_a?(Herb::AST::LiteralNode) && last.is_a?(Herb::AST::LiteralNode)
+          merged[-1] = literal(last.content.to_s + node.content.to_s)
+        else
+          merged << node
+        end
+      end
+    end
+
+    #: (String, String) -> { tag_opening: Herb::Token, content: Herb::Token, tag_closing: Herb::Token }
+    def erb_tag(opening, code)
+      {
+        tag_opening: token("TOKEN_ERB_START", opening),
+        content: token("TOKEN_ERB_CONTENT", " #{code.strip} "),
+        tag_closing: token("TOKEN_ERB_END", "%>"),
+      }
+    end
+  end
+end