asciidoctor-iso 0.0.1

data/.rubocop.yml

@@ -0,0 +1,15 @@
+# This project follows the Ribose OSS style guide.
+# https://github.com/riboseinc/oss-guides
+# All project-specific additions and overrides should be specified in this file.
+
+inherit_from:
+  # Thoughtbot's style guide from: https://github.com/thoughtbot/guides
+  - ".rubocop.tb.yml"
+  # Overrides from Ribose
+  - ".rubocop.ribose.yml"
+AllCops:
+  DisplayCopNames: false
+  StyleGuideCopsOnly: false
+  TargetRubyVersion: 2.4
+Rails:
+  Enabled: true

data/Gemfile

@@ -0,0 +1,4 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in ribose.gemspec
+gemspec

data/README.adoc

@@ -0,0 +1,96 @@
+# asciidoctor-iso
+= Asciidoctor processor for ISO standards
+
+image:https://img.shields.io/gem/v/asciidoctor-rfc.svg["Gem Version", link="https://rubygems.org/gems/asciidoctor-iso"]
+image:https://img.shields.io/travis/riboseinc/asciidoctor-rfc/master.svg["Build Status", link="https://travis-ci.org/riboseinc/asciidoctor-iso"]
+image:https://codeclimate.com/github/riboseinc/asciidoctor-rfc/badges/gpa.svg["Code Climate", link="https://codeclimate.com/github/riboseinc/asciidoctor-iso"]
+
+This gem processes http://asciidoctor.org/[Asciidoctor] documents and outputs an XML representation of the document, intended as a document model for ISO International Standards. The XML representation can then be processed in turn to generate PDF or Microsoft Word output (via DocBook).
+
+The document model intends to introduce rigour into the ISO standards authoring process; the existing https://www.iso.org/iso-templates.html[Microsoft Word template from ISO] do not support such rigour down to the element level. The ISO International Standard format is prescribed in http://www.iec.ch/members_experts/refdocs/iec/isoiecdir-2%7Bed7.0%7Den.pdf[ISO/IEC DIR 2 "Principles and rules for the structure and drafting of ISO and IEC documents"], to a level that is amenable to an explicit document model. A formal document model would allow checking for consistency in format and content, and expedite authoring and quality control of ISO standards.
+
+The document model ("ISO XML") is under development, but it already contains all the markup needed to render the https://www.iso.org/publication/PUB100407.html["Rice document"], the ISO's model document of an international standard. It is expressed as a link:lib/asciidoctor/iso/validate.rnc[Relax NG Compact schema]; actual validation occurs against its link:lib/asciidoctor/iso/validate.rng[full Relax NG counterpart]. A UML representation of the document model is given below. Note that the document model is currently still in the exploratory phase, and will likely be changing significantly.
+
+Asciidoctor has been selected as the authoring tool to generate the document model representation of ISO standards. It is a document formatting tool like Markdown and DocBook, which combines the relative ease of use of the former (using relatively lightweight markup), and the rigour and expressively of the latter (it has a well-defined syntax, and was in fact initially developed as a DocBook document authoring tool). Asciidoctor has built-in capability to output Text, DocBook and HTML; so it can be used to preview the file as it is being authored.
+
+Note that in order to generate output close to what is intended, the Asciidoc document includes a fair amount of formatting instructions (e.g. disabling section numbering where appropriate, the titling of Appendixes as Annexes), as well as ISO boilerplate text, and predefined section headers (sections are recognised by fixed titles such as `Normative References`). Authoring ISO standards in this fashion assumes that users will be populating an Asciidoc template, and not removing needed formatting instructions.
+
+== Features not visible in HTML preview
+
+The gem uses built-in Asciidoc formatting as much as possible, so that users can retain the ability to preview documents; for _Terms and Definitions_ clauses, which have a good deal of explicit structure, macros have been introduced for semantic markup (admitted terms, deprecated terms, etc). The default HTML output of an Asciidoc-formatted ISO document is quite close to the intended final output, with the following exceptions: 
+
+* _Terms and Definitions_: each term is marked up as an unnumbered subclause, the semantic markup of alternate and other terms is not rendered visually.
+* _Formuals_: Asciidoctor has no provision for the automated numbering of isolated block formulas ("stem"), and does not display the number assigned a block formula in its default HTML processor—although it does provide automated numbering of examples. The encoding of formulas may change in future versions, although the final numbering is meant to be provided by downstream tools processing the ISO XML output.
+* _Missing elements_: The document model does not yet include Asciidoc elements that do not occur in the Rice document: in particular, source code, definition lists (except when used as keys for formuals or figures), examples (as distinct from figures; examples within _Terms and Definitions_ are catered for), sidebars (as distinct from warnings), quotes.
+* _Markup_: Some connecting text which is used to convey markup structure is left out: in particular, `DEPRECATED` and `SOURCE` (replaced by formatting macros).
+* _Tables_: Table footnotes are treated like all other footnotes: they are rendered at the bottom of the document, rather than the bottom of the table, and they are not numbered separately.
+* _Crossreferences_: Footnoted crossreferences are indicated with the reference text `fn` in isolation, or `fn:` as a prefix to the reference text. The default HTML processor leaves these as is: if no reference text is given, only `fn` will be displayed (though it will still hyperlink to the right reference).
+* _References_: The convention for references is that ISO documents are cited without brackets by ISO number, and optionally year, whether they are normative or in the bibliography (e.g. `ISO 20483:2013`); while all other references are cited by bracketed number in the bibliography (e.g. `[1]`). The default HTML processor treats all references the same, and will bracket them (e.g. `[ISO 20483:2013]`). For the same reason, ISO references listed in the bibliography will be listed under an ISO reference, rather than a bracketed number.
+* _References_: References are rendered cited throughout, since they are automated. For that reason, if reference is to be made to both an undated and a dated version of an ISO reference, these need to be explicitly listed as separate references. (This is not done in the Rice model document, which lists ISO 6646, but under _Terms and Definitions_ cites the dated ISO 6646:2011.
+* _References_: ISO references that are undated but published have their date indicated under the ISO standards format in an explanatory footnote. Because of constraints introduced by Asciidoctor, that explanation is instead given in square brackets in Asciidoc format.
+* _Annexes_: Subheadings cannot preserve subsection numbering, while also appearing inline with their text (e.g. Rice document, Annex B.2): they appear as headings in separate lines.
+* _Annexes_: Crossreferences to Annex subclauses are automatically prefixed with `Clause` rather than `Annex` or nothing.
+* _Metadata_: Document metadata such as document numbers, technical committees and title wording are not rendered in the default HTML output.
+* _Patent Notice_: Patent notices are treated and rendered as a subsection of the introduction, with an explicit subheading.
+* _Numbering_: The numbering of figures and tables is sequential in the default HTML processor: it does not include the Clause or Annex number. This, _Figure 1_, not _Figure A.1_.
+* _Notes_: There is no automatic note numbering by the default HTML processor.
+* _Keys_: Keys to formulas and figures are expected to be marked up as definition lists consistently, rather than as inline prose.
+* _Figures_: Simple figures are marked up as images, figures containing subfigures as examples. Numbering by the default HTML processor may be inconsistent. Subfigures are automatically numbered as independent figures.
+* _Markup_: The default HTML processor does not support CSS extensions such as small caps or strike through, though these can be marked up as CSS classes through custom macros in Asciidoc: a custom CSS stylesheet will be needed to render them.
+
+TODO: May need to only encode figures as examples.
+
+== Document Attributes
+
+The gem also relies on Asciidoc document attributes to provide necessary metadata about the document. These include:
+
+`:docnumber:`:: The ISO document number (mandatory)
+`:tc-docnumber:`:: The document number assigned by the Technical committee
+`:ref-docnumber:`:: The reference document number (appearing in page headers)
+`:partnumber:`:: The ISO document part number
+`:edition:`:: The document edition
+`:revdate:`:: The date the document was last updated
+`:copyright-year:`:: The year which will be claimed as when the copyright for the document was issued
+`:title-intro-en:`:: The introductory component of the English title of the document
+`:title-main-en:`:: The main component of the English title of the document (mandatory). (The first line of the Asciidoc document, which contains the title introduced with `=`, is ignored)
+`:title-part-en:`:: The English title of the document part
+`:title-intro-fr:`:: The introductory component of the French title of the document. (This document template presupposes authoring in English; a different template will be needed for French, including French titles of document components such as annexes.)
+`:title-main-fr:`:: The main component of the French title of the document (mandatory). 
+`:title-part-fr:`:: The French title of the document part
+`:doctype:`:: The document type (see https://www.iso.org/deliverables-all.html[ISO deliverables: The different types of ISO publications]) (mandatory). The permitted types are: `international-standard, technical-specification, technical-report, publicly-available-specification, international-workshop-agreement, guide`.
+`:docstage:`:: The stage code for the document status (see https://www.iso.org/stage-codes.html[International harmonized stage codes])
+`:docsubstage:`:: The substage code for the document status (see https://www.iso.org/stage-codes.html[International harmonized stage codes])
+`:secretariat:`:: The national body acting as the secretariat for the document in the deafting stage
+`:technical-committee-number:`:: The number of the relevant ISO technical committee
+`:technical-committee:`:: The name of the relevant ISO technical committee (mandatory)
+`:subcommittee-number:`:: The number of the relevant ISO subcommittee
+`:subcommittee:`:: The name of the relevant ISO subcommittee
+`:workgroup-number:`:: The number of the relevant ISO workgroup
+`:workgroup:`:: The name of the relevant ISO workgroup
+`:language:` :: The language of the document (`en` or `fr`)  (mandatory)
+
+The gem translates the document into ISO XML format, and then validates its output against the ISO XML document model; errors are reported to console against the XML, and are intended for users to check that they have provided all necessary components of the document.
+
+The attribute `:draft:`, if present, includes review notes in the XML output; these are otherwise suppressed.
+
+== Usage
+[source,console]
+----
+$ asciidoctor a.adoc  # HTML output of Asciidoc file
+$ asciidoctor -b iso -r 'asciidoctor-iso' a.adoc  # ISO XML output
+----
+
+== Document model
+
+image::grammar1.gif[]
+image::grammar2.gif[]
+image::grammar3.gif[]
+image::grammar4.gif[]
+
+
+== Examples
+The gem has been tested to date against the https://www.iso.org/publication/PUB100407.html["Rice document"], the ISO's model document of an international standard. This repository includes:
+
+* the link:spec/examples/rice.adoc[Asciidoc version of the Rice document].
+* the link:spec/examples/rice.html[Asciidoc rendering of the Rice document as HTML].
+* the link:spec/examples/rice.xml[ISO XML rendering of the Rice document].

data/asciidoctor-iso.gemspec

@@ -0,0 +1,44 @@
+# coding: utf-8
+
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "asciidoctor/iso/version"
+
+Gem::Specification.new do |spec|
+  spec.name          = "asciidoctor-iso"
+  spec.version       = Asciidoctor::ISO::VERSION
+  spec.authors       = ["Ribose Inc."]
+  spec.email         = ["open.source@ribose.com"]
+
+  spec.summary       = "asciidoctor-iso lets you write ISO standards in AsciiDoc."
+  spec.description   = <<~DESCRIPTION
+    asciidoctor-iso lets you write ISO standards in AsciiDoc syntax.
+
+    This gem is in active development.
+  DESCRIPTION
+
+  spec.homepage      = "https://github.com/riboseinc/asciidoctor-iso"
+  spec.license       = "MIT"
+
+  spec.bindir        = "bin"
+  spec.require_paths = ["lib"]
+  spec.files         = `git ls-files`.split("\n")
+  spec.test_files    = `git ls-files -- {spec}/*`.split("\n")
+  spec.required_ruby_version = Gem::Requirement.new(">= 2.3.0")
+
+  spec.add_dependency "asciidoctor", "~> 1.5.6"
+  spec.add_dependency "htmlentities", "~> 4.3.4"
+  spec.add_dependency "nokogiri", "~> 1.8.1"
+  spec.add_dependency "thread_safe"
+
+  spec.add_development_dependency "bundler", "~> 1.15"
+  spec.add_development_dependency "byebug", "~> 9.1"
+  spec.add_development_dependency "equivalent-xml", "~> 0.6"
+  spec.add_development_dependency "guard", "~> 2.14"
+  spec.add_development_dependency "guard-rspec", "~> 4.7"
+  spec.add_development_dependency "rake", "~> 12.0"
+  spec.add_development_dependency "rspec", "~> 3.6"
+  spec.add_development_dependency "rubocop", "~> 0.50"
+  spec.add_development_dependency "simplecov", "~> 0.15"
+  spec.add_development_dependency "timecop", "~> 0.9"
+end

data/lib/asciidoctor-iso.rb

@@ -0,0 +1,3 @@
+require "asciidoctor" unless defined? Asciidoctor::Converter
+require_relative "asciidoctor/iso/converter"
+require_relative "asciidoctor/iso/version"

data/lib/asciidoctor/iso/base.rb

@@ -0,0 +1,145 @@
+require "date"
+require "nokogiri"
+require "htmlentities"
+require "json"
+require "pathname"
+require "open-uri"
+require "pp"
+
+module Asciidoctor
+  module ISO
+    module Base
+      def content(node)
+        node.content
+      end
+
+      def skip(node, name = nil)
+        warn %(asciidoctor: WARNING (#{current_location(node)}): \
+          converter missing for #{name || node.node_name} node in ISO backend)
+        nil
+      end
+
+      def document(node)
+        result = ["<?xml version='1.0' encoding='UTF-8'?>\n<iso-standard#{document_ns_attributes node}>"]
+        $draft = node.attributes.has_key?("draft")
+        result << noko { |ixml| front node, ixml }
+        result << noko { |ixml| middle node, ixml }
+        result << "</iso-standard>"
+        ret = result.flatten * "\n"
+        ret1 = cleanup(Nokogiri::XML(ret))
+        Validate::validate(ret1)
+        ret1.to_xml(indent: 2)
+      end
+
+      def front(node, xml)
+        xml.front do |xml_front|
+          title node, xml_front
+          metadata node, xml_front
+        end
+      end
+
+      def middle(node, xml)
+        xml.middle do |xml_middle|
+          xml_middle << node.content if node.blocks?
+        end
+      end
+
+      def termsource(node)
+        result = []
+        result << noko do |xml|
+          xml.termref do |xml_t|
+            matched = /^(?<xref><xref[^>]+>)
+            (,\s(?<section>.[^, ]+))?
+              (,\s(?<text>.*))?$/x.match node.content
+            if matched.nil?
+              warn %(asciidoctor: WARNING (#{current_location(node)}): term reference not in expected format: #{node.content})
+            else
+              seen_xref = Nokogiri::XML.fragment(matched[:xref])
+              attr = {
+                target: seen_xref.children[0]["target"], 
+                format: seen_xref.children[0]["format"],
+              }
+              xml_t.xref seen_xref.children[0].content, **attr_code(attr)
+              xml_t.isosection matched[:section] if matched[:section]
+              xml_t.modification { |m| m << matched[:text] } if matched[:text]
+            end
+          end
+        end
+        result
+      end
+
+      def paragraph(node)
+        return termsource(node) if node.role == "source"
+        result = []
+        result << noko do |xml|
+          xml.p do |xml_t|
+            xml_t << node.content
+          end
+        end
+        result
+      end
+
+      def inline_footnote(node)
+        noko do |xml|
+          xml.fn do |xml_t|
+            xml_t << node.text
+          end
+        end.join
+      end
+
+      def open(node)
+        # open block is a container of multiple blocks,
+        # treated as a single block.
+        # We append each contained block to its parent
+        result = []
+        if node.blocks?
+          node.blocks.each do |b|
+            result << send(b.context, b)
+          end
+        else
+          result = paragraph(node)
+        end
+        result
+      end
+
+      def inline_break(node)
+        noko do |xml|
+          xml << node.text
+          xml.br
+        end.join
+      end
+
+      def page_break(node)
+        noko do |xml|
+          xml << node.text
+          xml.pagebreak
+        end.join
+      end
+
+      def inline_quoted(node)
+        noko do |xml|
+          case node.type
+          when :emphasis then xml.em node.text
+          when :strong then xml.strong node.text
+          when :monospaced then xml.tt node.text
+          when :double then xml << "\"#{node.text}\""
+          when :single then xml << "'#{node.text}'"
+          when :superscript then xml.sup node.text
+          when :subscript then xml.sub node.text
+          when :asciimath then xml.stem node.text
+          else
+            if node.role == "alt"
+              xml.admitted_term { |a| a << node.text }
+            elsif node.role == "deprecated"
+              xml.deprecated_term { |a| a << node.text }
+            elsif node.role == "domain"
+              xml.termdomain { |a| a << node.text }
+            else
+              xml << node.text
+            end
+          end
+        end.join
+      end
+    end
+  end
+end

data/lib/asciidoctor/iso/blocks.rb

@@ -0,0 +1,185 @@
+require "htmlentities"
+require "uri"
+
+module Asciidoctor
+  module ISO
+    module Blocks
+      def stem(node)
+        stem_attributes = {
+          anchor: node.id,
+        }
+        # NOTE: html escaping is performed by Nokogiri
+        stem_content = node.lines.join("\n")
+
+        noko do |xml|
+          xml.formula **attr_code(stem_attributes) do |s|
+            s.stem stem_content
+          end
+        end
+      end
+
+      def sidebar(node)
+        if $draft
+          note_attributes = {
+            color: node.attr("color") ? node.attr("color") : "red",
+          }
+          content = flatten_rawtext(node.content).join("\n")
+          noko do |xml|
+            xml.review_note content, **attr_code(note_attributes)
+          end
+        end
+      end
+
+      def termnote(node)
+        note_attributes = { anchor: node.id }
+
+        warn <<~WARNING_MESSAGE if node.blocks?
+          asciidoctor: WARNING (#{current_location(node)}): \
+          comment can not contain blocks of text in XML RFC:\n #{node.content}
+        WARNING_MESSAGE
+
+        noko do |xml|
+          xml.termnote **attr_code(note_attributes) do |xml_cref|
+            xml_cref << node.content
+          end
+        end.join
+      end
+
+      def admonition(node)
+        return termnote(node) if $term_def
+        noko do |xml|
+          xml.note **attr_code(anchor: node.id) do |xml_cref|
+            if node.blocks?
+              xml_cref << node.content
+            else
+              xml_cref.p { |p| p << node.content }
+            end
+          end
+        end.join
+      end
+
+      def term_example(node)
+        noko do |xml|
+          xml.termexample **attr_code(anchor: node.id) do |ex|
+            ex << node.content
+          end
+        end.join
+      end
+
+      def example(node)
+        return term_example(node) if $term_def
+        noko do |xml|
+          xml.example **attr_code(anchor: node.id) do |ex|
+            ex << node.content
+          end
+        end.join
+      end
+
+      def preamble(node)
+        result = []
+        result << noko do |xml|
+          xml.foreword do |xml_abstract|
+            xml_abstract << node.content
+          end
+        end
+        result
+      end
+
+      def section(node)
+        attr = { anchor: node.id.empty? ? nil : node.id }
+        noko do |xml|
+          case node.title.downcase
+          when "introduction"
+            xml.introduction **attr_code(attr) do |xml_section|
+              xml_section << node.content
+            end
+          when "patent notice"
+            xml.patent_notice **attr_code(attr) do |xml_section|
+              xml_section << node.content
+            end
+          when "scope"
+            xml.scope **attr_code(attr) do |xml_section|
+              xml_section << node.content
+            end
+          when "normative references"
+            $norm_ref = true
+            xml.norm_ref **attr_code(attr) do |xml_section|
+              xml_section << node.content
+            end
+            $norm_ref = false
+          when "terms and definitions"
+            $term_def = true
+            xml.terms_defs **attr_code(attr) do |xml_section|
+              xml_section << node.content
+            end
+            $term_def = false
+          when "bibliography"
+            $biblio = true
+            xml.bibliography **attr_code(attr) do |xml_section|
+              xml_section << node.content
+            end
+            $biblio = true
+          else
+            if $term_def
+              xml.termdef **attr_code(attr) do |xml_section|
+                xml_section.term { |name| name << node.title }
+                xml_section << node.content
+              end
+            elsif node.attr("style") == "appendix"
+              xml.annex **attr_code(attr) do |xml_section|
+                xml_section.name { |name| name << node.title }
+                xml_section << node.content
+              end
+            else
+              xml.clause **attr_code(attr) do |xml_section|
+                unless node.title.nil?
+                  xml_section.name { |name| name << node.title }
+                end
+                xml_section << node.content
+              end
+            end
+          end
+        end.join
+      end
+
+      def image(node)
+        uri = node.image_uri node.attr("target")
+        artwork_attributes = {
+          anchor: node.id,
+          src: uri,
+        }
+
+        noko do |xml|
+          xml.figure **attr_code(artwork_attributes) do |f|
+            f.name { |name| name << node.title } unless node.title.nil?
+          end
+        end
+      end
+
+      def quote(node)
+        noko do |xml|
+          xml.quote **attr_code(anchor: node.id) do |xml_blockquote|
+            if node.blocks?
+              xml_blockquote << node.content
+            else
+              xml_blockquote.p { |p| p << node.content }
+            end
+          end
+        end
+      end
+
+      def listing(node)
+        # NOTE: html escaping is performed by Nokogiri
+        noko do |xml|
+          if node.parent.context != :example
+            xml.figure do |xml_figure|
+              xml_figure.sourcecode { |s| s << node.content }
+            end
+          else
+            xml.sourcecode { |s| s << node.content }
+          end
+        end
+      end
+    end
+  end
+end