tediparse 2.0.0

data/doc/Parsing.md

@@ -0,0 +1,133 @@
+Parsing X12
+===========
+
+Parsing turns the token stream from the [reader](Tokenizing.md) into a typed
+**parse tree** — a tree of interchange, functional-group, transaction-set,
+table, loop, segment, and element values — by walking it against a grammar you
+have registered. Where [tokenizing](Tokenizing.md) is purely lexical, parsing is
+grammar-aware: it knows which segments may occur where, which loops they open,
+and which elements are composite or repeating.
+
+This page covers getting from input to a tree. Once you have the tree, see
+[Navigating](Navigating.md) to traverse it and [Serializing](Serializing.md) to
+write it back out.
+
+You bring the grammar
+---------------------
+
+Tediparse ships the engine, not the grammars. Before parsing real documents you
+must register your own grammar against a `Stupidedi::Config` — see
+[Defining](Defining.md) (authoring by hand) or
+[Generating-Grammars](Generating-Grammars.md) (generating from ASC X12 Table
+Data). The examples below use `Synthetic.config`, the worked grammar in
+`spec/support/synthetic/`; substitute your own populated config (e.g.
+`MyApp::EDI.config`).
+
+Building a parser and reading input
+-----------------------------------
+
+`Stupidedi::Parser.build(config)` returns a `Parser::StateMachine`. Feed it a
+[reader](Tokenizing.md) with `#read`, which returns a **`[machine, result]`
+pair**: the updated state machine (now holding the parse tree) and a
+`Reader::Result` describing how the read ended.
+
+```ruby
+require "tediparse"
+
+config = Synthetic.config                       # your populated Stupidedi::Config
+parser = Stupidedi::Parser.build(config)
+
+input  = File.read("path/to/your.edi")          # or File.open(..., :encoding => "ISO-8859-1")
+machine, result = parser.read(Stupidedi::Reader.build(input))
+```
+
+Both the parser and the input are values: `read` does not mutate `parser`, it
+returns a new machine. The original is still usable, which is what makes the
+[navigation](Navigating.md) combinators safe to chain.
+
+Checking the result
+--------------------
+
+The second element of the pair is a `Reader::Result`. Ask it whether the read
+ended fatally, and use `explain` to surface the reason with its position:
+
+```ruby
+if result.fatal?
+  result.explain { |reason| raise "#{reason} at #{result.position.inspect}" }
+end
+```
+
+`result.position` (offset / line / column) points into the input, so errors are
+locatable. A non-fatal result is the normal end-of-input case.
+
+What "fatal" does and doesn't mean
+----------------------------------
+
+A non-fatal result does **not** mean the document was fully valid — it means
+tokenizing and the overall walk completed. Individual segments that could not be
+placed in the grammar are recorded *in the tree* as `InvalidSegmentVal` /
+`InvalidEnvelopeVal` nodes rather than aborting the parse. Walk the tree to find
+them; see [Validating](Validating.md) for how to collect and interpret these.
+
+Missing grammar
+---------------
+
+If you parse a document whose interchange, functional-group, or transaction-set
+identifiers are not registered on the config, the parser does **not** raise — it
+produces a `FailureState` whose reason is
+`Stupidedi::Exceptions::MissingGrammarError::DEFAULT_MESSAGE` (a pointer to
+register a grammar). That surfaces as an `InvalidSegmentVal` in the tree at the
+envelope segment that couldn't be resolved. So an empty or partial config yields
+locatable failures, not a stack trace. (Reaching for a removed per-era constant
+like `Stupidedi::Versions::FiftyTen` in *code*, by contrast, raises
+`MissingGrammarError` directly.)
+
+Immutability and non-determinism
+--------------------------------
+
+The state machine is immutable and may track **more than one parse hypothesis at
+once**. Internally `machine.active` is a list of cursors — when the grammar is
+momentarily ambiguous (a segment that could open more than one loop, say), the
+parser keeps every live interpretation rather than guessing. `machine` answers
+`#deterministic?` with `false` while more than one hypothesis is live.
+
+`read` accepts a `:nondeterminism` option capping how many simultaneous
+hypotheses are tolerated (**default `1`**). Exceeding the cap ends the read with
+a fatal result ("too much non-determinism"):
+
+```ruby
+# Allow up to 8 concurrent parse hypotheses before giving up
+machine, result = parser.read(Stupidedi::Reader.build(input), :nondeterminism => 8)
+```
+
+Most well-formed documents stay deterministic; raise the limit only for grammars
+that are genuinely ambiguous at points the input later disambiguates. See
+[Navigating → Non-determinism](Navigating.md) for how traversal behaves while
+multiple trees are live, and [Validating](Validating.md) for catching ambiguity
+in a *grammar definition* up front.
+
+Getting at the tree
+-------------------
+
+The parse tree hangs off the machine. `machine.zipper` returns an
+[`Either`](Navigating.md) wrapping a cursor at the root of the value tree;
+`machine.zipper.fetch.root` gives you the root node (e.g. for the
+[writer](Serializing.md)). In practice you navigate with the higher-level
+combinators rather than touching the zipper directly:
+
+```ruby
+machine.first
+  .flatmap { |m| m.find(:GS) }
+  .flatmap { |m| m.find(:ST) }
+  .tap     { |m| puts m.segment.fetch.node.id }
+```
+
+From here, traversal is its own topic — continue with
+[Navigating the Parse Tree](Navigating.md).
+
+Where to go next
+----------------
+
+- [Navigating](Navigating.md) — find, iterate, and read values from the tree.
+- [Validating](Validating.md) — interpret invalid nodes and check grammars.
+- [Serializing](Serializing.md) — write a tree back to X12 text.

data/doc/README.md

@@ -0,0 +1,51 @@
+Tediparse Documentation
+========================
+
+Tediparse is a library for **parsing, generating, and validating** ASC X12 EDI
+documents. It ships the engine — the schema vocabulary, the immutable
+parser/state machine, the reader/writer, and the tree cursor — but **not** any
+X12 transaction-set grammars: that material is X12 IP you supply or generate.
+See the [project README](../README.md) for scope, licensing, and installation.
+
+These pages are the human documentation. They run roughly in the order you'd
+use the library: define or generate a grammar, then read, traverse, validate,
+and write documents against it. The runnable worked example every page refers to
+is the synthetic grammar under `spec/support/synthetic/` (`Synthetic.config`).
+
+Authoring a grammar
+-------------------
+
+You bring the grammar; these cover the two ways to get one.
+
+- [Defining](Defining.md) — transcribe a grammar by hand from a purchased X12
+  implementation guide (`TableDef`, `LoopDef`, `SegmentDef`, elements).
+- [Generating Grammars](Generating-Grammars.md) — generate the whole definition
+  tree for a release from licensed ASC X12 Table Data flat files.
+
+Generating documents
+--------------------
+
+- [Generating](Generating.md) — build well-formed X12 documents with the
+  `BuilderDsl` writer DSL, with validation on every segment.
+
+Reading & parsing
+-----------------
+
+- [Tokenizing](Tokenizing.md) — the lexical layer: bytes → segment tokens,
+  separators, and positions (no grammar required).
+- [Parsing](Parsing.md) — tokens → a typed parse tree, against a registered
+  grammar; results, immutability, and non-determinism.
+- [Navigating](Navigating.md) — traverse and read values from the parse tree
+  (`first`/`find`/`next`/`parent`, the `Either` combinators, and more).
+
+Validating
+----------
+
+- [Validating](Validating.md) — the three moments validation happens: while
+  generating, while parsing, and when auditing a grammar for ambiguity.
+
+Serializing
+-----------
+
+- [Serializing](Serializing.md) — write a parse tree back to X12 text
+  (`Writer::Default`) or HTML (`Writer::Claredi`).

data/doc/Serializing.md

@@ -0,0 +1,100 @@
+Serializing X12
+===============
+
+Serializing is the inverse of [parsing](Parsing.md): taking a parse tree and
+writing it back out. Once you have a tree — whether you
+[generated](Generating.md) it with the builder DSL or [parsed](Parsing.md) it
+from input — you serialize it with one of the writers under
+`Stupidedi::Writer`:
+
+- **`Writer::Default`** writes X12 text using a set of [separators](Tokenizing.md).
+- **`Writer::Claredi`** writes a formatted HTML rendering of the tree.
+
+Getting a tree to serialize
+---------------------------
+
+Both writers take the **root node** of the value tree. You reach it through the
+state machine's zipper, `machine.zipper.fetch.root`, in either of the two ways a
+tree comes to exist:
+
+```ruby
+# (a) After generating with the builder DSL
+b = Stupidedi::Parser::BuilderDsl.build(config)
+b.ISA(...); b.GS(...); b.ST(...); # ... ; b.SE(...); b.GE(...); b.IEA(...)
+root = b.machine.zipper.fetch.root
+
+# (b) After parsing input
+machine, _result = Stupidedi::Parser.build(config).read(Stupidedi::Reader.build(input))
+root = machine.zipper.fetch.root
+```
+
+`zipper` returns an [`Either`](Navigating.md); `.fetch` unwraps it (or raises if
+the tree is empty) and `.root` rewinds to the top of the tree.
+
+Writing X12 text
+----------------
+
+`Writer::Default.new(root, separators)` writes the tree as X12, using the
+separators you pass for the element, component, repetition, and segment
+delimiters. `#write` returns the serialized `String`:
+
+```ruby
+separators = Stupidedi::Reader::Separators.new(":", "^", "*", "~")
+#                                              comp  rep  elem  seg
+
+x12 = Stupidedi::Writer::Default.new(root, separators).write
+puts x12
+```
+
+You can serialize with **different separators than the input used** — the writer
+re-stamps the envelope's `ISA11`/`ISA16` to match the separators you request, so
+the output is internally consistent. Round-tripping with the *same* separators
+reproduces the original document (the synthetic smoke tests in
+`spec/lib/stupidedi/synthetic/smoke_spec.rb` assert this byte-for-byte).
+
+The role of separators
+----------------------
+
+`Stupidedi::Reader::Separators` (the same class the [reader](Tokenizing.md)
+discovers from `ISA`) drives every delimiter the writer emits. A few rules the
+writer enforces:
+
+- The segment terminator and element separator must be non-blank when writing an
+  interchange — a blank one raises `Stupidedi::Exceptions::OutputError`
+  (`"separators.segment cannot be blank"`).
+- A separator character must not collide with the data: if a value contains a
+  character you've chosen as a delimiter, the writer raises `OutputError`
+  (`'characters "~" occur as data'`) rather than emit an unparseable document.
+
+`Separators.default` (`(":", "^", "*", "~")`) is a convenient starting point.
+
+Writing HTML
+------------
+
+`Writer::Claredi` renders the tree as an HTML document — useful for inspection
+or display. It takes just the root node (no separators) and `#write` returns the
+output (an `IO`/`StringIO` you can read or write to a file):
+
+```ruby
+File.open("output.html", "w") do |f|
+  f.write Stupidedi::Writer::Claredi.new(root).write.string
+end
+```
+
+Serializing a subtree
+---------------------
+
+The writers don't require the interchange root — you can serialize any subtree.
+Navigate to the node you want, take its zipper, and write that:
+
+```ruby
+st = b.machine.first.flatmap { |m| m.sequence(:GS, :ST) }.fetch.zipper.fetch
+fragment = Stupidedi::Writer::Default.new(st, separators).write
+```
+
+Where to go next
+----------------
+
+- [Generating](Generating.md) — build a tree to serialize.
+- [Parsing](Parsing.md) — parse input into a tree to round-trip.
+- [Navigating](Navigating.md) — locate the subtree you want to write.

data/doc/Tokenizing.md

@@ -0,0 +1,136 @@
+Tokenizing X12
+==============
+
+Tokenizing is the first stage of reading an X12 document: turning a stream of
+bytes into a sequence of **segment tokens**, before any grammar is consulted.
+The reader knows about X12's *lexical* structure — segments, elements,
+components, repetitions, and the four delimiters — but nothing about what a
+particular transaction set means. That separation is deliberate: you can
+tokenize a document with no grammar registered at all, and the same tokenizer
+feeds the grammar-aware [parser](Parsing.md).
+
+The reader lives under `Stupidedi::Reader`. To go from tokens to a typed parse
+tree, see [Parsing](Parsing.md); this page covers the lexical layer only.
+
+Building a reader
+-----------------
+
+`Stupidedi::Reader.build(input)` returns a tokenizer. `input` may be a `String`,
+an `IO` (e.g. an open `File`), or any `Stupidedi::Reader::Input`:
+
+```ruby
+require "tediparse"
+
+input  = File.open("path/to/your.edi", :encoding => "ISO-8859-1")
+reader = Stupidedi::Reader.build(input)   # => Reader::StreamReader
+```
+
+X12 is an ISO-8859-1 (Latin-1) format, not UTF-8 — open files with that
+encoding to avoid corrupting extended characters.
+
+Two phases: StreamReader and TokenReader
+----------------------------------------
+
+Tokenizing happens in two phases, because the delimiters are not known until
+the `ISA` segment has been read:
+
+- **`StreamReader`** is what `Reader.build` returns. It skips any out-of-band
+  data (whitespace, line breaks, or other content between interchanges) until
+  it finds a literal `ISA`, then tokenizes that fixed-width segment. The `ISA`
+  segment is special: its layout is fixed by the standard, so the reader can
+  parse it byte-for-byte even before it knows the delimiters.
+- **`TokenReader`** takes over once `ISA` has been read. It is delimiter-aware
+  and tokenizes every following segment (`GS`, `ST`, …) using the separators
+  discovered from `ISA`.
+
+You rarely name these classes directly — you call `read_segment` and thread the
+reader it hands back, and the transition from `StreamReader` to `TokenReader`
+happens for you.
+
+Reading segments
+----------------
+
+A reader has no `each`; you pull one segment at a time with `read_segment`,
+which returns an [`Either`](Navigating.md) wrapping the token **and the next
+reader**. Thread that reader through `Either#flatmap` to consume the stream,
+stopping when the `Either` becomes a failure (end of input):
+
+```ruby
+reader = Stupidedi::Reader.build(input)
+
+drain = lambda do |r|
+  r.read_segment.flatmap do |segment_tok, next_reader|
+    puts segment_tok.id              # => :ISA, :GS, :ST, ...
+    drain.call(next_reader)          # recurse on the new reader
+  end
+end
+
+drain.call(reader)
+```
+
+Each success yields a `Reader::SegmentTok` plus the next reader; the previous
+reader is untouched (the readers are immutable, like the rest of the library).
+The parser's own read loop is exactly this pattern — see
+`lib/stupidedi/parser/generation.rb` for the production version.
+
+Tokens
+------
+
+- **`SegmentTok`** — one segment. `#id` is the segment identifier as a `Symbol`
+  (`:ISA`, `:ST`, …), `#element_toks` is the ordered list of element tokens,
+  and `#position` is where it started in the input (see below).
+- **`SimpleElementTok`** — one ordinary element; `#value` is its string value.
+- **`CompositeElementTok`**, **`ComponentElementTok`**, **`RepeatedElementTok`**
+  — composites, their components, and repeated elements, when the grammar (or
+  the raw delimiters) calls for them.
+
+Without a grammar, the `TokenReader` treats every element as simple — it has no
+way to know which elements are composite or repeating. Supply a grammar through
+the [parser](Parsing.md) to get composite/repeated structure.
+
+Separators
+----------
+
+`Stupidedi::Reader::Separators` carries the four X12 delimiters, each available
+as an accessor:
+
+| Accessor      | X12 role                 | Typical character |
+| ------------- | ------------------------ | ----------------- |
+| `#element`    | separates elements       | `*`               |
+| `#component`  | separates components     | `:`               |
+| `#repetition` | separates repetitions    | `^`               |
+| `#segment`    | terminates a segment     | `~`               |
+
+The delimiters are **discovered from the document**, not assumed — handling an
+unexpected choice (say, swapping `:` and `~`) is one of the things a hand-rolled
+parser usually gets wrong. They become known in stages:
+
+1. While reading `ISA`, `StreamReader` takes the character immediately after
+   `ISA` as the **element** separator, and the character after the 16th element
+   as the **segment** terminator.
+2. The **component** and **repetition** separators are version-dependent and are
+   resolved only when the parser enters the interchange envelope and the
+   grammar's `InterchangeDef#separators(isa)` reads them out of `ISA16` and
+   `ISA11`.
+
+So **standalone tokenizing (no grammar) gives you the element and segment
+delimiters**; full component/repetition resolution needs the envelope grammar,
+i.e. the [parser](Parsing.md). You can also construct separators explicitly —
+`Separators.default` is `(":", "^", "*", "~")` — which the [writer](Serializing.md)
+uses to serialize a tree back to X12.
+
+Positions
+---------
+
+Every token carries a `Stupidedi::Reader::Position` (`#position`) recording
+where it began: `#offset`, `#line`, `#column`, and `#path`. Positions are what
+make error messages actionable — when the [parser](Parsing.md) reports a
+failure, `result.position` points at the exact spot in the input.
+
+Where to go next
+----------------
+
+- [Parsing](Parsing.md) — feed the tokens to a grammar and build a typed parse
+  tree.
+- [Navigating](Navigating.md) — traverse the tree the parser builds.
+- [Serializing](Serializing.md) — turn a tree back into X12 text.

data/doc/Validating.md

@@ -0,0 +1,140 @@
+Validating X12
+==============
+
+Validation in tediparse happens at three distinct moments, and it helps to keep
+them separate because they report problems in different ways:
+
+1. **Generating** a document — the writer DSL validates each segment as you
+   build it and *raises* on the first violation.
+2. **Parsing** a document — segments that don't fit the grammar become
+   *invalid-value nodes in the tree* rather than exceptions, so one bad segment
+   doesn't abort the read.
+3. **Auditing a grammar** — the `Ambiguity` analyzer statically checks a
+   transaction-set definition for places the parser couldn't deterministically
+   resolve, and raises before you ship the grammar.
+
+This page covers all three. The examples use `Synthetic.config` and the
+synthetic grammar in `spec/support/synthetic/`; substitute your own grammar.
+
+Validation while generating
+---------------------------
+
+`Stupidedi::Parser::BuilderDsl` validates incrementally. Built with
+`strict = true` (the default), it checks each segment the instant you add it and
+raises `Stupidedi::Exceptions::ParseError` the moment your code violates the
+specification — with a stack trace pointing at the offending call, not after the
+whole document is assembled.
+
+```ruby
+b = Stupidedi::Parser::BuilderDsl.build(config)   # strict: true by default
+
+b.ISA(...)
+b.GS(...)
+b.ST("100", "0001")
+b.HH("Synthetic heading text")
+# b.IT("P:item-1", "3", "125.5")   # would raise ParseError: required segment
+                                    #   missing, value not allowed, too long, etc.
+```
+
+What `critique` checks as each node completes
+(`lib/stupidedi/parser/builder_dsl.rb`):
+
+- **Element type** — the value is valid for its element type (numeric, date,
+  time, …).
+- **Requirement** — a required element/segment isn't blank; a forbidden
+  ("not used") one isn't present.
+- **Code lists** — an `ID` element's value is one of its allowed values.
+- **Length** — the value isn't shorter or longer than the element's min/max.
+- **Syntax notes** — paired/conditional/exclusion rules (e.g. the `P` paired
+  note) are satisfied on composites and segments.
+- **Occurrence bounds** — a segment or loop occurs at least the required number
+  of times and no more than its repeat count allows.
+
+Passing `strict = false` to `BuilderDsl.build(config, false)` skips these checks
+— useful when you deliberately need to emit a non-conforming document for a
+trading partner that demands one.
+
+Validation while parsing
+------------------------
+
+When *reading* (see [Parsing](Parsing.md)), the parser is forgiving by design: a
+segment that can't be placed in the grammar — or an envelope whose grammar isn't
+registered — becomes an `InvalidSegmentVal` / `InvalidEnvelopeVal` node in the
+tree instead of raising. `result.fatal?` stays `false`; the document parses, and
+you inspect the tree to find what didn't fit.
+
+Walk the value tree and collect the invalid nodes:
+
+```ruby
+machine, result = Stupidedi::Parser.build(config).read(Stupidedi::Reader.build(input))
+
+def collect_failures(machine)
+  failures = []
+  walk = lambda do |z|
+    node = z.node
+    failures << node if node.is_a?(Stupidedi::Values::InvalidSegmentVal) ||
+                        node.is_a?(Stupidedi::Values::InvalidEnvelopeVal)
+    z.children.each { |c| walk.call(c) } if z.respond_to?(:children) && !z.leaf?
+  end
+  walk.call(machine.zipper.fetch.root)
+  failures
+end
+
+failures = collect_failures(machine)
+```
+
+An `InvalidSegmentVal` carries the reason it couldn't be placed and the original
+segment token (with its [position](Tokenizing.md)). A transaction set whose
+`ST01` code isn't registered, for example, yields an `InvalidSegmentVal` at the
+`ST` segment carrying the missing-grammar message (see
+[Parsing → Missing grammar](Parsing.md)). This is the pattern the smoke tests in
+`spec/lib/stupidedi/synthetic/smoke_spec.rb` use.
+
+Auditing a grammar for ambiguity
+--------------------------------
+
+The third kind of validation is about the *grammar itself*, not a document. A
+grammar is ambiguous when the parser, reading a given segment, can't
+deterministically decide which slot it fills — for instance two sibling loops
+that open on the same segment with overlapping allowed values on the
+discriminating element. Ambiguity is what forces the parser into the
+multiple-hypothesis state described in [Parsing](Parsing.md); catching it at the
+definition stage is far better than discovering it at runtime.
+
+`Stupidedi::TransactionSets::Validation::Ambiguity` audits a `TransactionSetDef`
+and raises `Stupidedi::Exceptions::InvalidSchemaError` when it finds an
+unresolvable choice:
+
+```ruby
+Stupidedi::TransactionSets::Validation::Ambiguity.build(
+  transaction_set_def,    # the TransactionSetDef to audit
+  functional_group_def,   # the GS/GE envelope it lives in
+  interchange_def         # the ISA/IEA envelope
+).audit
+# raises InvalidSchemaError if the definition is ambiguous
+```
+
+The synthetic suite exercises this against a deliberately ambiguous grammar —
+`spec/support/synthetic/ambiguous_demo.rb` defines two `LO` loops at the same
+position that both accept the discriminator value `"P"`, and
+`spec/lib/stupidedi/synthetic/ambiguity_spec.rb` asserts the audit raises an
+`InvalidSchemaError` whose message names the overlapping `ValueBased` choice:
+
+```ruby
+Stupidedi::TransactionSets::Validation::Ambiguity.build(
+  Synthetic::AmbiguousDemo,
+  Synthetic::FunctionalGroupDef,
+  Synthetic::InterchangeDef
+).audit
+#=> raises InvalidSchemaError: ... overlapping ... ValueBased ...
+```
+
+Run the audit over your transaction-set definitions in your own test suite to
+catch ambiguity before it reaches a parser.
+
+Where to go next
+----------------
+
+- [Generating](Generating.md) — the full writer DSL these checks guard.
+- [Parsing](Parsing.md) — how invalid nodes get into the tree.
+- [Navigating](Navigating.md) — traversing a tree that may contain invalid nodes.