dsv7-parser 7.0.0

data/README.md

@@ -0,0 +1,370 @@
+# dsv7-parser
+
+Ruby gem for a SAX parser targeting the DSV7 swim file format.
+
+Requirements
+
+- Ruby >= 2.7.0
+
+## Specification
+
+- DSV-Standard zur Datenübermittlung "Format 7" (external, DE): https://www.dsv.de/de/service/formulare/schwimmen/
+
+## Validator
+
+Basic envelope checks plus element validation for all four list types (WKDL, VML, ERG, VRL) are available via one entrypoint:
+
+```
+require 'dsv7/parser'
+
+# Pass a path, IO, or a String with file content
+result = Dsv7::Validator.validate('path/to/file.DSV7')
+
+puts "valid?     #{result.valid?}"
+puts "list_type: #{result.list_type}"
+puts "version:   #{result.version}"
+puts "errors:    #{result.errors.inspect}"
+puts "warnings:  #{result.warnings.inspect}"
+```
+
+Accepted inputs:
+
+- File path String: streamed from disk
+- IO object (e.g., `File.open` or `StringIO`): streamed
+- Content String: streamed via `StringIO`
+
+Structural checks (all list types):
+
+- First effective line is `FORMAT:<Listentyp>;7;` (whitespace tolerated)
+- List type is one of: `Wettkampfdefinitionsliste`, `Vereinsmeldeliste`,
+  `Wettkampfergebnisliste`, `Vereinsergebnisliste`
+- UTF‑8 encoding, BOM detection (BOM is an error)
+- Inline comments `(* ... *)` stripped; unbalanced `(*`/`*)` on a line is an error
+- Non‑empty data lines after FORMAT must contain at least one `;`
+- Terminator `DATEIENDE` present; no effective content after it
+
+Filename guidance (when validating by path):
+
+- Warns if the filename does not match `JJJJ-MM-TT-Ort-Zusatz.DSV7`
+
+Minimal example (generic list type):
+
+```
+content = <<~DSV
+  FORMAT:Vereinsmeldeliste;7;
+  DATA;ok
+  DATEIENDE
+DSV
+
+result = Dsv7::Validator.validate(content)
+puts result.valid? # => true
+```
+
+Wettkampfdefinitionsliste validation (cardinality + attribute types):
+
+```
+wkdl = <<~DSV
+  FORMAT:Wettkampfdefinitionsliste;7;
+  ERZEUGER:Soft;1.0;mail@example.com;
+  VERANSTALTUNG:Name;Ort;25;HANDZEIT;
+  VERANSTALTUNGSORT:Schwimmstadion;Strasse;12345;Ort;GER;tel;fax;mail@ex.amp.le;
+  AUSSCHREIBUNGIMNETZ:;
+  VERANSTALTER:Club;
+  AUSRICHTER:Verein;Kontakt;;;Ort;GER;;;kontakt@example.com;
+  MELDEADRESSE:Kontakt;;;;;;;kontakt@example.com;
+  MELDESCHLUSS:01.01.2024;12:00;
+  ABSCHNITT:1;01.01.2024;;;10:00;;
+  WETTKAMPF:1;V;1;;100;F;GL;M;SW;;;
+  MELDEGELD:EINZELMELDEGELD;2,00;;
+  DATEIENDE
+DSV
+
+wk_result = Dsv7::Validator.validate(wkdl)
+puts wk_result.valid?      # => true
+puts wk_result.errors      # => []
+puts wk_result.warnings    # => []
+```
+
+Vereinsmeldeliste validation (cardinality + attribute types):
+
+```
+vml = <<~DSV
+  FORMAT:Vereinsmeldeliste;7;
+  ERZEUGER:Soft;1.0;mail@example.com;
+  VERANSTALTUNG:Name;Ort;25;HANDZEIT;
+  ABSCHNITT:1;01.01.2024;10:00;N;
+  WETTKAMPF:1;V;1;;100;F;GL;M;;;
+  VEREIN:Mein Verein;1234;17;GER;
+  ANSPRECHPARTNER:Beispiel, Alice;;;;;;;alice@example.com;
+  DATEIENDE
+DSV
+
+vml_result = Dsv7::Validator.validate(vml)
+puts vml_result.valid?    # => true
+```
+
+Validated VML elements: ERZEUGER, VERANSTALTUNG, ABSCHNITT, WETTKAMPF, VEREIN,
+ANSPRECHPARTNER, KARIMELDUNG, KARIABSCHNITT, TRAINER, PNMELDUNG, HANDICAP,
+STARTPN, STMELDUNG, STARTST, STAFFELPERSON.
+
+Validated WKDL elements: ERZEUGER, VERANSTALTUNG, VERANSTALTUNGSORT, AUSSCHREIBUNGIMNETZ,
+VERANSTALTER, AUSRICHTER, MELDEADRESSE, MELDESCHLUSS, BANKVERBINDUNG, BESONDERES,
+NACHWEIS, ABSCHNITT, WETTKAMPF, WERTUNG, MELDEGELD.
+
+Validated ERG elements: ERZEUGER, VERANSTALTUNG, VERANSTALTER, AUSRICHTER, ABSCHNITT,
+KAMPFGERICHT, WETTKAMPF, WERTUNG, VEREIN, PNERGEBNIS, PNZWISCHENZEIT, PNREAKTION,
+STAFFELERGEBNIS/STERGEBNIS, STAFFELPERSON, STZWISCHENZEIT, STABLOESE.
+
+Validated VRL elements: ERZEUGER, VERANSTALTUNG, VERANSTALTER, AUSRICHTER, ABSCHNITT,
+KAMPFGERICHT, WETTKAMPF, WERTUNG, VEREIN, PERSON, PERSONENERGEBNIS, PNZWISCHENZEIT,
+PNREAKTION, STAFFEL, STAFFELPERSON, STAFFELERGEBNIS/STERGEBNIS, STZWISCHENZEIT, STABLOESE.
+
+Common error and warning examples:
+
+```
+# 1) Unknown list type and missing DATEIENDE
+bad = "FORMAT:Unbekannt;7;\n"
+r = Dsv7::Validator.validate(bad)
+r.errors.any? { |e| e.include?("Unknown list type in FORMAT: 'Unbekannt'") }
+r.errors.include?("Missing 'DATEIENDE' terminator line")
+
+# 2) Unsupported version
+r = Dsv7::Validator.validate("FORMAT:Vereinsergebnisliste;6;\nDATEIENDE\n")
+r.errors.any? { |e| e.include?("Unsupported format version '6', expected '7'") }
+
+# 3) Unbalanced comment delimiters
+r = Dsv7::Validator.validate("FORMAT:Vereinsmeldeliste;7; (* open\nDATEIENDE\n")
+r.errors.any? { |e| e.include?('Unmatched comment delimiters') }
+
+# 4) CRLF detection (warning only)
+crlf = "FORMAT:Vereinsmeldeliste;7;\r\nDATEIENDE\r\n"
+r = Dsv7::Validator.validate(crlf)
+r.valid?          # => true
+r.warnings        # => ['CRLF line endings detected']
+
+# 5) Missing delimiter ';' in a data line
+r = Dsv7::Validator.validate("FORMAT:Vereinsmeldeliste;7;\nDATA no semicolon\nDATEIENDE\n")
+r.errors.any? { |e| e.include?("Missing attribute delimiter ';'") }
+
+# 6) Filename pattern warning
+File.write('tmp/badname.txt', "FORMAT:Vereinsmeldeliste;7;\nDATEIENDE\n")
+begin
+  r = Dsv7::Validator.validate('tmp/badname.txt')
+  r.warnings.first.include?("does not follow 'JJJJ-MM-TT-Ort-Zusatz.DSV7'")
+ensure
+  File.delete('tmp/badname.txt')
+end
+```
+
+## Parser (Streaming: WKDL, VML, ERG, VRL)
+
+The parser provides a streaming API via one generic entrypoint and four type-specific helpers.
+
+- Generic (auto-detects list type): `Dsv7::Parser.parse(...)`
+
+Type-specific helpers (enforce list type):
+
+- Wettkampfdefinitionsliste (WKDL): `Dsv7::Parser.parse_wettkampfdefinitionsliste(...)`
+- Vereinsmeldeliste (VML): `Dsv7::Parser.parse_vereinsmeldeliste(...)`
+- Wettkampfergebnisliste (ERG): `Dsv7::Parser.parse_wettkampfergebnisliste(...)`
+- Vereinsergebnisliste (VRL): `Dsv7::Parser.parse_vereinsergebnisliste(...)`
+
+It is tolerant and focuses on extracting elements efficiently; use the validator for strict checks.
+
+Generic example (auto-detect list type):
+
+```
+enum = Dsv7::Parser.parse('path/to/file.DSV7')
+enum.each do |type, payload, line_number|
+  case type
+  when :format
+    # payload = { list_type: 'Vereinsmeldeliste', version: '7' }
+  when :element
+    # payload = { name: 'ERZEUGER', attrs: [...] }
+  when :end
+    # reached DATEIENDE
+  end
+end
+```
+
+Key points:
+
+- Input: pass a file path, an IO, or a String with file content.
+- Yields events: `:format`, `:element`, `:end` along with payload and line number.
+- Strips inline comments `(* ... *)` and scrubs invalid UTF‑8 in lines.
+- Accepts UTF‑8 with or without BOM (validator will still report BOM as an error).
+
+Basic example (block style):
+
+```
+require 'dsv7/parser'
+
+content = <<~DSV
+  (* header comment *)
+  FORMAT:Wettkampfdefinitionsliste;7;
+  ERZEUGER:Soft;1.0;mail@example.com;
+  VERANSTALTUNG:Name;Ort;25;HANDZEIT; (* inline *)
+  MELDESCHLUSS:01.01.2024;12:00;
+  DATEIENDE
+DSV
+
+Dsv7::Parser.parse(content) do |type, payload, line_number|
+  case type
+  when :format
+    # { list_type: 'Wettkampfdefinitionsliste', version: '7' }
+    p [:format, payload, line_number]
+  when :element
+    # { name: 'ERZEUGER', attrs: ['Soft','1.0','mail@example.com'] }
+    p [:element, payload, line_number]
+  when :end
+    p [:end, line_number]
+  end
+end
+```
+
+Enumerator style:
+
+```
+enum = Dsv7::Parser.parse('path/to/2002-03-10-Duisburg-Wk.DSV7')
+enum.each do |type, payload, line_number|
+  # same triplets as the block example
+end
+```
+
+Building a simple structure (header + elements) from the stream:
+
+```
+data = { format: nil, elements: [] }
+
+Dsv7::Parser.parse(content) do |type, payload, line_number|
+  case type
+  when :format
+    data[:format] = payload # { list_type: 'Wettkampfdefinitionsliste', version: '7' }
+  when :element
+    data[:elements] << { name: payload[:name], attrs: payload[:attrs], line_number: line_number }
+  end
+end
+
+# Example: pick only WETTKAMPF rows
+wettkaempfe = data[:elements]
+  .select { |e| e[:name] == 'WETTKAMPF' }
+  .map { |e| e[:attrs] } # arrays of attributes per row
+```
+
+Combining validation with parsing:
+
+```
+result = Dsv7::Validator.validate('path/to/file.DSV7')
+if result.valid?
+  Dsv7::Parser.parse('path/to/file.DSV7') do |type, payload, line_number|
+    # consume events
+  end
+else
+  warn "Invalid DSV7: #{result.errors.join('; ')}"
+end
+```
+
+VML usage mirrors WKDL:
+
+```
+content = <<~DSV
+  FORMAT:Vereinsmeldeliste;7;
+  ERZEUGER:Soft;1.0;mail@example.com;
+  VERANSTALTUNG:Name;Ort;25;HANDZEIT;
+  ABSCHNITT:1;01.01.2024;10:00;N;
+  WETTKAMPF:1;V;1;;100;F;GL;M;;;
+  VEREIN:Mein Verein;1234;17;GER;
+  ANSPRECHPARTNER:Beispiel, Alice;;;;;;;alice@example.com;
+  DATEIENDE
+DSV
+
+Dsv7::Parser.parse(content) do |type, payload, line_number|
+  # same :format, :element, :end semantics
+end
+```
+
+ERG usage mirrors WKDL as well:
+
+```
+content = <<~DSV
+  FORMAT:Wettkampfergebnisliste;7;
+  ERZEUGER:Soft;1.0;mail@example.com;
+  VERANSTALTUNG:Name;Ort;25;HANDZEIT;
+  ABSCHNITT:1;01.01.2024;10:00;N;
+  WETTKAMPF:1;A;1;;100;F;GL;M;SW;;;
+  VEREIN:SV Hansa Adorf;1234;17;GER;
+  DATEIENDE
+DSV
+
+Dsv7::Parser.parse(content) do |type, payload, line_number|
+  # same :format, :element, :end semantics
+end
+```
+
+VRL usage mirrors WKDL as well:
+
+```
+content = <<~DSV
+  FORMAT:Vereinsergebnisliste;7;
+  ERZEUGER:Soft;1.0;mail@example.com;
+  VERANSTALTUNG:Name;Ort;25;HANDZEIT;
+  ABSCHNITT:1;01.01.2024;10:00;N;
+  WETTKAMPF:1;A;1;;100;F;GL;M;SW;;;
+  VEREIN:SV Hansa Adorf;1234;17;GER;
+  DATEIENDE
+DSV
+
+Dsv7::Parser.parse(content) do |type, payload, line_number|
+  # same :format, :element, :end semantics
+end
+```
+
+Errors and edge cases:
+
+- Raises `Dsv7::Parser::Error` if the first effective line is not a `FORMAT` line.
+- Raises `Dsv7::Parser::Error` if the list type does not match the parser method
+  (WKDL expects `Wettkampfdefinitionsliste`, VML expects `Vereinsmeldeliste`,
+  ERG expects `Wettkampfergebnisliste`, VRL expects `Vereinsergebnisliste`).
+- Stops at `DATEIENDE`. Whitespace/comments after `DATEIENDE` are ignored by the parser (validator permits only comments/whitespace after it).
+
+## Development
+
+- Tests use Minitest and live under `test/dsv7/`.
+- Version is defined in `lib/dsv7/parser/version.rb`.
+
+## Compact ERG Example
+
+Minimal Wettkampfergebnisliste validation and parsing in one go:
+
+```
+require 'dsv7/parser'
+
+content = <<~DSV
+  FORMAT:Wettkampfergebnisliste;7;
+  ERZEUGER:Soft;1.0;mail@example.com;
+  VERANSTALTUNG:Name;Ort;25;HANDZEIT;
+  VERANSTALTER:Club;
+  AUSRICHTER:Verein;Kontakt;;;;;;kontakt@example.com;
+  ABSCHNITT:1;01.01.2024;10:00;N;
+  WETTKAMPF:1;A;1;;100;F;GL;M;SW;;;
+  WERTUNG:1;V;1;JG;0;;;OFFEN;
+  VEREIN:SV Hansa Adorf;1234;17;GER;
+  DATEIENDE
+DSV
+
+result = Dsv7::Validator.validate(content)
+if result.valid?
+  Dsv7::Parser.parse(content) do |type, payload, line_number|
+    case type
+    when :format
+      # { list_type: 'Wettkampfergebnisliste', version: '7' }
+    when :element
+      # payload: { name: 'ERZEUGER', attrs: ['Soft','1.0','mail@example.com'] }
+    when :end
+      # reached DATEIENDE
+    end
+  end
+else
+  warn "Invalid ERG: #{result.errors.join('; ')}"
+end
+```

data/Rakefile

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rake/testtask'
+require 'rubocop/rake_task'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.test_files = FileList['test/**/*_test.rb']
+  # Do not echo the underlying ruby command with the test file list
+  t.verbose = false
+end
+
+# Run tests and lint by default
+task default: %i[test lint]
+
+desc 'Run RuboCop'
+RuboCop::RakeTask.new(:rubocop)
+
+desc 'Run all linters'
+task lint: :rubocop
+
+desc 'CI: run tests and lint'
+task ci: %i[test lint]

data/TODO.md

@@ -0,0 +1,20 @@
+# TODO
+
+* Cardinalities are all in one file (lib/dsv7/validator/cardinality.rb). Split by list type (e.g., wk_cardinality.rb, vml_cardinality.rb, …) to keep files small and grep‑friendly.
+
+- Extracting allowed enum sets to named constants to reuse across modules
+
+- Add a tiny unit test for Lex.element edge cases (e.g., multiple trailing ;, empty attributes in the middle) to lock in splitting semantics.
+
+- Parser resource handling test: add a test that parses a file path and verifies the   underlying File is closed after enumeration.
+
+- In README.md, add a short “Supported validations” matrix per list type (what’s checked today vs. planned).
+
+- Add fuzz/robustness slices around:
+  Extremely long lines and many inline comments on a single line.
+
+- Garbage bytes mixed into attributes (parser should still emit sanitized attributes; validator should flag encoding).
+
+- Add a tiny CLI (bin/dsv7-validate) that reads from a path/stdin and prints errors/warnings. It improves adoption and makes quick checks easy.
+
+- Remove Gemfile.lock from VCS for a library gem.

data/dsv7-parser.gemspec

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require_relative 'lib/dsv7/parser/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'dsv7-parser'
+  spec.version       = Dsv7::Parser::VERSION
+  spec.authors       = ['bigcurl']
+  spec.email         = ['maintheme@gmail.com']
+
+  spec.summary       = 'SAX parser for the DSV7 swim file format'
+  spec.description   = 'Ruby gem for a DSV7 SAX parser.'
+  spec.homepage      = 'https://github.com/bigcurl/dsv7-parser'
+  spec.license       = 'MIT'
+
+  spec.required_ruby_version = Gem::Requirement.new('>= 2.7.0')
+
+  spec.metadata['homepage_uri'] = spec.homepage
+  spec.metadata['source_code_uri'] = spec.homepage
+  spec.metadata['changelog_uri'] = spec.homepage
+  spec.metadata['rubygems_mfa_required'] = 'true'
+
+  spec.files = Dir.chdir(File.expand_path(__dir__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|features)/}) }
+  end
+
+  spec.require_paths = ['lib']
+
+  # Runtime dependencies (none yet)
+  # spec.add_dependency "nokogiri", ">= 1.14"
+
+  # Development dependencies are declared in Gemfile
+end

data/lib/dsv7/lex.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Dsv7
+  module Lex
+    module_function
+
+    # Parses a FORMAT line. Returns [list_type, version] or nil if not a FORMAT line.
+    def parse_format(line)
+      m = line.match(/^FORMAT:([^;]+);([^;]+);$/)
+      return nil unless m
+
+      [m[1], m[2]]
+    end
+
+    # Splits an element content line into name and attributes.
+    # Returns [name, attrs] or nil if the line is not an element line.
+    def element(content)
+      return nil unless content.include?(':')
+
+      name, rest = content.split(':', 2)
+      name = name.strip if name
+      attrs = rest.split(';', -1)
+      attrs.pop if attrs.last == ''
+      [name, attrs]
+    end
+  end
+end

data/lib/dsv7/parser/engine.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+require 'stringio'
+require_relative '../stream'
+require_relative '../lex'
+require_relative 'io_util'
+
+module Dsv7
+  module Parser
+    # Internal engine that implements the streaming mechanics.
+    class Engine
+      def self.stream_any(input, emitter)
+        new(input, emitter).stream_any
+      end
+
+      def self.stream_list(input, emitter, expected_list_type)
+        new(input, emitter).stream_list(expected_list_type)
+      end
+
+      def initialize(input, emitter)
+        @input = input
+        @emitter = emitter
+      end
+
+      def stream_any
+        state = { ln: 0, saw: false }
+        IoUtil.with_io(@input) do |io|
+          IoUtil.each_content_line(io) do |content, ln|
+            state[:ln] = ln
+            break if handle_first_or_emit_any?(content, ln, @emitter, state)
+          end
+        end
+        @emitter << [:end, nil, state[:ln]]
+      end
+
+      def stream_list(expected_list_type)
+        state = { ln: 0, saw: false }
+        IoUtil.with_io(@input) do |io|
+          IoUtil.each_content_line(io) do |content, ln|
+            state[:ln] = ln
+            break if handle_first_or_emit?(content, ln, @emitter, state, expected_list_type)
+          end
+        end
+        @emitter << [:end, nil, state[:ln]]
+      end
+
+      private
+
+      def handle_first_or_emit?(content, line_number, emitter, state, expected_list_type)
+        unless state[:saw]
+          lt, ver = parse_format_expect(content, line_number, expected_list_type)
+          emitter << [:format, { list_type: lt, version: ver }, line_number]
+          state[:saw] = true
+          return false
+        end
+        return true if content == 'DATEIENDE'
+
+        emit_element(content, line_number, emitter)
+        false
+      end
+
+      def handle_first_or_emit_any?(content, line_number, emitter, state)
+        unless state[:saw]
+          lt, ver = parse_format_any(content, line_number)
+          emitter << [:format, { list_type: lt, version: ver }, line_number]
+          state[:saw] = true
+          return false
+        end
+        return true if content == 'DATEIENDE'
+
+        emit_element(content, line_number, emitter)
+        false
+      end
+
+      def parse_format_expect(content, line_number, expected_list_type)
+        m = Dsv7::Lex.parse_format(content)
+        format_required!(line_number) unless m
+
+        list_type, version = m
+        unless list_type == expected_list_type
+          short = parser_short_name(expected_list_type)
+          raise Dsv7::Parser::Error, "Unsupported list type '#{list_type}' for #{short} parser"
+        end
+
+        [list_type, version]
+      end
+
+      def parse_format_any(content, line_number)
+        pair = Dsv7::Lex.parse_format(content)
+        format_required!(line_number) unless pair
+        pair
+      end
+
+      def format_required!(line_number)
+        raise Dsv7::Parser::Error, "First non-empty line must be FORMAT (line #{line_number})"
+      end
+
+      def emit_element(content, line_number, emitter)
+        pair = Dsv7::Lex.element(content)
+        return unless pair
+
+        name, attrs = pair
+        emitter << [:element, { name: name, attrs: attrs }, line_number]
+      end
+
+      def parser_short_name(expected_list_type)
+        case expected_list_type
+        when 'Wettkampfdefinitionsliste' then 'WKDL'
+        when 'Vereinsmeldeliste' then 'VML'
+        when 'Wettkampfergebnisliste' then 'ERG'
+        else expected_list_type
+        end
+      end
+    end
+  end
+end

data/lib/dsv7/parser/io_util.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require 'stringio'
+require_relative '../stream'
+
+module Dsv7
+  module Parser
+    module IoUtil
+      module_function
+
+      def to_io(input)
+        return input if input.respond_to?(:read)
+        return File.open(input, 'rb') if input.is_a?(String) && File.file?(input)
+        return StringIO.new(String(input).b) if input.is_a?(String)
+
+        raise ArgumentError, 'Unsupported input; pass IO, file path String, or content String'
+      end
+
+      def with_io(input)
+        close_after = input.is_a?(String) && File.file?(input)
+        io = to_io(input)
+        Dsv7::Stream.binmode_if_possible(io)
+        Dsv7::Stream.read_bom?(io) # parser tolerates BOM
+        yield io
+      ensure
+        io&.close if close_after
+      end
+
+      def each_content_line(io)
+        ln = 0
+        io.each_line("\n") do |raw|
+          ln += 1
+          line = Dsv7::Stream.sanitize_line(raw)
+          content = Dsv7::Stream.strip_inline_comment(line).strip
+          next if content.empty?
+
+          yield content, ln
+        end
+      end
+    end
+  end
+end

data/lib/dsv7/parser/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Dsv7
+  module Parser
+    VERSION = '7.0.0'
+  end
+end