csvops 0.5.0.alpha → 0.6.0.alpha

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b96fb7e03fa0629d3412a97d3abff5414492ac46ad08ede2c872e2176fcbfc62
-  data.tar.gz: 856b7735a472b5810d5f19dff6371a565a7fcc538ce5b6eba52260fff0028760
+  metadata.gz: f7db22cb84c1d08c58b473368f9ad37575a217d6293539309277ed2b032a2852
+  data.tar.gz: 124bebc822fefa5d1f71286701959876260c82164067c36ff94b712a0b4cc1b3
 SHA512:
-  metadata.gz: 5f643d331c6b54cb5feb0fe5db4ff7f8f7bc5c28461f74e3bfca5cf93d25703b84f497e72377302874b2b6302ef0fb542995c72d2d21798e3a998f6d5b294704
-  data.tar.gz: 0e254fa75780ce0605054c24b28301d8786535a0f2bbff7adfb45a75f09e60e5315e950648208fa5772d08cdd6abce95ea382838f568947af05ceaa77ba1888f
+  metadata.gz: a8b8dbcfb66073f46f0ecc625267081fbe730e69ef9295f5d2303af6b831a9d71ef564f78f5b44212eb33c4ad7a5fdb78b54fa98e21dd58669e9494a5d3325fb
+  data.tar.gz: 05cbcaa2ca3116ad463413e53600d32a53df0941ceb8873ed22c2ef2d4cfe1afc8f90e44c7ff4400212ebbd5083a2ceb6a983281291436e9478d5087cc98b9ad

data/README.md

@@ -37,11 +37,12 @@ CSV Tool Menu
 2. Extract rows (range)
 3. Randomize rows
 4. Dedupe using another CSV
-5. Exit
+5. Validate parity
+6. Exit
 >
 ```
 
-Select `1` for column extraction, `2` for row-range extraction, `3` for row randomization, or `4` for cross-CSV dedupe.
+Select `1` for column extraction, `2` for row-range extraction, `3` for row randomization, `4` for cross-CSV dedupe, or `5` for parity validation.
 
 ### 3. Follow prompts
 
@@ -59,6 +60,7 @@ Prompt flow by action:
 - `Extract rows (range)`: file path, separator, start row, end row, output destination.
 - `Randomize rows`: file path, separator, headers present, optional seed, output destination.
 - `Dedupe using another CSV`: source/reference files, separators, header modes, key selectors, match options, output destination.
+- `Validate parity`: left/right files, separator, header mode, parity summary, mismatch samples.
 
 ### 4. Example interaction (console output)
 
@@ -129,7 +131,8 @@ Legend: ` ` = prompt/menu, `+` = user input, `-` = tool output
  2. Extract rows (range)
  3. Randomize rows
  4. Dedupe using another CSV
- 5. Exit
+ 5. Validate parity
+ 6. Exit
 +> 4
  CSV file path: /tmp/source.csv
  Source CSV separator:
@@ -166,6 +169,45 @@ Legend: ` ` = prompt/menu, `+` = user input, `-` = tool output
 -Summary: source_rows=5 removed_rows=3 kept_rows=2
 ```
 
+### 8. Parity interaction example
+
+Legend: ` ` = prompt/menu, `+` = user input, `-` = tool output
+
+```diff
+ CSV Tool Menu
+ 1. Extract column
+ 2. Extract rows (range)
+ 3. Randomize rows
+ 4. Dedupe using another CSV
+ 5. Validate parity
+ 6. Exit
++> 5
+ Left CSV file path: /tmp/left.csv
+ Right CSV file path: /tmp/right.csv
+ Choose separator:
+ 1. comma (,)
+ 2. tab (\t)
+ 3. semicolon (;)
+ 4. pipe (|)
+ 5. custom
++Separator choice [1]: 1
+ Headers present? [Y/n]:
+-MISMATCH
+-Summary: left_rows=10 right_rows=10 left_only=2 right_only=2
+-Left-only examples:
+- 4,Dina (count +1)
+-Right-only examples:
+- 4,Dina-Updated (count +1)
+```
+
+### 9. Parity large-file behavior
+
+- Parity uses a streaming count-delta strategy:
+  - Stream left rows and increment row-key counts.
+  - Stream right rows and decrement row-key counts.
+- Exact duplicate semantics are preserved by count deltas per normalized row value.
+- Memory scales with the number of distinct row keys in the parity map, not the total input row count.
+
 ## Testing
 
 Run tests:

data/docs/architecture.md

@@ -3,14 +3,14 @@
 The codebase follows a DDD-lite layered structure:
 
 - `domain/`: core domain models and invariants (`ColumnSession`, `RowSession`, `RandomizationSession`, and `CrossCsvDedupeSession` aggregates + supporting entities/value objects).
-- `application/`: use-case orchestration (`RunExtraction`, `RunRowExtraction`, `RunRowRandomization`, `RunCrossCsvDedupe`).
-- `infrastructure/`: CSV reading/streaming and output adapters (console/file), plus cross-CSV dedupe adapter.
+- `application/`: use-case orchestration (`RunExtraction`, `RunRowExtraction`, `RunRowRandomization`, `RunCrossCsvDedupe`, `RunCsvParity`).
+- `infrastructure/`: CSV reading/streaming/comparison and output adapters (console/file).
 - `interface/cli/`: menu, prompts, workflows, and user-facing error presentation.
 - `Csvtool::CLI`: entrypoint wiring from command args to interface/application flow.
 
 ## Workflow boundary (standardized)
 
-For all interactive domains (`Column Extraction`, `Row Extraction`, `Row Randomization`, `Cross-CSV Dedupe`), the boundary is:
+For all interactive domains (`Column Extraction`, `Row Extraction`, `Row Randomization`, `Cross-CSV Dedupe`, `CSV Parity`), the boundary is:
 
 - `interface/cli/workflows/*`: owns prompts, stdout rendering, and user-facing error presentation.
 - `interface/cli/workflows/builders/*`: builds domain sessions/aggregates from prompt results.
@@ -32,6 +32,7 @@ Current usage:
 - `RunRowExtractionWorkflow` uses `WorkflowStepPipeline` + `Steps::RowExtraction::*`.
 - `RunRowRandomizationWorkflow` uses `WorkflowStepPipeline` + `Steps::RowRandomization::*`.
 - `RunCrossCsvDedupeWorkflow` uses `WorkflowStepPipeline` + `Steps::CrossCsvDedupe::*`.
+- `RunCsvParityWorkflow` uses `WorkflowStepPipeline` + `Steps::Parity::*`.
 
 ## Adding New Concepts
 
@@ -107,7 +108,7 @@ For a new function type, prefer one of these patterns:
 
 ## Domain model
 
-Bounded contexts: `Column Extraction`, `Row Extraction`, `Row Randomization`, and `Cross-CSV Dedupe`.
+Bounded contexts: `Column Extraction`, `Row Extraction`, `Row Randomization`, `Cross-CSV Dedupe`, and `CSV Parity`.
 
 ### Cross-CSV Dedupe (Large-file behavior)
 
@@ -366,6 +367,60 @@ classDiagram
   RunCrossCsvDedupe --> CsvCrossCsvDedupeFileWriter
 ```
 
+### CSV Parity
+
+Core DDD structure:
+
+- Aggregate root: `ParitySession`
+  - Captures one parity check request.
+  - Holds left/right source paths and parity options.
+- Entities:
+  - `SourcePair` (left and right file paths)
+- Value objects:
+  - `ParityOptions` (separator + header mode)
+- Application service:
+  - `Application::UseCases::RunCsvParity` orchestrates parity validation and returns request/result style payloads.
+- Infrastructure adapters:
+  - `Infrastructure::CSV::HeaderReader`
+  - `Infrastructure::CSV::CsvParityComparator` (streaming count-delta strategy with duplicate-aware semantics)
+- Interface adapters:
+  - `Interface::CLI::MenuLoop`
+  - `Interface::CLI::Workflows::RunCsvParityWorkflow`
+  - `Interface::CLI::Workflows::Builders::CsvParitySessionBuilder`
+  - `Interface::CLI::Workflows::Steps::WorkflowStepPipeline`
+  - `Interface::CLI::Workflows::Steps::Parity::*`
+  - `Interface::CLI::Workflows::Presenters::CsvParityPresenter`
+  - `Interface::CLI::Workflows::Support::ResultErrorHandler`
+  - `Interface::CLI::Prompts::*`
+  - `Interface::CLI::Errors::Presenter`
+
+```mermaid
+classDiagram
+  direction LR
+  class MenuLoop
+  class RunCsvParityWorkflow
+  class Prompts
+  class Errors
+  class RunCsvParity
+  class ParitySession
+  class SourcePair
+  class ParityOptions
+  class HeaderReader
+  class CsvParityComparator
+  class CsvParityPresenter
+
+  MenuLoop --> RunCsvParityWorkflow : invokes
+  RunCsvParityWorkflow --> Prompts : uses
+  RunCsvParityWorkflow --> Errors : reports failures
+  RunCsvParityWorkflow --> CsvParityPresenter : renders
+  RunCsvParityWorkflow --> RunCsvParity : calls
+  RunCsvParity --> ParitySession : orchestrates
+  ParitySession o-- SourcePair
+  ParitySession o-- ParityOptions
+  RunCsvParity --> HeaderReader
+  RunCsvParity --> CsvParityComparator
+```
+
 ## Project layout
 
 ```text
@@ -375,11 +430,13 @@ lib/csvtool/domain/column_session/*
 lib/csvtool/domain/row_session/*
 lib/csvtool/domain/row_randomization_session/*
 lib/csvtool/domain/cross_csv_dedupe_session/*
+lib/csvtool/domain/csv_parity_session/*
 lib/csvtool/domain/shared/output_destination.rb
 lib/csvtool/application/use_cases/run_extraction.rb
 lib/csvtool/application/use_cases/run_row_extraction.rb
 lib/csvtool/application/use_cases/run_row_randomization.rb
 lib/csvtool/application/use_cases/run_cross_csv_dedupe.rb
+lib/csvtool/application/use_cases/run_csv_parity.rb
 lib/csvtool/infrastructure/csv/*
 lib/csvtool/infrastructure/output/*
 lib/csvtool/interface/cli/menu_loop.rb

data/docs/release-v0.6.0-alpha.md

@@ -0,0 +1,84 @@
+# Release Checklist: v0.6.0-alpha
+
+## 1. Verify environment
+
+```bash
+ruby -v
+bundle -v
+```
+
+Expected:
+- Ruby `3.3.x`
+
+## 2. Install dependencies
+
+```bash
+bundle install
+```
+
+## 3. Run quality checks
+
+```bash
+bundle exec rake test
+```
+
+## 4. Smoke test CLI commands
+
+```bash
+bundle exec csvtool menu
+bundle exec csvtool column test/fixtures/sample_people.csv name
+```
+
+## 5. Smoke test workflows
+
+### CSV parity workflow
+
+Use menu option `5` (`Validate parity`) and verify:
+- matching files with reordered rows return parity success
+- mismatch files return friendly mismatch summary with sample deltas
+- separator and header-mode selections are respected
+
+### Existing workflows regression pass
+
+Run quick checks for menu options `1-4` and confirm:
+- column extraction still works
+- row-range extraction still works
+- row randomization still works
+- cross-CSV dedupe still works
+
+## 6. Build and validate gem package
+
+```bash
+gem build csvops.gemspec
+gem install ./csvops-0.6.0.alpha.gem
+csvtool menu
+```
+
+## 7. Commit release prep
+
+```bash
+git add -A
+git commit -m "chore(release): prepare v0.6.0-alpha"
+```
+
+## 8. Tag release
+
+```bash
+git tag -a v0.6.0-alpha -m "v0.6.0-alpha"
+git push origin main --tags
+```
+
+## 9. Publish gem
+
+```bash
+gem push csvops-0.6.0.alpha.gem
+```
+
+## 10. Create GitHub release
+
+Create release `v0.6.0-alpha` with:
+- Dedicated CSV parity validation workflow
+- Header/separator parity options
+- Friendly parity mismatch reporting
+- Streaming delta-count parity comparator
+- Parity architecture convergence (session model, workflow steps, presenter, docs)

data/lib/csvtool/application/use_cases/run_csv_parity.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+require "csv"
+require "csvtool/infrastructure/csv/csv_parity_comparator"
+require "csvtool/infrastructure/csv/header_reader"
+
+module Csvtool
+  module Application
+    module UseCases
+      class RunCsvParity
+        Result = Struct.new(:ok, :error, :data, keyword_init: true) do
+          def ok?
+            ok
+          end
+        end
+
+        def initialize(
+          comparator: Infrastructure::CSV::CsvParityComparator.new,
+          header_reader: Infrastructure::CSV::HeaderReader.new
+        )
+          @comparator = comparator
+          @header_reader = header_reader
+        end
+
+        def call(session:)
+          left_path = session.source_pair.left_path
+          right_path = session.source_pair.right_path
+          col_sep = session.options.separator
+          headers_present = session.options.headers_present?
+
+          return failure(:file_not_found, path: left_path) unless File.file?(left_path)
+          return failure(:file_not_found, path: right_path) unless File.file?(right_path)
+
+          if headers_present
+            left_headers = @header_reader.call(file_path: left_path, col_sep: col_sep)
+            return failure(:no_headers, path: left_path) if left_headers.empty?
+
+            right_headers = @header_reader.call(file_path: right_path, col_sep: col_sep)
+            return failure(:no_headers, path: right_path) if right_headers.empty?
+
+            return failure(:header_mismatch, left_headers: left_headers, right_headers: right_headers) unless left_headers == right_headers
+          end
+
+          stats = @comparator.call(
+            left_path: left_path,
+            right_path: right_path,
+            col_sep: col_sep,
+            headers_present: headers_present
+          )
+
+          success(stats)
+        rescue CSV::MalformedCSVError
+          failure(:could_not_parse_csv)
+        rescue Errno::EACCES => e
+          failure(:cannot_read_file, path: e.respond_to?(:path) ? e.path : left_path)
+        end
+
+        private
+
+        def success(data)
+          Result.new(ok: true, error: nil, data: data)
+        end
+
+        def failure(code, data = {})
+          Result.new(ok: false, error: code, data: data)
+        end
+      end
+    end
+  end
+end

data/lib/csvtool/cli.rb

@@ -6,6 +6,7 @@ require "csvtool/interface/cli/workflows/run_extraction_workflow"
 require "csvtool/interface/cli/workflows/run_row_extraction_workflow"
 require "csvtool/interface/cli/workflows/run_row_randomization_workflow"
 require "csvtool/interface/cli/workflows/run_cross_csv_dedupe_workflow"
+require "csvtool/interface/cli/workflows/run_csv_parity_workflow"
 require "csvtool/interface/cli/errors/presenter"
 require "csvtool/infrastructure/csv/header_reader"
 require "csvtool/infrastructure/csv/value_streamer"
@@ -18,6 +19,7 @@ module Csvtool
       "Extract rows (range)",
       "Randomize rows",
       "Dedupe using another CSV",
+      "Validate parity",
       "Exit"
     ].freeze
 
@@ -51,6 +53,7 @@ module Csvtool
       extract_rows_action = -> { Interface::CLI::Workflows::RunRowExtractionWorkflow.new(stdin: @stdin, stdout: @stdout).call }
       randomize_rows_action = -> { Interface::CLI::Workflows::RunRowRandomizationWorkflow.new(stdin: @stdin, stdout: @stdout).call }
       dedupe_action = -> { Interface::CLI::Workflows::RunCrossCsvDedupeWorkflow.new(stdin: @stdin, stdout: @stdout).call }
+      parity_action = -> { Interface::CLI::Workflows::RunCsvParityWorkflow.new(stdin: @stdin, stdout: @stdout).call }
       Interface::CLI::MenuLoop.new(
         stdin: @stdin,
         stdout: @stdout,
@@ -58,7 +61,8 @@ module Csvtool
         extract_column_action: extract_column_action,
         extract_rows_action: extract_rows_action,
         randomize_rows_action: randomize_rows_action,
-        dedupe_action: dedupe_action
+        dedupe_action: dedupe_action,
+        parity_action: parity_action
       ).run
     end
 

data/lib/csvtool/domain/csv_parity_session/parity_options.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module Csvtool
+  module Domain
+    module CsvParitySession
+      class ParityOptions
+        attr_reader :separator
+
+        def initialize(separator:, headers_present:)
+          raise ArgumentError, "separator cannot be empty" if separator.to_s.empty?
+
+          @separator = separator
+          @headers_present = headers_present
+        end
+
+        def headers_present?
+          @headers_present
+        end
+      end
+    end
+  end
+end

data/lib/csvtool/domain/csv_parity_session/parity_session.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module Csvtool
+  module Domain
+    module CsvParitySession
+      class ParitySession
+        attr_reader :source_pair, :options
+
+        def self.start(source_pair:, options:)
+          new(source_pair: source_pair, options: options)
+        end
+
+        def initialize(source_pair:, options:)
+          @source_pair = source_pair
+          @options = options
+        end
+      end
+    end
+  end
+end

data/lib/csvtool/domain/csv_parity_session/source_pair.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Csvtool
+  module Domain
+    module CsvParitySession
+      class SourcePair
+        attr_reader :left_path, :right_path
+
+        def initialize(left_path:, right_path:)
+          raise ArgumentError, "left_path cannot be empty" if left_path.to_s.empty?
+          raise ArgumentError, "right_path cannot be empty" if right_path.to_s.empty?
+
+          @left_path = left_path
+          @right_path = right_path
+        end
+      end
+    end
+  end
+end

data/lib/csvtool/infrastructure/csv/csv_parity_comparator.rb

@@ -0,0 +1,71 @@
+# frozen_string_literal: true
+
+require "csv"
+
+module Csvtool
+  module Infrastructure
+    module CSV
+      class CsvParityComparator
+        def call(left_path:, right_path:, col_sep:, headers_present:, sample_limit: 5)
+          deltas = Hash.new(0)
+          left_rows = stream_rows(path: left_path, col_sep: col_sep, headers_present: headers_present) do |key|
+            deltas[key] += 1
+          end
+          right_rows = stream_rows(path: right_path, col_sep: col_sep, headers_present: headers_present) do |key|
+            deltas[key] -= 1
+          end
+
+          left_only_count, right_only_count, left_only_examples, right_only_examples =
+            mismatch_totals_and_samples(deltas: deltas, sample_limit: sample_limit)
+
+          {
+            match: left_only_count.zero? && right_only_count.zero?,
+            left_rows: left_rows,
+            right_rows: right_rows,
+            left_only_count: left_only_count,
+            right_only_count: right_only_count,
+            left_only_examples: left_only_examples,
+            right_only_examples: right_only_examples
+          }
+        end
+
+        private
+
+        def stream_rows(path:, col_sep:, headers_present:)
+          rows = 0
+
+          ::CSV.foreach(path, headers: headers_present, col_sep: col_sep) do |row|
+            fields = headers_present ? row.fields : row
+            yield serialize(fields: fields, col_sep: col_sep)
+            rows += 1
+          end
+
+          rows
+        end
+
+        def mismatch_totals_and_samples(deltas:, sample_limit:)
+          left_only_count = 0
+          right_only_count = 0
+          left_only_examples = []
+          right_only_examples = []
+
+          deltas.each do |key, delta|
+            if delta.positive?
+              left_only_count += delta
+              left_only_examples << { row: key, count_delta: delta } if left_only_examples.length < sample_limit
+            elsif delta.negative?
+              right_only_count += -delta
+              right_only_examples << { row: key, count_delta: -delta } if right_only_examples.length < sample_limit
+            end
+          end
+
+          [left_only_count, right_only_count, left_only_examples, right_only_examples]
+        end
+
+        def serialize(fields:, col_sep:)
+          ::CSV.generate_line(fields, row_sep: "", col_sep: col_sep).chomp
+        end
+      end
+    end
+  end
+end

data/lib/csvtool/interface/cli/errors/presenter.rb

@@ -72,6 +72,10 @@ module Csvtool
           def row_range_out_of_bounds(total_rows)
             @stdout.puts "Row range is out of bounds. File has #{total_rows} data rows."
           end
+
+          def header_mismatch
+            @stdout.puts "CSV headers do not match."
+          end
         end
       end
     end

data/lib/csvtool/interface/cli/menu_loop.rb

@@ -4,7 +4,7 @@ module Csvtool
   module Interface
     module CLI
       class MenuLoop
-        def initialize(stdin:, stdout:, menu_options:, extract_column_action:, extract_rows_action:, randomize_rows_action:, dedupe_action:)
+        def initialize(stdin:, stdout:, menu_options:, extract_column_action:, extract_rows_action:, randomize_rows_action:, dedupe_action:, parity_action:)
           @stdin = stdin
           @stdout = stdout
           @menu_options = menu_options
@@ -12,6 +12,7 @@ module Csvtool
           @extract_rows_action = extract_rows_action
           @randomize_rows_action = randomize_rows_action
           @dedupe_action = dedupe_action
+          @parity_action = parity_action
         end
 
         def run
@@ -31,9 +32,11 @@ module Csvtool
             when "4"
               @dedupe_action.call
             when "5"
+              @parity_action.call
+            when "6"
               return 0
             else
-              @stdout.puts "Please choose 1, 2, 3, 4, or 5."
+              @stdout.puts "Please choose 1, 2, 3, 4, 5, or 6."
             end
           end
         end

data/lib/csvtool/interface/cli/workflows/builders/csv_parity_session_builder.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require "csvtool/domain/csv_parity_session/source_pair"
+require "csvtool/domain/csv_parity_session/parity_options"
+require "csvtool/domain/csv_parity_session/parity_session"
+
+module Csvtool
+  module Interface
+    module CLI
+      module Workflows
+        module Builders
+          class CsvParitySessionBuilder
+            def call(left_path:, right_path:, col_sep:, headers_present:)
+              source_pair = Domain::CsvParitySession::SourcePair.new(
+                left_path: left_path,
+                right_path: right_path
+              )
+              options = Domain::CsvParitySession::ParityOptions.new(
+                separator: col_sep,
+                headers_present: headers_present
+              )
+
+              Domain::CsvParitySession::ParitySession.start(
+                source_pair: source_pair,
+                options: options
+              )
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/csvtool/interface/cli/workflows/presenters/csv_parity_presenter.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module Csvtool
+  module Interface
+    module CLI
+      module Workflows
+        module Presenters
+          class CsvParityPresenter
+            def initialize(stdout:)
+              @stdout = stdout
+            end
+
+            def print_summary(data)
+              @stdout.puts(data[:match] ? "MATCH" : "MISMATCH")
+              @stdout.puts "Summary: left_rows=#{data[:left_rows]} right_rows=#{data[:right_rows]} " \
+                           "left_only=#{data[:left_only_count]} right_only=#{data[:right_only_count]}"
+              return if data[:match]
+
+              print_examples("Left-only examples", data[:left_only_examples])
+              print_examples("Right-only examples", data[:right_only_examples])
+            end
+
+            private
+
+            def print_examples(label, examples)
+              return if examples.nil? || examples.empty?
+
+              @stdout.puts "#{label}:"
+              examples.each do |example|
+                @stdout.puts "  #{example[:row]} (count +#{example[:count_delta]})"
+              end
+            end
+          end
+        end
+      end
+    end
+  end
+end