csvpath 0.0.488__tar.gz → 0.0.490__tar.gz

{csvpath-0.0.488 → csvpath-0.0.490}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.1
 Name: csvpath
-Version: 0.0.488
-Summary: A declarative language for validation of CSV files
+Version: 0.0.490
+Summary: A declarative language for validating CSV, Excel, and other tabular data files
 Author: David Kershaw
 Author-email: dk107dk@hotmail.com
 Requires-Python: >=3.9,<4.0
@@ -21,37 +21,40 @@ Classifier: Topic :: Software Development :: Quality Assurance
 Classifier: Topic :: Software Development :: Testing
 Classifier: Topic :: Text Processing
 Classifier: Topic :: Utilities
+Provides-Extra: pandas
+Provides-Extra: smartopen
 Requires-Dist: inflect (>=7.3.1,<8.0.0)
 Requires-Dist: jinja2 (>=3.1.4,<4.0.0)
 Requires-Dist: lark (>=1.2.2,<2.0.0)
 Requires-Dist: metaphone (>=0.6,<0.7)
+Requires-Dist: pandas (>=2.2.3,<3.0.0) ; extra == "pandas"
 Requires-Dist: ply (>=3.11,<4.0)
 Requires-Dist: pylightxl (>=1.61,<2.0)
 Requires-Dist: python-dateutil (>=2.9.0.post0,<3.0.0)
-Requires-Dist: smart-open[s3] (>=7.0.5,<8.0.0)
+Requires-Dist: smart-open[s3] (>=7.0.5,<8.0.0) ; extra == "smartopen"
 Requires-Dist: tabulate (>=0.9.0,<0.10.0)
 Project-URL: Csvpath.org, https://www.csvpath.org
 Project-URL: Github, https://github.com/csvpath/csvpath.git
 Description-Content-Type: text/markdown
 
 
-# <img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/> About CsvPath
+# <a href='https://www.csvpath.org/'><img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/></a> About CsvPath
 
-CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files.
+CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files, and other tabular data.
 
-CsvPath' goal is to make it easy to:
-- Analyze the content and structure of a CSV or Excel file
-- Validate that the file matches expectations
-- Report on the content or validity
-- Create new derived CSV files
+CsvPath's goal is to make it easy to setup a Collect, Store, Validate-pattern flat-file landing zone that:
+- Analyzes the content and structure of flat files
+- Validates that files match expectations
+- Reports on content validity
+- Creates new derived files using copy-on-write
 
-And do it all in an automation-friendly way.
+And does it all in an automation-friendly way.
 
-CsvPath is inspired by:
+CsvPath's validation is inspired by:
 - XPath for XML files
 - The ISO standard <a href='https://schematron.com/'>Schematron validation</a>
 
-CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. New functions are easy to create.
+CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. Metadata is plentiful. New functions are easy to create.
 
 Read more about CsvPath and see realistic CSV and Excel validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
 
@@ -63,6 +66,7 @@ If you need help, use the <a href='https://www.csvpath.org/getting-started/get-h
 # Contents
 
 - [Motivation](#motivation)
+- [Install](#install)
 - [High-level Description](#description)
 - [Running CsvPath](#running)
    - [Validation](#validating)
@@ -92,10 +96,37 @@ CSV files are everywhere!
 
 A surprisingly large number of companies depend on CSV processing for significant amounts of revenue. Research organizations are awash in CSV. And everyone's favorite issue tracker, database GUI, spreadsheet, APM platform, and most any other type of tool we use day to day uses CSV for sharing. CSV is the lowest of common dominators. Many CSVs are invalid or broken in some way. Often times a lot of manual effort goes into finding problems and fixing them.
 
-CsvPath is first and foremost a validation language. It is intended to describe CSV files in simple declarative rules that indicate if a file is as expected. CsvPath can also extract data, create reports, and in other ways have useful side effects.
+CsvPath is first and foremost a validation language. It describes tabular data in simple declarative rules that define what valid means for that data. CsvPath can also extract data, create reports, and do other useful things.
 
 CsvPath's goal is to make simple validations almost trivial and more complex situations more manageable. It is a library, not a system, so it relies on being easy to integrate with other DataOps tools.
 
+
+<a name="install"></a>
+# Install
+
+<a href='https://pypi.org/project/csvpath/'>CsvPath is available on PyPi</a>. Install with
+```
+    pip install csvpath
+```
+
+CsvPath has two optional dependencies:
+
+- <a target='_blank' href='https://pypi.org/project/pandas/'>Pandas</a>
+- <a target='_blank' href='https://pypi.org/project/smart-open/'>Smart-open</a>
+
+Pandas data frames can be used as a data source, much like Excel or CSV files. Install CsvPath with the Pandas option:
+```
+    pip install csvpath[pandas]
+```
+
+Smart-open is an option for loading data files directly from S3. Install the Smart Open extra with:
+```
+    pip install csvpath[smart-open]
+```
+
+Both of these optional dependencies can make it harder to use CsvPath in certain specific use cases. For e.g., using Pandas in an AWS Lambda layer may be less straightforward. If you need the capabilities, they are easy to install, but if you don't CsvPath is lighter weight without.
+
+
 # Description
 <a name="description"></a>
 
@@ -152,26 +183,25 @@ Two classes provide the functionality: CsvPath and CsvPaths. Each has only a few
 ### CsvPath
 (<a href='https://github.com/csvpath/csvpath/blob/main/csvpath/csvpath.py'>code</a>)
 The CsvPath class is the basic entry point for running csvpaths.
-|method                      |function                                                        |
-|----------------------------|----------------------------------------------------------------|
-| parse(csvpath)             | applies a csvpath                                              |
-| next()                     | iterates over matched rows returning each matched row as a list|
-| fast_forward()             | iterates over the file collecting variables and side effects   |
-| advance(n)                 | skips forward n rows from within a `for row in path.next()` loop|
-| collect(n)                 | processes n rows and collects the lines that matched as lists  |
+|method                      |function                                                         |
+|----------------------------|-----------------------------------------------------------------|
+| next()                     | iterates over matched rows returning each matched row as a list |
+| fast_forward()             | iterates over the file collecting variables and side effects    |
+| advance()                  | skips forward n rows from within a `for row in path.next()` loop|
+| collect()                  | processes n rows and collects the lines that matched as lists   |
 
 ### CsvPaths
 (<a href='https://github.com/dk107dk/csvpath/blob/main/csvpath/csvpaths.py'>code</a>)
 The CsvPaths class helps you manage validations of multiple files and/or multiple csvpaths. It coordinates the work of multiple CsvPath instances.
-|method                |function                                                         |
-|----------------------|-----------------------------------------------------------------|
-| csvpath()            | gets a CsvPath object that knows all the file names available   |
-| collect_paths()      | Same as CsvPath.collect() but for all paths sequentially        |
-| fast_forward_paths() | Same as CsvPath.fast_forward() but for all paths sequentially   |
-| next_paths()         | Same as CsvPath.next() but for all paths sequentially           |
-| collect_by_line()    | Same as CsvPath.collect() but for all paths breadth first       |
-| fast_forward_by_line()| Same as CsvPath.fast_forward() but for all paths breadth first |
-| next_by_line()       | Same as CsvPath.next() but for all paths breadth first          |
+|method                  |function                                                         |
+|------------------------|-----------------------------------------------------------------|
+| csvpath()              | gets a CsvPath object that knows all the file names available   |
+| collect_paths()        | Same as CsvPath.collect() but for all paths sequentially        |
+| fast_forward_paths()   | Same as CsvPath.fast_forward() but for all paths sequentially   |
+| next_paths()           | Same as CsvPath.next() but for all paths sequentially           |
+| collect_by_line()      | Same as CsvPath.collect() but for all paths breadth first       |
+| fast_forward_by_line() | Same as CsvPath.fast_forward() but for all paths breadth first  |
+| next_by_line()         | Same as CsvPath.next() but for all paths breadth first          |
 
 To be clear, the purpose of `CsvPaths` is to apply multiple csvpaths per CSV file. Its breadth-first versions of the `collect()`, `fast_forward()`, and `next()` methods attempt to match each csvpath to each row of a CSV file before continuing to the next row. As you can imagine, for very large files this approach is a must.
 
@@ -307,9 +337,9 @@ The match part is also bracketed. Matches have space separated components or "va
 ## Term
 A string, number, or regular expression value.
 
-|Returns | Matches | Examples      |
-|--------|---------|---------------|
-|A value | Always true | `"a value"` |
+|Returns | Matches     | Examples        |
+|--------|-------------|-----------------|
+|A value | Always true | `"a value"`     |
 
 <a href='https://github.com/dk107dk/csvpath/blob/main/docs/terms.md'>Read about terms here</a>.
 
@@ -317,9 +347,9 @@ A string, number, or regular expression value.
 ## Function
 A composable unit of functionality called once for every row scanned.
 
-|Returns | Matches | Examples      |
-|--------|---------|---------------|
-|Calculated | Calculated | `count()` |
+|Returns    | Matches    | Examples      |
+|-----------|------------|---------------|
+|Calculated | Calculated | `count()`     |
 
 <a href='https://github.com/dk107dk/csvpath/blob/main/docs/functions.md'>Read about functions here</a>.
 

{csvpath-0.0.488 → csvpath-0.0.490}/README.md

@@ -1,21 +1,21 @@
 
-# <img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/> About CsvPath
+# <a href='https://www.csvpath.org/'><img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/></a> About CsvPath
 
-CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files.
+CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files, and other tabular data.
 
-CsvPath' goal is to make it easy to:
-- Analyze the content and structure of a CSV or Excel file
-- Validate that the file matches expectations
-- Report on the content or validity
-- Create new derived CSV files
+CsvPath's goal is to make it easy to setup a Collect, Store, Validate-pattern flat-file landing zone that:
+- Analyzes the content and structure of flat files
+- Validates that files match expectations
+- Reports on content validity
+- Creates new derived files using copy-on-write
 
-And do it all in an automation-friendly way.
+And does it all in an automation-friendly way.
 
-CsvPath is inspired by:
+CsvPath's validation is inspired by:
 - XPath for XML files
 - The ISO standard <a href='https://schematron.com/'>Schematron validation</a>
 
-CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. New functions are easy to create.
+CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. Metadata is plentiful. New functions are easy to create.
 
 Read more about CsvPath and see realistic CSV and Excel validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
 
@@ -27,6 +27,7 @@ If you need help, use the <a href='https://www.csvpath.org/getting-started/get-h
 # Contents
 
 - [Motivation](#motivation)
+- [Install](#install)
 - [High-level Description](#description)
 - [Running CsvPath](#running)
    - [Validation](#validating)
@@ -56,10 +57,37 @@ CSV files are everywhere!
 
 A surprisingly large number of companies depend on CSV processing for significant amounts of revenue. Research organizations are awash in CSV. And everyone's favorite issue tracker, database GUI, spreadsheet, APM platform, and most any other type of tool we use day to day uses CSV for sharing. CSV is the lowest of common dominators. Many CSVs are invalid or broken in some way. Often times a lot of manual effort goes into finding problems and fixing them.
 
-CsvPath is first and foremost a validation language. It is intended to describe CSV files in simple declarative rules that indicate if a file is as expected. CsvPath can also extract data, create reports, and in other ways have useful side effects.
+CsvPath is first and foremost a validation language. It describes tabular data in simple declarative rules that define what valid means for that data. CsvPath can also extract data, create reports, and do other useful things.
 
 CsvPath's goal is to make simple validations almost trivial and more complex situations more manageable. It is a library, not a system, so it relies on being easy to integrate with other DataOps tools.
 
+
+<a name="install"></a>
+# Install
+
+<a href='https://pypi.org/project/csvpath/'>CsvPath is available on PyPi</a>. Install with
+```
+    pip install csvpath
+```
+
+CsvPath has two optional dependencies:
+
+- <a target='_blank' href='https://pypi.org/project/pandas/'>Pandas</a>
+- <a target='_blank' href='https://pypi.org/project/smart-open/'>Smart-open</a>
+
+Pandas data frames can be used as a data source, much like Excel or CSV files. Install CsvPath with the Pandas option:
+```
+    pip install csvpath[pandas]
+```
+
+Smart-open is an option for loading data files directly from S3. Install the Smart Open extra with:
+```
+    pip install csvpath[smart-open]
+```
+
+Both of these optional dependencies can make it harder to use CsvPath in certain specific use cases. For e.g., using Pandas in an AWS Lambda layer may be less straightforward. If you need the capabilities, they are easy to install, but if you don't CsvPath is lighter weight without.
+
+
 # Description
 <a name="description"></a>
 
@@ -116,26 +144,25 @@ Two classes provide the functionality: CsvPath and CsvPaths. Each has only a few
 ### CsvPath
 (<a href='https://github.com/csvpath/csvpath/blob/main/csvpath/csvpath.py'>code</a>)
 The CsvPath class is the basic entry point for running csvpaths.
-|method                      |function                                                        |
-|----------------------------|----------------------------------------------------------------|
-| parse(csvpath)             | applies a csvpath                                              |
-| next()                     | iterates over matched rows returning each matched row as a list|
-| fast_forward()             | iterates over the file collecting variables and side effects   |
-| advance(n)                 | skips forward n rows from within a `for row in path.next()` loop|
-| collect(n)                 | processes n rows and collects the lines that matched as lists  |
+|method                      |function                                                         |
+|----------------------------|-----------------------------------------------------------------|
+| next()                     | iterates over matched rows returning each matched row as a list |
+| fast_forward()             | iterates over the file collecting variables and side effects    |
+| advance()                  | skips forward n rows from within a `for row in path.next()` loop|
+| collect()                  | processes n rows and collects the lines that matched as lists   |
 
 ### CsvPaths
 (<a href='https://github.com/dk107dk/csvpath/blob/main/csvpath/csvpaths.py'>code</a>)
 The CsvPaths class helps you manage validations of multiple files and/or multiple csvpaths. It coordinates the work of multiple CsvPath instances.
-|method                |function                                                         |
-|----------------------|-----------------------------------------------------------------|
-| csvpath()            | gets a CsvPath object that knows all the file names available   |
-| collect_paths()      | Same as CsvPath.collect() but for all paths sequentially        |
-| fast_forward_paths() | Same as CsvPath.fast_forward() but for all paths sequentially   |
-| next_paths()         | Same as CsvPath.next() but for all paths sequentially           |
-| collect_by_line()    | Same as CsvPath.collect() but for all paths breadth first       |
-| fast_forward_by_line()| Same as CsvPath.fast_forward() but for all paths breadth first |
-| next_by_line()       | Same as CsvPath.next() but for all paths breadth first          |
+|method                  |function                                                         |
+|------------------------|-----------------------------------------------------------------|
+| csvpath()              | gets a CsvPath object that knows all the file names available   |
+| collect_paths()        | Same as CsvPath.collect() but for all paths sequentially        |
+| fast_forward_paths()   | Same as CsvPath.fast_forward() but for all paths sequentially   |
+| next_paths()           | Same as CsvPath.next() but for all paths sequentially           |
+| collect_by_line()      | Same as CsvPath.collect() but for all paths breadth first       |
+| fast_forward_by_line() | Same as CsvPath.fast_forward() but for all paths breadth first  |
+| next_by_line()         | Same as CsvPath.next() but for all paths breadth first          |
 
 To be clear, the purpose of `CsvPaths` is to apply multiple csvpaths per CSV file. Its breadth-first versions of the `collect()`, `fast_forward()`, and `next()` methods attempt to match each csvpath to each row of a CSV file before continuing to the next row. As you can imagine, for very large files this approach is a must.
 
@@ -271,9 +298,9 @@ The match part is also bracketed. Matches have space separated components or "va
 ## Term
 A string, number, or regular expression value.
 
-|Returns | Matches | Examples      |
-|--------|---------|---------------|
-|A value | Always true | `"a value"` |
+|Returns | Matches     | Examples        |
+|--------|-------------|-----------------|
+|A value | Always true | `"a value"`     |
 
 <a href='https://github.com/dk107dk/csvpath/blob/main/docs/terms.md'>Read about terms here</a>.
 
@@ -281,9 +308,9 @@ A string, number, or regular expression value.
 ## Function
 A composable unit of functionality called once for every row scanned.
 
-|Returns | Matches | Examples      |
-|--------|---------|---------------|
-|Calculated | Calculated | `count()` |
+|Returns    | Matches    | Examples      |
+|-----------|------------|---------------|
+|Calculated | Calculated | `count()`     |
 
 <a href='https://github.com/dk107dk/csvpath/blob/main/docs/functions.md'>Read about functions here</a>.
 

{csvpath-0.0.488 → csvpath-0.0.490}/csvpath/csvpath.py

@@ -27,7 +27,7 @@ from .util.exceptions import (
 )
 from .matching.util.exceptions import MatchException
 from csvpath.util.printer import Printer
-from csvpath.util.file_readers import CsvDataFileReader
+from csvpath.util.file_readers import DataFileReader
 
 
 class CsvPathPublic(ABC):
@@ -35,7 +35,10 @@ class CsvPathPublic(ABC):
 
     @abstractmethod
     def parse(self, csvpath):  # pragma: no cover
-        """Reads a csvpath prepares to match against CSV file lines"""
+        """Reads a csvpath prepares to match against CSV file lines. This
+        method is an alternative to simply passing the csvpath string to the
+        collect, next, or fast_forward methods. You don't do both.
+        """
 
     @abstractmethod
     def parse_named_path(
@@ -55,24 +58,31 @@ class CsvPathPublic(ABC):
         using the stop() function"""
 
     @abstractmethod
-    def collect(self, nexts: int = -1) -> List[List[Any]]:  # pragma: no cover
-        """Returns the lines of a CSV file that match the csvpath"""
+    def collect(
+        self, csvpath: str = None, *, nexts: int = -1
+    ) -> List[List[Any]]:  # pragma: no cover
+        """Returns the lines of a CSV file that match the csvpath. Pass
+        nexts to limit a run to collecting only N lines; the default
+        is -1 for collecting all. If you do not pass the csvpath
+        string here you must first use the parse method."""
 
     @abstractmethod
     def advance(self, ff: int = -1) -> None:  # pragma: no cover
         """Advances the iteration by ff rows. -1 means to the end of the file."""
 
     @abstractmethod
-    def fast_forward(self) -> None:  # pragma: no cover
+    def fast_forward(self, csvpath: str = None) -> None:  # pragma: no cover
         """Scans to the end of the CSV file. All scanned rows will be
         considered for match and variables and side effects will happen,
         but no rows will be returned or stored. -1 means to the end of
-        the file."""
+        the file. If you do not pass the csvpath string here you must first
+        use the parse method."""
 
     @abstractmethod
-    def next(self):  # pragma: no cover
+    def next(self, csvpath: str = None):  # pragma: no cover
         """A generator function that steps through the CSV file returning
-        matching rows"""
+        matching rows. If you do not pass the csvpath string here you must
+        first use the parse method."""
 
 
 class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902, R0904
@@ -290,6 +300,52 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
         self._created_at = datetime.now()
         self._run_started_at = None
 
+        self._collecting = False
+        self._unmatched = None
+        self._unmatched_available = False
+        self._data_from_preceding = False
+
+    @property
+    def data_from_preceding(self) -> bool:
+        return self._data_from_preceding
+
+    @data_from_preceding.setter
+    def data_from_preceding(self, dfp: bool) -> None:
+        self._data_from_preceding = dfp
+
+    @property
+    def unmatched(self) -> list[list[Any]]:
+        return self._unmatched
+
+    @unmatched.setter
+    def unmatched(self, lines: list[list[Any]]) -> None:
+        self._unmatched = lines
+
+    @property
+    def collecting(self) -> bool:
+        return self._collecting
+
+    @collecting.setter
+    def collecting(self, c: bool) -> None:
+        self._collecting = c
+
+    def set_unmatched_availability(self) -> None:
+        um = self.metadata.get("unmatched-mode")
+        if um is not None and um.find("no-keep") > -1:
+            self.unmatched_available = False
+        elif um is not None and um.find("keep") > -1:
+            self.unmatched_available = True
+        else:
+            self.unmatched_available = False
+
+    @property
+    def unmatched_available(self) -> bool:
+        return self._unmatched_available
+
+    @unmatched_available.setter
+    def unmatched_available(self, ua: bool) -> None:
+        self._unmatched_available = ua
+
     @property
     def created_at(self) -> datetime:
         return self._created_at
@@ -652,6 +708,9 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
         #   - return-mode: matches | no-matches
         #   - print-mode: default | no-default
         #   - validation-mode: (no-)print | log | (no-)raise | quiet | (no-)match
+        #   - run-mode: no-run | run
+        #   - unmatched-mode: no-keep | keep
+        #   - source-mode: preceding | origin
         #
         self.update_logic_mode_if()
         self.update_run_mode_if()
@@ -659,6 +718,18 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
         self.update_print_mode_if()
         self.update_explain_mode_if()
         self.update_arg_validation_mode_if()
+        self.update_unmatched_mode_if()
+        self.update_data_from_preceding_if()
+
+    def update_data_from_preceding_if(self) -> None:
+        if self.metadata and "source-mode" in self.metadata:
+            dfp = self.metadata["source-mode"]
+            self.data_from_preceding = dfp == "preceding"
+        else:
+            self.data_from_preceding = False
+
+    def update_unmatched_mode_if(self) -> None:
+        self.set_unmatched_availability()
 
     def update_arg_validation_mode_if(self) -> None:
         if self.metadata and "validation-mode" in self.metadata:
@@ -938,13 +1009,16 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
     # collect(), fast_forward(), and next() are the central methods of CsvPath.
     #
     #
-    def collect(self, nexts: int = -1) -> List[List[Any]]:
+    def collect(self, csvpath: str = None, *, nexts: int = -1) -> List[List[Any]]:
         """Runs the csvpath forward and returns the matching lines seen as
         a list of lists"""
+        if self.scanner is None and csvpath is not None:
+            self.parse(csvpath)
         if nexts < -1:
             raise ProcessingException(
                 "Input must be >= -1. -1 means collect to the end of the file."
             )
+        self.collecting = True
         lines = []
         for _ in self.next():
             _ = _[:]
@@ -957,17 +1031,21 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
                 break
         return lines
 
-    def fast_forward(self) -> None:
+    def fast_forward(self, csvpath=None) -> None:
         """Runs the path for all rows of the file. Variables are collected
         and side effects like print happen. No lines are collected.
         """
+        if self.scanner is None and csvpath is not None:
+            self.parse(csvpath)
         for _ in self.next():
             pass
 
-    def next(self):
+    def next(self, csvpath=None):
         """Iterates over the lines in the CSV file returning those that match
         the csvpath. collect() and fast_forward() call next() behind the scenes.
         """
+        if self.scanner is None and csvpath is not None:
+            self.parse(csvpath)
         start = time.time()
         if self.run_mode is True:
             for line in self._next_line():
@@ -983,6 +1061,12 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
                         self.logger.error(msg)
                         raise MatchException(msg)
                     yield line
+                elif self.collecting and self.unmatched_available:
+                    if self.unmatched is None:
+                        self.unmatched = []
+                    line = self.limit_collection(line)
+                    # we aren't None and 0 checking as above. needed?
+                    self.unmatched.append(line)
                 if self.stopped:
                     self.logger.info(
                         "CsvPath has been stopped at line %s",
@@ -1018,14 +1102,7 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
         #
         if self.scanner.filename is None:
             raise FileException("There is no filename")
-        """
-        with open(self.scanner.filename, "r", encoding="utf-8") as file:
-            reader = csv.reader(
-                file, delimiter=self.delimiter, quotechar=self.quotechar
-            )
-            for line in reader:
-        """
-        reader = CsvDataFileReader(
+        reader = DataFileReader(
             self.scanner.filename, delimiter=self.delimiter, quotechar=self.quotechar
         )
         for line in reader.next():
@@ -1173,7 +1250,12 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
             return line
         ls = []
         for k in self.limit_collection_to:
-            ls.append(line[k])
+            if k is None or k >= len(line):
+                raise InputException(
+                    f"[{self.identity}] Line {self.line_monitor.physical_line_number}: unknown header name: {k}"
+                )
+            else:
+                ls.append(line[k])
         return ls
 
     def advance(self, ff: int = -1) -> None: