csvpath 0.0.485__tar.gz → 0.0.487__tar.gz

{csvpath-0.0.485 → csvpath-0.0.487}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: csvpath
-Version: 0.0.485
+Version: 0.0.487
 Summary: A declarative language for validation of CSV files
 Author: David Kershaw
 Author-email: dk107dk@hotmail.com
@@ -28,6 +28,7 @@ Requires-Dist: metaphone (>=0.6,<0.7)
 Requires-Dist: ply (>=3.11,<4.0)
 Requires-Dist: pylightxl (>=1.61,<2.0)
 Requires-Dist: python-dateutil (>=2.9.0.post0,<3.0.0)
+Requires-Dist: smart-open[s3] (>=7.0.5,<8.0.0)
 Requires-Dist: tabulate (>=0.9.0,<0.10.0)
 Project-URL: Csvpath.org, https://www.csvpath.org
 Project-URL: Github, https://github.com/csvpath/csvpath.git
@@ -36,23 +37,25 @@ Description-Content-Type: text/markdown
 
 # <img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/> About CsvPath
 
-CsvPath defines a declarative syntax for inspecting and validating CSV files.
+CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files.
 
 CsvPath' goal is to make it easy to:
-- Analyze the content and structure of a CSV
+- Analyze the content and structure of a CSV or Excel file
 - Validate that the file matches expectations
 - Report on the content or validity
 - Create new derived CSV files
 
 And do it all in an automation-friendly way.
 
-Though much simpler, it is inspired by:
-- XPath. CsvPath is to CSV files like XPath is to XML files.
-- Validation of XML using <a href='https://schematron.com/'>Schematron rules</a>
+CsvPath is inspired by:
+- XPath for XML files
+- The ISO standard <a href='https://schematron.com/'>Schematron validation</a>
 
 CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. New functions are easy to create.
 
-Read more about CsvPath and see realistic CSV validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
+Read more about CsvPath and see realistic CSV and Excel validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
+
+If you need help, use the <a href='https://www.csvpath.org/getting-started/get-help'>contact form</a> or the <a href='https://github.com/csvpath/csvpath/issues'>issue tracker</a> or talk to one of our [sponsors](#sponsors).
 
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/csvpath?logoColor=green&color=green) ![GitHub commit activity](https://img.shields.io/github/commit-activity/m/dk107dk/csvpath) ![PyPI - Version](https://img.shields.io/pypi/v/csvpath)
 
@@ -80,6 +83,7 @@ Read more about CsvPath and see realistic CSV validation examples at <a href='ht
    - [Error Handling](#errors)
 - [More Examples](#examples)
 - [Grammar](#grammar)
+- [Sponsors](#sponsors)
 
 <a name="motivation"></a>
 # Motivation
@@ -451,10 +455,12 @@ To create example CsvPaths from your own data, try <a href='https://autogen.csvp
 Read <a href='https://github.com/dk107dk/csvpath/blob/main/docs/grammar.md'>more about the CsvPath grammar definition here</a>.
 
 
+<a name="more-info"></a>
 # More Info
 
 Visit <a href="https://www.csvpath.org">https://www.csvpath.org</a>
 
+<a name="sponsors"></a>
 # Sponsors
 
 <a href='https://www.atestaanalytics.com/' >

{csvpath-0.0.485 → csvpath-0.0.487}/README.md

@@ -1,23 +1,25 @@
 
 # <img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/> About CsvPath
 
-CsvPath defines a declarative syntax for inspecting and validating CSV files.
+CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files.
 
 CsvPath' goal is to make it easy to:
-- Analyze the content and structure of a CSV
+- Analyze the content and structure of a CSV or Excel file
 - Validate that the file matches expectations
 - Report on the content or validity
 - Create new derived CSV files
 
 And do it all in an automation-friendly way.
 
-Though much simpler, it is inspired by:
-- XPath. CsvPath is to CSV files like XPath is to XML files.
-- Validation of XML using <a href='https://schematron.com/'>Schematron rules</a>
+CsvPath is inspired by:
+- XPath for XML files
+- The ISO standard <a href='https://schematron.com/'>Schematron validation</a>
 
 CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. New functions are easy to create.
 
-Read more about CsvPath and see realistic CSV validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
+Read more about CsvPath and see realistic CSV and Excel validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
+
+If you need help, use the <a href='https://www.csvpath.org/getting-started/get-help'>contact form</a> or the <a href='https://github.com/csvpath/csvpath/issues'>issue tracker</a> or talk to one of our [sponsors](#sponsors).
 
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/csvpath?logoColor=green&color=green) ![GitHub commit activity](https://img.shields.io/github/commit-activity/m/dk107dk/csvpath) ![PyPI - Version](https://img.shields.io/pypi/v/csvpath)
 
@@ -45,6 +47,7 @@ Read more about CsvPath and see realistic CSV validation examples at <a href='ht
    - [Error Handling](#errors)
 - [More Examples](#examples)
 - [Grammar](#grammar)
+- [Sponsors](#sponsors)
 
 <a name="motivation"></a>
 # Motivation
@@ -416,10 +419,12 @@ To create example CsvPaths from your own data, try <a href='https://autogen.csvp
 Read <a href='https://github.com/dk107dk/csvpath/blob/main/docs/grammar.md'>more about the CsvPath grammar definition here</a>.
 
 
+<a name="more-info"></a>
 # More Info
 
 Visit <a href="https://www.csvpath.org">https://www.csvpath.org</a>
 
+<a name="sponsors"></a>
 # Sponsors
 
 <a href='https://www.atestaanalytics.com/' >

{csvpath-0.0.485 → csvpath-0.0.487}/config/config.ini

@@ -24,3 +24,6 @@ path = cache
 [functions]
 imports = config/functions.imports
 
+[results]
+archive = archive
+

{csvpath-0.0.485 → csvpath-0.0.487}/csvpath/csvpath.py

@@ -5,6 +5,7 @@ import csv
 import time
 import os
 import hashlib
+from datetime import datetime
 from typing import List, Dict, Any
 from collections.abc import Iterator
 from abc import ABC, abstractmethod
@@ -286,6 +287,16 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
         self._ecoms = ErrorCommsManager(csvpath=self)
         self._function_times_match = {}
         self._function_times_value = {}
+        self._created_at = datetime.now()
+        self._run_started_at = None
+
+    @property
+    def created_at(self) -> datetime:
+        return self._created_at
+
+    @property
+    def run_started_at(self) -> datetime:
+        return self._run_started_at
 
     @property
     def run_mode(self) -> bool:
@@ -1071,6 +1082,8 @@ class CsvPath(CsvPathPublic, ErrorCollector, Printer):  # pylint: disable=R0902,
         if self.matcher:
             last_line = self.matcher.line
         self.line_monitor.next_line(last_line=last_line, data=line)
+        if self.line_monitor.physical_line_number == 0:
+            self._run_started_at = datetime.now()
 
     def _consider_line(self, line):  # pylint: disable=R0912, R0911
         # re: R0912: this method has already been refactored but maybe

{csvpath-0.0.485 → csvpath-0.0.487}/csvpath/csvpaths.py

@@ -5,6 +5,7 @@ from abc import ABC, abstractmethod
 from typing import List, Any
 import csv
 import traceback
+from datetime import datetime
 from .util.error import ErrorHandler, ErrorCollector, Error
 from .util.config import Config
 from .util.log_utility import LogUtility
@@ -146,6 +147,13 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
         self._fail_all = False
         self._skip_all = False
         self._advance_all = 0
+        self._current_run_time = None
+
+    @property
+    def current_run_time(self) -> datetime:
+        if self._current_run_time is None:
+            self._current_run_time = datetime.now()
+        return self._current_run_time
 
     def clear_run_coordination(self) -> None:
         """run coordination is the set of signals that csvpaths send to affect
@@ -154,6 +162,7 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
         self._fail_all = False
         self._skip_all = False
         self._advance_all = 0
+        self._current_run_time = None
         self.logger.debug("Cleared run coordination")
 
     def csvpath(self) -> CsvPath:
@@ -224,9 +233,17 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
         self.logger.info(
             "Beginning collect_paths %s with %s paths", pathsname, len(paths)
         )
-        for path in paths:
+        crt = self.results_manager.get_run_time_str(pathsname, self.current_run_time)
+        for i, path in enumerate(paths):
             csvpath = self.csvpath()
-            result = Result(csvpath=csvpath, file_name=filename, paths_name=pathsname)
+            result = Result(
+                csvpath=csvpath,
+                file_name=filename,
+                paths_name=pathsname,
+                run_index=i,
+                run_time=self.current_run_time,
+                run_dir=crt,
+            )
             # casting a broad net because if "raise" not in the error policy we
             # want to never fail during a run
             try:
@@ -249,7 +266,9 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
             except Exception as ex:  # pylint: disable=W0718
                 ex.trace = traceback.format_exc()
                 ex.source = self
+                self.results_manager.save(result)
                 ErrorHandler(csvpaths=self, error_collector=result).handle_error(ex)
+            self.results_manager.save(result)
         self.clear_run_coordination()
         self.logger.info(
             "Completed collect_paths %s with %s paths", pathsname, len(paths)
@@ -280,10 +299,18 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
             len(paths),
             filename,
         )
+        crt = self.results_manager.get_run_time_str(pathsname, self.current_run_time)
         for i, path in enumerate(paths):
             csvpath = self.csvpath()
             self.logger.debug("Beginning to FF CsvPath instance: %s", csvpath)
-            result = Result(csvpath=csvpath, file_name=filename, paths_name=pathsname)
+            result = Result(
+                csvpath=csvpath,
+                file_name=filename,
+                paths_name=pathsname,
+                run_index=i,
+                run_time=self.current_run_time,
+                run_dir=crt,
+            )
             try:
                 self.results_manager.add_named_result(result)
                 self._load_csvpath(csvpath, path=path, file=file)
@@ -299,7 +326,9 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
             except Exception as ex:  # pylint: disable=W0718
                 ex.trace = traceback.format_exc()
                 ex.source = self
+                self.results_manager.save(result)
                 ErrorHandler(csvpaths=self, error_collector=result).handle_error(ex)
+            self.results_manager.save(result)
         self.clear_run_coordination()
         self.logger.info(
             "Completed fast_forward_paths %s with %s paths", pathsname, len(paths)
@@ -315,7 +344,8 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
         self.logger.info("Cleaning out any %s and %s results", filename, pathsname)
         self.clean(paths=pathsname)
         self.logger.info("Beginning next_paths with %s paths", len(paths))
-        for path in paths:
+        crt = self.results_manager.get_run_time_str(pathsname, self.current_run_time)
+        for i, path in enumerate(paths):
             if self._skip_all:
                 skip_err = "Found the skip-all signal set. skip_all() is"
                 skip_err = f"{skip_err} only for breadth-first runs using the"
@@ -334,7 +364,14 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
                 advance_err = f"{advance_err} serial run like this one."
                 self.logger.error(advance_err)
             csvpath = self.csvpath()
-            result = Result(csvpath=csvpath, file_name=filename, paths_name=pathsname)
+            result = Result(
+                csvpath=csvpath,
+                file_name=filename,
+                paths_name=pathsname,
+                run_index=i,
+                run_time=self.current_run_time,
+                run_dir=crt,
+            )
             if self._fail_all:
                 self.logger.warning(
                     "Fail-all set. Failing all remaining CsvPath instances in the run."
@@ -351,7 +388,9 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
             except Exception as ex:  # pylint: disable=W0718
                 ex.trace = traceback.format_exc()
                 ex.source = self
+                self.results_manager.save(result)
                 ErrorHandler(csvpaths=self, error_collector=result).handle_error(ex)
+            self.results_manager.save(result)
         self.clear_run_coordination()
 
     # =============== breadth first processing ================
@@ -541,6 +580,9 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
             except Exception as ex:  # pylint: disable=W0718
                 ex.trace = traceback.format_exc()
                 ex.source = self
+                for r in csvpath_objects:
+                    r = r[1]
+                    self.results_manager.save(r)
                 ErrorHandler(
                     csvpaths=self, error_collector=self.current_matcher
                 ).handle_error(ex)
@@ -553,6 +595,9 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
                 yield line
             if sum(stopped_count) == len(csvpath_objects):
                 break
+        for r in csvpath_objects:
+            r = r[1]
+            self.results_manager.save(r)
         self.clear_run_coordination()
 
     def _load_csvpath_objects(
@@ -577,7 +622,8 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
         return csvpath_objects
 
     def _prep_csvpath_results(self, *, csvpath_objects, filename, pathsname):
-        for csvpath in csvpath_objects:
+        crt = self.results_manager.get_run_time_str(pathsname, self.current_run_time)
+        for i, csvpath in enumerate(csvpath_objects):
             try:
                 #
                 # Result will set itself into its CsvPath as error collector
@@ -588,6 +634,9 @@ class CsvPaths(CsvPathsPublic, CsvPathsCoordinator, ErrorCollector):
                     file_name=filename,
                     paths_name=pathsname,
                     lines=csvpath[1],
+                    run_index=i,
+                    run_time=self.current_run_time,
+                    run_dir=crt,
                 )
                 csvpath[1] = result
                 self.results_manager.add_named_result(result)

{csvpath-0.0.485 → csvpath-0.0.487}/csvpath/managers/result.py

@@ -1,4 +1,5 @@
 # pylint: disable=C0114
+from datetime import datetime
 from typing import Dict, List, Any
 from ..util.error import Error, ErrorCollector
 from ..util.printer import Printer
@@ -20,9 +21,14 @@ class Result(ErrorCollector, Printer):  # pylint: disable=R0902
         csvpath: CsvPath,
         file_name: str,
         paths_name: str,
+        run_index: int,
+        run_time: datetime,
+        run_dir: str,
+        runtime_data: dict = None,
     ):
         self._lines: List[List[Any]] = None
         self._csvpath = None
+        self._runtime_data = runtime_data
         self._paths_name = paths_name
         self._file_name = file_name
         self._errors = []
@@ -32,6 +38,28 @@ class Result(ErrorCollector, Printer):  # pylint: disable=R0902
         # use the properties so error_collector, etc. is set correctly
         self.csvpath = csvpath
         self.lines = lines
+        self.run_index = f"{run_index}"
+        self._run_time = run_time
+        self._run_dir = run_dir
+
+    @property
+    def run_time(self) -> datetime:
+        return self._run_time
+
+    @property
+    def run_dir(self) -> str:
+        return self._run_dir
+
+    @run_dir.setter
+    def run_dir(self, d: str) -> None:
+        self._run_dir = d
+
+    @property
+    def identity_or_index(self) -> str:
+        s = self._csvpath.identity
+        if f"{s}".strip() == "":
+            s = self.run_index
+        return s
 
     @property
     def metadata(self) -> Dict[str, Any]:  # pylint: disable=C0116
@@ -99,6 +127,10 @@ class Result(ErrorCollector, Printer):  # pylint: disable=R0902
     def errors(self) -> List[Error]:  # pylint: disable=C0116
         return self._errors
 
+    @errors.setter
+    def errors(self, errors: List[Error]) -> None:
+        self._errors = errors
+
     @property
     def errors_count(self) -> int:  # pylint: disable=C0116
         return len(self._errors)
@@ -112,8 +144,12 @@ class Result(ErrorCollector, Printer):  # pylint: disable=R0902
 
     @property
     def is_valid(self) -> bool:  # pylint: disable=C0116
-        if self._csvpath:
+        # if the csvpath has not been run -- e.g. because it represents results that were
+        # saved to disk and reloaded -- it won't have a run started time.
+        if self._csvpath and self._csvpath.run_started_at is not None:
             return self._csvpath.is_valid
+        elif self._runtime_data and "valid" in self._runtime_data:
+            return self._runtime_data["valid"]
         return False
 
     @property
@@ -124,6 +160,14 @@ class Result(ErrorCollector, Printer):  # pylint: disable=R0902
             self._printouts = []
         return self._printouts["default"] if "default" in self._printouts else []
 
+    def get_printouts(self) -> dict[str, list[str]]:
+        return self._printouts
+
+    def set_printouts(self, name: str, lines: List[str]) -> None:
+        if self._printouts is None:
+            self._printouts = {}
+        self._printouts[name] = lines
+
     def get_printout_by_name(self, name: str) -> List[str]:  # pylint: disable=C0116
         if self._printouts is None:
             self._printouts = []

csvpath-0.0.487/csvpath/managers/result_serializer.py

@@ -0,0 +1,165 @@
+import os
+import json
+import csv
+from typing import NewType, List, Dict, Optional, Union
+from datetime import datetime
+from .result import Result
+from ..matching.util.runtime_data_collector import RuntimeDataCollector
+from csvpath import CsvPath
+
+Simpledata = NewType("Simpledata", Union[None | str | int | float | bool])
+Listdata = NewType("Listdata", list[None | str | int | float | bool])
+Csvdata = NewType("Csvdata", list[List[str]])
+Metadata = NewType("Metadata", Dict[str, Simpledata])
+
+
+class ResultSerializer:
+    def __init__(self, base_dir: str):
+        self.base_dir = base_dir
+
+    def save_result(self, result: Result) -> None:
+        runtime_data = {}
+        RuntimeDataCollector.collect(result.csvpath, runtime_data, local=True)
+        runtime_data["run_index"] = result.run_index
+        self._save(
+            metadata=result.csvpath.metadata,
+            errors=result.errors,
+            variables=result.variables,
+            lines=result.lines,
+            printouts=result.get_printouts(),
+            runtime_data=runtime_data,
+            paths_name=result.paths_name,
+            file_name=result.file_name,
+            identity=result.identity_or_index,
+            run_time=result.run_time,
+            run_dir=result.run_dir,
+            run_index=result.run_index,
+        )
+
+    def _save(
+        self,
+        *,
+        metadata: Metadata,
+        runtime_data: Metadata,
+        errors: List[Metadata],
+        variables: dict[str, Simpledata | Listdata | Metadata],
+        lines: Csvdata,
+        printouts: dict[str, list[str]],
+        paths_name: str,
+        file_name: str,
+        identity: str,
+        run_time: datetime,
+        run_dir: str,
+        run_index: int,
+    ) -> None:
+        """Save a single Result object to basedir/paths_name/run_time/identity_or_index."""
+        meta = {
+            "paths_name": paths_name,
+            "file_name": file_name,
+            "run_time": f"{run_time}",
+            "run_index": run_index,
+            "identity": identity,
+            "metadata": metadata,
+            "runtime_data": runtime_data,
+        }
+        print(f"\nresult_serializer: meta: {meta}")
+        run_dir = self.get_instance_dir(run_dir=run_dir, identity=identity)
+        # Save the JSON files
+        with open(os.path.join(run_dir, "meta.json"), "w") as f:
+            json.dump(meta, f, indent=2)
+        with open(os.path.join(run_dir, "errors.json"), "w") as f:
+            json.dump(errors, f, indent=2)
+        with open(os.path.join(run_dir, "vars.json"), "w") as f:
+            json.dump(variables, f, indent=2)
+
+        # Save lines returned as a CSV file
+        if lines is None:
+            lines = []
+        with open(os.path.join(run_dir, "data.csv"), "w") as f:
+            writer = csv.writer(f)
+            writer.writerows(lines)
+
+        # Save the printout lines
+        with open(os.path.join(run_dir, "printouts.txt"), "w") as f:
+            for k, v in printouts.items():
+                f.write(f"---- PRINTOUT: {k}\n")
+                for _ in v:
+                    f.write(f"{_}\n")
+
+    def get_run_dir(self, *, paths_name, run_time):
+        run_dir = os.path.join(self.base_dir, paths_name)
+        if not isinstance(run_time, str):
+            run_time = run_time.strftime("%Y-%m-%d_%I-%M-%S")
+        run_dir = os.path.join(run_dir, f"{run_time}")
+        # the path existing for a different named-paths run in progress
+        # or having completed less than 1000ms ago is expected to be
+        # uncommon in real world usage. CsvPaths are single user instances
+        # atm. a server process would namespace each CsvPaths instance
+        # to prevent conflicts. if there is a conflict the two runs would
+        # overwrite each other. this prevents that.
+        if os.path.exists(run_dir):
+            i = 0
+            adir = f"{run_dir}.{i}"
+            while os.path.exists(adir):
+                i += 1
+                adir = f"{run_dir}.{i}"
+            run_dir = adir
+        return run_dir
+
+    def get_instance_dir(self, run_dir, identity) -> str:
+        run_dir = os.path.join(run_dir, identity)
+        os.makedirs(run_dir, exist_ok=True)
+        return run_dir
+
+    def load_result(
+        self, paths_name: str, run_time: str, identity: str
+    ) -> Optional[Result]:
+        """Load a single Result object from the base directory."""
+        run_dir = self._run_dir(
+            paths_name=paths_name, run_time=run_time, identity=identity
+        )
+        if not os.path.exists(run_dir):
+            return None
+        return self._load_result(run_dir)
+
+    def _load_result(self, run_dir: str) -> Optional[Result]:
+        if not os.path.exists(run_dir):
+            return None
+        try:
+            meta = None
+            variables = None
+            errors = None
+            data = None
+            printouts = None
+
+            with open(os.path.join(run_dir, "meta.json"), "r") as f:
+                meta = json.load(f)
+            with open(os.path.join(run_dir, "vars.json"), "r") as f:
+                variables = json.load(f)
+            with open(os.path.join(run_dir, "errors.json"), "r") as f:
+                errors = json.load(f)
+            with open(os.path.join(run_dir, "data.csv"), "r") as f:
+                reader = csv.reader(f)
+                data = [",".join(row) for row in reader]
+            with open(os.path.join(run_dir, "printouts.txt"), "r") as f:
+                printouts = f.readlines()
+
+            c = CsvPath()
+            c.variables = variables
+            c.metadata = meta["metadata"]
+            c.identity = meta["identity"]
+            result = Result(
+                lines=data,
+                csvpath=c,
+                file_name=meta["file_name"],
+                paths_name=meta["paths_name"],
+                run_index=meta["run_index"],
+                run_time=meta["run_time"],
+                runtime_data=meta["runtime_data"],
+            )
+            result.errors = errors
+            result.set_printouts("all", printouts)
+
+            return result
+        except (FileNotFoundError, ValueError, IOError):
+            return None

{csvpath-0.0.485 → csvpath-0.0.487}/csvpath/managers/results_manager.py

@@ -3,7 +3,8 @@ from __future__ import annotations
 from typing import Dict, List, Any
 from abc import ABC, abstractmethod
 from .result import Result
-from ..util.exceptions import InputException
+from ..util.exceptions import InputException, CsvPathsException
+from .result_serializer import ResultSerializer
 
 
 class CsvPathsResultsManager(ABC):
@@ -15,6 +16,14 @@ class CsvPathsResultsManager(ABC):
     CsvPaths clears the named results from the ResultsManager.
     """
 
+    #
+    # - printout lines
+    # - lines of captured data
+    # - variables
+    # - csvpath.metadata
+    # - csvpath.csvpath data
+    #
+
     @abstractmethod
     def get_variables(self, name: str) -> bool:
         """gets all the variables from all csvpaths in one dict. variables may
@@ -191,6 +200,19 @@ class ResultsManager(CsvPathsResultsManager):  # pylint: disable=C0115
         for r in results:
             self.add_named_result(r)
 
+    def save(self, result: Result) -> None:
+        if self._csvpaths is None:
+            raise CsvPathsException(
+                "Cannot save result because there is no CsvPaths instance"
+            )
+        rs = ResultSerializer(self._csvpaths.config.archive_path)
+        rs.save_result(result)
+
+    def get_run_time_str(self, name, run_time) -> str:
+        rs = ResultSerializer(self._csvpaths.config.archive_path)
+        t = rs.get_run_dir(paths_name=name, run_time=run_time)
+        return t
+
     def remove_named_results(self, name: str) -> None:
         if name in self.named_results:
             del self.named_results[name]
@@ -210,6 +232,9 @@ class ResultsManager(CsvPathsResultsManager):  # pylint: disable=C0115
     def clean_named_results(self, name: str) -> None:
         if name in self.named_results:
             self.remove_named_results(name)
+            #
+            # clean from filesystem too?
+            #
 
     def get_named_results(self, name) -> List[List[Any]]:
         if name in self.named_results:

{csvpath-0.0.485 → csvpath-0.0.487}/csvpath/matching/functions/args.py

@@ -456,16 +456,9 @@ class Args:
     def handle_errors_if(self, mismatch_count, mismatches):
         if mismatch_count == len(self._argsets):
             self._args_match = False
-            pm = f"mismatch in {self.matchable.my_chain}: {mismatches}"
-            # when would we not have a csvpath?
-            # pln = (
-            #    self._csvpath.line_monitor.physical_line_number if self._csvpath else 0
-            # )
-            # csvpathid = f"{self._csvpath_id()} " if self._csvpath_id() else ""
-            # ei = ExpressionUtility.get_my_expressions_index(self._matchable)
-            # pm = f"{csvpathid}Wrong value in match component {ei} at line {pln}: {pm}"
-            # raise ChildrenValidationException(pm)
-            #
+            pm = f"mismatch in {self.matchable.my_chain}"
             ei = ExpressionUtility.get_my_expressions_index(self._matchable)
             pm = f"Wrong value in match component {ei}: {pm}"
+            lpm = f"{pm}: {mismatches}"
+            self._matchable.matcher.csvpath.logger.error(lpm)
             self._matchable.raiseChildrenException(pm)

{csvpath-0.0.485 → csvpath-0.0.487}/csvpath/matching/functions/boolean/inf.py

@@ -31,6 +31,10 @@ class In(MatchDecider):
                 v = f"{v}".strip()
                 nvs = [_.strip() for _ in v.split("|")]
                 inf += nvs
+            # elif isinstance(s, Reference) and s.is_header():
+            #
+            # do lookup here
+            #
             else:
                 # tuple would mean vars were frozen. this would not be
                 # surprising from a reference