csvpath 0.0.45__tar.gz → 0.0.451__tar.gz

{csvpath-0.0.45 → csvpath-0.0.451}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: csvpath
-Version: 0.0.45
+Version: 0.0.451
 Summary: A declarative language for data extraction and validation of CSV files
 Author: David Kershaw
 Author-email: dk107dk@hotmail.com
@@ -88,7 +88,7 @@ Two classes do all the work: CsvPath and CsvPaths. Each has only a few external
     - a single .csv file or
     - a directory of .csv files
 
-There are several ways to set up csvpath file references. Read <a href='docs/files.md'>more about filenames</a>.
+There are several ways to set up csvpath file references. Read <a href='https://github.com/dk107dk/csvpath/blob/main/docs/files.md'>more about filenames</a>.
 
 This is a very basic programmatic use of CsvPath. For lots more examples, see the unit tests.
 
@@ -178,7 +178,7 @@ A string, number, or regular expression value.
 |--------|---------|---------------|
 |A value | Always true | `"a value"` |
 
-<a href='docs/terms.md'>Read about terms here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/terms.md'>Read about terms here</a>.
 
 ## Function
 A composable unit of functionality called once for every row scanned.
@@ -187,7 +187,7 @@ A composable unit of functionality called once for every row scanned.
 |--------|---------|---------------|
 |Calculated | Calculated | `count()` |
 
-<a href='docs/functions.md'>Read about functions here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/functions.md'>Read about functions here</a>.
 
 ## Variable
 A stored value that is set or retrieved once per row scanned.
@@ -196,7 +196,7 @@ A stored value that is set or retrieved once per row scanned.
 |--------|---------|---------------|
 |A value | True when set, unless `onchange`. Used alone it is an existence test. | `@firstname` |
 
-<a href='docs/variables.md'>Read about variables here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/variables.md'>Read about variables here</a>.
 
 ## Header
 
@@ -206,7 +206,7 @@ A named column or a column identified by 0-based index.
 |--------|---------|---------------|
 |A value | Calculated. Used alone it is an existence test. | `#area_code` |
 
-<a href='docs/variables.md'>Read about headers here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/headers.md'>Read about headers here</a>.
 
 ## Equality
 Two of the other types joined with an "=" or "==".
@@ -245,7 +245,7 @@ Says to set the `firstname` variable to the value of the first column when the f
 
 Qualifiers are tokens added to variable, header, and function names. They are separated from the names and each other with `.` characters. Each qualifier causes the qualified match component to behave in a different way than it otherwise would.
 
-<a href='docs/qualifiers.md'>Read about qualifiers here.</a>
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/qualifiers.md'>Read about qualifiers here.</a>
 
 <a name="examples"></a>
 ## More Examples
@@ -261,9 +261,9 @@ In the path above, the rules applied are:
 
 There are more examples scattered throughout the documentation. Good places to look include:
 
-- The individual <a href='docs/functions.md'>function descriptions</a>
-- The <a href='tests'>unit tests</a> _(not realistic, but a good source of ideas)_
-- A few <a href='docs/examples.md'>more real-looking examples</a>
+- The individual <a href='https://github.com/dk107dk/csvpath/blob/main/docs/functions.md'>function descriptions</a>
+- The <a href='https://github.com/dk107dk/csvpath/tree/main/tests'>unit tests</a> _(not realistic, but a good source of ideas)_
+- A few <a href='https://github.com/dk107dk/csvpath/blob/main/docs/examples.md'>more real-looking examples</a>
 
 # Not Ready For Production
 Anything could change and performance could be better. This project is a hobby.

{csvpath-0.0.45 → csvpath-0.0.451}/README.md

@@ -58,7 +58,7 @@ Two classes do all the work: CsvPath and CsvPaths. Each has only a few external
     - a single .csv file or
     - a directory of .csv files
 
-There are several ways to set up csvpath file references. Read <a href='docs/files.md'>more about filenames</a>.
+There are several ways to set up csvpath file references. Read <a href='https://github.com/dk107dk/csvpath/blob/main/docs/files.md'>more about filenames</a>.
 
 This is a very basic programmatic use of CsvPath. For lots more examples, see the unit tests.
 
@@ -148,7 +148,7 @@ A string, number, or regular expression value.
 |--------|---------|---------------|
 |A value | Always true | `"a value"` |
 
-<a href='docs/terms.md'>Read about terms here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/terms.md'>Read about terms here</a>.
 
 ## Function
 A composable unit of functionality called once for every row scanned.
@@ -157,7 +157,7 @@ A composable unit of functionality called once for every row scanned.
 |--------|---------|---------------|
 |Calculated | Calculated | `count()` |
 
-<a href='docs/functions.md'>Read about functions here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/functions.md'>Read about functions here</a>.
 
 ## Variable
 A stored value that is set or retrieved once per row scanned.
@@ -166,7 +166,7 @@ A stored value that is set or retrieved once per row scanned.
 |--------|---------|---------------|
 |A value | True when set, unless `onchange`. Used alone it is an existence test. | `@firstname` |
 
-<a href='docs/variables.md'>Read about variables here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/variables.md'>Read about variables here</a>.
 
 ## Header
 
@@ -176,7 +176,7 @@ A named column or a column identified by 0-based index.
 |--------|---------|---------------|
 |A value | Calculated. Used alone it is an existence test. | `#area_code` |
 
-<a href='docs/variables.md'>Read about headers here</a>.
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/headers.md'>Read about headers here</a>.
 
 ## Equality
 Two of the other types joined with an "=" or "==".
@@ -215,7 +215,7 @@ Says to set the `firstname` variable to the value of the first column when the f
 
 Qualifiers are tokens added to variable, header, and function names. They are separated from the names and each other with `.` characters. Each qualifier causes the qualified match component to behave in a different way than it otherwise would.
 
-<a href='docs/qualifiers.md'>Read about qualifiers here.</a>
+<a href='https://github.com/dk107dk/csvpath/blob/main/docs/qualifiers.md'>Read about qualifiers here.</a>
 
 <a name="examples"></a>
 ## More Examples
@@ -231,9 +231,9 @@ In the path above, the rules applied are:
 
 There are more examples scattered throughout the documentation. Good places to look include:
 
-- The individual <a href='docs/functions.md'>function descriptions</a>
-- The <a href='tests'>unit tests</a> _(not realistic, but a good source of ideas)_
-- A few <a href='docs/examples.md'>more real-looking examples</a>
+- The individual <a href='https://github.com/dk107dk/csvpath/blob/main/docs/functions.md'>function descriptions</a>
+- The <a href='https://github.com/dk107dk/csvpath/tree/main/tests'>unit tests</a> _(not realistic, but a good source of ideas)_
+- A few <a href='https://github.com/dk107dk/csvpath/blob/main/docs/examples.md'>more real-looking examples</a>
 
 # Not Ready For Production
 Anything could change and performance could be better. This project is a hobby.

csvpath-0.0.451/csvpath/matching/functions/above.py

@@ -0,0 +1,35 @@
+from typing import Any
+from .function import Function, ChildrenException
+
+
+class Above(Function):
+    def to_value(self, *, skip=[]) -> Any:
+        if self in skip:
+            return True
+        if len(self.children) != 1:
+            raise ChildrenException("Above function must have 1 child")
+        if self.children[0].op != ",":
+            raise ChildrenException(
+                f"Above function must have an equality with the ',' operation, not {self.children[0].op}"
+            )
+        if self.value is None:
+
+            thischild = self.children[0].children[0]
+            abovethatchild = self.children[0].children[1]
+
+            this_is = thischild.to_value(skip=skip)
+            above_that = abovethatchild.to_value(skip=skip)
+            this = -1
+            that = -1
+            try:
+                this = float(this_is)
+                that = float(above_that)
+            except Exception:
+                raise Exception(
+                    f"Above.to_value: this: {this}, a {this.__class__}, and {that}, a {that.__class__}"
+                )
+            self.value = this > that
+        return self.value
+
+    def matches(self, *, skip=[]) -> bool:
+        return self.to_value(skip=skip)

csvpath-0.0.451/csvpath/matching/functions/correlate.md

@@ -0,0 +1,31 @@
+
+# Correlate
+
+Correlate calculates the correlation between two columns of numbers. It takes each column as a list of floats.If a row does not have a value in either of the columns, the tuple is captured to a `correlate_gap` variable with the row number as tracking key. The result of correlate is updated for each row seen.
+
+The correlation value is given in a tuple with related data:
+- The line number
+- Variance left
+- Variance right
+- The covariance
+- The correlation
+
+The running track of the column values needed for the calculations is stored as a list of float in `correlate_left` and `correlate_right`.
+
+Correlate takes `onmatch` and can have a name qualifier to set the key names of its data. For e.g. using:
+
+    [ correlate.cor(#0, #1) ]
+
+You would have variables:
+- cor_left
+- cor_right
+- cor_gap
+- cor
+
+## Example
+
+    $file.csv[1*][ correlate.cor(#years, #salary) ]
+
+This path gives a correlation of experience to salary.
+
+

csvpath-0.0.451/csvpath/matching/functions/correlate.py

@@ -0,0 +1,112 @@
+from typing import Any
+from .function import Function, ChildrenException
+from math import sqrt
+
+
+class Correlate(Function):
+    def _float(self, v: Any) -> float:
+        try:
+            return float(v)
+        except Exception:
+            return None
+
+    def to_value(self, *, skip=[]) -> Any:
+        if self in skip:
+            return self.value
+        if len(self.children) != 1:
+            raise ChildrenException("Correlate function must have 1 child")
+        if self.children[0].op != ",":
+            raise ChildrenException(
+                f"Correlate function must have an equality with the ',' operation, not {self.children[0].op}"
+            )
+        if self.value is None:
+            om = self.has_onmatch()
+            if not om or self.line_matches():
+                child = self.children[0]
+                left = child.left
+                right = child.right
+                leftv = left.to_value()
+                rightv = right.to_value()
+
+                lv = self._float(leftv)
+                rv = self._float(rightv)
+
+                name = self.first_non_term_qualifier("correlate")
+                if lv is None or rv is None:
+                    #
+                    # variables[name][lineno]=(left,right)
+                    self.matcher.set_variable(
+                        f"{name}_gap",
+                        value=(lv, rv),
+                        tracking=self.matcher.csvpath.line_number,
+                    )
+                else:
+                    nl = f"{name}_left"
+                    nr = f"{name}_right"
+                    leftlist = self.matcher.get_variable(nl)
+                    rightlist = self.matcher.get_variable(nr)
+
+                    if leftlist is None:
+                        leftlist = []
+                    leftlist.append(lv)
+
+                    if rightlist is None:
+                        rightlist = []
+                    rightlist.append(rv)
+
+                    self.matcher.set_variable(nl, value=leftlist)
+                    self.matcher.set_variable(nr, value=rightlist)
+
+                    if len(leftlist) == 1:
+                        #
+                        # not a lot to go on. skip the rest.
+                        return self.value
+                    elif len(leftlist) != len(rightlist):
+                        #
+                        # how could this happen?
+                        raise Exception(
+                            "Number of values to calculate correlation from must be the same"
+                        )
+
+                    mean_left = sum(leftlist) / len(leftlist)
+                    mean_right = sum(rightlist) / len(rightlist)
+
+                    var_left = sum((li - mean_left) ** 2 for li in leftlist)
+                    var_right = sum((ri - mean_right) ** 2 for ri in rightlist)
+
+                    if var_left == 0 or var_right == 0:
+                        #
+                        # how do we want to handle this? how likely is it?
+                        print(f"skipping because 0: {var_left}, {var_right}")
+                        return None
+
+                    cov_lr = sum(
+                        (li - mean_left) * (ri - mean_right)
+                        for li, ri in zip(leftlist, rightlist)
+                    )
+                    cor = cov_lr / (sqrt(var_left) * sqrt(var_right))
+                    #
+                    # store:
+                    #   - line number
+                    #   - variance left
+                    #   - variance right
+                    #   - covariance
+                    #   - correlation
+                    #
+                    vs = (
+                        self.matcher.csvpath.line_number,
+                        var_left,
+                        var_right,
+                        cov_lr,
+                        cor,
+                    )
+
+                    self.matcher.set_variable(name, value=vs)
+
+                    self.value = cor
+
+        return self.value
+
+    def matches(self, *, skip=[]) -> bool:
+        self.to_value(skip=skip)
+        return True

{csvpath-0.0.45 → csvpath-0.0.451}/csvpath/matching/functions/exists.py

@@ -27,7 +27,7 @@ class Exists(Function):
                     self.match = v
                 except Exception:
                     self.match = False
-            elif v is not None and v.strip() != "":
+            elif v is not None and f"{v}".strip() != "":
                 self.match = True
             else:
                 self.match = False

{csvpath-0.0.45 → csvpath-0.0.451}/csvpath/matching/functions/function_factory.py

@@ -42,6 +42,7 @@ from .mod import Mod
 from .equals import Equals
 from .strip import Strip
 from .jinjaf import Jinjaf
+from .correlate import Correlate
 
 
 class UnknownFunctionException(Exception):
@@ -164,6 +165,8 @@ class FunctionFactory:
             f = Strip(matcher, name, child)
         elif name == "jinja":
             f = Jinjaf(matcher, name, child)
+        elif name == "correlate":
+            f = Correlate(matcher, name, child)
         else:
             raise UnknownFunctionException(f"{name}")
         if child:

{csvpath-0.0.45 → csvpath-0.0.451}/csvpath/matching/functions/stop.md

@@ -13,12 +13,8 @@ Stops the scan immediately on a condition or by being match-activated by an encl
 
 This path stops the scan when the match count hits 5.
 
-    $file.csv[*][ when(
-                    above(
-                        count(),
-                        5),
-                    stop())
-                ]
+    $file.csv[*][
+         above(count(), 5) -> stop() ]
 
 This path stops scanning if its match count goes above 5.
 

{csvpath-0.0.45 → csvpath-0.0.451}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "csvpath"
-version = "0.0.45"
+version = "0.0.451"
 description = "A declarative language for data extraction and validation of CSV files"
 authors = ["David Kershaw <dk107dk@hotmail.com>"]
 readme = "README.md"
@@ -15,7 +15,7 @@ classifiers = [
     "Topic :: Software Development :: Quality Assurance",
     "Topic :: Software Development :: Testing",
     "Topic :: Text Processing",
-    "Topic :: Utilities",
+    "Topic :: Utilities"
 ]
 
 [tool.poetry.urls]

csvpath-0.0.45/csvpath/matching/functions/above.py

@@ -1,36 +0,0 @@
-from typing import Any
-from .function import Function, ChildrenException
-
-
-class Above(Function):
-    def to_value(self, *, skip=[]) -> Any:
-        if self in skip:
-            return True
-        if len(self.children) != 1:
-            self.matcher.print(
-                f"Above.to_value: must have 1 equality child: {self.children}"
-            )
-            raise ChildrenException("Above function must have 1 child")
-        if self.children[0].op != ",":
-            raise ChildrenException(
-                f"Above function must have an equality with the ',' operation, not {self.children[0].op}"
-            )
-        thischild = self.children[0].children[0]
-        abovethatchild = self.children[0].children[1]
-
-        this_is = thischild.to_value(skip=skip)
-        above_that = abovethatchild.to_value(skip=skip)
-        this = -1
-        that = -1
-        try:
-            this = float(this_is)
-            that = float(above_that)
-        except Exception:
-            raise Exception(
-                f"Above.to_value: this: {this}, a {this.__class__}, and {that}, a {that.__class__}"
-            )
-        b = this > that
-        return b
-
-    def matches(self, *, skip=[]) -> bool:
-        return self.to_value(skip=skip)