lcpcli 0.2.8__tar.gz → 0.3.0__tar.gz

{lcpcli-0.2.8 → lcpcli-0.3.0}/.gitignore

@@ -2,6 +2,7 @@ __pycache__/
 lcpcli/__pycache__/
 tmp.py
 *.pyc
+.env
 build/
 dist/
 dist/

{lcpcli-0.2.8 → lcpcli-0.3.0}/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright 2025 LiRI - UZH
+Copyright 2026 LiRI - UZH
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

lcpcli-0.3.0/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: lcpcli
+Version: 0.3.0
+Summary: Helper for converting CONLLU files and uploading the corpus to LiRI Corpus Platform (LCP)
+Project-URL: Homepage, https://github.com/liri-uzh/lcpcli
+Project-URL: Issues, https://github.com/liri-uzh/lcpcli/issues
+Author-email: Danny McDonald <daniel.mcdonald@uzh.ch>, Igor Mustač <igor.mustac@uzh.ch>, Jeremy Zehr <jeremy.zehr@uzh.ch>, Jonathan Schaber <jeremy.schaber@uzh.ch>
+License: MIT
+License-File: LICENSE.txt
+Keywords: CONLL,TEI,VERT,corpora,corpus,linguistics
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Requires-Dist: diskcache>=5.6.3
+Requires-Dist: jsonpickle>=3.0
+Requires-Dist: jsonschema>=4.21
+Requires-Dist: lxml>=4.7.1
+Requires-Dist: pandas>=2.2.2
+Requires-Dist: py7zr>=0.20.5
+Requires-Dist: requests>=2.30.0
+Requires-Dist: tqdm>=4.65.0
+Requires-Dist: types-requests>=2.30.0.0
+Requires-Dist: types-tqdm>=4.65.0.1
+Requires-Dist: xmltodict>=0.13
+Description-Content-Type: text/markdown
+
+# LCP CLI module
+
+> Command-line tool for converting CONLLU files and uploading the corpus to LCP
+
+## Installation
+
+Make sure you have python 3.11+ with `pip` installed in your local environment, then run:
+
+```bash
+pip install lcpcli
+```
+
+## Usage
+
+**Examples:**
+
+Conversion of a CoNLL-U (Plus) corpus:
+
+```bash
+lcpcli -i ~/conll_ext/ -o ~/upload/
+```
+
+Data upload:
+
+```bash
+lcpcli -c ~/upload/ -k $API_KEY -s $API_SECRET -p "my project" --live
+```
+
+Including `--live` points the upload to the live instance of LCP. Leave it out if you want to add a corpus to an instance of LCP running on `localhost`.
+
+**Help:**
+
+```bash
+lcpcli --help
+```
+
+`lcpcli` can take a corpus of CoNLL-U (PLUS) files and import it to a collection created on LCP.
+
+Besides the standard token-level CoNLL-U fields (`form`, `lemma`, `upos`, `xpos`, `feats`, `head`, `deprel`, `deps`) one can also provide document-, paragraph- and sentence-level annotations using comment lines in the files (see [the CoNLL-U Format section](#conll-u-format)).
+
+### CoNLL-U Format
+
+The CoNLL-U format is documented at: https://universaldependencies.org/format.html
+
+The LCP CLI converter will treat all the comments that start with `# newdoc KEY = VALUE` as document-level attributes, and all the comments that start with `# newpar KEY = VALUE` as paragraph-level attributes. All other comment lines following the format `# key = value` will be treated sentence-level attributes.
+
+The key-value pairs in the `FEATS` and `MISC` columns of a token line will be mapped to corresponding attributes in the LCP corpus. Additionally, if the `MISC` cell includes `SpaceAfter=Yes` or `SpaceAfter=No` (case senstive) the token will be represented with (respectively, without) a trailing space character in the database.
+
+#### CoNLL-U Plus
+
+CoNLL-U Plus is an extension to the CoNLLU-U format documented at: https://universaldependencies.org/ext-format.html
+
+If your files start with a comment line of the form `# global.columns = ID FORM LEMMA UPOS XPOS FEATS HEAD DEPREL DEPS MISC`, `lcpcli` will treat them as CoNLL-U PLUS files and process the columns according to the names you set in that line.
+
+### CoNLL-U conversion and upload
+
+1. Create a directory in which you have all your properly-fromatted CoNLL-U files.
+
+2. Visit an LCP instance (e.g. _catchphrase_) and create a new collection if you don't already have one where your corpus should go.
+
+3. Retrieve the API key and secret for your project by clicking on the button that says: "Create API Key".
+
+4. Once you have your API key and secret, you can start converting and uploading your corpus by running the following command:
+
+```
+lcpcli -i $CONLLU_FOLDER -o $OUTPUT_FOLDER -k $API_KEY -s $API_SECRET -p $PROJECT_NAME --live
+```
+
+- `$CONLLU_FOLDER` should point to the folder that contains your CONLLU files
+- `$OUTPUT_FOLDER` should point to *another* folder that will be used to store the converted files to be uploaded
+- `$API_KEY` is the key you copied from your project on LCP (still visible when you visit the page)
+- `$API_SECRET` is the secret you copied from your project on LCP (only visible upon API Key creation)
+- `$PROJECT_NAME` is the name of the project exactly as displayed on LCP -- it is case-sensitive, and space characters should be escaped
+
+### Other input formats, rich data
+
+Previous versions of `lcpcli` defined procedures to include rich annotations in CoNLL-U files, including time-anchored media files, in combination with annex non-CoNLL-U files. These methods are no longer supported -- use an older version of `lcpcli` if you require those features.
+
+`lcpcli` now ships with a Python module called `lcpcli.builder` that you can use to convert any input format. The default CoNLL-U converter included in `lcpcli` uses `lcpcli.builder` under the hood.
+
+You can find a short tutorial on how to use the module [in BUILDER.md](BUILDER.md). Further information can be found in [the LCP documentation](https://lcp.linguistik.uzh.ch/manual/builder.html).

lcpcli-0.3.0/README.md

@@ -0,0 +1,81 @@
+# LCP CLI module
+
+> Command-line tool for converting CONLLU files and uploading the corpus to LCP
+
+## Installation
+
+Make sure you have python 3.11+ with `pip` installed in your local environment, then run:
+
+```bash
+pip install lcpcli
+```
+
+## Usage
+
+**Examples:**
+
+Conversion of a CoNLL-U (Plus) corpus:
+
+```bash
+lcpcli -i ~/conll_ext/ -o ~/upload/
+```
+
+Data upload:
+
+```bash
+lcpcli -c ~/upload/ -k $API_KEY -s $API_SECRET -p "my project" --live
+```
+
+Including `--live` points the upload to the live instance of LCP. Leave it out if you want to add a corpus to an instance of LCP running on `localhost`.
+
+**Help:**
+
+```bash
+lcpcli --help
+```
+
+`lcpcli` can take a corpus of CoNLL-U (PLUS) files and import it to a collection created on LCP.
+
+Besides the standard token-level CoNLL-U fields (`form`, `lemma`, `upos`, `xpos`, `feats`, `head`, `deprel`, `deps`) one can also provide document-, paragraph- and sentence-level annotations using comment lines in the files (see [the CoNLL-U Format section](#conll-u-format)).
+
+### CoNLL-U Format
+
+The CoNLL-U format is documented at: https://universaldependencies.org/format.html
+
+The LCP CLI converter will treat all the comments that start with `# newdoc KEY = VALUE` as document-level attributes, and all the comments that start with `# newpar KEY = VALUE` as paragraph-level attributes. All other comment lines following the format `# key = value` will be treated sentence-level attributes.
+
+The key-value pairs in the `FEATS` and `MISC` columns of a token line will be mapped to corresponding attributes in the LCP corpus. Additionally, if the `MISC` cell includes `SpaceAfter=Yes` or `SpaceAfter=No` (case senstive) the token will be represented with (respectively, without) a trailing space character in the database.
+
+#### CoNLL-U Plus
+
+CoNLL-U Plus is an extension to the CoNLLU-U format documented at: https://universaldependencies.org/ext-format.html
+
+If your files start with a comment line of the form `# global.columns = ID FORM LEMMA UPOS XPOS FEATS HEAD DEPREL DEPS MISC`, `lcpcli` will treat them as CoNLL-U PLUS files and process the columns according to the names you set in that line.
+
+### CoNLL-U conversion and upload
+
+1. Create a directory in which you have all your properly-fromatted CoNLL-U files.
+
+2. Visit an LCP instance (e.g. _catchphrase_) and create a new collection if you don't already have one where your corpus should go.
+
+3. Retrieve the API key and secret for your project by clicking on the button that says: "Create API Key".
+
+4. Once you have your API key and secret, you can start converting and uploading your corpus by running the following command:
+
+```
+lcpcli -i $CONLLU_FOLDER -o $OUTPUT_FOLDER -k $API_KEY -s $API_SECRET -p $PROJECT_NAME --live
+```
+
+- `$CONLLU_FOLDER` should point to the folder that contains your CONLLU files
+- `$OUTPUT_FOLDER` should point to *another* folder that will be used to store the converted files to be uploaded
+- `$API_KEY` is the key you copied from your project on LCP (still visible when you visit the page)
+- `$API_SECRET` is the secret you copied from your project on LCP (only visible upon API Key creation)
+- `$PROJECT_NAME` is the name of the project exactly as displayed on LCP -- it is case-sensitive, and space characters should be escaped
+
+### Other input formats, rich data
+
+Previous versions of `lcpcli` defined procedures to include rich annotations in CoNLL-U files, including time-anchored media files, in combination with annex non-CoNLL-U files. These methods are no longer supported -- use an older version of `lcpcli` if you require those features.
+
+`lcpcli` now ships with a Python module called `lcpcli.builder` that you can use to convert any input format. The default CoNLL-U converter included in `lcpcli` uses `lcpcli.builder` under the hood.
+
+You can find a short tutorial on how to use the module [in BUILDER.md](BUILDER.md). Further information can be found in [the LCP documentation](https://lcp.linguistik.uzh.ch/manual/builder.html).

{lcpcli-0.2.8 → lcpcli-0.3.0}/lcpcli/__init__.py

@@ -1,3 +1,3 @@
-__version__ = "0.2.8"
+__version__ = "0.3.0"
 
 from .lcpcli import Lcpcli  # noqa: F401

{lcpcli-0.2.8 → lcpcli-0.3.0}/lcpcli/builder.py

@@ -4,8 +4,8 @@ import csv
 import json
 import os
 import re
-import tempfile
 import shutil
+import tempfile
 
 from typing import Any
 from uuid import uuid4
@@ -16,9 +16,7 @@ ANCHORINGS = ("stream", "time", "location")
 # ATYPES = ("text", "categorical", "number", "dict", "labels")
 ATYPES_LOOKUP = ("text", "dict", "labels")
 NAMEDATALEN = 63
-PATTERN_TXT = (
-    "(must start with a lower case and only contain alpha-numerical characters)"
-)
+PATTERN_TXT = "(must start with a lower case, be at leat 2 characters long and only contain alpha-numerical characters)"
 
 
 def meta_subattr(meta: dict, k: str, v: Any) -> dict:
@@ -53,11 +51,13 @@ def get_layer_method(layer: "Layer"):
             corpus._files.pop(fname)
             return GlobalAttribute(corpus, layer._name, args[0])
         largs = [a for a in args]
-        if layer._name == corpus._token and isinstance(largs[0], str):
+        if layer._name == corpus._token and largs and isinstance(largs[0], str):
             form = largs.pop(0)
             layer.form = form
         if len(largs) > 0:
-            assert all(isinstance(c, Layer) for c in largs), RuntimeError()
+            assert all(isinstance(c, Layer) for c in largs), RuntimeError(
+                "Tried to pass non-layers as arguments of a layer"
+            )
             layer.add(*largs)
         for aname, avalue in kwargs.items():
             setattr(layer, aname, avalue)
@@ -85,10 +85,23 @@ def get_layer_method(layer: "Layer"):
                 a.make()
             return
         # source is the attribute that's missing in at least one layer
-        source_a = next(
-            ra for ra in relation_attrs if any(ra not in a._attributes for a in args)
-        )
-        target_a = next(ra for ra in relation_attrs if ra != source_a)
+        try:
+            source_a = next(
+                ra
+                for ra in relation_attrs
+                if any(ra not in a._attributes for a in args)
+            )
+        except:
+            # if can't identify a missing attribute,
+            # look for "head"/"source" or use the first one
+            source_a = next(
+                (ra for ra in relation_attrs if ra in ("head", "source")),
+                next(ra for ra in relation_attrs),
+            )
+        target_a = next((ra for ra in relation_attrs if ra != source_a), None)
+        if not target_a:
+            # no target: possibly a one-token sentence, return
+            return
         # reference nested sets by target's id
         nested_sets = {
             a._attributes[target_a]._value._id: NestedSet(
@@ -246,7 +259,7 @@ class Corpus:
                     can_categorize = (
                         not (is_token and aname in ("form", "lemma"))
                         and len(lookup) <= 100
-                        and all(len(v) < NAMEDATALEN for v in lookup)
+                        and all(len(str(v)) < NAMEDATALEN for v in lookup)
                     )
                     if can_categorize:
                         texts_to_categorical[na] = aname
@@ -374,7 +387,7 @@ class Corpus:
                 if ais_global:
                     aopts["isGlobal"] = True
                 if aopts["type"] == "categorical" and not ais_global:
-                    aopts["values"] = [v for v in mapping.lookups[aname] if v]
+                    aopts["values"] = [str(v) for v in mapping.lookups[aname] if v]
                 elif aopts["type"] == "ref":
                     aopts.pop("type")
                     aopts.pop("nullable", "")
@@ -418,6 +431,16 @@ class Layer:
             assert re.match(r"[a-z][a-zA-Z0-9_]+$", name), RuntimeError(
                 f"The attribute '{name}' on the layer {self._name} does not match the pattern {PATTERN_TXT}"
             )
+            # Disallow linebreak in token string values because it messes with CSV's (in particular, FTS)
+            if (
+                self._name == self._corpus._token
+                and isinstance(value, str)
+                and ("\n" in value or "\r" in value)
+            ):
+                print(
+                    f"Warning: a token attribute contains a linebreak; this is not allowed, removing the linebreaks from the value {value}."
+                )
+                value = value.replace("\n", "").replace("\r", "")
             Attribute(self, name, value)
 
     def __getattribute__(self, name: str):
@@ -494,6 +517,9 @@ class Layer:
             rows.append(doc_name)
             rows.append(json.dumps(self._media))
         if is_token:
+            assert "form" in self._attributes, RuntimeError(
+                "Tried to make a token with no form"
+            )
             seg_parent = self._find_in_parents(corpus._segment)
             rows.append(seg_parent._id)
             char_low = corpus._char_counter
@@ -569,6 +595,10 @@ class Layer:
         # Add any new attribute to mapping
         for aname, attr in self._attributes.items():
             if aname in mapping.attributes:
+                try:
+                    mapping.attributes[aname]["subtype"] = attr._subtype
+                except:
+                    pass
                 continue
             atype = attr._type
             mapping.attributes[aname] = {
@@ -577,6 +607,10 @@ class Layer:
                     True if mapping.counter > 1 else False
                 ),  # adding a new attribute
             }
+            try:
+                mapping.attributes[aname]["subtype"] = attr._subtype
+            except:
+                pass
             if atype == "ref":
                 mapping.attributes[aname]["ref"] = attr._ref.lower()
             elif atype in ATYPES_LOOKUP and aname != "meta":
@@ -724,6 +758,8 @@ class Layer:
 class Attribute:
     def __init__(self, layer: Layer, name: str, value: Any = None):
         self._name = name
+        if name not in layer._attributes:
+            layer._attributes[name] = self
         self._value = value
         self._layer = layer
         self._ref = None
@@ -739,6 +775,10 @@ class Attribute:
             self._value = json.dumps(sorted_dict(value))
         elif isinstance(value, (int, float)):
             atype = "number"
+            if isinstance(value, float):
+                self._subtype = "float"
+                # overwrite to ensure subtype is taken into consideration
+                layer._attributes[name] = self
         elif isinstance(value, GlobalAttribute):
             atype = "ref"
             self._ref = value._name
@@ -747,7 +787,6 @@ class Attribute:
             atype = "entity"
             self._ref = value._name
         self._type: str = atype
-        layer._attributes[name] = self
 
 
 class GlobalAttribute:

{lcpcli-0.2.8 → lcpcli-0.3.0}/lcpcli/check_files.py

@@ -68,6 +68,8 @@ class Checker:
                 subtyps = aprops
             elif aprops.get("type") == "dict":
                 subtyps = {k: v.get("type") for k, v in aprops.get("keys", {}).items()}
+            if typ == "number" and aprops.get("subtype", "") == "float":
+                typ = "float"
             ret[aname] = (acol, typ, subtyps)
         return ret
 
@@ -83,6 +85,21 @@ class Checker:
     def check_uuid(self, uuid: str) -> None:
         assert UUID(uuid, version=4), SyntaxError(f"Invalid UUID ({uuid})")
 
+    def check_number(self, value: str, aname: str = "") -> None:
+        is_digit = value.strip().isdigit()
+        is_almost_digit = not is_digit and value.strip().replace(".", "", 1).isdigit()
+        if is_almost_digit:
+            raise TypeError(
+                f"Number attribute {aname} appears to contain floating values ({value}); add 'subtype': 'float' to its properties."
+            )
+        assert is_digit, TypeError(f"Number attribute {aname} is ill-formed: {value}")
+
+    def check_float(self, value: str, aname: str = "") -> None:
+        is_numerical = value.strip().replace(".", "", 1).isdigit()
+        assert is_numerical, TypeError(
+            f"Float number attribute {aname} is ill-formed: {value}"
+        )
+
     def check_categorical(self, value: str, values: None | list[str]) -> None:
         assert len(value.encode("utf-8")) <= NAMEDATALEN, ValueError(
             f"Found a categorical value ('{value}') that exceeds the database's limit of {NAMEDATALEN} bytes on enum values"
@@ -132,7 +149,7 @@ class Checker:
         return None
 
     def check_ftsvector(self, vector: str) -> None:
-        whole_pattern = r"^('\d+([^']|'')*':\d+(\s|$))+$"
+        whole_pattern = r"^('\d+([^']|'')*':\d+(,\d+)*(\s|$))+$"
         simple_unit_pattern = r"('([^']|'')*':[^\s]+)(\s|$)"
         units = findall(simple_unit_pattern, vector)
         for n, (unit, *_) in enumerate(units):
@@ -142,7 +159,7 @@ class Checker:
             assert match(r"'\d+", unit), SyntaxError(
                 f"Each value in the tsvector must start with a single quote character followed by an integer index ({unit} -- {n})"
             )
-            m = match(r"'\d+(.*)':\d+\s?$", unit)
+            m = match(r"'\d+(.*)':\d+(,\d+)*\s?$", unit)
             assert m, SyntaxError(
                 f"Each value in the tsvector must end with a single quote followed by a colon and an integer index ({unit} -- {n})"
             )
@@ -387,8 +404,11 @@ class Checker:
                 "vector": "ftsvector",
             }
         elif "_" in no_ext:
-            lname, aname, *remainder = no_ext.split("_")
-            assert not remainder, SyntaxError(f"Invalid filename: {filename}")
+            # lname, aname, *remainder = no_ext.split("_")
+            # assert not remainder, SyntaxError(
+            #     f"Invalid filename: {filename} (cannot contain more than one underscore character)"
+            # )
+            lname, aname = no_ext.split("_", 1)
             props = next(
                 (v for k, v in layer.items() if k.lower() == lname.lower()), None
             )
@@ -490,7 +510,7 @@ class Checker:
                     typ = columns[headers[n]]
                     if not col:
                         assert headers[n] in nullables, ValueError(
-                            f"Found an empty value for column #{n+1} ({headers[n]}) on line {counter} in {filename} even though the configuration does not reported it as nullable"
+                            f"Found an empty value for column #{n+1} ({headers[n]}) on line {counter} in {filename} even though the configuration does not report it as nullable"
                         )
                         continue
                     if typ == "int":
@@ -523,6 +543,10 @@ class Checker:
                                 self.check_uuid(col)
                             elif typ == "ftsvector":
                                 self.check_ftsvector(col)
+                            elif typ == "number":
+                                self.check_number(col, headers[n])
+                            elif typ == "float":
+                                self.check_float(col, headers[n])
                             elif typ == "categorical":
                                 assert layer_name, NotImplementedError(
                                     f"Attributes of type 'categorical' are only supported on layers ({filename})"

{lcpcli-0.2.8 → lcpcli-0.3.0}/lcpcli/cli.py

@@ -1,5 +1,4 @@
 import argparse
-
 from typing import Any
 
 BOOL_KWARGS: dict[str, Any]
@@ -24,12 +23,6 @@ def _parse_cmd_line():
     parser.add_argument(
         "-e", "--extension", type=str, help="Output format when output is a directory"
     )
-    parser.add_argument(
-        "-f", "--filter", required=False, type=str, help="Path to a Python filter file"
-    )
-    parser.add_argument(
-        "-u", "--lua-filter", required=False, type=str, help="Path to a Lua filter file"
-    )
     parser.add_argument(
         "-x",
         "--example",
@@ -87,7 +80,7 @@ def _parse_cmd_line():
         "--escape",
         type=str,
         required=False,
-        help="The character used to escape a character in the uploaded files (default is backslash \)",
+        help="The character used to escape a character in the uploaded files (default is backslash \\)",
     )
     parser.add_argument(
         "-j",
@@ -121,6 +114,42 @@ def _parse_cmd_line():
                 help="Run the pre-import check without importing.",
                 **BOOL_KWARGS,
             )
+            parser.add_argument(
+                "--force-corpus-overwrite",
+                required=False,
+                default=False,
+                help="Will overwrite any corpus with the same name in the destination collection.",
+                **BOOL_KWARGS,
+            )
+            parser.add_argument(
+                "--force-output-overwrite",
+                required=False,
+                default=False,
+                help="Will overwrite any file in the destination folder with the same names as the newly generated ones.",
+                **BOOL_KWARGS,
+            )
+            parser.add_argument(
+                "--conll-only",
+                required=False,
+                default=False,
+                help="Used in conjunction with -i/-o, will ignore any file not ending in .conll/.conllu from the input directory.",
+                **BOOL_KWARGS,
+            )
+            parser.add_argument(
+                "--skip-check",
+                required=False,
+                default=False,
+                help="Skip the local checks before uploading the corpus (the server may still run checks).",
+                **BOOL_KWARGS,
+            )
+            parser.add_argument(
+                "-v",
+                "--version",
+                required=False,
+                default=False,
+                help="The current version of LCPCLI",
+                **BOOL_KWARGS,
+            )
             break
         except Exception as e:
             if "type" in BOOL_KWARGS: