pyconll 3.1.0.dev3__tar.gz → 3.3.0__tar.gz

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/MANIFEST.in

@@ -1,4 +1,2 @@
 include LICENSE
 include README.md
-include README.rst
-include README

pyconll-3.3.0/PKG-INFO

@@ -0,0 +1,138 @@
+Metadata-Version: 2.4
+Name: pyconll
+Version: 3.3.0
+Summary: Read and manipulate CoNLL files
+Home-page: https://github.com/pyconll/pyconll
+Author: Matias Grioni
+Author-email: matgrioni@gmail.com
+License: MIT
+Keywords: nlp,conllu,conll,universal dependencies
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Utilities
+Requires-Python: ~=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-python
+Dynamic: summary
+
+[![Build Status](https://github.com/pyconll/pyconll/workflows/CI/badge.svg?branch=master)](https://github.com/pyconll/pyconll)
+[![Coverage Status](https://coveralls.io/repos/github/pyconll/pyconll/badge.svg?branch=master)](https://coveralls.io/github/pyconll/pyconll?branch=master)
+[![Documentation Status](https://readthedocs.org/projects/pyconll/badge/?version=stable)](https://pyconll.readthedocs.io/en/stable)
+[![Version](https://img.shields.io/github/v/release/pyconll/pyconll)](https://github.com/pyconll/pyconll/releases)
+[![gitter](https://badges.gitter.im/pyconll/pyconll.svg)](https://gitter.im/pyconll/pyconll?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+
+## pyconll
+
+*Easily work with **CoNLL** files using the familiar syntax of **python**.*
+
+<img src="res/logo.svg" width="256px" height="256px">
+
+##### Links
+- [Homepage](https://pyconll.github.io)
+- [Documentation](https://pyconll.readthedocs.io/)
+
+
+### Installation
+
+As with most python packages, simply use `pip` to install from PyPi.
+
+```
+pip install pyconll
+```
+
+`pyconll` is also available as a conda package on the `pyconll` channel. Only packages 2.2.0 and newer are available on conda at the moment.
+
+```
+conda install -c pyconll pyconll
+```
+
+pyconll supports Python 3.10 or newer. In general, pyconll will focus development efforts on officially supported python versions.
+
+
+### Use
+
+This tool is intended to be a **minimal**, **low level**, **expressive** and **pragmatic** library in a widely used programming language. pyconll creates a thin API on top of raw CoNLL annotations that is simple and intuitive.
+
+It offers the following features:
+* Regular CI testing and validation against all UD v2.x versions.
+* A strong domain model that includes CoNLL sources, Sentences, Tokens, Trees, etc.
+* A typed API for better development experience and better semantics.
+* A focus on usability and simplicity in design (no dependencies)
+* Performance optimizations for a smooth development workflow no matter the dataset size (performs about 25%-35% faster than other comparable packages)
+
+See the following code example to understand the basics of the API.
+
+```python
+# This snippet finds sentences where a token marked with part of speech 'AUX' are
+# governed by a NOUN. For example, in French this is a less common construction
+# and we may want to validate these examples because we have previously found some
+# problematic examples of this construction.
+import pyconll
+
+train = pyconll.load_from_file('./ud/train.conllu')
+
+review_sentences = []
+
+# Conll objects are iterable over their sentences, and sentences are iterable
+# over their tokens. Sentences also de/serialize comment information.
+for sentence in train:
+    for token in sentence:
+
+        # Tokens have attributes such as upos, head, id, deprel, etc, and sentences
+        # can be indexed by a token's id. We must check that the token is not the
+        # root token, whose id, '0', cannot be looked up.
+        if token.upos == 'AUX' and (token.head != '0' and sentence[token.head].upos == 'NOUN'):
+            review_sentences.append(sentence)
+
+print('Review the following sentences:')
+for sent in review_sentences:
+    print(sent.id)
+```
+
+A full definition of the API can be found in the [documentation](https://pyconll.readthedocs.io/) or use the [quick start](https://pyconll.readthedocs.io/en/stable/starting.html) guide for a focused introduction.
+
+
+### Uses and Limitations
+
+This package edits CoNLL-U annotations. This does not include the annotated text itself. Word forms on Tokens are not editable and Sentence Tokens cannot be reassigned or reordered. `pyconll` focuses on editing CoNLL-U annotation rather than creating it or changing the underlying text that is annotated. If there is interest in this functionality area, please create a GitHub issue for more visibility.
+
+This package also is only validated against the CoNLL-U format. The CoNLL and CoNLL-X format are not supported, but are very similar. I originally intended to support these formats as well, but their format is not as well defined as CoNLL-U so they are not included. Please create an issue for visibility if this feature interests you.
+
+Lastly, linguistic data can often be very large and this package attempts to keep that in mind. pyconll provides methods for creating in memory conll objects along with an iterate only version in case a corpus is too large to store in memory (the size of the memory structure is several times larger than the actual corpus file). The iterate only version can parse upwards of 100,000 words per second on a 16gb ram machine, so for most datasets to be used on a local dev machine, this package will perform well. The 2.2.0 release also improves parse time and memory footprint by about 25%!
+
+
+### Contributing
+
+Contributions to this project are welcome and encouraged! If you are unsure how to contribute, here is a [guide](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork) from Github explaining the basic workflow. After cloning this repo, please run `pip install -r requirements.txt` to properly setup locally. Some of these tools like yapf, pylint, and mypy do not have to be run locally, but CI builds will fail without their successful running. Some other release dependencies like twine and sphinx are also installed.
+
+For packaging new versions, use setuptools version 24.2.0 or greater for creating the appropriate packaging that recognizes the `python_requires` metadata. Final packaging and release is now done with Github actions so this is less of a concern.
+
+
+#### README and CHANGELOG
+
+When changing either of these files, please change the Markdown version and run ``make gendocs`` so that the other versions stay in sync.
+
+#### Release Checklist
+
+Below enumerates the general release process explicitly. This section is for internal use and most people do not have to worry about this. First note, that the dev branch is always a direct extension of master with the latest changes since the last release. That is, it is essentially a staging release branch.
+
+* Change the version in `pyconll/_version.py` appropriately.
+* Merge dev into master **locally**. Github does not offer a fast forward merge and explicitly uses --no-ff. So to keep the linear nature of changes, merge locally to fast forward. This is assuming that the dev branch looks good on CI tests which do not automatically run in this situation.
+* Push the master branch. This should start some CI tests specifically for master. After validating these results, create a tag corresponding to the next version number and push the tag.
+* Create a new release from this tag from the [Releases page](https://github.com/pyconll/pyconll/releases). On creating this release, two workflows will start. One releases to pypi, and the other releases to conda.
+* Validate these workflows pass, and the package is properly released on both platforms.

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/README.md

@@ -29,7 +29,7 @@ pip install pyconll
 conda install -c pyconll pyconll
 ```
 
-pyconll supports Python 3.6 and greater, starting in version 3.0.0. In general pyconll will focus development efforts on officially supported python versions. Python 3.5 reached end of support in October 2020.
+pyconll supports Python 3.10 or newer. In general, pyconll will focus development efforts on officially supported python versions.
 
 
 ### Use

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/__init__.py

@@ -5,6 +5,6 @@ and python code.
 
 __all__ = ['conllable', 'exception', 'load', 'tree', 'unit', 'util']
 
-from .load import load_from_string, load_from_file, iter_from_string, \
-       iter_from_file
+from .load import load_from_string, load_from_file, load_from_resource, \
+       iter_from_string, iter_from_file, iter_from_resource
 from ._version import __version__

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/_parser.py

@@ -6,15 +6,18 @@ can then be used in the Conll class or in pyconll.load.
 
 from typing import Iterable, Iterator
 
+from pyconll.exception import ParseError
 from pyconll.unit.sentence import Sentence
 
 
-def _create_sentence(sent_lines: Iterable[str]) -> Sentence:
+def _create_sentence(sent_lines: Iterable[str], line_num: int) -> Sentence:
     """
     Creates a Sentence object given the current state of the source iteration.
 
     Args:
         sent_lines: An iterable of the lines that make up the source.
+        line_num: The current line number the sentence starts at, for logging
+            purposes.
 
     Returns:
         The created Sentence.
@@ -23,7 +26,11 @@ def _create_sentence(sent_lines: Iterable[str]) -> Sentence:
         ParseError: If the sentence source is not valid.
     """
     sent_source = '\n'.join(sent_lines)
-    sentence = Sentence(sent_source)
+    try:
+        sentence = Sentence(sent_source)
+    except ParseError as err:
+        raise ParseError(
+            f'Failed to create sentence at line {line_num}') from err
 
     return sentence
 
@@ -44,19 +51,22 @@ def iter_sentences(lines_it: Iterable[str]) -> Iterator[Sentence]:
         ValueError: If there is an error constructing the Sentence.
     """
     sent_lines = []
-    for line in lines_it:
+    last_empty_line = -1
+    for i, line in enumerate(lines_it):
         line = line.strip()
 
         # Collect all lines until there is a blank line. Then all the
         # collected lines were between blank lines and are a sentence.
         if line:
             sent_lines.append(line)
-        elif sent_lines:
-            sentence = _create_sentence(sent_lines)
-            sent_lines.clear()
+        else:
+            if sent_lines:
+                sentence = _create_sentence(sent_lines, last_empty_line + 2)
+                sent_lines.clear()
+                yield sentence
 
-            yield sentence
+            last_empty_line = i
 
     if sent_lines:
-        sentence = _create_sentence(sent_lines)
+        sentence = _create_sentence(sent_lines, last_empty_line)
         yield sentence

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/_version.py

@@ -5,4 +5,4 @@ This is surfaced publicly via __init__.py and keeps the version specification
 separate from other module logic to allow for easy parsing when packaging.
 """
 
-__version__ = '3.1.0.dev3'
+__version__ = '3.3.0'

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/conllable.py

@@ -12,6 +12,7 @@ class Conllable(metaclass=abc.ABCMeta):
     A Conllable mixin to indicate that the component can be converted into a
     CoNLL representation.
     """
+
     @abc.abstractmethod
     def conll(self) -> str:
         """

pyconll-3.3.0/pyconll/load.py

@@ -0,0 +1,132 @@
+"""
+A wrapper around the Conll class to easily load treebanks from multiple formats.
+This module can also load resources by iterating over treebank data without
+storing Conll objects in memory. This module is the main entrance to pyconll's
+functionalities.
+"""
+
+import os
+from typing import Iterable, Iterator
+
+from pyconll._parser import iter_sentences
+from pyconll.unit.conll import Conll
+from pyconll.unit.sentence import Sentence
+
+PathLike = str | bytes | os.PathLike
+
+
+def load_from_string(source: str) -> Conll:
+    """
+    Load the CoNLL-U source in a string into a Conll object.
+
+    Args:
+        source: The CoNLL-U formatted string.
+
+    Returns:
+        A Conll object equivalent to the provided source.
+
+    Raises:
+        ParseError: If there is an error parsing the input into a Conll object.
+    """
+    lines = source.splitlines()
+    c = Conll(lines)
+
+    return c
+
+
+def load_from_file(file_descriptor: PathLike) -> Conll:
+    """
+    Load a CoNLL-U file given its location.
+
+    Args:
+        file_descriptor: The file to load the CoNLL-U data from. This can be a
+            filepath as a Path object, or string, or a file descriptor.
+
+    Returns:
+        A Conll object equivalent to the provided file.
+
+    Raises:
+        IOError: If there is an error opening the given filename.
+        ParseError: If there is an error parsing the input into a Conll object.
+    """
+    with open(file_descriptor, encoding='utf-8') as f:
+        c = Conll(f)
+
+    return c
+
+
+def load_from_resource(resource: Iterable[str]) -> Conll:
+    """
+    Load a CoNLL-U file from a generic string resource.
+
+    Args:
+        resource: The generic string resource. Each string from the resource is
+            assumed to be a line in a CoNLL-U formatted resource.
+
+    Returns:
+        A Conll object equivalent to the string resource provided.
+
+    Raises:
+        ParseError: If there is an error parsing the input into a Conll object.
+    """
+    return Conll(resource)
+
+
+def iter_from_string(source: str) -> Iterator[Sentence]:
+    """
+    Iterate over a CoNLL-U string's sentences.
+
+    Use this method if you only need to iterate over the CoNLL-U file once and
+    do not need to create or store the Conll object.
+
+    Args:
+        source: The CoNLL-U string.
+
+    Yields:
+        The sentences that make up the CoNLL-U file.
+
+    Raises:
+        ParseError: If there is an error parsing the input into a Conll object.
+    """
+    lines = source.splitlines()
+    yield from iter_sentences(lines)
+
+
+def iter_from_file(file_descriptor: PathLike) -> Iterator[Sentence]:
+    """
+    Iterate over a CoNLL-U file's sentences.
+
+    Args:
+        file_descriptor: The file to iterate the CoNLL-U data from. This can be a
+            filepath as a Path object, or string, or a file descriptor.
+
+    Yields:
+        The sentences that make up the CoNLL-U file.
+
+    Raises:
+        IOError: If there is an error opening the file.
+        ParseError: If there is an error parsing the input into a Conll object.
+    """
+    with open(file_descriptor, encoding='utf-8') as f:
+        yield from iter_sentences(f)
+
+
+def iter_from_resource(resource: Iterable[str]) -> Iterator[Sentence]:
+    """
+    Iterate over the sentences from an iterable string resource.
+
+    This is a generic method that allows for any general resource that can
+    provide data (like a streaming network request or memory mapped data) to be
+    parsed as a CoNLL-U data source.
+
+    Args:
+        resource: The line source. Each iterated string should be a line in a
+            CoNLL-U formatted file.
+
+    Yields:
+        The sentences that make up the CoNLL-U file.
+
+    Raises:
+        ParseError: If there is an error parsing the input into a Conll object.
+    """
+    yield from iter_sentences(resource)

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/tree/_treebuilder.py

@@ -23,6 +23,7 @@ class TreeBuilder(Generic[T]):
     created from the same TreeBuilder, Tree nodes will be unique, but data on
     the nodes will be shallow copies.
     """
+
     def __init__(self) -> None:
         """
         Creates a new empty TreeBuilder, with no internal data.
@@ -76,8 +77,8 @@ class TreeBuilder(Generic[T]):
             self.current = self.current[i]
         except IndexError as e:
             raise IndexError(
-                '{}-th child is out of range. There are {} children on this node'
-                .format(i, len(self.current))) from e
+                f'{i}-th child is out of range. There are {len(self.current)} children on this node'
+            ) from e
 
     def move_to_root(self) -> None:
         """
@@ -121,8 +122,8 @@ class TreeBuilder(Generic[T]):
             del self.current._children[i]
         except IndexError as e:
             raise IndexError(
-                '{}-th child is out of range. There are {} children on this node'
-                .format(i, len(self.current))) from e
+                f'{i}-th child is out of range. There are {len(self.current)} children on this node'
+            ) from e
 
     def add_child(self, data: T, move: bool = False) -> None:
         """

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/tree/tree.py

@@ -18,6 +18,7 @@ class Tree(Generic[T]):
     module which is a sort of friend class of Tree to maintain its immutable
     public contract.
     """
+
     def __init__(self, data: T) -> None:
         """
         Create a tree holding the value. Create a larger Tree, with TreeBuilder.
@@ -70,8 +71,7 @@ class Tree(Generic[T]):
         """
         Provides an iterator over the children.
         """
-        for child in self._children:
-            yield child
+        yield from self._children
 
     def __len__(self) -> int:
         """

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/unit/conll.py

@@ -2,7 +2,7 @@
 Defines the Conll type and the associated parsing and output logic.
 """
 
-from typing import Any, Iterable, Iterator, List, Union, MutableSequence, overload
+from typing import Any, Iterable, Iterator, MutableSequence, overload
 
 import pyconll._parser
 from pyconll.conllable import Conllable
@@ -17,6 +17,7 @@ class Conll(MutableSequence[Sentence], Conllable):
     specifies that the file must end in a new line but that requirement is
     relaxed here in parsing.
     """
+
     def __init__(self, it: Iterable[str]) -> None:
         """
         Create a CoNLL-U file collection of sentences.
@@ -28,7 +29,7 @@ class Conll(MutableSequence[Sentence], Conllable):
             ParseError: If there is an error constructing the sentences in the
                 iterator.
         """
-        self._sentences: List[Sentence] = []
+        self._sentences: list[Sentence] = []
 
         for sentence in pyconll._parser.iter_sentences(it):
             self._sentences.append(sentence)
@@ -39,6 +40,10 @@ class Conll(MutableSequence[Sentence], Conllable):
 
         Returns:
             The CoNLL-U object as a string. This string will end in a newline.
+
+        Raises:
+            FormatError: If there are issues converting the sentences to the
+                CoNLL format.
         """
         # Add newlines along with sentence strings so that there is no need to
         # slice potentially long lists or modify strings.
@@ -95,8 +100,7 @@ class Conll(MutableSequence[Sentence], Conllable):
         Yields:
             An iterator over the sentences in this Conll object.
         """
-        for sentence in self._sentences:
-            yield sentence
+        yield from self._sentences
 
     @overload
     def __getitem__(self, key: int) -> Sentence:
@@ -152,7 +156,7 @@ class Conll(MutableSequence[Sentence], Conllable):
         """
         self._sentences[key] = item
 
-    def __delitem__(self, key: Union[int, slice]) -> None:
+    def __delitem__(self, key: int | slice) -> None:
         """
         Delete the Sentence corresponding with the given key.
 

{pyconll-3.1.0.dev3 → pyconll-3.3.0}/pyconll/unit/sentence.py

@@ -4,9 +4,10 @@ Defines the Sentence type and the associated parsing and output logic.
 
 from collections import OrderedDict
 import re
-from typing import ClassVar, Dict, Iterator, List, Optional, Sequence, overload
+from typing import ClassVar, Iterator, Optional, Sequence, overload
 
 from pyconll.conllable import Conllable
+from pyconll.exception import FormatError, ParseError
 from pyconll.tree._treebuilder import TreeBuilder
 from pyconll.tree.tree import Tree
 from pyconll.unit.token import Token
@@ -61,10 +62,10 @@ class Sentence(Sequence[Token], Conllable):
         lines = source.split('\n')
 
         self._meta: OrderedDict[str, Optional[str]] = OrderedDict()  # pylint: disable=E1136
-        self._tokens: List[Token] = []
-        self._ids_to_indexes: Dict[str, int] = {}
+        self._tokens: list[Token] = []
+        self._ids_to_indexes: dict[str, int] = {}
 
-        for line in lines:
+        for i, line in enumerate(lines):
             if line:
                 if line[0] == Sentence.COMMENT_MARKER:
                     kv_match = re.match(Sentence.KEY_VALUE_COMMENT_PATTERN,
@@ -81,7 +82,13 @@ class Sentence(Sequence[Token], Conllable):
                             k = singleton_match.group(1)
                             self._meta[k] = None
                 else:
-                    token = Token(line)
+                    try:
+                        token = Token(line)
+                    except ParseError as err:
+                        raise ParseError(
+                            f'Error creating token on line {i} for the current sentence'
+                        ) from err
+
                     self._tokens.append(token)
 
                     if token.id is not None:
@@ -163,7 +170,7 @@ class Sentence(Sequence[Token], Conllable):
                 singleton, this field can be ignored or set to None.
         """
         if key == Sentence.TEXT_KEY:
-            raise ValueError('Key cannot be {}'.format(Sentence.TEXT_KEY))
+            raise ValueError(f'Key cannot be {Sentence.TEXT_KEY}')
 
         self._meta[key] = value
 
@@ -179,7 +186,7 @@ class Sentence(Sequence[Token], Conllable):
             ValueError: If the text key is provided, regardless of presence.
         """
         if key == Sentence.TEXT_KEY:
-            raise ValueError('Key cannot be {}'.format(Sentence.TEXT_KEY))
+            raise ValueError(f'Key cannot be {Sentence.TEXT_KEY}')
 
         del self._meta[key]
 
@@ -205,7 +212,7 @@ class Sentence(Sequence[Token], Conllable):
             ValueError: If the sentence can not be made into a tree because a
                 token has an empty head value or if there is no root token.
         """
-        children_tokens: Dict[str, List[Token]] = {}
+        children_tokens: dict[str, list[Token]] = {}
 
         for token in self:
             if token.head is not None:
@@ -215,8 +222,8 @@ class Sentence(Sequence[Token], Conllable):
                     children_tokens[token.head] = [token]
             elif not (token.is_multiword() or token.is_empty_node()):
                 raise ValueError(
-                    'The current sentence is not fully defined as a tree and ' \
-                    'has a token with an empty head at {}'.format(token.id))
+                    'The current sentence is not fully defined as a tree and has a token with an '
+                    f'empty head at {token.id}')
 
         builder: TreeBuilder[Token] = TreeBuilder()
         if '0' in children_tokens:
@@ -237,7 +244,7 @@ class Sentence(Sequence[Token], Conllable):
     @staticmethod
     def _create_tree_helper(builder: TreeBuilder, sentence: 'Sentence',
                             root: Token,
-                            children_tokens: Dict[str, List[Token]]) -> None:
+                            children_tokens: dict[str, list[Token]]) -> None:
         """
         Method to create a tree from a sentence given the root token.
 
@@ -267,19 +274,27 @@ class Sentence(Sequence[Token], Conllable):
 
         Returns:
             A string representing the Sentence in CoNLL-U format.
+
+        Raises:
+            FormatError: If the Sentence or underlying Tokens can not be
+                converted to the CoNLL format.
         """
         lines = []
         for meta in self._meta.items():
             if meta[1] is not None:
-                line = '{} {} = {}'.format(Sentence.COMMENT_MARKER, meta[0],
-                                           meta[1])
+                line = f'{Sentence.COMMENT_MARKER} {meta[0]} = {meta[1]}'
             else:
-                line = '{} {}'.format(Sentence.COMMENT_MARKER, meta[0])
+                line = f'{Sentence.COMMENT_MARKER} {meta[0]}'
 
             lines.append(line)
 
         for token in self._tokens:
-            lines.append(token.conll())
+            try:
+                lines.append(token.conll())
+            except FormatError as err:
+                raise FormatError(
+                    f'Error serializing sentence with id {self.id} on token \'{token.id}\'.'
+                ) from err
 
         return '\n'.join(lines)
 
@@ -288,8 +303,7 @@ class Sentence(Sequence[Token], Conllable):
         Iterate through all the tokens in the Sentence including multiword
         tokens.
         """
-        for token in self._tokens:
-            yield token
+        yield from self._tokens
 
     @overload
     def __getitem__(self, key: str) -> Token: