chemdataextractor-lite 1.0.0__tar.gz

chemdataextractor_lite-1.0.0/LICENSE

@@ -0,0 +1,50 @@
+ChemDataExtractor v2 is released under the MIT license.
+===========================================================
+
+The MIT License
+
+Copyright 2020 Juraj Mavračić, Callum J. Court, Taketomo Isazawa, Jacqueline M. Cole and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the “Software”), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE
+OR THE USE OR OTHER DEALINGS IN THE SOFTWARE
+
+
+
+ChemDataExtractor License
+===========================================================
+
+The MIT License
+
+Copyright 2017 Matt Swain and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

chemdataextractor_lite-1.0.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE
+include README.md
+recursive-include chemdataextractor/data *.pickle *.md
+recursive-include tests/data *
+global-exclude *.py[cod] __pycache__ *.so *.dylib

chemdataextractor_lite-1.0.0/PKG-INFO

@@ -0,0 +1,72 @@
+Metadata-Version: 2.4
+Name: chemdataextractor-lite
+Version: 1.0.0
+Summary: Lightweight scientific document loaders for HTML, XML, PDF, and plain text.
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cssselect>=1.0.1
+Requires-Dist: lxml>=4.9
+Requires-Dist: nltk>=3.8
+Requires-Dist: pdfminer.six>=20220524
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Dynamic: license-file
+
+# ChemDataExtractor Lite
+
+ChemDataExtractor Lite converts scientific HTML, XML, PDF, and plain-text files into a consistent `Document` object. It keeps document structure, metadata, tables, figures, references, sentence segmentation, and chemistry-aware word segmentation without the tagging and extraction stack from ChemDataExtractor 2.
+
+## Installation
+
+```bash
+pip install -e .
+```
+
+The chemistry-trained sentence model is included in the package, so document loading and segmentation work offline after installation.
+
+## Usage
+
+```python
+from chemdataextractor import Document
+
+document = Document.from_file("article.xml")
+
+for element in document:
+    if hasattr(element, "text"):
+        print(type(element).__name__, element.text)
+
+for sentence in document.sentences:
+    print(sentence.text)
+    print(sentence.raw_tokens)
+```
+
+`Document.from_string` accepts `bytes` or Unicode text. Both loading methods accept an optional `readers` list when explicit format handling is needed.
+
+## Supported readers
+
+- Generic HTML and XML
+- ACS, RSC, CSSP, and Springer HTML
+- Elsevier, NLM/JATS, Springer JATS, and USPTO XML
+- PDF through `pdfminer.six`
+- Plain text
+
+Publisher-specific readers run before generic fallbacks. Readers preserve titles, headings, paragraphs, citations, footnotes, metadata, figure captions and links, table captions, and raw table cells when the source exposes them.
+
+## Document model
+
+The public import namespace remains `chemdataextractor`. Structural classes are available from `chemdataextractor.doc`, and readers from `chemdataextractor.reader`.
+
+Extraction APIs from ChemDataExtractor 2 are intentionally absent: models, parsers, records, POS/NER tagging, chemical mention detection, relation extraction, scraping clients, and model downloads are outside this package.
+
+## Development
+
+```bash
+python -m pip install -e .
+python -m pip install pytest build
+python -m pytest
+python -m build
+```
+
+The project is released under the MIT license. The bundled `punkt_chem-1.0.pickle` sentence model is the chemistry Punkt model distributed by the ChemDataExtractor project.

chemdataextractor_lite-1.0.0/README.md

@@ -0,0 +1,56 @@
+# ChemDataExtractor Lite
+
+ChemDataExtractor Lite converts scientific HTML, XML, PDF, and plain-text files into a consistent `Document` object. It keeps document structure, metadata, tables, figures, references, sentence segmentation, and chemistry-aware word segmentation without the tagging and extraction stack from ChemDataExtractor 2.
+
+## Installation
+
+```bash
+pip install -e .
+```
+
+The chemistry-trained sentence model is included in the package, so document loading and segmentation work offline after installation.
+
+## Usage
+
+```python
+from chemdataextractor import Document
+
+document = Document.from_file("article.xml")
+
+for element in document:
+    if hasattr(element, "text"):
+        print(type(element).__name__, element.text)
+
+for sentence in document.sentences:
+    print(sentence.text)
+    print(sentence.raw_tokens)
+```
+
+`Document.from_string` accepts `bytes` or Unicode text. Both loading methods accept an optional `readers` list when explicit format handling is needed.
+
+## Supported readers
+
+- Generic HTML and XML
+- ACS, RSC, CSSP, and Springer HTML
+- Elsevier, NLM/JATS, Springer JATS, and USPTO XML
+- PDF through `pdfminer.six`
+- Plain text
+
+Publisher-specific readers run before generic fallbacks. Readers preserve titles, headings, paragraphs, citations, footnotes, metadata, figure captions and links, table captions, and raw table cells when the source exposes them.
+
+## Document model
+
+The public import namespace remains `chemdataextractor`. Structural classes are available from `chemdataextractor.doc`, and readers from `chemdataextractor.reader`.
+
+Extraction APIs from ChemDataExtractor 2 are intentionally absent: models, parsers, records, POS/NER tagging, chemical mention detection, relation extraction, scraping clients, and model downloads are outside this package.
+
+## Development
+
+```bash
+python -m pip install -e .
+python -m pip install pytest build
+python -m pytest
+python -m build
+```
+
+The project is released under the MIT license. The bundled `punkt_chem-1.0.pickle` sentence model is the chemistry Punkt model distributed by the ChemDataExtractor project.

chemdataextractor_lite-1.0.0/chemdataextractor/__init__.py

@@ -0,0 +1,13 @@
+"""Lightweight scientific document loading."""
+
+import logging
+
+from .doc.document import Document
+
+__title__ = "ChemDataExtractor Lite"
+__version__ = "1.0.0"
+__license__ = "MIT"
+
+logging.getLogger(__name__).addHandler(logging.NullHandler())
+
+__all__ = ["Document"]

chemdataextractor_lite-1.0.0/chemdataextractor/data/README.md

@@ -0,0 +1,3 @@
+# Bundled sentence model
+
+`punkt_chem-1.0.pickle` is the chemistry-trained Punkt sentence tokenizer distributed by the ChemDataExtractor project. It is bundled here so sentence segmentation does not perform network access at runtime.

chemdataextractor_lite-1.0.0/chemdataextractor/doc/__init__.py

@@ -0,0 +1,17 @@
+"""Structural document model."""
+
+from .document import Document
+from .element import BaseElement, CaptionedElement
+from .figure import Figure
+from .meta import MetaData
+from .table import Table
+from .text import (
+    Caption, Cell, Citation, Footnote, Heading, Paragraph, RichToken, Sentence,
+    Span, Text, Title, Token,
+)
+
+__all__ = [
+    "BaseElement", "Caption", "CaptionedElement", "Cell", "Citation", "Document",
+    "Figure", "Footnote", "Heading", "MetaData", "Paragraph", "RichToken",
+    "Sentence", "Span", "Table", "Text", "Title", "Token",
+]

chemdataextractor_lite-1.0.0/chemdataextractor/doc/document.py

@@ -0,0 +1,168 @@
+"""Document container and reader dispatch."""
+
+import collections
+import io
+import json
+import logging
+from pathlib import Path
+
+from ..errors import ReaderError
+from ..text import get_encoding
+from .element import CaptionedElement
+from .figure import Figure
+from .meta import MetaData
+from .table import Table
+from .text import Caption, Cell, Citation, Footnote, Heading, Paragraph, Sentence, Title
+
+log = logging.getLogger(__name__)
+
+
+class Document(collections.abc.Sequence):
+    """A format-neutral sequence of structural document elements."""
+
+    def __init__(self, *elements):
+        self._elements = []
+        for element in elements:
+            if isinstance(element, str):
+                element = Paragraph(element)
+            elif isinstance(element, bytes):
+                encoding = get_encoding(element)
+                log.warning("Guessed bytestring encoding as %s", encoding)
+                element = Paragraph(element.decode(encoding))
+            element.document = self
+            self._elements.append(element)
+
+    def __repr__(self):
+        return f"<Document: {len(self)} elements>"
+
+    __str__ = __repr__
+
+    def __getitem__(self, index):
+        return self.elements[index]
+
+    def __len__(self):
+        return len(self.elements)
+
+    @property
+    def elements(self):
+        return self._elements
+
+    @classmethod
+    def from_file(cls, file, fname=None, readers=None):
+        if isinstance(file, (str, bytes, Path)):
+            path = Path(file)
+            with path.open("rb") as stream:
+                return cls.from_string(stream.read(), fname=fname or str(path), readers=readers)
+        if fname is None and hasattr(file, "name"):
+            fname = file.name
+        return cls.from_string(file.read(), fname=fname, readers=readers)
+
+    @classmethod
+    def from_string(cls, content, fname=None, readers=None):
+        if isinstance(content, str):
+            content = content.encode("utf-8")
+        if not isinstance(content, bytes):
+            raise TypeError("content must be bytes or str")
+        if readers is None:
+            from ..reader import DEFAULT_READERS
+            readers = DEFAULT_READERS
+        errors = []
+        for reader in readers:
+            try:
+                if not reader.detect(content, fname=fname):
+                    continue
+                document = reader.readstring(content)
+                log.debug("Parsed document with %s", reader.__class__.__name__)
+                return document
+            except ReaderError as error:
+                errors.append(error)
+        message = "Unable to read document"
+        if errors:
+            message += f": {errors[-1]}"
+        raise ReaderError(message)
+
+    def get_element_with_id(self, id):
+        pending = list(self.elements)
+        while pending:
+            element = pending.pop(0)
+            if element.id == id:
+                return element
+            if element.elements:
+                pending.extend(element.elements)
+        return None
+
+    def _elements_of_type(self, element_type):
+        return [element for element in self.elements if isinstance(element, element_type)]
+
+    @property
+    def figures(self):
+        return self._elements_of_type(Figure)
+
+    @property
+    def tables(self):
+        return self._elements_of_type(Table)
+
+    @property
+    def citations(self):
+        return self._elements_of_type(Citation)
+
+    @property
+    def footnotes(self):
+        return self._elements_of_type(Footnote)
+
+    @property
+    def titles(self):
+        return self._elements_of_type(Title)
+
+    @property
+    def headings(self):
+        return self._elements_of_type(Heading)
+
+    @property
+    def paragraphs(self):
+        return self._elements_of_type(Paragraph)
+
+    @property
+    def captions(self):
+        return self._elements_of_type(Caption)
+
+    @property
+    def captioned_elements(self):
+        return self._elements_of_type(CaptionedElement)
+
+    @property
+    def metadata(self):
+        metadata = self._elements_of_type(MetaData)
+        return metadata[0] if metadata else None
+
+    @property
+    def sentences(self):
+        pending = list(self.elements)
+        sentences = []
+        while pending:
+            element = pending.pop(0)
+            if isinstance(element, Sentence) and not isinstance(element, Cell):
+                sentences.append(element)
+            elif element.elements:
+                pending[0:0] = element.elements
+        return sentences
+
+    def heading_for_sentence(self, sentence):
+        current_heading = None
+        for element in self.elements:
+            if isinstance(element, Heading):
+                current_heading = element
+                continue
+            if sentence in (element.elements or []):
+                return current_heading
+        return None
+
+    def serialize(self):
+        return {"type": "document", "elements": [element.serialize() for element in self.elements]}
+
+    def to_json(self, *args, **kwargs):
+        return json.dumps(self.serialize(), *args, **kwargs)
+
+    def _repr_html_(self):
+        body = "\n".join(element._repr_html_() for element in self.elements if hasattr(element, "_repr_html_"))
+        return f'<div class="cde-document">\n{body}\n</div>'

chemdataextractor_lite-1.0.0/chemdataextractor/doc/element.py

@@ -0,0 +1,59 @@
+"""Base classes for document elements."""
+
+import json
+from abc import ABCMeta, abstractmethod
+
+
+class BaseElement(metaclass=ABCMeta):
+    def __init__(self, document=None, references=None, id=None, **kwargs):
+        self._document = document
+        self.id = id
+        self.references = list(references or [])
+
+    def __repr__(self):
+        return f"<{self.__class__.__name__}>"
+
+    @property
+    def document(self):
+        return self._document
+
+    @document.setter
+    def document(self, document):
+        self._document = document
+
+    @property
+    def elements(self):
+        return None
+
+    @abstractmethod
+    def serialize(self):
+        raise NotImplementedError
+
+    def to_json(self, *args, **kwargs):
+        return json.dumps(self.serialize(), *args, **kwargs)
+
+
+class CaptionedElement(BaseElement):
+    def __init__(self, caption, label=None, **kwargs):
+        self.caption = caption
+        self.label = label
+        super().__init__(**kwargs)
+        self.caption.document = self.document
+
+    def __str__(self):
+        return self.caption.text
+
+    @BaseElement.document.setter
+    def document(self, document):
+        self._document = document
+        self.caption.document = document
+
+    @property
+    def elements(self):
+        return [self.caption]
+
+    def serialize(self):
+        data = {"type": self.__class__.__name__, "caption": self.caption.serialize()}
+        if self.label is not None:
+            data["label"] = self.label
+        return data

chemdataextractor_lite-1.0.0/chemdataextractor/doc/figure.py

@@ -0,0 +1,14 @@
+"""Figure document element."""
+
+from .element import CaptionedElement
+
+
+class Figure(CaptionedElement):
+    def __init__(self, caption, label=None, links=None, **kwargs):
+        super().__init__(caption=caption, label=label, **kwargs)
+        self.links = list(links or [])
+
+    def serialize(self):
+        data = super().serialize()
+        data["links"] = self.links
+        return data

chemdataextractor_lite-1.0.0/chemdataextractor/doc/meta.py

@@ -0,0 +1,30 @@
+"""Bibliographic metadata document element."""
+
+from .element import BaseElement
+
+
+class MetaData(BaseElement):
+    FIELDS = (
+        "title", "authors", "publisher", "journal", "volume", "issue",
+        "firstpage", "lastpage", "doi", "date", "language", "pdf_url", "html_url",
+    )
+
+    def __init__(self, data, **kwargs):
+        super().__init__(**kwargs)
+        normalized = {key.lstrip("_"): value for key, value in data.items()}
+        self._data = {key: normalized.get(key) for key in self.FIELDS}
+
+    def __repr__(self):
+        return repr({key: value for key, value in self.data.items() if value})
+
+    def __getattr__(self, name):
+        if name in self.FIELDS:
+            return self._data[name]
+        raise AttributeError(name)
+
+    @property
+    def data(self):
+        return dict(self._data)
+
+    def serialize(self):
+        return {"MetaData": {key: value for key, value in self._data.items() if value}}

chemdataextractor_lite-1.0.0/chemdataextractor/doc/table.py

@@ -0,0 +1,36 @@
+"""Structural table document element."""
+
+from .element import CaptionedElement
+from .text import Cell
+
+
+class Table(CaptionedElement):
+    def __init__(self, caption, label=None, table_data=None, footnotes=None, **kwargs):
+        super().__init__(caption=caption, label=label, **kwargs)
+        self.table_data = [
+            [cell if isinstance(cell, Cell) else Cell(str(cell)) for cell in row]
+            for row in (table_data or [])
+        ]
+        self.footnotes = list(footnotes or [])
+        self.document = self.document
+
+    @CaptionedElement.document.setter
+    def document(self, document):
+        self._document = document
+        self.caption.document = document
+        for row in getattr(self, "table_data", []):
+            for cell in row:
+                cell.document = document
+        for footnote in getattr(self, "footnotes", []):
+            footnote.document = document
+
+    @property
+    def elements(self):
+        return [cell for row in self.table_data for cell in row] + [self.caption] + self.footnotes
+
+    def serialize(self):
+        data = super().serialize()
+        data["rows"] = [[cell.text for cell in row] for row in self.table_data]
+        if self.footnotes:
+            data["footnotes"] = [footnote.serialize() for footnote in self.footnotes]
+        return data