datadock 1.0.2__tar.gz

datadock-1.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 PETER MBACHU
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datadock-1.0.2/PKG-INFO

@@ -0,0 +1,179 @@
+Metadata-Version: 2.1
+Name: datadock
+Version: 1.0.2
+Summary: DataDock python library to access SEC edgar fillings.
+License: MIT
+Keywords: datadockpy,datadockAI,python,SEC
+Author: Peter Mbachu
+Author-email: peter.mbachu@datadock.ai.com
+Requires-Python: >=3.10,<4.0
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Dist: azure-core (>=1.31.0,<2.0.0)
+Requires-Dist: azure-storage-blob (>=12.23.1,<13.0.0)
+Requires-Dist: beautifulsoup4 (>=4.12.3,<5.0.0)
+Requires-Dist: colorama (>=0.4.6,<0.5.0)
+Requires-Dist: fastcore (>=1.7.17,<2.0.0)
+Requires-Dist: humanize (>=4.11.0,<5.0.0)
+Requires-Dist: lxml (>=5.3.0,<6.0.0)
+Requires-Dist: pandas (>=2.2.3,<3.0.0)
+Requires-Dist: pyarrow (>=17.0.0,<18.0.0)
+Requires-Dist: python-decouple (>=3.8,<4.0)
+Requires-Dist: requests (>=2.32.3,<3.0.0)
+Requires-Dist: rich (>=13.8.1,<14.0.0)
+Requires-Dist: tabulate (>=0.9.0,<0.10.0)
+Project-URL: Bug Tracker, https://github.com/cla-bit/PayStackEase/issues
+Project-URL: Documentation, https://paystackease.readthedocs.io/en/latest/
+Project-URL: Homepage, https://github.com/cla-bit/PayStackEase
+Project-URL: Source Code, https://github.com/cla-bit/PayStackEase
+Description-Content-Type: text/markdown
+
+# DataDockPy Library  
+
+--------------
+
+![Python Versions](https://img.shields.io/badge/python-3.9|3.10|3.11|3.12-blue) 
+![License](https://img.shields.io/pypi/l/paystackease.svg) 
+![PyPi - Version](https://img.shields.io/pypi/v/paystackease.svg)
+
+<p align="center">    
+<a href="https://github.com/DataDock-AI/DataDockPy">
+    <img src="media/20241001135438-ezgif.com-video-to-gif-converter.gif" alt="DataDock demo" height="350">
+</a>
+</p>
+
+
+## Introduction
+
+DataDockPy is a powerful and user-friendly Python package designed to enhance data analysis. 
+It simplifies the extraction and presentation of information from the SEC Edgar database, 
+offering enriched data display for SEC form types, form controls, and financial analysis, 
+making it an essential tool for professionals working with regulatory filings.
+
+This library uses `Poetry` as a dependency manager.
+
+
+
+## Features
+
+- 📁 **Access any SEC filing**: You can access any filings on SEC forms of Form 8-K and Form 10-K
+- 📅 **List filings for any date range**: List filings by date e.g. or date range `2024-02-29:2024-03-15`
+- 🌟 **User Friendly library**: Uses **[rich](https://rich.readthedocs.io/en/stable/introduction.html) & [tabulate](https://github.com/astanin/python-tabulate)** library to display SEC Edgar data in a beautiful way.
+- 🔄 **Page through filings**: Use `filings.next()` and `filings.previous()` to page through filings
+- 🏗️ **Filter filings data**: Build data filtering by cik, accession number, form type, filing date
+- ✅ **Select a filing**: You can select a filing from the list of filings.
+- 🔍 **Preview the text data for a filing**: You can preview the filing (sections) in the terminal or a notebook.
+- 📊 **Parse to Dataframe**: You can parse filings to a dataframe.
+- 📈 **Financial Statement**: Get financial statements of Form 8-K and Form 10-K of various companies.
+
+
+## Get Started on Windows/MacOS/Linux Terminal
+
+1. Open your terminal and install poetry using `pip`.
+    
+    ```commandline
+   pip install poetry
+   ```
+   or
+
+    Install poetry using `pipx`.
+    
+    ```commandline 
+   pipx install poetry
+    ```
+
+2. Create a project and clone DataDockPy github repository.
+
+    ```git
+    git clone https://github.com/DataDock-AI/DataDockPy.git
+    ```
+
+3. Change directory to `DataDockPy`
+
+   ```commandline
+   cd DataDockPy
+   ```
+
+4. Run the poetry command to install dependencies: 
+
+    ```commandline
+   poetry install
+   ```
+
+5. Activate virtual environment using poetry: 
+
+    ```commandline
+    poetry shell
+    ```
+
+6. Set up your SEC_IDENTITY in an `.env` file
+
+   ```dotenv
+    SEC_IDENTITY=<your email or usernmae for SEC IDENTITY>
+   ```
+
+7. See the following scripts on how to use the package: `run_checks.py` and `run_checks2.py`
+
+
+## Use DataDockPy with Jupyter
+
+1. Open terminal and install poetry using `pip`.
+    
+    ```commandline
+   pip install poetry
+   ```
+   or
+
+    Install poetry using `pipx`.
+    
+    ```commandline 
+   pipx install poetry
+    ```
+
+2. Create a project and clone `DataDockPy` github repository 
+
+   ```git
+    git clone https://github.com/DataDock-AI/DataDockPy.git
+    ```
+
+3. Change directory to `DataDockPy`
+
+   ```commandline
+   cd DataDockPy
+   ```
+
+4. Open your project directory on Anaconda or Visual Studio Code.
+
+5. Choose a python environment (recommended python environment), where poetry was installed. `Ctrl+Shift+P`
+6. If asked to install `ipykernel`, see here on installation: [ipykernel installation](https://devinschumacher.com/how-to-setup-jupyter-notebook-virtual-environment-vs-code-kernels/)
+7. Check to see if poetry is installed:
+
+   ```bash
+   !poetry --version   
+   ```
+
+8. Run the poetry command to install dependencies: 
+
+    ```commandline
+   poetry install
+   ```
+
+
+## Download DataDockPy Source Code
+
+You can download any of the source codes: `zip` or `tar.gz` here: [DataDockPy Source Code](https://github.com/DataDock-AI/DataDockPy/releases/tag/v0.1.0).
+
+
+If you have any issue or contribution, please write an issue with this link: https://github.com/DataDock-AI/DataDockPy/issues
+
+
+

datadock-1.0.2/README.md

@@ -0,0 +1,139 @@
+# DataDockPy Library  
+
+--------------
+
+![Python Versions](https://img.shields.io/badge/python-3.9|3.10|3.11|3.12-blue) 
+![License](https://img.shields.io/pypi/l/paystackease.svg) 
+![PyPi - Version](https://img.shields.io/pypi/v/paystackease.svg)
+
+<p align="center">    
+<a href="https://github.com/DataDock-AI/DataDockPy">
+    <img src="media/20241001135438-ezgif.com-video-to-gif-converter.gif" alt="DataDock demo" height="350">
+</a>
+</p>
+
+
+## Introduction
+
+DataDockPy is a powerful and user-friendly Python package designed to enhance data analysis. 
+It simplifies the extraction and presentation of information from the SEC Edgar database, 
+offering enriched data display for SEC form types, form controls, and financial analysis, 
+making it an essential tool for professionals working with regulatory filings.
+
+This library uses `Poetry` as a dependency manager.
+
+
+
+## Features
+
+- 📁 **Access any SEC filing**: You can access any filings on SEC forms of Form 8-K and Form 10-K
+- 📅 **List filings for any date range**: List filings by date e.g. or date range `2024-02-29:2024-03-15`
+- 🌟 **User Friendly library**: Uses **[rich](https://rich.readthedocs.io/en/stable/introduction.html) & [tabulate](https://github.com/astanin/python-tabulate)** library to display SEC Edgar data in a beautiful way.
+- 🔄 **Page through filings**: Use `filings.next()` and `filings.previous()` to page through filings
+- 🏗️ **Filter filings data**: Build data filtering by cik, accession number, form type, filing date
+- ✅ **Select a filing**: You can select a filing from the list of filings.
+- 🔍 **Preview the text data for a filing**: You can preview the filing (sections) in the terminal or a notebook.
+- 📊 **Parse to Dataframe**: You can parse filings to a dataframe.
+- 📈 **Financial Statement**: Get financial statements of Form 8-K and Form 10-K of various companies.
+
+
+## Get Started on Windows/MacOS/Linux Terminal
+
+1. Open your terminal and install poetry using `pip`.
+    
+    ```commandline
+   pip install poetry
+   ```
+   or
+
+    Install poetry using `pipx`.
+    
+    ```commandline 
+   pipx install poetry
+    ```
+
+2. Create a project and clone DataDockPy github repository.
+
+    ```git
+    git clone https://github.com/DataDock-AI/DataDockPy.git
+    ```
+
+3. Change directory to `DataDockPy`
+
+   ```commandline
+   cd DataDockPy
+   ```
+
+4. Run the poetry command to install dependencies: 
+
+    ```commandline
+   poetry install
+   ```
+
+5. Activate virtual environment using poetry: 
+
+    ```commandline
+    poetry shell
+    ```
+
+6. Set up your SEC_IDENTITY in an `.env` file
+
+   ```dotenv
+    SEC_IDENTITY=<your email or usernmae for SEC IDENTITY>
+   ```
+
+7. See the following scripts on how to use the package: `run_checks.py` and `run_checks2.py`
+
+
+## Use DataDockPy with Jupyter
+
+1. Open terminal and install poetry using `pip`.
+    
+    ```commandline
+   pip install poetry
+   ```
+   or
+
+    Install poetry using `pipx`.
+    
+    ```commandline 
+   pipx install poetry
+    ```
+
+2. Create a project and clone `DataDockPy` github repository 
+
+   ```git
+    git clone https://github.com/DataDock-AI/DataDockPy.git
+    ```
+
+3. Change directory to `DataDockPy`
+
+   ```commandline
+   cd DataDockPy
+   ```
+
+4. Open your project directory on Anaconda or Visual Studio Code.
+
+5. Choose a python environment (recommended python environment), where poetry was installed. `Ctrl+Shift+P`
+6. If asked to install `ipykernel`, see here on installation: [ipykernel installation](https://devinschumacher.com/how-to-setup-jupyter-notebook-virtual-environment-vs-code-kernels/)
+7. Check to see if poetry is installed:
+
+   ```bash
+   !poetry --version   
+   ```
+
+8. Run the poetry command to install dependencies: 
+
+    ```commandline
+   poetry install
+   ```
+
+
+## Download DataDockPy Source Code
+
+You can download any of the source codes: `zip` or `tar.gz` here: [DataDockPy Source Code](https://github.com/DataDock-AI/DataDockPy/releases/tag/v0.1.0).
+
+
+If you have any issue or contribution, please write an issue with this link: https://github.com/DataDock-AI/DataDockPy/issues
+
+

datadock-1.0.2/datadock/__init__.py

@@ -0,0 +1,19 @@
+from datadock.src import (
+    DataDockError,
+    CustomLogger,
+    SECRequestHandler,
+    # CurrentFile,
+    display_table_with_rich,
+    # TextSummaryModel,
+    # SentimentAnalysisModel,
+    # EntityRecognitionModel,
+)
+from datadock.core import DataPager, table_array
+from datadock.filings import CurrentFilings
+from datadock.controllers import FormControl
+from datadock.financials import parse_html_new, DocumentFinancial, DataDocFiling
+from datadock.ddhtml import CompanyFilingInfo
+from datadock.metadata.__version__ import __version__
+
+
+VERSION = __version__

datadock-1.0.2/datadock/config.py

@@ -0,0 +1,4 @@
+from decouple import config
+
+
+sec_identity: str = config("SEC_IDENTITY")

datadock-1.0.2/datadock/controllers/__init__.py

@@ -0,0 +1,4 @@
+from datadock.controllers.control import FormControl
+
+
+__all__ = ["FormControl"]

datadock-1.0.2/datadock/controllers/_base_controllers.py

@@ -0,0 +1,120 @@
+from abc import ABC, abstractmethod
+from typing import Optional, List, Union
+import pyarrow as pa
+import pandas as pd
+from rich.console import Group
+from rich.panel import Panel
+from rich.text import Text
+
+from datadock.src import DocumentProcessor
+
+from datadock.src.custom_logger import CustomLogger
+from datadock.core import DataPager
+from datadock.src._rich_ import repr_rich
+from datadock.src.constants import IntString
+from datadock.src.filters import filter_by_section_titles
+from datadock.src.dataclasses import FilingsState
+
+
+class FormBaseController(ABC):
+    def __init__(
+        self,
+        cik: str,
+        accession_number: str,
+        form_type: str,
+        logger: CustomLogger,
+        document_processor: DocumentProcessor,
+    ):
+        self.cik = cik
+        self.accession_number = accession_number
+        self.form_type = form_type
+        self._logger = logger
+        self._document_processor = document_processor
+        self.data_table: Optional[pa.Table] = self._process()
+        self.data: FormDataSections = FormDataSections(self.data_table)
+
+    @abstractmethod
+    def _process(self) -> Optional[pa.Table]:
+        pass
+
+    def to_pandas(self, *columns) -> Optional[pd.DataFrame]:
+        if not self.data:
+            return None
+        data_frame = self.data.data.to_pandas()
+        return data_frame.filter(columns) if len(columns) > 0 else data_frame
+
+    """
+    Next step is to implement using rich or tabulate to display the pyarrow table result to users
+    """
+
+    def filter(
+        self,
+        titles: Optional[Union[str, List[IntString]]] = None,
+    ) -> Optional["FormDataSections"]:
+        filing_index = self.data.data
+        section_titles = titles
+
+        if isinstance(section_titles, list):
+            section_titles = [str(title) for title in section_titles]
+
+        # Filter by form
+        if section_titles:
+            filing_index = filter_by_section_titles(filing_index, titles=section_titles)
+        return FormDataSections(filing_index)
+
+    @property
+    def get_all_sections(self):
+        return self.data
+
+
+class FormDataSections:
+
+    def __init__(
+        self,
+        filing_index: pa.Table,
+        original_state: Optional[FilingsState] = None,
+        logger: Optional[CustomLogger] = None,
+    ) -> None:
+        self.data = filing_index
+        self.original_state = original_state or FilingsState(0, len(filing_index))
+        self.data_pager = DataPager(self.data)
+        self._logger: CustomLogger = logger or CustomLogger().logger
+
+    def _page_index(self) -> range:
+        """Create the range index to set on the page dataframe depending on where in the data we are"""
+        if self.original_state:
+            return range(
+                self.original_state.page_start,
+                self.original_state.page_start
+                + min(self.data_pager.page_size, len(self.data)),
+            )  # set the index to the size of the page
+        else:
+            return range(*self.data_pager.current_range)
+
+    def __rich__(self) -> Panel:
+        # Convert the PyArrow table to a pandas DataFrame for easier processing
+        df = self.data.to_pandas()
+
+        # Create a list to hold all sections
+        sections_content = []
+
+        # Add each section ID and its content to the list
+        for section_id, section_text in zip(df["Section Title"], df["Section Content"]):
+            section_id_text = Text(section_id, style="bold blue")
+            section_content_text = Text(section_text)
+
+            # Create a sub-panel for each section
+            section_sub_panel = Panel(section_content_text, title=section_id_text)
+            sections_content.append(section_sub_panel)
+
+        # Create a main panel that contains all sections
+        sections_panel = Panel(
+            Group(*sections_content),
+            title="DataDock Filings Sections and Contents",
+            border_style="green",
+        )
+
+        return sections_panel
+
+    def __repr__(self):
+        return repr_rich(self.__rich__())

datadock-1.0.2/datadock/controllers/_controller_10k.py

@@ -0,0 +1,157 @@
+import re
+import pyarrow as pa
+from typing import Optional, Dict
+
+from bs4 import BeautifulSoup
+
+from datadock.controllers._base_controllers import FormBaseController
+from datadock.src.utils import clean
+from datadock.core import table_array
+
+
+class Clean10KController(FormBaseController):
+    _SECTION_NAMES = [
+        ("item1", "Business"),
+        ("item1a", "Risk Factors"),
+        ("item1b", "Unresolved Staff Comments"),
+        ("item1c", "Cybersecurity"),
+        ("item2", "Properties"),
+        ("item3", "Legal Proceedings"),
+        ("item4", "Mine Safety Disclosures"),
+        (
+            "item5",
+            "Market for Registrant’s Common Equity, Related Stockholder Matters, and Issuer Purchases of Equity Securities",
+        ),
+        ("item6", "Selected Financial Data"),
+        (
+            "item7",
+            "Management’s Discussion and Analysis of Financial Condition and Results of Operations",
+        ),
+        ("item7a", "Quantitative and Qualitative Disclosures about Market Risk"),
+        ("item8", "Financial Statements and Supplementary Data"),
+        (
+            "item9",
+            "Changes in and Disagreements with Accountants on Accounting and Financial Disclosure",
+        ),
+        ("item9a", "Controls and Procedures"),
+        ("item9b", "Other Information"),
+        (
+            "item9c",
+            "Disclosure Regarding Foreign Jurisdictions that Prevent Inspections",
+        ),
+        ("item10", "Directors, Executive Officers, and Corporate Governance"),
+        ("item11", "Executive Compensation"),
+        (
+            "item12",
+            "Security Ownership of Certain Beneficial Owners and Management and Related Stockholder Matters",
+        ),
+        (
+            "item13",
+            "Certain Relationships and Related Transactions, and Director Independence",
+        ),
+        ("item14", "Principal Accountant Fees and Services"),
+        ("item15", "Exhibit and Financial Statement Schedules"),
+        ("item16", "Form 10-K Summary"),
+    ]
+
+    def _process(self) -> Optional[pa.Table]:
+        raw_sections = {}
+        doc_sections = self._document_processor.extract_sections("10-K")
+
+        if "10-K" in doc_sections:
+            document_10k = doc_sections["10-K"]
+
+            # Parse the document with BeautifulSoup
+            soup = BeautifulSoup(document_10k, "html.parser")
+
+            # Extract section names from <td> tags with an <a> tag
+            for td_tag in soup.find_all("td"):
+                a_tag = td_tag.find("a", href=True)
+                if not a_tag:
+                    continue  # Skip if there is no <a> tag
+
+                # Get the section name text from the <a> tag
+                section_text = a_tag.get_text(strip=True)
+
+                for item_pattern, section_name in self._SECTION_NAMES:
+                    if re.search(section_name, section_text, re.IGNORECASE):
+                        # Match the href with the corresponding content <p> tag by id
+                        href_value = a_tag["href"].lstrip("#")
+                        content_tag = soup.find(id=href_value)
+                        section_content = ""
+
+                        if content_tag:
+                            # Check if the content_tag is part of a <span> tag with no direct text
+                            if content_tag.name == "span" and not content_tag.get_text(
+                                strip=True
+                            ):
+                                # Check if the parent contains the text (e.g., a <p> tag or another wrapping element)
+                                parent_tag = content_tag.find_parent(["p", "span"])
+                                if parent_tag:
+                                    section_content = parent_tag.get_text(
+                                        separator=" ", strip=True
+                                    )
+                            else:
+                                section_content = content_tag.get_text(
+                                    separator=" ", strip=True
+                                )
+
+                            # Gather all the content until the next section id is encountered
+                            next_tag = content_tag.find_next_sibling()
+                            while next_tag and not self._is_new_section(next_tag):
+                                section_content += " " + next_tag.get_text(
+                                    separator=" ", strip=True
+                                )
+                                next_tag = next_tag.find_next_sibling()
+
+                            # Save the accumulated content for the current section
+                            raw_sections[
+                                f"{self._document_processor.unique_id}-{section_name}"
+                            ] = section_content
+                        break
+
+            # Clean up sections by removing overlaps where a section contains the next section's name
+            raw_sections = self._clean_overlapping_sections(raw_sections)
+
+            # Clean the extracted section contents before returning
+            for section_key, raw_content in raw_sections.items():
+                cleaned_text = clean(raw_content)
+                raw_sections[section_key] = cleaned_text
+
+        return table_array(raw_sections) if raw_sections else None
+
+    def _is_new_section(self, element) -> bool:
+        """
+        Checks if the current element contains the start of a new section.
+        """
+        if element.name in ["p", "span"] and element.get("id"):
+            # Check if the id matches any section header
+            for _, section_name in self._SECTION_NAMES:
+                if re.search(section_name, element.get_text(), re.IGNORECASE):
+                    return True
+        return False
+
+    def _clean_overlapping_sections(self, sections: Dict[str, str]) -> Dict[str, str]:
+        """
+        Ensure each section's content is unique and doesn't include the content of subsequent sections.
+        """
+        section_keys = list(sections.keys())
+        for i, current_section_key in enumerate(section_keys):
+            current_content = sections[current_section_key]
+
+            # Look ahead to the next section to find and remove overlaps
+            if i + 1 < len(section_keys):
+                next_section_name = self._SECTION_NAMES[i + 1][1]
+                next_section_pattern = re.escape(next_section_name)
+
+                # Remove anything from the current content that includes the next section name
+                next_section_match = re.search(
+                    next_section_pattern, current_content, re.IGNORECASE
+                )
+                if next_section_match:
+                    # Cut the content from the start of the next section's name
+                    sections[current_section_key] = current_content[
+                        : next_section_match.start()
+                    ].strip()
+
+        return sections