ipo-mine 0.0.0__tar.gz

ipo_mine-0.0.0/PKG-INFO

@@ -0,0 +1,150 @@
+Metadata-Version: 2.4
+Name: ipo-mine
+Version: 0.0.0
+Summary: Mining and parsing S-1 IPO filings
+Author: Michael Galarnyk
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=1.5
+Requires-Dist: numpy>=1.21
+Requires-Dist: beautifulsoup4>=4.11
+Requires-Dist: lxml>=4.9
+Requires-Dist: requests>=2.28
+Requires-Dist: fuzzywuzzy>=0.18.0
+Requires-Dist: python-levenshtein
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: jupyter; extra == "dev"
+
+# IPO-Mine: S-1 (IPO) Filings Toolkit
+
+GitHub Repository: https://github.com/gtfintechlab/S1-Filings  
+Project Website: https://ipo-mine.web.app/
+
+## Overview
+
+IPO-Mine is a Python package for downloading, parsing, and structuring S-1 IPO filings from the U.S. Securities and Exchange Commission (SEC) EDGAR system.
+
+This repository implements the data processing pipeline used to construct the IPO-Mine dataset, a section-structured corpus introduced in the research paper:
+
+IPO-Mine: A Section-Structured Dataset for Analyzing Long and Complex IPO Filings
+
+The objective of this project is to transform raw SEC filings into clean, standardized, and section-aligned textual representations suitable for large-scale analysis in natural language processing, information retrieval, and long-document modeling.
+
+## Motivation
+
+S-1 filings are among the most complex regulatory documents used in empirical research. They exhibit several challenges:
+
+- Extreme document length, often exceeding 100–300 pages
+- Substantial variation in section headers across firms and time
+- Heterogeneous formats, including HTML, plain text, and scanned images
+- Limited structural consistency despite regulatory guidance
+
+These characteristics complicate tasks such as section segmentation, cross-firm comparison, longitudinal analysis, and long-context modeling.
+
+IPO-Mine addresses these challenges by providing a unified and reproducible pipeline that converts raw EDGAR filings into structured, research-ready data.
+
+## Features
+
+- Automated downloading of S-1 and S-1/A filings from SEC EDGAR
+- Parsing of Tables of Contents (TOCs) for filings dating back to 1997
+- Extraction and normalization of key IPO sections, including:
+  - Risk Factors
+  - Business
+  - Use of Proceeds
+  - Management’s Discussion and Analysis (MD&A)
+  - Financial Statements
+- Support for multiple filing formats:
+  - HTML
+  - plain text
+  - image-based filings via OCR
+- Fuzzy matching of section headers using global section mappings
+- Deterministic outputs suitable for reproducible dataset construction
+
+## IPO-Mine Dataset
+
+Using this toolkit, the IPO-Mine dataset is constructed as a large-scale corpus of IPO filings with:
+
+- Section-aligned text across firms
+- Standardized section nomenclature
+- Clean document boundaries
+- Compatibility with long-document modeling and retrieval frameworks
+
+Additional details and examples are available at:
+
+https://ipo-mine.web.app/
+
+## Installation
+
+The package is available on PyPI under the name `ipo-mine`.
+
+```
+pip install ipo-mine
+```
+
+## OCR Dependency
+
+Parsing image-based filings requires a local installation of Tesseract OCR.
+
+### Tesseract Installation
+
+| Operating System | Installation Method |
+|------------------|---------------------|
+| macOS | `brew install tesseract` |
+| Ubuntu / Debian | `sudo apt install tesseract-ocr` |
+| Windows | UB Mannheim Tesseract installer |
+| Conda environments | Included automatically |
+
+## Example Usage
+
+```python
+from ipo_mine.download.company import Company
+from ipo_mine.download import S1Downloader
+from ipo_mine.parse.s1_parser import S1Parser
+from ipo_mine.resources import GLOBAL_SECTIONS_JSON
+from ipo_mine.utils.config import PARSED_DIR
+
+downloader = S1Downloader(
+    email="your_email@domain.com",
+    company="Your Institution"
+)
+
+ticker = "SNOW"
+filing = downloader.download_s1(Company.from_ticker(ticker))
+
+parser = S1Parser(
+    filing=filing,
+    mappings_path=GLOBAL_SECTIONS_JSON,
+    output_base_path=PARSED_DIR
+)
+
+risk_factors = parser.parse_section("Risk Factors", ticker)
+```
+
+## Research-Oriented Design
+
+This library is designed primarily for dataset construction and reproducible empirical research rather than ad-hoc scraping.
+
+Typical use cases include:
+
+- Building section-aligned IPO corpora
+- Comparing disclosure language across firms and time
+- Training and evaluation of long-document language models
+- Large-scale studies of regulatory disclosures
+
+## Citation
+
+If you use this package or the IPO-Mine dataset in your research, please cite:
+
+```
+@inproceedings{ipomine2025,
+  title     = {IPO-Mine: A Section-Structured Dataset for Analyzing Long and Complex IPO Filings},
+  author    = {Author names},
+  booktitle = {Proceedings of the ACM SIGKDD Conference},
+  year      = {2025}
+}
+```
+
+## License
+
+This project is released under the MIT License.

ipo_mine-0.0.0/README.md

@@ -0,0 +1,132 @@
+# IPO-Mine: S-1 (IPO) Filings Toolkit
+
+GitHub Repository: https://github.com/gtfintechlab/S1-Filings  
+Project Website: https://ipo-mine.web.app/
+
+## Overview
+
+IPO-Mine is a Python package for downloading, parsing, and structuring S-1 IPO filings from the U.S. Securities and Exchange Commission (SEC) EDGAR system.
+
+This repository implements the data processing pipeline used to construct the IPO-Mine dataset, a section-structured corpus introduced in the research paper:
+
+IPO-Mine: A Section-Structured Dataset for Analyzing Long and Complex IPO Filings
+
+The objective of this project is to transform raw SEC filings into clean, standardized, and section-aligned textual representations suitable for large-scale analysis in natural language processing, information retrieval, and long-document modeling.
+
+## Motivation
+
+S-1 filings are among the most complex regulatory documents used in empirical research. They exhibit several challenges:
+
+- Extreme document length, often exceeding 100–300 pages
+- Substantial variation in section headers across firms and time
+- Heterogeneous formats, including HTML, plain text, and scanned images
+- Limited structural consistency despite regulatory guidance
+
+These characteristics complicate tasks such as section segmentation, cross-firm comparison, longitudinal analysis, and long-context modeling.
+
+IPO-Mine addresses these challenges by providing a unified and reproducible pipeline that converts raw EDGAR filings into structured, research-ready data.
+
+## Features
+
+- Automated downloading of S-1 and S-1/A filings from SEC EDGAR
+- Parsing of Tables of Contents (TOCs) for filings dating back to 1997
+- Extraction and normalization of key IPO sections, including:
+  - Risk Factors
+  - Business
+  - Use of Proceeds
+  - Management’s Discussion and Analysis (MD&A)
+  - Financial Statements
+- Support for multiple filing formats:
+  - HTML
+  - plain text
+  - image-based filings via OCR
+- Fuzzy matching of section headers using global section mappings
+- Deterministic outputs suitable for reproducible dataset construction
+
+## IPO-Mine Dataset
+
+Using this toolkit, the IPO-Mine dataset is constructed as a large-scale corpus of IPO filings with:
+
+- Section-aligned text across firms
+- Standardized section nomenclature
+- Clean document boundaries
+- Compatibility with long-document modeling and retrieval frameworks
+
+Additional details and examples are available at:
+
+https://ipo-mine.web.app/
+
+## Installation
+
+The package is available on PyPI under the name `ipo-mine`.
+
+```
+pip install ipo-mine
+```
+
+## OCR Dependency
+
+Parsing image-based filings requires a local installation of Tesseract OCR.
+
+### Tesseract Installation
+
+| Operating System | Installation Method |
+|------------------|---------------------|
+| macOS | `brew install tesseract` |
+| Ubuntu / Debian | `sudo apt install tesseract-ocr` |
+| Windows | UB Mannheim Tesseract installer |
+| Conda environments | Included automatically |
+
+## Example Usage
+
+```python
+from ipo_mine.download.company import Company
+from ipo_mine.download import S1Downloader
+from ipo_mine.parse.s1_parser import S1Parser
+from ipo_mine.resources import GLOBAL_SECTIONS_JSON
+from ipo_mine.utils.config import PARSED_DIR
+
+downloader = S1Downloader(
+    email="your_email@domain.com",
+    company="Your Institution"
+)
+
+ticker = "SNOW"
+filing = downloader.download_s1(Company.from_ticker(ticker))
+
+parser = S1Parser(
+    filing=filing,
+    mappings_path=GLOBAL_SECTIONS_JSON,
+    output_base_path=PARSED_DIR
+)
+
+risk_factors = parser.parse_section("Risk Factors", ticker)
+```
+
+## Research-Oriented Design
+
+This library is designed primarily for dataset construction and reproducible empirical research rather than ad-hoc scraping.
+
+Typical use cases include:
+
+- Building section-aligned IPO corpora
+- Comparing disclosure language across firms and time
+- Training and evaluation of long-document language models
+- Large-scale studies of regulatory disclosures
+
+## Citation
+
+If you use this package or the IPO-Mine dataset in your research, please cite:
+
+```
+@inproceedings{ipomine2025,
+  title     = {IPO-Mine: A Section-Structured Dataset for Analyzing Long and Complex IPO Filings},
+  author    = {Author names},
+  booktitle = {Proceedings of the ACM SIGKDD Conference},
+  year      = {2025}
+}
+```
+
+## License
+
+This project is released under the MIT License.

ipo_mine-0.0.0/pyproject.toml

@@ -0,0 +1,36 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ipo-mine"
+version = "0.0.0"
+description = "Mining and parsing S-1 IPO filings"
+readme = "README.md"
+requires-python = ">=3.9"
+
+authors = [
+    { name = "Michael Galarnyk" }
+]
+
+dependencies = [
+    "pandas>=1.5",
+    "numpy>=1.21",
+    "beautifulsoup4>=4.11",
+    "lxml>=4.9",
+    "requests>=2.28",
+    "fuzzywuzzy>=0.18.0",
+    "python-levenshtein",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest",
+    "jupyter",
+]
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]

ipo_mine-0.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ipo_mine-0.0.0/src/ipo_mine/__init__.py

@@ -0,0 +1,3 @@
+"""IPO Mining toolkit for downloading and parsing S-1 SEC filings."""
+
+__version__ = "0.0.0"

ipo_mine-0.0.0/src/ipo_mine/download/__init__.py

@@ -0,0 +1,7 @@
+from .downloader import IPODownloader
+from .company import Company
+from .dataset import Dataset
+
+
+
+__all__ = ["IPODownloader", "Company", "Dataset"]

ipo_mine-0.0.0/src/ipo_mine/download/company.py

@@ -0,0 +1,112 @@
+import pandas as pd
+import requests
+from dataclasses import dataclass
+from typing import Optional, Dict, Tuple
+
+from resources import get_resource_path
+
+@dataclass
+class Company:
+    
+    ticker: str
+    cik: str
+    active: bool
+
+    _ticker_to_cik: Optional[Dict[str, Tuple[str, bool]]] = None
+    _cik_to_ticker: Optional[Dict[str, Tuple[str, bool]]] = None
+    
+    @classmethod
+    def _load_mapping(cls):
+        """
+        Loads and caches the ticker-CIK mapping from file(s) into dictionaries.
+        This method is designed to run only once.
+        """
+        if cls._ticker_to_cik is not None and cls._cik_to_ticker is not None:
+            return
+
+        # print("Loading and caching Ticker-CIK mapping for the first time...")
+        try:
+            csv_path = get_resource_path("company_tickers_to_cik.csv")
+            df = pd.read_csv(csv_path, dtype=str)
+            df['active'] = True # Mark entries from the SEC file as active
+        except FileNotFoundError:
+            cls._download_ticker_cik_mapping()
+            csv_path = get_resource_path("company_tickers_to_cik.csv")
+            df = pd.read_csv(csv_path, dtype=str)
+            df['active'] = True # Mark entries from the SEC file as active
+        
+        try:
+            upgraded_df = pd.read_csv("upgraded_mapping.csv", dtype=str)
+            upgraded_df['active'] = False # Mark entries from the custom file as inactive
+            # Combine the two DataFrames. If there are duplicate tickers,
+            # the one from the SEC (marked active=True) will be kept.
+            df = pd.concat([upgraded_df, df]).drop_duplicates(subset=['ticker'], keep='last')
+            # print("Successfully merged upgraded mapping.")
+        # except FileNotFoundError:
+            # print("Warning: 'upgraded_mapping' file not found. Using standard mapping only.")
+        except Exception as e:
+            pass
+            # print(f"Warning: Could not load 'upgraded_mapping' due to an error: {e}")
+
+        # Create fast dictionary lookups storing a tuple of (value, active_status)
+        cls._ticker_to_cik = {
+            str(row.ticker): (str(row.cik_str), bool(row.active))
+            for row in df.itertuples(index=False)
+        }
+
+        cls._cik_to_ticker = {
+            str(row.cik_str): (str(row.ticker), bool(row.active))
+            for row in df.itertuples(index=False)
+        }
+
+    @classmethod
+    def from_ticker(cls, ticker: str) -> 'Company':
+        """Create Company from ticker, looking up the CIK and active status."""
+        lookup_result = cls._lookup_cik_from_ticker(ticker.upper())
+        if not lookup_result:
+            raise ValueError(f"No CIK found for ticker: {ticker}")
+        
+        cik, active = lookup_result
+        return cls(ticker=ticker.upper(), cik=cik, active=active)
+    
+    @classmethod
+    def from_cik(cls, cik: str) -> 'Company':
+        """Create Company from CIK, looking up the ticker and active status."""
+        formatted_cik = cik.zfill(10)
+        lookup_result = cls._lookup_ticker_from_cik(formatted_cik)
+        if not lookup_result:
+            return cls(ticker="", cik=formatted_cik, active=False)
+            
+        ticker, active = lookup_result
+        return cls(ticker=ticker, cik=formatted_cik, active=active)
+    
+    @classmethod
+    def _lookup_cik_from_ticker(cls, ticker: str) -> Optional[str]:
+        """Look up CIK from the cached dictionary. 🚀"""
+        cls._load_mapping()  # Ensures mapping is loaded before lookup
+        return cls._ticker_to_cik.get(ticker)
+    
+    @classmethod
+    def _lookup_ticker_from_cik(cls, cik: str) -> Optional[str]:
+        """Look up ticker from the cached dictionary. 🚀"""
+        cls._load_mapping() # Ensures mapping is loaded before lookup
+        return cls._cik_to_ticker.get(cik)
+    
+    @staticmethod
+    def _download_ticker_cik_mapping():
+        """Download the ticker-CIK mapping from SEC."""
+        headers = {"User-Agent": "Company Lookup Tool contact@example.com"}
+        try:
+            response = requests.get(
+                "https://www.sec.gov/files/company_tickers.json", 
+                headers=headers
+            )
+            response.raise_for_status()
+            df = pd.DataFrame.from_dict(response.json(), orient="index")
+            df["cik_str"] = df["cik_str"].astype(str).str.zfill(10)
+    
+            save_path = get_resource_path("company_tickers_to_cik.csv")
+            df.to_csv(save_path, index=False)
+            print(f"✅ Successfully downloaded ticker-CIK mapping → {save_path}")
+        except Exception as e:
+            raise RuntimeError(f"Failed to download ticker-CIK mapping: {e}")

ipo_mine-0.0.0/src/ipo_mine/download/dataset.py

@@ -0,0 +1,193 @@
+import json
+from typing import Dict, Any, List, Union, Optional, Tuple
+from collections import defaultdict
+from entities import S1Filing, S1FilingImage, CompanyFilings, Filing, FilingImage, FormType
+import random
+
+class Dataset:
+    """
+    Manages the collection of CompanyFilings objects, handling JSON loading and sampling.
+    """
+    def __init__(self, raw_data_or_path: Union[str, Dict[str, Dict[str, Any]]]):
+        """
+        Initializes the dataset by either loading from a file path or processing 
+        an existing dictionary of raw data.
+        """
+        if isinstance(raw_data_or_path, str):
+            raw_data = self._load_json(raw_data_or_path)
+        else:
+            raw_data = raw_data_or_path
+            
+        self.companies: Dict[str, CompanyFilings] = self._parse_data(raw_data)
+
+    def _load_json(self, file_path: str) -> Dict[str, Dict[str, Any]]:
+        """Handles opening and parsing the JSON file."""
+        with open(file_path, 'r') as f:
+            return json.load(f)
+
+    def _parse_data(self, raw_data: Dict[str, Dict[str, Any]]) -> Dict[str, CompanyFilings]:
+        """Iterates through the raw data dictionary and creates CompanyFilings objects."""
+        parsed_companies = {}
+        
+        for cik, company_data in raw_data.items():
+            # Extract and parse filings
+            raw_filings = company_data.get("filings", [])
+            filings_list = []
+            
+            for filing_data in raw_filings:
+                # Parse images for this filing
+                raw_images = filing_data.get("images", [])
+                images_list = [
+                    FilingImage(
+                        img_name=img.get('img_name'),
+                        url=img.get('url'),
+                        local_path=img.get('local_path')
+                    ) for img in raw_images
+                ]
+                
+                # Parse form_type enum
+                form_type_str = filing_data.get('form_type', '')
+                try:
+                    form_type = FormType(form_type_str)
+                except ValueError:
+                    print(f"Warning: Unknown form type '{form_type_str}' for CIK {cik}")
+                    continue
+                
+                # Create Filing object
+                filing = Filing(
+                    form_type=form_type,
+                    acession_number=filing_data.get('acession_number', ''),
+                    filing_date=filing_data.get('filing_date', ''),
+                    primary_document=filing_data.get('primary_document', ''),
+                    filing_url=filing_data.get('filing_url', ''),
+                    local_path=filing_data.get('local_path'),
+                    images=images_list,
+                    raw_content=filing_data.get('raw_content')
+                )
+                filings_list.append(filing)
+            
+            # Create CompanyFilings object
+            company = CompanyFilings(
+                tickers=company_data.get('tickers', []),
+                cik=company_data.get('cik', cik),
+                name=company_data.get('name', ''),
+                sic=company_data.get('sic'),
+                industry=company_data.get('industry'),
+                office=company_data.get('office'),
+                exchanges=company_data.get('exchanges'),
+                filings=filings_list
+            )
+            
+            parsed_companies[cik] = company
+                
+        return parsed_companies
+
+    def sample_filings_by_year(self, num_samples: int, start_year: Optional[int] = None, end_year: Optional[int] = None) -> 'Dataset':
+        """
+        Samples up to 'num_samples' filings per year and returns a new 
+        Dataset instance containing only the sampled filings.
+
+        :param num_samples: The maximum number of filings to sample per year.
+        :param start_year: The start year to sample (inclusive)
+        :param end_year: The end year to sample (exclusive)
+        :return: A new Dataset instance with the sampled data.
+        """
+        
+        # Collect all filings across all companies, grouped by year
+        filings_by_year: Dict[int, List[Tuple[str, Filing]]] = defaultdict(list)
+        
+        for cik, company in self.companies.items():
+            for filing in company.filings:
+                # Extract year from filing_date (format: "YYYY-MM-DD")
+                try:
+                    year = int(filing.filing_date.split('-')[0])
+                except (ValueError, IndexError, AttributeError):
+                    print(f"Warning: Could not parse filing_date for CIK {cik}")
+                    continue
+                
+                if (start_year is not None and year < start_year):
+                    continue
+                if (end_year is not None and year >= end_year):
+                    continue
+                
+                filings_by_year[year].append((cik, filing))
+        
+        # Sample filings per year
+        sampled_filings: Dict[str, List[Filing]] = defaultdict(list)
+        sorted_years = sorted(filings_by_year.keys())
+        
+        for year in sorted_years:
+            filing_list = filings_by_year[year]
+            k = min(num_samples, len(filing_list))
+            yearly_sample = random.sample(filing_list, k)
+            
+            for cik, filing in yearly_sample:
+                sampled_filings[cik].append(filing)
+        
+        # Reconstruct raw data structure with only sampled filings
+        new_raw_data = {}
+        
+        for cik, filing_list in sampled_filings.items():
+            company = self.companies[cik]
+            
+            # Convert filings to dictionaries
+            filings_data = []
+            for filing in filing_list:
+                filing_dict = {
+                    'form_type': filing.form_type.value,
+                    'acession_number': filing.acession_number,
+                    'filing_date': filing.filing_date,
+                    'primary_document': filing.primary_document,
+                    'filing_url': filing.filing_url,
+                    'local_path': filing.local_path,
+                    'images': [
+                        {
+                            'img_name': img.img_name,
+                            'url': img.url,
+                            'local_path': img.local_path
+                        } for img in filing.images
+                    ],
+                    'raw_content': filing.raw_content
+                }
+                filings_data.append(filing_dict)
+            
+            # Reconstruct company data
+            new_raw_data[cik] = {
+                'tickers': company.tickers,
+                'cik': company.cik,
+                'name': company.name,
+                'sic': company.sic,
+                'industry': company.industry,
+                'office': company.office,
+                'exchanges': company.exchanges,
+                'filings': filings_data
+            }
+        
+        total_sampled = sum(len(filings) for filings in sampled_filings.values())
+        print(f"Sampled {total_sampled} filings across {len(filings_by_year)} years from {len(sampled_filings)} companies.")
+        
+        return Dataset(new_raw_data)
+
+    def get_company_by_cik(self, cik: str) -> Optional[CompanyFilings]:
+        """Retrieves the CompanyFilings object for a given CIK."""
+        return self.companies.get(cik)
+
+    def get_company_by_ticker(self, ticker: str) -> Optional[CompanyFilings]:
+        """Retrieves the CompanyFilings object for a given ticker."""
+        ticker_upper = ticker.upper()
+        for company in self.companies.values():
+            if ticker_upper in [t.upper() for t in company.tickers]:
+                return company
+        return None
+
+    def __len__(self) -> int:
+        """Returns the number of companies in the dataset."""
+        return len(self.companies)
+
+    def __iter__(self):
+        """Allows iteration over the CompanyFilings objects in the dataset."""
+        return iter(self.companies.values())
+
+    def __repr__(self):
+        total_filings = sum(len(company.filings) for company in self.companies.values())
+        return f"Dataset(total_companies={len(self.companies)}, total_filings={total_filings})"