finance_data_llm 0.1.0__tar.gz

finance_data_llm-0.1.0/PKG-INFO

@@ -0,0 +1,235 @@
+Metadata-Version: 2.4
+Name: finance_data_llm
+Version: 0.1.0
+Summary: Add your description here
+Requires-Python: <3.14,>=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: pydantic-settings>=2.6.0
+Requires-Dist: uvicorn[standard]>=0.32.0
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: ratelimit>=2.2.1
+Requires-Dist: playwright>=1.49
+Requires-Dist: yfinance>=1.2.0
+Requires-Dist: numpy>=1.26
+
+# SEC-filings-Markdown
+
+## Configuration
+
+Settings are loaded via Pydantic Settings from environment variables or a `.env` file:
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `SEC_API_ORGANIZATION` | Organization name for SEC API User-Agent | `Your-Organization` |
+| `SEC_API_EMAIL` | Contact email for SEC API User-Agent | `your-email@example.com` |
+| `OLMOCR_SERVER` | vLLM server URL for olmOCR | `http://localhost:8000/v1` |
+| `OLMOCR_MODEL` | Model name for olmOCR | `allenai/olmOCR-2-7B-1025-FP8` |
+| `OLMOCR_WORKSPACE` | Workspace directory for OCR output | `./localworkspace` |
+| `EARNINGS_TRANSCRIPTS_DIR` | Directory for fetched transcript JSONL files | `earnings_transcripts_data` |
+| `EMBEDDING_SERVER` | OpenAI-compatible embedding API (e.g. vLLM pooling) | `http://127.0.0.1:8888/v1` |
+| `EMBEDDING_MODEL` | Model id passed to the embedding server | `Qwen/Qwen3-Embedding-0.6B` |
+| `CHROMA_PERSIST_DIR` | ChromaDB persistence directory | `./chroma_db` |
+
+
+## MCP server
+
+This repository includes an MCP server at `mcp_server.py` that exposes the same operational functions as `server.py` (SEC fetch, OCR, embedding, and search), plus file exploration tools for PDFs, JSONL, markdown, and other artifacts under configured data roots.
+
+Run it with the MCP dependency group:
+
+```bash
+uv run --group mcp python mcp_server.py
+```
+
+Key exploration tools exposed to MCP clients:
+
+- `list_data_roots_tool`: shows root directories available for browsing.
+- `list_data_files_tool`: glob file listing (for example `**/*.pdf`, `**/*.jsonl`).
+- `read_data_file_tool`: reads text-based files directly and provides metadata/preview for binary files.
+
+## Docker
+
+### Build
+
+```bash
+docker build -t sec-filings-md .
+```
+
+The image now defaults to a smaller footprint by using the CUDA runtime base while still preinstalling Playwright Chromium for scraping.
+If you want to skip Playwright browser installation (to reduce image size further), build with:
+
+```bash
+docker build --build-arg INSTALL_PLAYWRIGHT_BROWSER=0 -t sec-filings-md .
+```
+
+Or via Makefile:
+
+```bash
+make docker-build
+```
+
+### Run
+
+```bash
+GPU_DEVICE=${GPU_DEVICE:-3}
+docker run --gpus device=${GPU_DEVICE} \
+  -e SEC_API_ORGANIZATION="Your-Organization" \
+  -e SEC_API_EMAIL="your-email@example.com" \
+  -v ./sec_data:/app/sec_data \
+  -v ./localworkspace:/app/localworkspace \
+  -p 8081:8081 \
+  sec-filings-md
+```
+
+Or via Makefile (build + run in one step):
+
+```bash
+make docker-start
+```
+
+Makefile overrides:
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `IMAGE_NAME` | Docker image name | `sec-filings-md` |
+| `GPU_DEVICE` | GPU device index | `0` |
+| `API_PORT` | Host port for API | `8081` |
+| `SEC_API_ORGANIZATION` | SEC API User-Agent org | `Your-Organization` |
+| `SEC_API_EMAIL` | SEC API contact email | `your-email@example.com` |
+
+Example with overrides:
+
+```bash
+make docker-start GPU_DEVICE=3 SEC_API_EMAIL="you@example.com"
+```
+
+The two volumes persist data across container restarts:
+
+| Volume | Container path | Purpose |
+|--------|---------------|---------|
+| `sec_data` | `/app/sec_data` | Downloaded SEC filing PDFs |
+| `localworkspace` | `/app/localworkspace` | OCR workspace and output markdown |
+
+Override the workspace path at runtime with `-e OLMOCR_WORKSPACE=/custom/path`.
+
+## Installation
+
+```bash
+uv sync
+playwright install chromium
+```
+
+Install OCR/markdown + embedding stack dependencies when you need those pipelines:
+
+```bash
+uv sync --group ocr-md
+```
+
+Package install (for publishing/consuming from PyPI):
+
+```bash
+pip install finance_data_llm
+```
+
+Use package functions directly from Python (no server process required):
+
+```python
+from finance_data import (
+    company_name_to_ticker,
+    fetch_sec_filings_sync,
+)
+
+ticker = company_name_to_ticker("Amazon")
+filings = fetch_sec_filings_sync(ticker=ticker or "AMZN", year="2025")
+```
+
+If you do want to run the API, use the packaged console script:
+
+```bash
+finance-data-llm-server
+```
+
+## Usage
+
+Start vLLM server:
+```bash
+make vllm-olmocr-serve
+```
+
+Benchmark vLLM with guidellm (start the vLLM server first, then in another terminal):
+```bash
+make guidellm-benchmark
+```
+
+Fetch SEC filings:
+```bash
+uv run python -m filings.sec_data --ticker AMZN --year 2025
+```
+
+Run OCR pipeline:
+```bash
+uv run python ocr/olmocr_pipeline.py --pdf-dir sec_data/AMZN-2025
+```
+
+## Earnings call transcripts
+
+Transcripts are scraped from [discountingcashflows.com](https://discountingcashflows.com) (Playwright + Chromium). Each quarter is saved as one JSONL file under `{EARNINGS_TRANSCRIPTS_DIR}/{TICKER}/{year}/Q{n}.jsonl`.
+
+### 1. Fetch transcripts
+
+**CLI** (writes files under `earnings_transcripts_data` by default):
+
+```bash
+uv run python -m earnings_transcripts.transcripts AMZN 2025
+```
+
+Optional: `--max-concurrency` (default `4`) to limit parallel quarter fetches.
+
+**HTTP** (same fetch + persist, with the API running):
+
+```bash
+curl -s -X POST "http://127.0.0.1:8081/earnings_transcripts/for_year" \
+  -H "Content-Type: application/json" \
+  -d '{"ticker":"AMZN","year":2025}'
+```
+
+Response body is a JSON array of transcript objects (`ticker`, `year`, `quarter_num`, `date`, `speaker_texts`, …).
+
+### 2. Start embedding server and API
+
+Transcript chunks are embedded with the same OpenAI-compatible embedding endpoint as SEC filings (`EMBEDDING_SERVER` / `EMBEDDING_MODEL`). In one terminal:
+
+```bash
+make vllm-embd-serve
+```
+
+In another:
+
+```bash
+make start-server
+```
+
+(Adjust `API_PORT` / `EMBD_PORT` in the `Makefile` or your environment if needed.)
+
+### 3. Index transcripts in Chroma
+
+```bash
+curl -s -X POST "http://127.0.0.1:8081/vector_store/embed_transcripts" \
+  -H "Content-Type: application/json" \
+  -d '{"ticker":"AMZN","year":"2025","force":false}'
+```
+
+Use `"force": true` to replace existing vectors for those quarters. Filing types in the index appear as `Q1`–`Q4`.
+
+### 4. Search across indexed quarters
+
+Search merges hits from all transcript quarters present for that ticker/year:
+
+```bash
+curl -s -X POST "http://127.0.0.1:8081/vector_store/search_transcripts" \
+  -H "Content-Type: application/json" \
+  -d '{"ticker":"AMZN","year":"2025","query":"AWS revenue growth","top_k":5}'
+```
+
+Each result includes `filing_type` (`Q1`, …) so you can see which call the chunk came from.

finance_data_llm-0.1.0/README.md

@@ -0,0 +1,220 @@
+# SEC-filings-Markdown
+
+## Configuration
+
+Settings are loaded via Pydantic Settings from environment variables or a `.env` file:
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `SEC_API_ORGANIZATION` | Organization name for SEC API User-Agent | `Your-Organization` |
+| `SEC_API_EMAIL` | Contact email for SEC API User-Agent | `your-email@example.com` |
+| `OLMOCR_SERVER` | vLLM server URL for olmOCR | `http://localhost:8000/v1` |
+| `OLMOCR_MODEL` | Model name for olmOCR | `allenai/olmOCR-2-7B-1025-FP8` |
+| `OLMOCR_WORKSPACE` | Workspace directory for OCR output | `./localworkspace` |
+| `EARNINGS_TRANSCRIPTS_DIR` | Directory for fetched transcript JSONL files | `earnings_transcripts_data` |
+| `EMBEDDING_SERVER` | OpenAI-compatible embedding API (e.g. vLLM pooling) | `http://127.0.0.1:8888/v1` |
+| `EMBEDDING_MODEL` | Model id passed to the embedding server | `Qwen/Qwen3-Embedding-0.6B` |
+| `CHROMA_PERSIST_DIR` | ChromaDB persistence directory | `./chroma_db` |
+
+
+## MCP server
+
+This repository includes an MCP server at `mcp_server.py` that exposes the same operational functions as `server.py` (SEC fetch, OCR, embedding, and search), plus file exploration tools for PDFs, JSONL, markdown, and other artifacts under configured data roots.
+
+Run it with the MCP dependency group:
+
+```bash
+uv run --group mcp python mcp_server.py
+```
+
+Key exploration tools exposed to MCP clients:
+
+- `list_data_roots_tool`: shows root directories available for browsing.
+- `list_data_files_tool`: glob file listing (for example `**/*.pdf`, `**/*.jsonl`).
+- `read_data_file_tool`: reads text-based files directly and provides metadata/preview for binary files.
+
+## Docker
+
+### Build
+
+```bash
+docker build -t sec-filings-md .
+```
+
+The image now defaults to a smaller footprint by using the CUDA runtime base while still preinstalling Playwright Chromium for scraping.
+If you want to skip Playwright browser installation (to reduce image size further), build with:
+
+```bash
+docker build --build-arg INSTALL_PLAYWRIGHT_BROWSER=0 -t sec-filings-md .
+```
+
+Or via Makefile:
+
+```bash
+make docker-build
+```
+
+### Run
+
+```bash
+GPU_DEVICE=${GPU_DEVICE:-3}
+docker run --gpus device=${GPU_DEVICE} \
+  -e SEC_API_ORGANIZATION="Your-Organization" \
+  -e SEC_API_EMAIL="your-email@example.com" \
+  -v ./sec_data:/app/sec_data \
+  -v ./localworkspace:/app/localworkspace \
+  -p 8081:8081 \
+  sec-filings-md
+```
+
+Or via Makefile (build + run in one step):
+
+```bash
+make docker-start
+```
+
+Makefile overrides:
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `IMAGE_NAME` | Docker image name | `sec-filings-md` |
+| `GPU_DEVICE` | GPU device index | `0` |
+| `API_PORT` | Host port for API | `8081` |
+| `SEC_API_ORGANIZATION` | SEC API User-Agent org | `Your-Organization` |
+| `SEC_API_EMAIL` | SEC API contact email | `your-email@example.com` |
+
+Example with overrides:
+
+```bash
+make docker-start GPU_DEVICE=3 SEC_API_EMAIL="you@example.com"
+```
+
+The two volumes persist data across container restarts:
+
+| Volume | Container path | Purpose |
+|--------|---------------|---------|
+| `sec_data` | `/app/sec_data` | Downloaded SEC filing PDFs |
+| `localworkspace` | `/app/localworkspace` | OCR workspace and output markdown |
+
+Override the workspace path at runtime with `-e OLMOCR_WORKSPACE=/custom/path`.
+
+## Installation
+
+```bash
+uv sync
+playwright install chromium
+```
+
+Install OCR/markdown + embedding stack dependencies when you need those pipelines:
+
+```bash
+uv sync --group ocr-md
+```
+
+Package install (for publishing/consuming from PyPI):
+
+```bash
+pip install finance_data_llm
+```
+
+Use package functions directly from Python (no server process required):
+
+```python
+from finance_data import (
+    company_name_to_ticker,
+    fetch_sec_filings_sync,
+)
+
+ticker = company_name_to_ticker("Amazon")
+filings = fetch_sec_filings_sync(ticker=ticker or "AMZN", year="2025")
+```
+
+If you do want to run the API, use the packaged console script:
+
+```bash
+finance-data-llm-server
+```
+
+## Usage
+
+Start vLLM server:
+```bash
+make vllm-olmocr-serve
+```
+
+Benchmark vLLM with guidellm (start the vLLM server first, then in another terminal):
+```bash
+make guidellm-benchmark
+```
+
+Fetch SEC filings:
+```bash
+uv run python -m filings.sec_data --ticker AMZN --year 2025
+```
+
+Run OCR pipeline:
+```bash
+uv run python ocr/olmocr_pipeline.py --pdf-dir sec_data/AMZN-2025
+```
+
+## Earnings call transcripts
+
+Transcripts are scraped from [discountingcashflows.com](https://discountingcashflows.com) (Playwright + Chromium). Each quarter is saved as one JSONL file under `{EARNINGS_TRANSCRIPTS_DIR}/{TICKER}/{year}/Q{n}.jsonl`.
+
+### 1. Fetch transcripts
+
+**CLI** (writes files under `earnings_transcripts_data` by default):
+
+```bash
+uv run python -m earnings_transcripts.transcripts AMZN 2025
+```
+
+Optional: `--max-concurrency` (default `4`) to limit parallel quarter fetches.
+
+**HTTP** (same fetch + persist, with the API running):
+
+```bash
+curl -s -X POST "http://127.0.0.1:8081/earnings_transcripts/for_year" \
+  -H "Content-Type: application/json" \
+  -d '{"ticker":"AMZN","year":2025}'
+```
+
+Response body is a JSON array of transcript objects (`ticker`, `year`, `quarter_num`, `date`, `speaker_texts`, …).
+
+### 2. Start embedding server and API
+
+Transcript chunks are embedded with the same OpenAI-compatible embedding endpoint as SEC filings (`EMBEDDING_SERVER` / `EMBEDDING_MODEL`). In one terminal:
+
+```bash
+make vllm-embd-serve
+```
+
+In another:
+
+```bash
+make start-server
+```
+
+(Adjust `API_PORT` / `EMBD_PORT` in the `Makefile` or your environment if needed.)
+
+### 3. Index transcripts in Chroma
+
+```bash
+curl -s -X POST "http://127.0.0.1:8081/vector_store/embed_transcripts" \
+  -H "Content-Type: application/json" \
+  -d '{"ticker":"AMZN","year":"2025","force":false}'
+```
+
+Use `"force": true` to replace existing vectors for those quarters. Filing types in the index appear as `Q1`–`Q4`.
+
+### 4. Search across indexed quarters
+
+Search merges hits from all transcript quarters present for that ticker/year:
+
+```bash
+curl -s -X POST "http://127.0.0.1:8081/vector_store/search_transcripts" \
+  -H "Content-Type: application/json" \
+  -d '{"ticker":"AMZN","year":"2025","query":"AWS revenue growth","top_k":5}'
+```
+
+Each result includes `filing_type` (`Q1`, …) so you can see which call the chunk came from.

finance_data_llm-0.1.0/dataloader/__init__.py

@@ -0,0 +1,23 @@
+"""Dataloader for SEC filings: fetch, OCR, embed, and vector search."""
+
+from filings.utils import company_to_ticker
+
+from .pipeline import ensure_sec_data, prepare_sec_filing_envs
+from .repl_env import MarkdownReplEnvironment, markdown_to_repl_env
+from .text_splitter import Chunk, chunk_markdown
+from .vector_store import (
+    FaissVectorIndex,
+    embed_chunks,
+)
+
+__all__ = [
+    "company_to_ticker",
+    "ensure_sec_data",
+    "prepare_sec_filing_envs",
+    "MarkdownReplEnvironment",
+    "markdown_to_repl_env",
+    "Chunk",
+    "FaissVectorIndex",
+    "chunk_markdown",
+    "embed_chunks",
+]

finance_data_llm-0.1.0/dataloader/pipeline.py

@@ -0,0 +1,132 @@
+"""Pipeline to fetch SEC filings, run OCR, and prepare REPL environments."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from filings.sec_data import (
+    SecResults,
+    get_sec_results,
+    save_sec_results_as_pdfs,
+)
+from ocr.olmocr_pipeline import get_markdown_path, run_olmo_ocr
+from settings import sec_settings
+
+from .repl_env import MarkdownReplEnvironment, markdown_to_repl_env
+
+
+def _matches_filing_type(sec_result: SecResults, filing_type: str) -> bool:
+    """Return True if sec_result matches the requested filing type."""
+    if filing_type == "10-K":
+        return sec_result.form_name == "10-K"
+    if filing_type == "10-Q":
+        return sec_result.form_name.startswith("10-Q")
+    return sec_result.form_name == filing_type
+
+
+async def ensure_sec_data(
+    ticker: str,
+    year: str,
+    filing_types: list[str],
+    include_amends: bool = True,
+) -> tuple[list[SecResults], list[Path]]:
+    """
+    Ensure SEC filing PDFs exist locally. Download only missing files.
+
+    PDFs are stored in sec_data/{ticker}-{year}/ (per filings.sec_data).
+
+    Returns:
+        (sec_results matching filing_types, paths to all PDFs)
+    """
+    sec_results = get_sec_results(
+        ticker=ticker,
+        year=year,
+        filing_types=filing_types,
+        include_amends=include_amends,
+    )
+    output_dir = Path("sec_data") / f"{ticker}-{year}"
+
+    filtered = [
+        sr
+        for sr in sec_results
+        if any(_matches_filing_type(sr, ft) for ft in filing_types)
+    ]
+    existing_paths: list[Path] = []
+    missing_results: list[SecResults] = []
+    for sr in filtered:
+        p = output_dir / f"{sr.form_name}.pdf"
+        if p.exists():
+            existing_paths.append(p)
+        else:
+            missing_results.append(sr)
+
+    if missing_results:
+        new_paths = await save_sec_results_as_pdfs(
+            sec_results=missing_results,
+            ticker=ticker,
+            year=year,
+        )
+        pdf_paths = existing_paths + new_paths
+    else:
+        pdf_paths = existing_paths
+
+    return filtered, pdf_paths
+
+
+async def prepare_sec_filing_envs(
+    ticker: str,
+    year: str,
+    filing_type: str,
+    include_amends: bool = True,
+    workspace: str | Path | None = None,
+) -> list[MarkdownReplEnvironment]:
+    """
+    Fetch SEC filings (if needed), run OCR, and return REPL environments.
+
+    Args:
+        ticker: Stock ticker symbol (e.g. "GOOG", "AAPL").
+        year: Filing year (e.g. "2025").
+        filing_type: One of "10-K" or "10-Q".
+        include_amends: Include amended filings.
+        workspace: olmOCR workspace (default from settings). Markdown is written
+            to workspace/markdown/sec_data/{ticker}-{year}/...
+
+    Returns:
+        List of MarkdownReplEnvironment, one per filing (e.g. 10-K or 10-Q1..10-Q4).
+    """
+    workspace_str = str(workspace or sec_settings.olmocr_workspace)
+    pdf_dir_str = f"sec_data/{ticker}-{year}"
+
+    filing_types = [filing_type]
+    sec_results, _pdf_paths = await ensure_sec_data(
+        ticker=ticker,
+        year=year,
+        filing_types=filing_types,
+        include_amends=include_amends,
+    )
+    if not sec_results:
+        return []
+
+    await run_olmo_ocr(
+        pdf_dir=pdf_dir_str,
+        workspace=workspace_str,
+    )
+
+    envs: list[MarkdownReplEnvironment] = []
+    rel_pdf_base = f"sec_data/{ticker}-{year}"
+    for sr in sec_results:
+        source_file = f"{rel_pdf_base}/{sr.form_name}.pdf"
+        markdown_path_str = get_markdown_path(workspace_str, source_file)
+        markdown_path = Path(markdown_path_str)
+
+        if not markdown_path.exists():
+            continue
+        env = markdown_to_repl_env(
+            markdown_path=markdown_path,
+            ticker=ticker,
+            year=year,
+            sec_result=sr,
+        )
+        envs.append(env)
+
+    return envs

finance_data_llm-0.1.0/dataloader/repl_env.py

@@ -0,0 +1,68 @@
+from __future__ import annotations
+from code import InteractiveConsole
+from dataclasses import dataclass
+from pathlib import Path
+import functools
+
+from filings.sec_data import SecResults
+
+
+@dataclass
+class MarkdownReplEnvironment:
+    ticker: str
+    year: str
+    filing_type: str
+    markdown_path: Path
+    markdown_text: str
+    namespace: dict[str, object]
+    console: InteractiveConsole
+    sec_result: SecResults
+
+
+@functools.lru_cache
+def markdown_to_repl_env(
+    markdown_path: Path,
+    ticker: str,
+    year: str,
+    sec_result: SecResults,
+) -> MarkdownReplEnvironment:
+    resolved_path = markdown_path.resolve()
+    markdown_text = resolved_path.read_text(encoding="utf-8")
+    filing_type = resolved_path.stem
+    namespace: dict[str, object] = {
+        "ticker": ticker,
+        "year": year,
+        "filing_type": filing_type,
+        "markdown_path": resolved_path,
+        "markdown_text": markdown_text,
+        "sec_result": sec_result,
+    }
+    console = InteractiveConsole(locals=namespace)
+
+    return MarkdownReplEnvironment(
+        ticker=ticker,
+        year=year,
+        filing_type=filing_type,
+        markdown_path=resolved_path,
+        markdown_text=markdown_text,
+        namespace=namespace,
+        console=console,
+        sec_result=sec_result,
+    )
+
+
+if __name__ == "__main__":
+    import asyncio
+
+    from dataloader.pipeline import prepare_sec_filing_envs
+
+    envs = asyncio.run(
+        prepare_sec_filing_envs(
+            ticker="AMZN",
+            year="2025",
+            filing_type="10-K",
+            include_amends=True,
+        )
+    )
+    for env in envs:
+        print(env.ticker, env.year, env.filing_type, env.markdown_path)