bigdata-smart-batching 0.1.0__tar.gz → 1.0.1__tar.gz

bigdata_smart_batching-1.0.1/.gitignore

@@ -0,0 +1,115 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# Ruff
+.ruff_cache/
+
+# IDEs and editors
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# Jupyter Notebook
+.ipynb_checkpoints/
+
+# pyenv
+.python-version
+
+# Local / OS
+.DS_Store
+Thumbs.db
+
+# CSV files
+*.csv
+
+benchmark/
+benchmark.py
+scatter_date_relevance.py
+simple.py

bigdata_smart_batching-1.0.1/CHANGELOG.md

@@ -0,0 +1,104 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.1] - 2026-04-17
+
+### Fixed
+
+- Republish on PyPI after an erroneous publish of v1.0.0; package contents match the intended release.
+
+
+## [1.0.0] - 2026-04-17
+
+First stable release. Redesigns the batching pipeline so every API request stays
+within service limits while keeping plans deterministic.
+
+### Added
+
+- `create_entity_groups` step: sorts companies by chunks desc and greedy-packs
+  under `MAX_CHUNKS_PER_BASKET` and `MAX_ENTITIES_IN_ANY_OF`, honoring
+  `min_entities_per_basket`, with tail-group merge when it fits the entity
+  limit.
+- Automatic splitting of co-mention volume queries into sub-ranges of up to 365
+  days, with per-company chunks summed across sub-ranges in both three-pass and
+  iterative modes. Iterative mode now fans out `(batch, sub-range)` work items
+  to the thread pool and derives `very_low` from companies never seen in any
+  sub-range.
+- Parallel execution of `get_volume_timeseries` via `ThreadPoolExecutor`.
+- `volume_bucket` metadata on every group/basket via `classify_volume_bucket`
+  and `majority_volume_bucket` (ties broken by `VOLUME_BUCKET_PRIORITY`).
+- Centralized `VOLUME_BUCKETS` in `smart_batching_config` with explicit
+  `[lower, upper)` semantics and new `VOLUME_BUCKET_PRIORITY` constant.
+- INCLUDE/EXCLUDE category filtering for planning and all Bigdata query paths
+  (co-mention, volume, search), validated against a fixed set of allowed
+  category strings. `CategoryFilter`, `CategoryMode`, and validators/builders
+  are exported from the package.
+- Shared payload builders `build_comention_payload`, `build_volume_payload`,
+  and `build_search_query`, which `SmartBatchingPlanner` now routes through
+  instead of constructing payloads inline.
+- `content_diversification` option in `ranking_params`, exposed on
+  `plan_search`.
+- Universe input accepts either a CSV path or an inline list of entity IDs
+  (`universe: str | list[str]`). Added `resolve_universe()` and `UniverseInput`,
+  both exported from the package.
+- `Makefile` test targets and coverage settings in `pyproject.toml`.
+
+### Changed
+
+- **Breaking:** `SmartBatchingPlanner` public API redesigned. `create_baskets`
+  is replaced by `create_entity_groups` (new signature:
+  `min_entities_per_group` instead of `max_chunks`/`min_entities_per_basket`);
+  returned dicts no longer include `basket_id` or `volume_range` and expose
+  `volume_bucket` instead.
+- **Breaking:** Removed helpers `group_by_volume`,
+  `consolidate_period_groups`, `sub_period_volumes_from_series`, and
+  `get_volume_timeseries_per_company`.
+- **Breaking:** `very_low` basket IDs are now
+  `basket_{n}_very_low_{start}_{end}` instead of `very_low_basket_{n}`;
+  downstream parsers must be updated.
+- **Breaking:** `VOLUME_BUCKETS` ranges are now enforced as `[lower, upper)`.
+- **Breaking:** `plan_search(..., universe_csv_path=)` renamed to
+  `universe=` (accepts `str | list[str]`). `SmartBatchingPlanner.load_universe`
+  and `plan_all_periods` now take a required `universe` argument and delegate
+  to `resolve_universe`; `UNIVERSE_CSV_PATH` and duplicate CSV parsing removed.
+- **Breaking:** `plan_search` / `plan_all_periods` now accept an optional
+  `text` parameter and parameter order has been reordered.
+- Entity filters now use `search_in=BODY` for co-mention and volume payloads;
+  `ranking_params` dropped from the volume query.
+- Omit `text` from co-mention, volume, and search payloads when topic/text is
+  `None`.
+- Planned baskets are sorted by `basket_id` for stable ordering.
+- Architecture documentation updated to reflect the new `plan_search` flow
+  (sub-range splitting, entity groups, pre-fetch only for heavy groups, one
+  basket per `(group, sub-period)`) and a new high-level flow diagram.
+- `load_universe_from_csv` rejects empty CSV paths.
+
+### Removed
+
+- Obsolete `VOLUME_BUCKETS` structure test.
+
+## [0.2.0] - 2026-04-14
+
+### Changed
+
+- **Breaking:** `plan_search` now returns `chunk_upper_bound_estimate` instead of `total_expected_chunks`. Callers, saved plans, and any code reading the plan dict must use the new key. `execute_search` reads `chunk_upper_bound_estimate` when deriving sampling from the plan.
+- Terminology and docs describe this value as a chunk upper bound estimate (module docstring, README, architecture diagram, and example output).
+- README installation: document PyPI consumption (`pip install bigdata-smart-batching`, `uv add bigdata-smart-batching`) and keep a separate **Development** section for cloning the repo and `uv sync`.
+
+### Fixed
+
+- Align `uv.lock` `requires-python` with `pyproject.toml` (`>=3.11,<4.0`).
+
+## [0.1.0] - 2026-04-14
+
+### Added
+
+- Initial release of **bigdata-smart-batching**: high-performance semantic search with intelligent company grouping and parallel execution.
+- Planning API (`plan_search`) to organize searches via smart batching and report expected chunk counts.
+- Execution API (`execute_search`) with rate-limited parallel requests and proportional sampling to preserve result distribution.
+- Utilities: document deduplication, DataFrame conversion, and configurable search/batching behavior via `smart_batching` and related modules.
+- Python 3.11+ support; dependencies include `requests`, `pandas`, `numpy`, `matplotlib`, and `python-dotenv`.

bigdata_smart_batching-1.0.1/LICENSE

@@ -0,0 +1,5 @@
+This software is licensed for use solely under the terms agreed upon in the applicable Master Agreement and
+Order Schedule between the parties. For trials, the applicable legal documents are the Mutual Non-Disclosure
+Agreement, or if applicable the Trial Agreement. No other rights or licenses are granted by implication,
+estoppel, or otherwise. For further details, please refer to your specific Master Agreement and Order
+Schedule or contact us at legal@ravenpack.com.

{bigdata_smart_batching-0.1.0 → bigdata_smart_batching-1.0.1}/Makefile

@@ -1,4 +1,4 @@
-.PHONY: lint lint-check format format-check
+.PHONY: lint lint-check format format-check test test-cov
 
 lint:
 	uvx ruff check --fix src/ tests/
@@ -11,3 +11,9 @@ format:
 
 format-check:
 	uvx ruff format --check src/ tests/
+
+test:
+	uv run pytest
+
+test-cov:
+	uv run pytest --cov --cov-report=term-missing --cov-report=html

bigdata_smart_batching-0.1.0/README.md → bigdata_smart_batching-1.0.1/PKG-INFO

@@ -1,9 +1,25 @@
+Metadata-Version: 2.4
+Name: bigdata-smart-batching
+Version: 1.0.1
+Summary: High-performance semantic search with intelligent company grouping and parallel execution
+Project-URL: homepage, https://bigdata.com/api
+Author-email: "Bigdata.com" <support@ravenpack.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: <4.0,>=3.11
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: requests>=2.31.0
+Description-Content-Type: text/markdown
+
 # Smart Batching Search
 
 A high-performance semantic search system that reduces API queries by **67-99%** (varies by topic specificity) through intelligent company grouping and parallel execution.
 
 This module provides a two-step system for efficient semantic search:
-1. **Planning**: Organize search using smart batching and return total expected chunks
+1. **Planning**: Organize search using smart batching and return Chunk upper bound estimate
 2. **Execution**: Perform search with proportional sampling to preserve distribution
 
 ## Key Benefits
@@ -16,17 +32,24 @@ This module provides a two-step system for efficient semantic search:
 
 ## Installation
 
-### Using uv (Recommended)
+Install the package from PyPI (Python 3.11+):
 
 ```bash
-# Clone or navigate to the project
-cd bigdata-smart-batching
+pip install bigdata-smart-batching
+```
 
-# Install in development mode
-uv sync
+With [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv add bigdata-smart-batching
+```
+
+### Development
+
+To work on this repository locally, from the project root:
 
-# Or with dev dependencies
-uv sync --all-extras
+```bash
+uv sync
 ```
 
 ### Environment Setup
@@ -45,6 +68,21 @@ BIGDATA_API_KEY=your_api_key_here
 BIGDATA_API_BASE_URL=https://api.bigdata.com
 ```
 
+## Universe (CSV path or entity list)
+
+`plan_search()` takes a **`universe`** argument: either a path to a UTF-8 CSV file, or a `list[str]` of entity IDs. IDs must match the identifiers used by the Bigdata API for your dataset.
+
+When using a CSV, two layouts are supported:
+
+**1. Header row with an `id` column** (optional extra columns such as `name` are ignored):
+
+```csv
+id,name
+B8EF97,Example Corp A
+BB07E4,Example Corp B
+3461CF,Example Corp C
+```
+
 ## Quick Start
 
 ```python
@@ -58,13 +96,13 @@ from bigdata_smart_batching import (
 # Step 1: Plan the search
 plan = plan_search(
     text="earnings revenue profit",
-    universe_csv_path="id_name_mapping_us_top_3000.csv",
+    universe="id_name_mapping_us_top_3000.csv",
     start_date="2023-01-01",
     end_date="2023-12-31",
     api_key="your_api_key",  # or set BIGDATA_API_KEY env var
 )
 
-print(f"Total expected chunks: {plan['total_expected_chunks']:,}")
+print(f"Chunk upper bound estimate: {plan['chunk_upper_bound_estimate']:,}")
 
 # Step 2: Execute search with 10% of total chunks (preserves distribution)
 results_raw = execute_search(
@@ -88,7 +126,7 @@ from bigdata_smart_batching import plan_search, execute_search, save_plan, load_
 # Create and save a plan
 plan = plan_search(
     text="merger acquisition",
-    universe_csv_path="id_name_mapping_us_top_3000.csv",
+    universe="id_name_mapping_us_top_3000.csv",
     start_date="2023-01-01",
     end_date="2023-12-31",
 )
@@ -114,11 +152,11 @@ Step 2: EXECUTION
 
 ### Planning (`plan_search()`)
 
-1. Loads the universe of companies from CSV
+1. Loads the universe of companies from a CSV path or inline list
 2. Queries the comention endpoint to get chunk volumes per company
 3. Splits date ranges by volume when a company exceeds the chunk limit
 4. Creates optimized baskets grouped by volume
-5. Returns a plan with total expected chunks and basket configurations
+5. Returns a plan with Chunk upper bound estimate and basket configurations
 
 ### Execution (`execute_search()`)
 
@@ -134,7 +172,7 @@ Step 2: EXECUTION
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `text` | `str` | required | Search query text |
-| `universe_csv_path` | `str` | required | Path to CSV with entity IDs |
+| `universe` | `str \| list[str]` | required | CSV path or list of entity IDs |
 | `start_date` | `str` | required | Start date (YYYY-MM-DD) |
 | `end_date` | `str` | required | End date (YYYY-MM-DD) |
 | `api_key` | `str` | env var | API key |
@@ -156,7 +194,8 @@ Step 2: EXECUTION
 ### Helper Functions
 
 - **`deduplicate_documents(documents)`** -- Merges duplicate documents by `id`
-- **`load_universe_from_csv(csv_path)`** -- Loads entity IDs from CSV
+- **`resolve_universe(universe)`** -- Loads entity IDs from a CSV path or validates a `list[str]`
+- **`load_universe_from_csv(csv_path)`** -- Loads entity IDs from CSV only
 - **`convert_to_dataframe(raw_results)`** -- Converts documents to DataFrame (one row per chunk)
 - **`save_plan(plan, path)`** / **`load_plan(path)`** -- Persist plans as JSON
 - **`portfolio_backtesting_pipeline(...)`** -- Long-short portfolio backtesting
@@ -213,6 +252,6 @@ bigdata-smart-batching/
 
 ## License
 
-This project is part of the Bigdata.com and WorldQuant Challenge.
+This project is part of Bigdata.com.
 
 **Disclaimer**: This software is provided "as is" without warranty of any kind, express or implied. The authors and contributors assume no responsibility for the accuracy, completeness, or usefulness of any information, results, or processes provided. This software is for educational and research purposes only and is not intended to be used as financial advice.

bigdata_smart_batching-0.1.0/PKG-INFO → bigdata_smart_batching-1.0.1/README.md

@@ -1,22 +1,9 @@
-Metadata-Version: 2.4
-Name: bigdata-smart-batching
-Version: 0.1.0
-Summary: High-performance semantic search with intelligent company grouping and parallel execution
-License: MIT
-Requires-Python: >=3.11
-Requires-Dist: matplotlib>=3.7
-Requires-Dist: numpy>=1.24
-Requires-Dist: pandas>=2.0
-Requires-Dist: python-dotenv>=1.2.2
-Requires-Dist: requests>=2.31.0
-Description-Content-Type: text/markdown
-
 # Smart Batching Search
 
 A high-performance semantic search system that reduces API queries by **67-99%** (varies by topic specificity) through intelligent company grouping and parallel execution.
 
 This module provides a two-step system for efficient semantic search:
-1. **Planning**: Organize search using smart batching and return total expected chunks
+1. **Planning**: Organize search using smart batching and return Chunk upper bound estimate
 2. **Execution**: Perform search with proportional sampling to preserve distribution
 
 ## Key Benefits
@@ -29,17 +16,24 @@ This module provides a two-step system for efficient semantic search:
 
 ## Installation
 
-### Using uv (Recommended)
+Install the package from PyPI (Python 3.11+):
 
 ```bash
-# Clone or navigate to the project
-cd bigdata-smart-batching
+pip install bigdata-smart-batching
+```
 
-# Install in development mode
-uv sync
+With [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv add bigdata-smart-batching
+```
+
+### Development
+
+To work on this repository locally, from the project root:
 
-# Or with dev dependencies
-uv sync --all-extras
+```bash
+uv sync
 ```
 
 ### Environment Setup
@@ -58,6 +52,21 @@ BIGDATA_API_KEY=your_api_key_here
 BIGDATA_API_BASE_URL=https://api.bigdata.com
 ```
 
+## Universe (CSV path or entity list)
+
+`plan_search()` takes a **`universe`** argument: either a path to a UTF-8 CSV file, or a `list[str]` of entity IDs. IDs must match the identifiers used by the Bigdata API for your dataset.
+
+When using a CSV, two layouts are supported:
+
+**1. Header row with an `id` column** (optional extra columns such as `name` are ignored):
+
+```csv
+id,name
+B8EF97,Example Corp A
+BB07E4,Example Corp B
+3461CF,Example Corp C
+```
+
 ## Quick Start
 
 ```python
@@ -71,13 +80,13 @@ from bigdata_smart_batching import (
 # Step 1: Plan the search
 plan = plan_search(
     text="earnings revenue profit",
-    universe_csv_path="id_name_mapping_us_top_3000.csv",
+    universe="id_name_mapping_us_top_3000.csv",
     start_date="2023-01-01",
     end_date="2023-12-31",
     api_key="your_api_key",  # or set BIGDATA_API_KEY env var
 )
 
-print(f"Total expected chunks: {plan['total_expected_chunks']:,}")
+print(f"Chunk upper bound estimate: {plan['chunk_upper_bound_estimate']:,}")
 
 # Step 2: Execute search with 10% of total chunks (preserves distribution)
 results_raw = execute_search(
@@ -101,7 +110,7 @@ from bigdata_smart_batching import plan_search, execute_search, save_plan, load_
 # Create and save a plan
 plan = plan_search(
     text="merger acquisition",
-    universe_csv_path="id_name_mapping_us_top_3000.csv",
+    universe="id_name_mapping_us_top_3000.csv",
     start_date="2023-01-01",
     end_date="2023-12-31",
 )
@@ -127,11 +136,11 @@ Step 2: EXECUTION
 
 ### Planning (`plan_search()`)
 
-1. Loads the universe of companies from CSV
+1. Loads the universe of companies from a CSV path or inline list
 2. Queries the comention endpoint to get chunk volumes per company
 3. Splits date ranges by volume when a company exceeds the chunk limit
 4. Creates optimized baskets grouped by volume
-5. Returns a plan with total expected chunks and basket configurations
+5. Returns a plan with Chunk upper bound estimate and basket configurations
 
 ### Execution (`execute_search()`)
 
@@ -147,7 +156,7 @@ Step 2: EXECUTION
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `text` | `str` | required | Search query text |
-| `universe_csv_path` | `str` | required | Path to CSV with entity IDs |
+| `universe` | `str \| list[str]` | required | CSV path or list of entity IDs |
 | `start_date` | `str` | required | Start date (YYYY-MM-DD) |
 | `end_date` | `str` | required | End date (YYYY-MM-DD) |
 | `api_key` | `str` | env var | API key |
@@ -169,7 +178,8 @@ Step 2: EXECUTION
 ### Helper Functions
 
 - **`deduplicate_documents(documents)`** -- Merges duplicate documents by `id`
-- **`load_universe_from_csv(csv_path)`** -- Loads entity IDs from CSV
+- **`resolve_universe(universe)`** -- Loads entity IDs from a CSV path or validates a `list[str]`
+- **`load_universe_from_csv(csv_path)`** -- Loads entity IDs from CSV only
 - **`convert_to_dataframe(raw_results)`** -- Converts documents to DataFrame (one row per chunk)
 - **`save_plan(plan, path)`** / **`load_plan(path)`** -- Persist plans as JSON
 - **`portfolio_backtesting_pipeline(...)`** -- Long-short portfolio backtesting
@@ -226,6 +236,6 @@ bigdata-smart-batching/
 
 ## License
 
-This project is part of the Bigdata.com and WorldQuant Challenge.
+This project is part of Bigdata.com.
 
 **Disclaimer**: This software is provided "as is" without warranty of any kind, express or implied. The authors and contributors assume no responsibility for the accuracy, completeness, or usefulness of any information, results, or processes provided. This software is for educational and research purposes only and is not intended to be used as financial advice.