bigdata-smart-batching 0.1.0__tar.gz → 0.2.0__tar.gz

bigdata_smart_batching-0.2.0/.gitignore

@@ -0,0 +1,110 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# Ruff
+.ruff_cache/
+
+# IDEs and editors
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# Jupyter Notebook
+.ipynb_checkpoints/
+
+# pyenv
+.python-version
+
+# Local / OS
+.DS_Store
+Thumbs.db
+
+# CSV files
+*.csv

bigdata_smart_batching-0.2.0/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] - 2026-04-14
+
+### Changed
+
+- **Breaking:** `plan_search` now returns `chunk_upper_bound_estimate` instead of `total_expected_chunks`. Callers, saved plans, and any code reading the plan dict must use the new key. `execute_search` reads `chunk_upper_bound_estimate` when deriving sampling from the plan.
+- Terminology and docs describe this value as a chunk upper bound estimate (module docstring, README, architecture diagram, and example output).
+- README installation: document PyPI consumption (`pip install bigdata-smart-batching`, `uv add bigdata-smart-batching`) and keep a separate **Development** section for cloning the repo and `uv sync`.
+
+### Fixed
+
+- Align `uv.lock` `requires-python` with `pyproject.toml` (`>=3.11,<4.0`).
+
+## [0.1.0] - 2026-04-14
+
+### Added
+
+- Initial release of **bigdata-smart-batching**: high-performance semantic search with intelligent company grouping and parallel execution.
+- Planning API (`plan_search`) to organize searches via smart batching and report expected chunk counts.
+- Execution API (`execute_search`) with rate-limited parallel requests and proportional sampling to preserve result distribution.
+- Utilities: document deduplication, DataFrame conversion, and configurable search/batching behavior via `smart_batching` and related modules.
+- Python 3.11+ support; dependencies include `requests`, `pandas`, `numpy`, `matplotlib`, and `python-dotenv`.

bigdata_smart_batching-0.2.0/LICENSE

@@ -0,0 +1,5 @@
+This software is licensed for use solely under the terms agreed upon in the applicable Master Agreement and
+Order Schedule between the parties. For trials, the applicable legal documents are the Mutual Non-Disclosure
+Agreement, or if applicable the Trial Agreement. No other rights or licenses are granted by implication,
+estoppel, or otherwise. For further details, please refer to your specific Master Agreement and Order
+Schedule or contact us at legal@ravenpack.com.

bigdata_smart_batching-0.1.0/README.md → bigdata_smart_batching-0.2.0/PKG-INFO

@@ -1,9 +1,25 @@
+Metadata-Version: 2.4
+Name: bigdata-smart-batching
+Version: 0.2.0
+Summary: High-performance semantic search with intelligent company grouping and parallel execution
+Project-URL: homepage, https://bigdata.com/api
+Author-email: "Bigdata.com" <support@ravenpack.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: <4.0,>=3.11
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: requests>=2.31.0
+Description-Content-Type: text/markdown
+
 # Smart Batching Search
 
 A high-performance semantic search system that reduces API queries by **67-99%** (varies by topic specificity) through intelligent company grouping and parallel execution.
 
 This module provides a two-step system for efficient semantic search:
-1. **Planning**: Organize search using smart batching and return total expected chunks
+1. **Planning**: Organize search using smart batching and return Chunk upper bound estimate
 2. **Execution**: Perform search with proportional sampling to preserve distribution
 
 ## Key Benefits
@@ -16,17 +32,24 @@ This module provides a two-step system for efficient semantic search:
 
 ## Installation
 
-### Using uv (Recommended)
+Install the package from PyPI (Python 3.11+):
 
 ```bash
-# Clone or navigate to the project
-cd bigdata-smart-batching
+pip install bigdata-smart-batching
+```
 
-# Install in development mode
-uv sync
+With [uv](https://docs.astral.sh/uv/):
 
-# Or with dev dependencies
-uv sync --all-extras
+```bash
+uv add bigdata-smart-batching
+```
+
+### Development
+
+To work on this repository locally, from the project root:
+
+```bash
+uv sync
 ```
 
 ### Environment Setup
@@ -45,6 +68,19 @@ BIGDATA_API_KEY=your_api_key_here
 BIGDATA_API_BASE_URL=https://api.bigdata.com
 ```
 
+## Universe CSV file
+
+`plan_search()` loads company **entity IDs** from a UTF-8 CSV. IDs must match the identifiers used by the Bigdata API for your dataset. Two layouts are supported:
+
+**1. Header row with an `id` column** (optional extra columns such as `name` are ignored):
+
+```csv
+id,name
+B8EF97,Example Corp A
+BB07E4,Example Corp B
+3461CF,Example Corp C
+```
+
 ## Quick Start
 
 ```python
@@ -64,7 +100,7 @@ plan = plan_search(
     api_key="your_api_key",  # or set BIGDATA_API_KEY env var
 )
 
-print(f"Total expected chunks: {plan['total_expected_chunks']:,}")
+print(f"Chunk upper bound estimate: {plan['chunk_upper_bound_estimate']:,}")
 
 # Step 2: Execute search with 10% of total chunks (preserves distribution)
 results_raw = execute_search(
@@ -118,7 +154,7 @@ Step 2: EXECUTION
 2. Queries the comention endpoint to get chunk volumes per company
 3. Splits date ranges by volume when a company exceeds the chunk limit
 4. Creates optimized baskets grouped by volume
-5. Returns a plan with total expected chunks and basket configurations
+5. Returns a plan with Chunk upper bound estimate and basket configurations
 
 ### Execution (`execute_search()`)
 

bigdata_smart_batching-0.1.0/PKG-INFO → bigdata_smart_batching-0.2.0/README.md

@@ -1,22 +1,9 @@
-Metadata-Version: 2.4
-Name: bigdata-smart-batching
-Version: 0.1.0
-Summary: High-performance semantic search with intelligent company grouping and parallel execution
-License: MIT
-Requires-Python: >=3.11
-Requires-Dist: matplotlib>=3.7
-Requires-Dist: numpy>=1.24
-Requires-Dist: pandas>=2.0
-Requires-Dist: python-dotenv>=1.2.2
-Requires-Dist: requests>=2.31.0
-Description-Content-Type: text/markdown
-
 # Smart Batching Search
 
 A high-performance semantic search system that reduces API queries by **67-99%** (varies by topic specificity) through intelligent company grouping and parallel execution.
 
 This module provides a two-step system for efficient semantic search:
-1. **Planning**: Organize search using smart batching and return total expected chunks
+1. **Planning**: Organize search using smart batching and return Chunk upper bound estimate
 2. **Execution**: Perform search with proportional sampling to preserve distribution
 
 ## Key Benefits
@@ -29,17 +16,24 @@ This module provides a two-step system for efficient semantic search:
 
 ## Installation
 
-### Using uv (Recommended)
+Install the package from PyPI (Python 3.11+):
 
 ```bash
-# Clone or navigate to the project
-cd bigdata-smart-batching
+pip install bigdata-smart-batching
+```
 
-# Install in development mode
-uv sync
+With [uv](https://docs.astral.sh/uv/):
 
-# Or with dev dependencies
-uv sync --all-extras
+```bash
+uv add bigdata-smart-batching
+```
+
+### Development
+
+To work on this repository locally, from the project root:
+
+```bash
+uv sync
 ```
 
 ### Environment Setup
@@ -58,6 +52,19 @@ BIGDATA_API_KEY=your_api_key_here
 BIGDATA_API_BASE_URL=https://api.bigdata.com
 ```
 
+## Universe CSV file
+
+`plan_search()` loads company **entity IDs** from a UTF-8 CSV. IDs must match the identifiers used by the Bigdata API for your dataset. Two layouts are supported:
+
+**1. Header row with an `id` column** (optional extra columns such as `name` are ignored):
+
+```csv
+id,name
+B8EF97,Example Corp A
+BB07E4,Example Corp B
+3461CF,Example Corp C
+```
+
 ## Quick Start
 
 ```python
@@ -77,7 +84,7 @@ plan = plan_search(
     api_key="your_api_key",  # or set BIGDATA_API_KEY env var
 )
 
-print(f"Total expected chunks: {plan['total_expected_chunks']:,}")
+print(f"Chunk upper bound estimate: {plan['chunk_upper_bound_estimate']:,}")
 
 # Step 2: Execute search with 10% of total chunks (preserves distribution)
 results_raw = execute_search(
@@ -131,7 +138,7 @@ Step 2: EXECUTION
 2. Queries the comention endpoint to get chunk volumes per company
 3. Splits date ranges by volume when a company exceeds the chunk limit
 4. Creates optimized baskets grouped by volume
-5. Returns a plan with total expected chunks and basket configurations
+5. Returns a plan with Chunk upper bound estimate and basket configurations
 
 ### Execution (`execute_search()`)
 

{bigdata_smart_batching-0.1.0 → bigdata_smart_batching-0.2.0}/docs/smart-batching-architecture.md

@@ -98,7 +98,7 @@ flowchart TD
 
     PHASE2 --> ASSEMBLE["Assemble plan:<br/>Convert baskets → query payloads<br/>Add very_low baskets<br/>(0-chunk companies, max_chunks=100)"]
 
-    ASSEMBLE --> PLAN_OUT["Plan Output:<br/>total_expected_chunks,<br/>baskets[ ] with embedded queries,<br/>planning_metadata"]
+    ASSEMBLE --> PLAN_OUT["Plan Output:<br/>chunk_upper_bound_estimate,<br/>baskets[ ] with embedded queries,<br/>planning_metadata"]
 
     PLAN_OUT --> EXEC_START(["Start: execute_search()"])
 

{bigdata_smart_batching-0.1.0 → bigdata_smart_batching-0.2.0}/examples/example.py

@@ -55,7 +55,7 @@ def main() -> None:
         min_entities_per_basket=10,
     )
 
-    print(f"Total expected chunks: {plan['total_expected_chunks']:,}")
+    print(f"Chunk upper bound estimate: {plan['chunk_upper_bound_estimate']:,}")
 
     # Step 2: Execute search with 10% of total chunks (preserves distribution)
     results_raw = execute_search(

{bigdata_smart_batching-0.1.0 → bigdata_smart_batching-0.2.0}/pyproject.toml

@@ -1,9 +1,10 @@
 [project]
 name = "bigdata-smart-batching"
-version = "0.1.0"
+version = "0.2.0"
 description = "High-performance semantic search with intelligent company grouping and parallel execution"
 readme = "README.md"
-requires-python = ">=3.11"
+authors = [{ name = "Bigdata.com", email = "support@ravenpack.com" }]
+requires-python = ">=3.11,<4.0"
 license = { text = "MIT" }
 dependencies = [
     "requests>=2.31.0",
@@ -28,6 +29,9 @@ build-backend = "hatchling.build"
 [tool.hatch.build.targets.wheel]
 packages = ["src/bigdata_smart_batching"]
 
+[project.urls]
+homepage = "https://bigdata.com/api"
+
 [tool.ruff]
 line-length = 100
 target-version = "py311"

{bigdata_smart_batching-0.1.0 → bigdata_smart_batching-0.2.0}/src/bigdata_smart_batching/search_function.py

@@ -2,7 +2,7 @@
 Smart Batching Search Function
 
 This module provides a two-step system for efficient semantic search:
-1. Planning: Organize search using smart batching and return total expected chunks
+1. Planning: Organize search using smart batching and return Chunk upper bound estimate
 2. Execution: Perform search with proportional sampling to preserve distribution
 """
 
@@ -411,7 +411,7 @@ def plan_search(
                 (no consolidation).
     Returns:
         Dict with planning results:
-        - total_expected_chunks: Total chunks expected
+        - chunk_upper_bound_estimate: Total chunks expected
         - baskets: List of basket configs with complete query structures
         - planning_metadata: Additional metadata
     """
@@ -479,7 +479,7 @@ def plan_search(
         }
 
         plan = {
-            "total_expected_chunks": estimated_chunks,
+            "chunk_upper_bound_estimate": estimated_chunks,
             "baskets": [basket],
             "planning_metadata": {
                 "total_companies": len(companies),
@@ -589,7 +589,7 @@ def plan_search(
                 baskets.append(basket)
 
         plan = {
-            "total_expected_chunks": total_expected,
+            "chunk_upper_bound_estimate": total_expected,
             "baskets": baskets,
             "planning_metadata": {
                 "total_companies": len(companies),
@@ -603,7 +603,7 @@ def plan_search(
         }
 
     logger.info(
-        f"Planning complete: {plan['total_expected_chunks']:,} expected chunks in {len(plan['baskets'])} baskets"
+        f"Planning complete: {plan['chunk_upper_bound_estimate']:,} expected chunks in {len(plan['baskets'])} baskets"
     )
     return plan
 
@@ -778,7 +778,7 @@ def execute_search(
 
     api_base_url = api_base_url or os.getenv("BIGDATA_API_BASE_URL", DEFAULT_API_BASE_URL)
 
-    total_expected = search_plan.get("total_expected_chunks", 0)
+    total_expected = search_plan.get("chunk_upper_bound_estimate", 0)
     target_chunks = int(total_expected * chunk_percentage)
     logger.info(f"Executing search with {chunk_percentage * 100:.1f}% of chunks")
     logger.info(f"Total maximum expected chunks: {target_chunks:,}")

{bigdata_smart_batching-0.1.0 → bigdata_smart_batching-0.2.0}/uv.lock

@@ -1,6 +1,6 @@
 version = 1
 revision = 2
-requires-python = ">=3.11"
+requires-python = ">=3.11, <4.0"
 resolution-markers = [
     "python_full_version >= '3.14' and sys_platform == 'win32'",
     "python_full_version >= '3.14' and sys_platform == 'emscripten'",

bigdata_smart_batching-0.1.0/.env

@@ -1 +0,0 @@
-BIGDATA_API_KEY=bd_v1_UPdNGgq44wBO9XR8THfiGvHd7Km9TCIvDmnT5AiatA9T56

bigdata_smart_batching-0.1.0/.python-version

@@ -1 +0,0 @@
-3.11

bigdata_smart_batching-0.1.0/.vscode/launch.json

@@ -1,19 +0,0 @@
-{
-    // Use IntelliSense to learn about possible attributes.
-    // Hover to view descriptions of existing attributes.
-    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
-    "version": "0.2.0",
-    "configurations": [
-        {
-            "name": "example",
-            "type": "debugpy",
-            "request": "launch",
-            "python": ".venv/bin/python",
-            "program": "examples/example.py",
-            "console": "integratedTerminal",
-            //"env": {"LOGNAME": "amartinezg",
-                    //"AWS_DS_S3_DEFAULT_BUCKET": "s3://com.ravenpack.dsteam.datalake"},
-            "justMyCode": false,
-        },
-    ]
-}