bigdata-smart-batching 1.4.1__tar.gz → 1.5.0__tar.gz

bigdata_smart_batching-1.4.1/README.md → bigdata_smart_batching-1.5.0/PKG-INFO

@@ -1,3 +1,19 @@
+Metadata-Version: 2.5
+Name: bigdata-smart-batching
+Version: 1.5.0
+Summary: High-performance semantic search with intelligent company grouping and parallel execution
+Project-URL: homepage, https://bigdata.com/api
+Author-email: "Bigdata.com" <support@ravenpack.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: <4.0,>=3.11
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: requests>=2.31.0
+Description-Content-Type: text/markdown
+
 # Smart Batching Search
 
 A high-performance semantic search system that reduces API queries by **67-99%** (varies by topic specificity) through intelligent company grouping and parallel execution.
@@ -267,6 +283,8 @@ Step 2: EXECUTION
 3. Optionally performs a **second pass**, removing already-retrieved entities from each basket and re-querying for the remainder.
 4. Enriches each chunk with `entity_ids` and `primary_entity_id`, and returns the raw documents.
 
+Every search request is sent with `response_detail: "extended"`, so each document includes its filer in `reporting_entity` (a list of entity IDs). The filer is used as a chunk's `entity_ids` when the chunk has no detections, and the second pass counts it as an entity already retrieved.
+
 ## API Reference
 
 ### `plan_search()`
@@ -308,7 +326,8 @@ Step 2: EXECUTION
 ### Helper Functions
 
 - `deduplicate_documents(documents)` -- merge duplicate documents by `id`, combining their chunks.
-- `convert_to_dataframe(raw_results)` -- convert documents to a DataFrame with one row per chunk.
+- `convert_to_dataframe(raw_results)` -- convert documents to a DataFrame with one row per chunk. The `reporting_entities` column holds the document's filer ids.
+- `get_reporting_entities(document)` -- return the filer ids of a search document (`reporting_entity`; the legacy `reporting_entities` / `reportingEntities` keys are also accepted).
 - `resolve_universe(universe)` -- load entity IDs from a CSV path or validate a `list[str]`.
 - `load_universe_from_csv(csv_path)` -- load entity IDs from a CSV only.
 - `save_plan(plan, path)` / `load_plan(path)` -- persist plans as JSON.

bigdata_smart_batching-1.4.1/PKG-INFO → bigdata_smart_batching-1.5.0/README.md

@@ -1,19 +1,3 @@
-Metadata-Version: 2.4
-Name: bigdata-smart-batching
-Version: 1.4.1
-Summary: High-performance semantic search with intelligent company grouping and parallel execution
-Project-URL: homepage, https://bigdata.com/api
-Author-email: "Bigdata.com" <support@ravenpack.com>
-License: MIT
-License-File: LICENSE
-Requires-Python: <4.0,>=3.11
-Requires-Dist: matplotlib>=3.7
-Requires-Dist: numpy>=1.24
-Requires-Dist: pandas>=2.0
-Requires-Dist: python-dotenv>=1.2.2
-Requires-Dist: requests>=2.31.0
-Description-Content-Type: text/markdown
-
 # Smart Batching Search
 
 A high-performance semantic search system that reduces API queries by **67-99%** (varies by topic specificity) through intelligent company grouping and parallel execution.
@@ -283,6 +267,8 @@ Step 2: EXECUTION
 3. Optionally performs a **second pass**, removing already-retrieved entities from each basket and re-querying for the remainder.
 4. Enriches each chunk with `entity_ids` and `primary_entity_id`, and returns the raw documents.
 
+Every search request is sent with `response_detail: "extended"`, so each document includes its filer in `reporting_entity` (a list of entity IDs). The filer is used as a chunk's `entity_ids` when the chunk has no detections, and the second pass counts it as an entity already retrieved.
+
 ## API Reference
 
 ### `plan_search()`
@@ -324,7 +310,8 @@ Step 2: EXECUTION
 ### Helper Functions
 
 - `deduplicate_documents(documents)` -- merge duplicate documents by `id`, combining their chunks.
-- `convert_to_dataframe(raw_results)` -- convert documents to a DataFrame with one row per chunk.
+- `convert_to_dataframe(raw_results)` -- convert documents to a DataFrame with one row per chunk. The `reporting_entities` column holds the document's filer ids.
+- `get_reporting_entities(document)` -- return the filer ids of a search document (`reporting_entity`; the legacy `reporting_entities` / `reportingEntities` keys are also accepted).
 - `resolve_universe(universe)` -- load entity IDs from a CSV path or validate a `list[str]`.
 - `load_universe_from_csv(csv_path)` -- load entity IDs from a CSV only.
 - `save_plan(plan, path)` / `load_plan(path)` -- persist plans as JSON.

{bigdata_smart_batching-1.4.1 → bigdata_smart_batching-1.5.0}/examples/example.py

@@ -70,28 +70,28 @@ def main() -> None:
         print(f"% overlapping chunks: {(len(merged[merged._merge!='both'])/2)/len(merged)*100:.2f}%")
         print('a')        
         
-
-    plan = load_plan("pricing_power_evidence.json")
-    # Set the query.ranking.params.content_diversification to False
-    for basket in plan['baskets']:
-        basket['query']['ranking_params']['content_diversification']['enabled'] = False
-        #Remove sentiment filter, do not set it to None, remove the key instead
-        basket['query']['filters'].pop('sentiment', None)
-    
-    # Keep only basket basket_6_high_20250924_20251022
-    plan['baskets'] = [basket for basket in plan['baskets'] if basket['basket_id'] == "basket_6_high_20250924_20251022"]
-    results_raw = execute_search(
-        search_plan=plan,
-        chunk_percentage=1,
-        requests_per_minute=450,
-        basket_filtered_entities=True,
-    )
+    if False:
+        plan = load_plan("pricing_power_evidence.json")
+        # Set the query.ranking.params.content_diversification to False
+        for basket in plan['baskets']:
+            basket['query']['ranking_params']['content_diversification']['enabled'] = False
+            #Remove sentiment filter, do not set it to None, remove the key instead
+            basket['query']['filters'].pop('sentiment', None)
+        
+        # Keep only basket basket_6_high_20250924_20251022
+        plan['baskets'] = [basket for basket in plan['baskets'] if basket['basket_id'] == "basket_6_high_20250924_20251022"]
+        results_raw = execute_search(
+            search_plan=plan,
+            chunk_percentage=1,
+            requests_per_minute=450,
+            basket_filtered_entities=True,
+        )
 
     # Store the results in a file but in parquet format
     import json
     import pandas as pd
-    df = pd.DataFrame(results_raw)
-    df.to_parquet(f"results_raw_{time.time()}_100pct_basket_6_without_sentiment.parquet")
+    #df = pd.DataFrame(results_raw)
+    #df.to_parquet(f"results_raw_{time.time()}_100pct_basket_6_without_sentiment.parquet")
 
     plans = []
     results_dfs = []
@@ -99,7 +99,7 @@ def main() -> None:
         # Step 1: Plan the search
         # Try to get plan from file
         plan_t0 = time.perf_counter()
-        if False or Path("plan.json").is_file():
+        if False: #or Path("plan.json").is_file():
             plan = load_plan("plan.json")
         else:
             plan = plan_search(
@@ -136,6 +136,8 @@ def main() -> None:
                 chunk_percentage=0.1,
                 requests_per_minute=450,
             )
+            df = convert_to_dataframe(results_raw)
+            df.to_csv('example_out.csv',index=False)
             search_elapsed_s = time.perf_counter() - search_t0
             print(f"Search execution time: {search_elapsed_s:.3f}s")
 

{bigdata_smart_batching-1.4.1 → bigdata_smart_batching-1.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "bigdata-smart-batching"
-version = "1.4.1"
+version = "1.5.0"
 description = "High-performance semantic search with intelligent company grouping and parallel execution"
 readme = "README.md"
 authors = [{ name = "Bigdata.com", email = "support@ravenpack.com" }]
@@ -29,6 +29,14 @@ build-backend = "hatchling.build"
 [tool.hatch.build.targets.wheel]
 packages = ["src/bigdata_smart_batching"]
 
+# Explicit allow-list: never ship untracked local files (.env, notes, scratch examples).
+# Hatchling always adds pyproject.toml, README.md, LICENSE and .gitignore.
+[tool.hatch.build.targets.sdist]
+include = [
+  "src/",
+  "examples/example.py",
+]
+
 [project.urls]
 homepage = "https://bigdata.com/api"
 

{bigdata_smart_batching-1.4.1 → bigdata_smart_batching-1.5.0}/src/bigdata_smart_batching/__init__.py

@@ -24,6 +24,7 @@ from .search_function import (
     build_volume_payload,
     deduplicate_documents,
     execute_search,
+    get_reporting_entities,
     load_plan,
     load_universe_from_csv,
     plan_search,
@@ -51,6 +52,7 @@ __all__ = [
     "convert_to_dataframe",
     "deduplicate_documents",
     "execute_search",
+    "get_reporting_entities",
     "load_plan",
     "load_universe_from_csv",
     "plan_search",

{bigdata_smart_batching-1.4.1 → bigdata_smart_batching-1.5.0}/src/bigdata_smart_batching/output_converter.py

@@ -1,5 +1,7 @@
 import pandas as pd
 
+from .search_function import get_reporting_entities
+
 
 def convert_to_dataframe(raw_results: list[dict]) -> pd.DataFrame:
     """
@@ -25,7 +27,7 @@ def convert_to_dataframe(raw_results: list[dict]) -> pd.DataFrame:
         headline = raw_doc.get("headline", "")
         timestamp = raw_doc.get("timestamp", "")
         url = raw_doc.get("url", "")
-        reporting_entities = raw_doc.get("reporting_entities", [])
+        reporting_entities = get_reporting_entities(raw_doc)
 
         # Extract source info
         raw_source = raw_doc.get("source", {})

{bigdata_smart_batching-1.4.1 → bigdata_smart_batching-1.5.0}/src/bigdata_smart_batching/search_function.py

@@ -42,6 +42,8 @@ logger = logging.getLogger(__name__)
 DEFAULT_API_BASE_URL = os.getenv("BIGDATA_API_BASE_URL", "https://api.bigdata.com")
 SEARCH_ENDPOINT = "/v1/search"
 COMENTION_ENDPOINT = "/v1/search/co-mentions/entities"
+# Only "extended" returns the filer (``reporting_entity``). Bigdata API change of 2026-10-22.
+SEARCH_RESPONSE_DETAIL = "extended"
 MAX_ENTITIES_IN_ANY_OF = 500
 MAX_CHUNKS_PER_BASKET = 1000
 DEFAULT_REQUESTS_PER_MINUTE = int(os.getenv("BIGDATA_REQUESTS_PER_MINUTE", "450"))
@@ -1114,6 +1116,22 @@ def exponential_backoff(
     return max(0.0, min(delay, cap))
 
 
+def get_reporting_entities(document: dict) -> list[str]:
+    """Filer ids of a /v1/search document.
+
+    The API returns them as ``reporting_entity`` (a list) only with
+    ``response_detail="extended"`` (since 2026-10-22). The legacy plural keys
+    ``reporting_entities`` / ``reportingEntities`` are still accepted.
+    """
+    value = (
+        document.get("reporting_entity")
+        or document.get("reporting_entities")
+        or document.get("reportingEntities")
+        or []
+    )
+    return [e for e in (value if isinstance(value, list) else [value]) if e]
+
+
 def make_search_request(
     query: dict,
     api_key: str,
@@ -1125,6 +1143,9 @@ def make_search_request(
     """
     Make a search API request with rate limiting and retry logic.
 
+    Always sends ``response_detail="extended"`` at the top level of the body
+    (next to ``query``, not inside it) so documents include ``reporting_entity``.
+
     Args:
         query: Complete query structure
         api_key: API key
@@ -1139,7 +1160,7 @@ def make_search_request(
     url = f"{api_base_url}{SEARCH_ENDPOINT}"
     headers = {"X-API-KEY": api_key, "Content-Type": "application/json"}
 
-    payload = {"query": query}
+    payload = {"query": query, "response_detail": SEARCH_RESPONSE_DETAIL}
 
     retry_count = 0
     while retry_count < max_retries:
@@ -1368,12 +1389,8 @@ def execute_search(
                 for document in documents:
                     document_chunks = document.get("chunks", [])
                     total_chunks += len(document_chunks)
-                    # Get document-level entity information (try both snake_case and camelCase)
-                    reporting_entities = (
-                        document.get("reporting_entities")
-                        or document.get("reportingEntities")
-                        or []
-                    )
+                    # Document-level filer ids (reporting_entity)
+                    reporting_entities = get_reporting_entities(document)
 
                     # Debug: Log first document and chunk structure
                     if not first_doc_logged and document_chunks:
@@ -1418,12 +1435,9 @@ def execute_search(
                                     if entity_id:
                                         entity_ids.append(entity_id)
 
-                        # Try document-level reporting_entities
+                        # Fall back to the document's filer (reporting_entity)
                         if not entity_ids and reporting_entities:
-                            if isinstance(reporting_entities, list):
-                                entity_ids = [e for e in reporting_entities if e]
-                            else:
-                                entity_ids = [reporting_entities] if reporting_entities else []
+                            entity_ids = list(reporting_entities)
 
                         # Store entity information (always set, even if empty)
                         if basket_filtered_entities:
@@ -1503,7 +1517,7 @@ def execute_search(
                 for chunk in i.get("chunks", [])
                 for entity in chunk.get("entity_ids", [])
             ]
-            + [entity for i in all_documents for entity in i.get("reporting_entities", [])]
+            + [entity for i in all_documents for entity in get_reporting_entities(i)]
             + [
                 chunk.get("primary_entity_id")
                 for i in all_documents

bigdata_smart_batching-1.4.1/CHANGELOG.md

@@ -1,165 +0,0 @@
-# Changelog
-
-All notable changes to this project are documented in this file.
-
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-
-
-## [1.4.1] - 2026-06-02
-
-### Fixed
-
-- `SmartBatchingPlanner._post_with_retry`: retry with exponential backoff on any non-200 HTTP status or transient network error. Previously only `429` and `403` were retried; other statuses (e.g. `500`) failed immediately.
-
-## [1.4.0] - 2026-05-20
-
-### Added
-
-- `execute_search(..., basket_filtered_entities=True)`: when enabled, each chunk's `entity_ids` list is filtered to only include entity IDs present in the basket's `query.filters.entity.any_of` list, removing co-mentioned entities that were not explicitly targeted by the basket. Entity IDs are always deduplicated.
-
-## [1.3.1] - 2026-05-19
-
-### Fixed
-
-- Now we always retry requests, before we were only retrying on 429 and 403 errors.
-
-## [1.3.0] - 2026-05-12
-
-### Added
-
-- Document-level **source** filters: `INCLUDE` / `EXCLUDE` with a non-empty list of source ID strings, validated with `validate_source_filter`, modeled as `SourceFilter` / `SourceFilterInput` with `SourceMode` (type alias of `CategoryMode`). Serialized to `query.filters.source` on co-mention, volume, and search payloads. `plan_search(..., source=...)` and `source_filter` plumbed through `SmartBatchingPlanner` co-mention and volume flows. Payload builders `build_comention_payload`, `build_volume_payload`, and `build_search_query` accept optional `source_filter`. Types and validator re-exported from the package root.
-
-## [1.2.1] - 2026-05-12
-
-### Fixed
-
-- Modified README.md file to add latest changes
-
-## [1.2.0] - 2026-05-12
-
-### Added
-
-- Document-level **sentiment** filters: inclusive `min` / `max` range(s), validated with `validate_sentiment_range`, modeled as `SentimentRange` / `SentimentRangeInput`, serialized to `query.filters.sentiment.ranges` on co-mention, volume, and search payloads. `plan_search(..., sentiment=...)` and `sentiment_filter` plumbed through `SmartBatchingPlanner` co-mention and volume flows. Types and validator re-exported from the package root.
-- **Planning throughput controls**: `plan_search(..., requests_per_minute=..., max_workers=...)` and `SmartBatchingPlanner(..., requests_per_minute=..., max_workers=..., rate_limiter=..., concurrency_limiter=...)` so all planner HTTP work shares one sliding-window rate limiter and one concurrency semaphore across internal thread pools.
-- **`exponential_backoff`** helper in `search_function` (used by retry paths; capped via `MAX_RETRY_BACKOFF_SECONDS`).
-
-### Changed
-
-- **Retries:** co-mention / volume POST retries now use **shared exponential backoff** for `429`, `403`, timeouts, and other failures, instead of parsing `Retry-After` for `429` and separate ad-hoc `403` handling (constant renamed/clamped to `MAX_RETRY_BACKOFF_SECONDS`).
-- **Volume time-series merge:** when aggregating batch results per date, a **`sentiment`** value on each point is the arithmetic mean of batch sentiments for that date (batches without sentiment for that date are omitted from the mean).
-- **`SlidingWindowRateLimiter`:** window accounting aligned with **ceil-based** `max_per_window` expectations (tests updated for low rates and 450 rpm-style cases).
-
-
-
-## [1.1.0] - 2026-04-20
-
-### Added
-
-- `convert_to_dataframe` now includes a `detections` column: for each chunk row,
-  entity detections are taken from `chunk["detections"]` (entries with
-  `type == "entity"`); rows without chunks use an empty list.
-
-## [1.0.2] - 2026-04-17
-
-### Fixed
-
-- Corrected README file
-
-## [1.0.1] - 2026-04-17
-
-### Fixed
-
-- Republish on PyPI after an erroneous publish of v1.0.0; package contents match the intended release.
-
-
-## [1.0.0] - 2026-04-17
-
-First stable release. Redesigns the batching pipeline so every API request stays
-within service limits while keeping plans deterministic.
-
-### Added
-
-- `create_entity_groups` step: sorts companies by chunks desc and greedy-packs
-  under `MAX_CHUNKS_PER_BASKET` and `MAX_ENTITIES_IN_ANY_OF`, honoring
-  `min_entities_per_basket`, with tail-group merge when it fits the entity
-  limit.
-- Automatic splitting of co-mention volume queries into sub-ranges of up to 365
-  days, with per-company chunks summed across sub-ranges in both three-pass and
-  iterative modes. Iterative mode now fans out `(batch, sub-range)` work items
-  to the thread pool and derives `very_low` from companies never seen in any
-  sub-range.
-- Parallel execution of `get_volume_timeseries` via `ThreadPoolExecutor`.
-- `volume_bucket` metadata on every group/basket via `classify_volume_bucket`
-  and `majority_volume_bucket` (ties broken by `VOLUME_BUCKET_PRIORITY`).
-- Centralized `VOLUME_BUCKETS` in `smart_batching_config` with explicit
-  `[lower, upper)` semantics and new `VOLUME_BUCKET_PRIORITY` constant.
-- INCLUDE/EXCLUDE category filtering for planning and all Bigdata query paths
-  (co-mention, volume, search), validated against a fixed set of allowed
-  category strings. `CategoryFilter`, `CategoryMode`, and validators/builders
-  are exported from the package.
-- Shared payload builders `build_comention_payload`, `build_volume_payload`,
-  and `build_search_query`, which `SmartBatchingPlanner` now routes through
-  instead of constructing payloads inline.
-- `content_diversification` option in `ranking_params`, exposed on
-  `plan_search`.
-- Universe input accepts either a CSV path or an inline list of entity IDs
-  (`universe: str | list[str]`). Added `resolve_universe()` and `UniverseInput`,
-  both exported from the package.
-- `Makefile` test targets and coverage settings in `pyproject.toml`.
-
-### Changed
-
-- **Breaking:** `SmartBatchingPlanner` public API redesigned. `create_baskets`
-  is replaced by `create_entity_groups` (new signature:
-  `min_entities_per_group` instead of `max_chunks`/`min_entities_per_basket`);
-  returned dicts no longer include `basket_id` or `volume_range` and expose
-  `volume_bucket` instead.
-- **Breaking:** Removed helpers `group_by_volume`,
-  `consolidate_period_groups`, `sub_period_volumes_from_series`, and
-  `get_volume_timeseries_per_company`.
-- **Breaking:** `very_low` basket IDs are now
-  `basket_{n}_very_low_{start}_{end}` instead of `very_low_basket_{n}`;
-  downstream parsers must be updated.
-- **Breaking:** `VOLUME_BUCKETS` ranges are now enforced as `[lower, upper)`.
-- **Breaking:** `plan_search(..., universe_csv_path=)` renamed to
-  `universe=` (accepts `str | list[str]`). `SmartBatchingPlanner.load_universe`
-  and `plan_all_periods` now take a required `universe` argument and delegate
-  to `resolve_universe`; `UNIVERSE_CSV_PATH` and duplicate CSV parsing removed.
-- **Breaking:** `plan_search` / `plan_all_periods` now accept an optional
-  `text` parameter and parameter order has been reordered.
-- Entity filters now use `search_in=BODY` for co-mention and volume payloads;
-  `ranking_params` dropped from the volume query.
-- Omit `text` from co-mention, volume, and search payloads when topic/text is
-  `None`.
-- Planned baskets are sorted by `basket_id` for stable ordering.
-- Architecture documentation updated to reflect the new `plan_search` flow
-  (sub-range splitting, entity groups, pre-fetch only for heavy groups, one
-  basket per `(group, sub-period)`) and a new high-level flow diagram.
-- `load_universe_from_csv` rejects empty CSV paths.
-
-### Removed
-
-- Obsolete `VOLUME_BUCKETS` structure test.
-
-## [0.2.0] - 2026-04-14
-
-### Changed
-
-- **Breaking:** `plan_search` now returns `chunk_upper_bound_estimate` instead of `total_expected_chunks`. Callers, saved plans, and any code reading the plan dict must use the new key. `execute_search` reads `chunk_upper_bound_estimate` when deriving sampling from the plan.
-- Terminology and docs describe this value as a chunk upper bound estimate (module docstring, README, architecture diagram, and example output).
-- README installation: document PyPI consumption (`pip install bigdata-smart-batching`, `uv add bigdata-smart-batching`) and keep a separate **Development** section for cloning the repo and `uv sync`.
-
-### Fixed
-
-- Align `uv.lock` `requires-python` with `pyproject.toml` (`>=3.11,<4.0`).
-
-## [0.1.0] - 2026-04-14
-
-### Added
-
-- Initial release of **bigdata-smart-batching**: high-performance semantic search with intelligent company grouping and parallel execution.
-- Planning API (`plan_search`) to organize searches via smart batching and report expected chunk counts.
-- Execution API (`execute_search`) with rate-limited parallel requests and proportional sampling to preserve result distribution.
-- Utilities: document deduplication, DataFrame conversion, and configurable search/batching behavior via `smart_batching` and related modules.
-- Python 3.11+ support; dependencies include `requests`, `pandas`, `numpy`, `matplotlib`, and `python-dotenv`.

bigdata_smart_batching-1.4.1/Makefile

@@ -1,19 +0,0 @@
-.PHONY: lint lint-check format format-check test test-cov
-
-lint:
-	uvx ruff check --fix src/ tests/
-
-lint-check:
-	uvx ruff check --extend-select I src/ tests/
-
-format:
-	uvx ruff format src/ tests/
-
-format-check:
-	uvx ruff format --check src/ tests/
-
-test:
-	uv run pytest
-
-test-cov:
-	uv run pytest --cov --cov-report=term-missing --cov-report=html