pyproc 0.3.1__tar.gz → 0.3.2__tar.gz

{pyproc-0.3.1 → pyproc-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyproc
-Version: 0.3.1
+Version: 0.3.2
 Summary: MCP tools for real-time Indonesian public procurement data from SPSE/Inaproc
 Project-URL: Homepage, https://github.com/wakataw/pyproc
 Project-URL: Issues, https://github.com/wakataw/pyproc/issues
@@ -36,7 +36,7 @@ Description-Content-Type: text/markdown
 # PyProc MCP
 
 <p align="center">
-  <img src="docs/assets/logo.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/logo.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="200">
 </p>
@@ -46,7 +46,7 @@ Description-Content-Type: text/markdown
 </p>
 
 <p align="center">
-  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.1-blue" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.2-blue" alt="PyPI version"></a>
   <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/python-≥3.9-yellow" alt="Python ≥3.9"></a>
   <a href="https://github.com/wakataw/pyproc/actions/workflows/test.yml?query=branch%3Amaster"><img src="https://github.com/wakataw/pyproc/actions/workflows/test.yml/badge.svg?branch=master" alt="Test Status"></a>
   <a href="https://github.com/wakataw/pyproc/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT"></a>
@@ -60,7 +60,7 @@ PyProc MCP turns public **SPSE/Inaproc** procurement data into **MCP tools** tha
 ## Why PyProc MCP?
 
 <p align="center">
-  <img src="docs/assets/pyproc-mcp-banner.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/pyproc-mcp-banner.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="800">
 </p>

{pyproc-0.3.1 → pyproc-0.3.2}/README.md

@@ -1,7 +1,7 @@
 # PyProc MCP
 
 <p align="center">
-  <img src="docs/assets/logo.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/logo.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="200">
 </p>
@@ -11,7 +11,7 @@
 </p>
 
 <p align="center">
-  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.1-blue" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.2-blue" alt="PyPI version"></a>
   <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/python-≥3.9-yellow" alt="Python ≥3.9"></a>
   <a href="https://github.com/wakataw/pyproc/actions/workflows/test.yml?query=branch%3Amaster"><img src="https://github.com/wakataw/pyproc/actions/workflows/test.yml/badge.svg?branch=master" alt="Test Status"></a>
   <a href="https://github.com/wakataw/pyproc/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT"></a>
@@ -25,7 +25,7 @@ PyProc MCP turns public **SPSE/Inaproc** procurement data into **MCP tools** tha
 ## Why PyProc MCP?
 
 <p align="center">
-  <img src="docs/assets/pyproc-mcp-banner.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/pyproc-mcp-banner.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="800">
 </p>

pyproc-0.3.2/plan/SEARCH_INDEX_LIMIT_REMOVAL_AND_PROGRESS.md

@@ -0,0 +1,87 @@
+# Plan: Remove max_packages cap, default to all data, paginate at 100/req, add progress
+
+## Context
+
+The `create_procurement_search_index` MCP tool has `MAX_INDEX_PACKAGES = 500` capping the **total** number of packages that can be downloaded. Remove this cap so users can download all available data for a host/year. Default behavior becomes "download all available data." Per-request batch size stays fixed at 100 (internal, not exposed). Progress is reported as a running count via stderr logging — no percentage since we don't rely on `recordsTotal`/`recordsFiltered`.
+
+## Parameter design
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `max_packages` | int | `0` | `0` = download all (paginate until exhausted). Positive value caps the total. No upper bound. |
+| Batch size | int (internal) | `100` | Fixed `CHUNK_SIZE`, not a user-facing parameter. Matches CLI `--chunk-size` default. |
+
+## Changes
+
+### 1. `pyproc/mcp/schemas.py`
+
+- Delete `MAX_INDEX_PACKAGES = 500` (line 33)
+- Change `DEFAULT_INDEX_MAX_PACKAGES = 100` → `DEFAULT_INDEX_MAX_PACKAGES = 0` (0 = all)
+- Line 410: `max(1, min(max_packages, MAX_INDEX_PACKAGES))` → `max(0, max_packages)` (0 means unlimited)
+- `CREATE_SEARCH_INDEX_SCHEMA` (lines 952-960):
+  - Remove `"maximum": MAX_INDEX_PACKAGES`
+  - Update description to `"Maximum packages to download. 0 = all available. Default 0."`
+
+### 2. `pyproc/mcp/search_index.py`
+
+- Add `import logging` → `logger = logging.getLogger(__name__)`
+- Add `CHUNK_SIZE = 100`
+
+Replace the single-request block (lines 127-139) with a pagination loop:
+
+```python
+CHUNK_SIZE = 100
+all_rows = []
+start = 0
+limit = max_packages if max_packages > 0 else float('inf')
+
+while len(all_rows) < limit:
+    remaining = limit - len(all_rows)
+    req_length = min(CHUNK_SIZE, int(remaining))
+    chunk_kwargs = {**kwargs, "start": start, "length": req_length}
+    chunk = search_method(**chunk_kwargs)
+    if not chunk:
+        break
+    all_rows.extend(chunk)
+    logger.info("Scrolled: %d rows from %s (TA %s)", len(all_rows), lpse_host, tahun_anggaran)
+    start += len(chunk)
+    if len(chunk) < req_length:
+        break  # partial page → end of data
+
+to_process = len(all_rows)
+logger.info("Will index %d packages from %s (%s, %s)", to_process, lpse_host, package_type, tahun_anggaran)
+```
+
+Progress logging in the detail loop:
+
+```python
+for idx, row in enumerate(all_rows, start=1):
+    title = _package_title(row)
+    logger.info("[%d/%d] Indexing: %s", idx, to_process, title)
+    # ... existing fetch + index + rate_limit ...
+```
+
+### 3. `pyproc/mcp/tools.py`
+
+- Update tool description string to mention "0 = all" default
+- No code changes needed — progress is handled in `search_index.py`'s logger
+
+### 4. Tests
+
+**`tests/test_mcp_schemas.py`** (`TestValidateSearchIndexParams`):
+- Default `max_packages` → 0 (all)
+- `max_packages=5000` → 5000 (not clamped)
+- `max_packages=-1` → 0 (floor at 0)
+
+**`tests/test_mcp_search_index.py`** (`TestSearchIndex`):
+- Pagination: mock returns [100, 100, 50] → all 250 collected
+- Stops on empty chunk
+- Stops on partial chunk (< requested length)
+- Respects max_packages limit when positive
+
+## Verification
+
+```bash
+python -m pytest tests/test_mcp_schemas.py::TestValidateSearchIndexParams tests/test_mcp_search_index.py -v
+python -m pytest tests/ --ignore=tests/test_lpse.py --ignore=tests/test_downloader.py -v
+```

{pyproc-0.3.1 → pyproc-0.3.2}/pyproc/__init__.py

@@ -1,6 +1,6 @@
 from .lpse import Lpse, JenisPengadaan, TipeSwakelola, By, KontrakStatus
 
-__version__ = '0.3.1'
+__version__ = '0.3.2'
 __author__ = 'Agung Pratama'
 __all__ = [
     'Lpse',

{pyproc-0.3.1 → pyproc-0.3.2}/pyproc/mcp/schemas.py

@@ -29,8 +29,7 @@ VALID_ORDER_BY = {
 }
 VALID_ORDER_DIR = {"asc", "desc"}
 VALID_KONTRAK_STATUS = {0, 1, 2}
-DEFAULT_INDEX_MAX_PACKAGES = 100
-MAX_INDEX_PACKAGES = 500
+DEFAULT_INDEX_MAX_PACKAGES = 0
 DEFAULT_INDEX_SEARCH_LIMIT = 20
 MAX_INDEX_SEARCH_LIMIT = 100
 MAX_BULK_DETAIL_PACKAGE_IDS = 20
@@ -407,7 +406,7 @@ def validate_search_index_create_params(params: dict) -> dict:
         "tahun_anggaran": validate_tahun_anggaran(params.get("tahun_anggaran")),
         "kategori": validate_kategori(params.get("kategori")),
         "keyword_seed": str(params.get("keyword_seed") or "").strip() or None,
-        "max_packages": max(1, min(max_packages, MAX_INDEX_PACKAGES)),
+        "max_packages": max(0, max_packages),
         "confirm_download": True,
     }
 
@@ -952,17 +951,17 @@ CREATE_SEARCH_INDEX_SCHEMA = {
         "max_packages": {
             "type": "integer",
             "description": (
-                "Maximum packages to download and index. Default "
-                f"{DEFAULT_INDEX_MAX_PACKAGES}, maximum {MAX_INDEX_PACKAGES}."
+                "Maximum packages to download and index. 0 = all available. "
+                f"Default {DEFAULT_INDEX_MAX_PACKAGES}."
             ),
             "default": DEFAULT_INDEX_MAX_PACKAGES,
-            "maximum": MAX_INDEX_PACKAGES,
         },
         "confirm_download": {
             "type": "boolean",
             "description": (
-                "Must be true. This confirms the user agreed to download "
-                "package details into a local disposable full-text index."
+                "Must be true. Confirms user consent to download package details "
+                "into a local disposable full-text index. May involve many "
+                "SPSE requests when max_packages is 0 or large."
             ),
         },
     },

{pyproc-0.3.1 → pyproc-0.3.2}/pyproc/mcp/search_index.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import logging
 import os
 import shutil
 import sqlite3
@@ -12,6 +13,10 @@ from uuid import uuid4
 
 from pyproc import Lpse, JenisPengadaan
 
+logger = logging.getLogger(__name__)
+
+CHUNK_SIZE = 100
+
 PACKAGE_METHODS = {
     "tender": ("get_paket_tender", "detil_paket_tender"),
     "non_tender": ("get_paket_non_tender", "detil_paket_non_tender"),
@@ -97,11 +102,18 @@ def create_procurement_search_index(
     tahun_anggaran: int | None = None,
     kategori: str | None = None,
     keyword_seed: str | None = None,
-    max_packages: int = 100,
+    max_packages: int = 0,
     timeout: int = 30,
     rate_limit_callback=None,
+    progress_callback=None,
 ) -> dict:
-    """Download a bounded package set and index details into local SQLite FTS."""
+    """Download a bounded package set and index details into local SQLite FTS.
+
+    Args:
+        progress_callback: Optional callable(step, current, total, message)
+            called at progress milestones.  step is one of 'scroll',
+            'index_start', 'index_package', 'complete'.
+    """
     index_id = f"{lpse_host}-{package_type}-{int(time.time())}-{uuid4().hex[:8]}"
     path = _index_path(index_id)
     metadata = {
@@ -125,22 +137,66 @@ def create_procurement_search_index(
             if rate_limit_callback:
                 rate_limit_callback()
             search_method = getattr(lpse, PACKAGE_METHODS[package_type][0])
-            kwargs = {
-                "start": 0,
-                "length": max_packages,
+            base_kwargs = {
                 "data_only": True,
                 "search_keyword": keyword_seed,
                 "tahun": tahun_anggaran,
             }
             if package_type != "swakelola":
-                kwargs["kategori"] = kategori_enum
-            rows = search_method(**kwargs)
-
-            for row in rows[:max_packages]:
+                base_kwargs["kategori"] = kategori_enum
+
+            # Paginate: scroll until max_packages reached or SPSE exhausted
+            all_rows = []
+            start = 0
+            unlimited = max_packages <= 0
+            limit = max_packages if not unlimited else float('inf')
+
+            while len(all_rows) < limit:
+                if unlimited:
+                    req_length = CHUNK_SIZE
+                else:
+                    req_length = min(CHUNK_SIZE, max(1, max_packages - len(all_rows)))
+                chunk_kwargs = {**base_kwargs, "start": start, "length": req_length}
+                chunk = search_method(**chunk_kwargs)
+                if not chunk:
+                    break
+                all_rows.extend(chunk)
+                logger.info(
+                    "Scrolled: %d rows from %s (TA %s)",
+                    len(all_rows), lpse_host, tahun_anggaran,
+                )
+                if progress_callback:
+                    progress_callback(
+                        "scroll", len(all_rows), None,
+                        f"Scrolled {len(all_rows)} packages from {lpse_host}",
+                    )
+                start += len(chunk)
+                if len(chunk) < req_length:
+                    break  # partial page -> end of data
+
+            # Respect max_packages cap (only matters if SPSE returned more than requested)
+            to_process = len(all_rows) if unlimited else min(len(all_rows), max_packages)
+            logger.info(
+                "Will index %d packages from %s (%s, %s)",
+                to_process, lpse_host, package_type, tahun_anggaran,
+            )
+            if progress_callback:
+                progress_callback(
+                    "index_start", 0, to_process,
+                    f"Indexing {to_process} packages from {lpse_host}",
+                )
+
+            for idx, row in enumerate(all_rows[:to_process], start=1):
                 id_paket = _package_id(row)
                 if not id_paket:
                     continue
                 title = _package_title(row)
+                logger.info("[%d/%d] Indexing: %s", idx, to_process, title)
+                if progress_callback:
+                    progress_callback(
+                        "index_package", idx, to_process,
+                        f"Indexing package {idx}/{to_process}: {title}",
+                    )
                 try:
                     if rate_limit_callback:
                         rate_limit_callback()
@@ -160,11 +216,24 @@ def create_procurement_search_index(
                     indexed += 1
                 except Exception:
                     failed += 1
+                    logger.warning(
+                        "Failed to index package %s (%s)", id_paket, title,
+                    )
                     continue
     finally:
         db.commit()
         db.close()
 
+    logger.info(
+        "Index complete: %d indexed, %d failed from %s (%s, %s)",
+        indexed, failed, lpse_host, package_type, tahun_anggaran,
+    )
+    if progress_callback:
+        progress_callback(
+            "complete", indexed + failed, indexed + failed,
+            f"Index complete: {indexed} indexed, {failed} failed",
+        )
+
     return {
         **metadata,
         "path": str(path),

{pyproc-0.3.1 → pyproc-0.3.2}/pyproc/mcp/tools.py

@@ -6,8 +6,10 @@ normalizes output, and returns MCP TextContent responses.
 This module auto-registers all tools with the server on import.
 """
 
+import anyio
 import json
 import logging
+import queue
 import time
 
 from mcp import types as mcp_types
@@ -42,7 +44,7 @@ from pyproc.mcp.schemas import (
     LIST_SEARCH_INDEXES_SCHEMA,
     DELETE_SEARCH_INDEX_SCHEMA,
 )
-from pyproc.mcp.server import register_tool, TIMEOUT, RATE_LIMIT_DELAY
+from pyproc.mcp.server import register_tool, server, TIMEOUT, RATE_LIMIT_DELAY
 from pyproc.mcp.hosts import (
     HostMetadataError,
     search_lpse_hosts,
@@ -651,7 +653,10 @@ async def handle_validate_lpse_host(
 async def handle_create_procurement_search_index(
     name: str, arguments: dict
 ) -> list[mcp_types.TextContent]:
-    """Create a bounded local SQLite FTS index from public procurement data."""
+    """Create a bounded local SQLite FTS index from public procurement data.
+
+    Sends MCP progress notifications when the client provides a progressToken.
+    """
     try:
         params = validate_search_index_create_params(arguments)
     except ValueError as exc:
@@ -665,12 +670,87 @@ async def handle_create_procurement_search_index(
     )
     params = dict(params)
     params.pop("confirm_download", None)
-    output = create_procurement_search_index(
-        timeout=TIMEOUT,
-        rate_limit_callback=_rate_limit,
-        **params,
-    )
-    return _make_json_response(output)
+
+    # ── check for MCP progress token ──────────────────────────────────────
+    progress_token = None
+    session = None
+    try:
+        ctx = server.request_context
+        if ctx.meta is not None:
+            progress_token = ctx.meta.progressToken
+            session = ctx.session
+    except LookupError:
+        pass  # Not inside an MCP request (e.g., direct test call)
+
+    if progress_token is None or session is None:
+        # No progress support — call directly (preserves existing behaviour)
+        output = create_procurement_search_index(
+            timeout=TIMEOUT,
+            rate_limit_callback=_rate_limit,
+            **params,
+        )
+        return _make_json_response(output)
+
+    # ── progress requested — run indexing in worker thread, send notifications ──
+    progress_queue: queue.Queue = queue.Queue(maxsize=100)
+    _SENTINEL = object()
+
+    def _progress_callback(step: str, current: int, total: int | None,
+                           message: str) -> None:
+        """Thread-safe: push progress data to queue (non-blocking)."""
+        try:
+            progress_queue.put_nowait((step, current, total, message))
+        except queue.Full:
+            pass  # Non-critical; discard if queue is overwhelmed
+
+    result_container: list[dict] = []
+
+    async def _run_index() -> None:
+        """Run the synchronous indexing function in a worker thread."""
+        def _sync_index():
+            return create_procurement_search_index(
+                timeout=TIMEOUT,
+                rate_limit_callback=_rate_limit,
+                progress_callback=_progress_callback,
+                **params,
+            )
+        result = await anyio.to_thread.run_sync(_sync_index)
+        result_container.append(result)
+        progress_queue.put(_SENTINEL)
+
+    async def _drain_progress() -> None:
+        """Drain progress events from the queue and send MCP notifications."""
+        sentinel_seen = False
+        while not sentinel_seen:
+            # Drain all currently queued items without blocking
+            while True:
+                try:
+                    item = progress_queue.get_nowait()
+                except queue.Empty:
+                    break
+                if item is _SENTINEL:
+                    sentinel_seen = True
+                    break
+                _step, current, total, message = item
+                try:
+                    await session.send_progress_notification(
+                        progress_token=progress_token,
+                        progress=float(current),
+                        total=float(total) if total is not None else None,
+                        message=message,
+                    )
+                except Exception:
+                    logger.debug(
+                        "Failed to send progress notification", exc_info=True,
+                    )
+            if not sentinel_seen:
+                await anyio.sleep(0.5)
+
+    async with anyio.create_task_group() as tg:
+        tg.start_soon(_drain_progress)
+        tg.start_soon(_run_index)
+
+    return _make_json_response(result_container[0])
 
 
 async def handle_search_procurement_index(
@@ -836,11 +916,14 @@ TOOL_DESCRIPTIONS = {
         "accessible. Useful before using a host with search or detail tools."
     ),
     "create_procurement_search_index": (
-        "Create a bounded local SQLite full-text search index by downloading "
-        "public package details for one LPSE host. Use only after the user "
-        "chooses a broader local full-text search. This can make many SPSE "
-        "requests, so keep max_packages small and prefer year/category/seed "
-        "filters. Does not call the CLI."
+        "Create a local SQLite full-text search index by downloading "
+        "public package details for one LPSE host. By default downloads all "
+        "available packages (max_packages=0). Set a positive max_packages to "
+        "limit scope. Prefer year/category/seed filters to narrow results. "
+        "This can make many SPSE requests. "
+        "Progress notifications are sent during index creation when the "
+        "client supports the notifications/progress protocol. "
+        "Does not call the CLI."
     ),
     "search_procurement_index": (
         "Search a previously created local SQLite full-text procurement index. "

{pyproc-0.3.1 → pyproc-0.3.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pyproc"
-version = "0.3.1"
+version = "0.3.2"
 authors = [
     { name="Agung Pratama", email="workwithagung@gmail.com"}
 ]

{pyproc-0.3.1 → pyproc-0.3.2}/tests/test_mcp_schemas.py

@@ -329,7 +329,7 @@ class TestValidateSearchIndexParams(unittest.TestCase):
         })
         self.assertEqual(result["lpse_host"], "kemenkeu")
         self.assertEqual(result["package_type"], "tender")
-        self.assertEqual(result["max_packages"], 100)
+        self.assertEqual(result["max_packages"], 0)
         self.assertTrue(result["confirm_download"])
 
     def test_create_index_params_full(self):
@@ -356,6 +356,33 @@ class TestValidateSearchIndexParams(unittest.TestCase):
                 "confirm_download": True,
             })
 
+    def test_create_index_max_packages_large(self):
+        """max_packages above old 500 cap is no longer clamped."""
+        result = validate_search_index_create_params({
+            "lpse_host": "kemenkeu",
+            "max_packages": "5000",
+            "confirm_download": True,
+        })
+        self.assertEqual(result["max_packages"], 5000)
+
+    def test_create_index_max_packages_zero_means_all(self):
+        """max_packages=0 means download all."""
+        result = validate_search_index_create_params({
+            "lpse_host": "kemenkeu",
+            "max_packages": "0",
+            "confirm_download": True,
+        })
+        self.assertEqual(result["max_packages"], 0)
+
+    def test_create_index_max_packages_negative(self):
+        """Negative max_packages is floored to 0."""
+        result = validate_search_index_create_params({
+            "lpse_host": "kemenkeu",
+            "max_packages": "-5",
+            "confirm_download": True,
+        })
+        self.assertEqual(result["max_packages"], 0)
+
 
 class TestValidateMasterKlpdParams(unittest.TestCase):
 

pyproc-0.3.2/tests/test_mcp_search_index.py

@@ -0,0 +1,169 @@
+"""Tests for pyproc.mcp.search_index — local SQLite FTS indexes."""
+
+import os
+import tempfile
+import unittest
+from unittest.mock import MagicMock, patch
+
+
+class TestSearchIndex(unittest.TestCase):
+
+    def test_create_search_and_delete_index(self):
+        from pyproc.mcp import search_index
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            mock_detail = MagicMock()
+            mock_detail.get_all_detil.return_value = {"error": False, "error_message": []}
+            mock_detail.todict.return_value = {
+                "id_paket": "100",
+                "pengumuman": {
+                    "nama_tender": "Pengadaan Laptop",
+                    "spesifikasi": "Laptop untuk pekerjaan kantor",
+                },
+            }
+
+            mock_lpse = MagicMock()
+            mock_lpse.get_paket_tender.return_value = [
+                ["100", "Pengadaan Laptop", "KEMENTERIAN KEUANGAN"],
+            ]
+            mock_lpse.detil_paket_tender.return_value = mock_detail
+            mock_lpse.__enter__ = MagicMock(return_value=mock_lpse)
+            mock_lpse.__exit__ = MagicMock(return_value=False)
+
+            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
+                with patch("pyproc.mcp.search_index.Lpse", return_value=mock_lpse):
+                    created = search_index.create_procurement_search_index(
+                        lpse_host="kemenkeu",
+                        package_type="tender",
+                        tahun_anggaran=2025,
+                        keyword_seed="laptop",
+                        max_packages=1,
+                        timeout=1,
+                    )
+
+                self.assertEqual(created["indexed_packages"], 1)
+
+                result = search_index.search_procurement_index(
+                    created["index_id"],
+                    "laptop",
+                )
+                self.assertEqual(result["count"], 1)
+                self.assertEqual(result["matches"][0]["id_paket"], "100")
+
+                listed = search_index.list_procurement_indexes()
+                self.assertEqual(listed["count"], 1)
+
+                deleted = search_index.delete_procurement_index(created["index_id"])
+                self.assertTrue(deleted["deleted"])
+
+    def test_search_missing_index(self):
+        from pyproc.mcp import search_index
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
+                with self.assertRaises(ValueError):
+                    search_index.search_procurement_index("missing", "laptop")
+
+    def _make_mock_lpse(self, chunks, detail=None):
+        """Build a mock Lpse that returns paginated chunks from get_paket_tender."""
+        if detail is None:
+            detail = MagicMock()
+            detail.get_all_detil.return_value = {"error": False, "error_message": []}
+            detail.todict.return_value = {"id_paket": "100"}
+
+        mock_lpse = MagicMock()
+        mock_lpse.get_paket_tender.side_effect = list(chunks)
+        mock_lpse.detil_paket_tender.return_value = detail
+        mock_lpse.__enter__ = MagicMock(return_value=mock_lpse)
+        mock_lpse.__exit__ = MagicMock(return_value=False)
+        return mock_lpse
+
+    def _make_row(self, pkg_id, title="Test Package"):
+        return [str(pkg_id), title, "TEST INSTANSI"]
+
+    def test_pagination_multiple_chunks(self):
+        """All rows from multiple chunks are collected."""
+        from pyproc.mcp import search_index
+
+        chunk1 = [self._make_row(i) for i in range(1, 101)]   # 100 rows
+        chunk2 = [self._make_row(i) for i in range(101, 201)]  # 100 rows
+        chunk3 = [self._make_row(i) for i in range(201, 251)]  # 50 rows (partial)
+        mock_lpse = self._make_mock_lpse([chunk1, chunk2, chunk3])
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
+                with patch("pyproc.mcp.search_index.Lpse", return_value=mock_lpse):
+                    created = search_index.create_procurement_search_index(
+                        lpse_host="kemenkeu",
+                        max_packages=0,
+                        timeout=1,
+                    )
+
+                self.assertEqual(created["indexed_packages"], 250)
+                self.assertEqual(created["failed_packages"], 0)
+                # Should have made exactly 3 requests
+                self.assertEqual(mock_lpse.get_paket_tender.call_count, 3)
+
+    def test_pagination_stops_on_empty(self):
+        """Pagination stops when SPSE returns an empty chunk after a full page."""
+        from pyproc.mcp import search_index
+
+        chunk1 = [self._make_row(i) for i in range(1, 101)]  # 100 rows (full page)
+        chunk2 = []  # empty -> stop
+        mock_lpse = self._make_mock_lpse([chunk1, chunk2])
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
+                with patch("pyproc.mcp.search_index.Lpse", return_value=mock_lpse):
+                    created = search_index.create_procurement_search_index(
+                        lpse_host="kemenkeu",
+                        max_packages=0,
+                        timeout=1,
+                    )
+
+                self.assertEqual(created["indexed_packages"], 100)
+                self.assertEqual(mock_lpse.get_paket_tender.call_count, 2)
+
+    def test_pagination_respects_max_packages(self):
+        """max_packages > 0 caps total rows collected."""
+        from pyproc.mcp import search_index
+
+        chunk1 = [self._make_row(i) for i in range(1, 101)]  # 100 rows
+        mock_lpse = self._make_mock_lpse([chunk1])
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
+                with patch("pyproc.mcp.search_index.Lpse", return_value=mock_lpse):
+                    created = search_index.create_procurement_search_index(
+                        lpse_host="kemenkeu",
+                        max_packages=30,
+                        timeout=1,
+                    )
+
+                # Should only index 30 (limited by max_packages)
+                self.assertEqual(created["indexed_packages"], 30)
+
+    def test_pagination_small_max_packages_single_request(self):
+        """max_packages <= CHUNK_SIZE makes a single request of that size."""
+        from pyproc.mcp import search_index
+
+        chunk = [self._make_row(i) for i in range(1, 51)]
+        mock_lpse = self._make_mock_lpse([chunk])
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
+                with patch("pyproc.mcp.search_index.Lpse", return_value=mock_lpse):
+                    created = search_index.create_procurement_search_index(
+                        lpse_host="kemenkeu",
+                        max_packages=50,
+                        timeout=1,
+                    )
+
+                self.assertEqual(created["indexed_packages"], 50)
+                # Single request with length=50
+                call_kwargs = mock_lpse.get_paket_tender.call_args[1]
+                self.assertEqual(call_kwargs["length"], 50)
+
+
+if __name__ == "__main__":
+    unittest.main()

pyproc-0.3.1/tests/test_mcp_search_index.py

@@ -1,69 +0,0 @@
-"""Tests for pyproc.mcp.search_index — local SQLite FTS indexes."""
-
-import os
-import tempfile
-import unittest
-from unittest.mock import MagicMock, patch
-
-
-class TestSearchIndex(unittest.TestCase):
-
-    def test_create_search_and_delete_index(self):
-        from pyproc.mcp import search_index
-
-        with tempfile.TemporaryDirectory() as tmpdir:
-            mock_detail = MagicMock()
-            mock_detail.get_all_detil.return_value = {"error": False, "error_message": []}
-            mock_detail.todict.return_value = {
-                "id_paket": "100",
-                "pengumuman": {
-                    "nama_tender": "Pengadaan Laptop",
-                    "spesifikasi": "Laptop untuk pekerjaan kantor",
-                },
-            }
-
-            mock_lpse = MagicMock()
-            mock_lpse.get_paket_tender.return_value = [
-                ["100", "Pengadaan Laptop", "KEMENTERIAN KEUANGAN"],
-            ]
-            mock_lpse.detil_paket_tender.return_value = mock_detail
-            mock_lpse.__enter__ = MagicMock(return_value=mock_lpse)
-            mock_lpse.__exit__ = MagicMock(return_value=False)
-
-            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
-                with patch("pyproc.mcp.search_index.Lpse", return_value=mock_lpse):
-                    created = search_index.create_procurement_search_index(
-                        lpse_host="kemenkeu",
-                        package_type="tender",
-                        tahun_anggaran=2025,
-                        keyword_seed="laptop",
-                        max_packages=1,
-                        timeout=1,
-                    )
-
-                self.assertEqual(created["indexed_packages"], 1)
-
-                result = search_index.search_procurement_index(
-                    created["index_id"],
-                    "laptop",
-                )
-                self.assertEqual(result["count"], 1)
-                self.assertEqual(result["matches"][0]["id_paket"], "100")
-
-                listed = search_index.list_procurement_indexes()
-                self.assertEqual(listed["count"], 1)
-
-                deleted = search_index.delete_procurement_index(created["index_id"])
-                self.assertTrue(deleted["deleted"])
-
-    def test_search_missing_index(self):
-        from pyproc.mcp import search_index
-
-        with tempfile.TemporaryDirectory() as tmpdir:
-            with patch.dict(os.environ, {"PYPROC_MCP_INDEX_DIR": tmpdir}):
-                with self.assertRaises(ValueError):
-                    search_index.search_procurement_index("missing", "laptop")
-
-
-if __name__ == "__main__":
-    unittest.main()