pyproc 0.3.1__tar.gz → 0.3.3__tar.gz

{pyproc-0.3.1 → pyproc-0.3.3}/.gitignore

@@ -54,4 +54,4 @@ coverage.xml
 .pytest_cache/
 *.csv
 *.idx
-statistic.txt
+statistic.txt.claude/

{pyproc-0.3.1 → pyproc-0.3.3}/CHANGELOG.md

@@ -4,6 +4,59 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.3.2
+
+### Added
+- [mcp] Tambah tool `get_master_klpd` untuk referensi K/L/PD dari LKPP ISB Satu Data
+- [mcp] Tambah tool `get_master_lpse` untuk referensi LPSE dari LKPP ISB Satu Data
+- [mcp] Tambah tool `get_tender_umum_publik` untuk unduh data tender dari ISB Satu Data (sumber alternatif)
+- [mcp] Tambah tool `set_ssl_verify` untuk mengaktifkan/menonaktifkan verifikasi sertifikat TLS/SSL
+- [mcp] Tambah tool `get_ssl_verify` untuk mengecek status verifikasi sertifikat TLS/SSL
+- [mcp] Tambah notifikasi progress untuk `create_procurement_search_index`
+- [cli] Subcommand `satudata` grouping `masterlpse` dan `tenderumum` untuk akses ISB Satu Data
+- [env] Tambah variabel lingkungan `PYPROC_SSL_VERIFY` untuk konfigurasi SSL saat startup
+
+## 0.3.1
+
+### Changed
+- [mcp] Hapus batas `max_packages` pada `create_procurement_search_index`, tambah pagination dan progress logging
+
+## 0.3
+
+### Added
+- [mcp] Server MCP dengan dukungan penuh untuk MCP protocol (stdio transport)
+- [mcp] Tool pencarian paket tender, non-tender, pencatatan, swakelola, dan darurat
+- [mcp] Tool detil paket individu dan bulk (sampai 20 paket per panggilan)
+- [mcp] Tool pencarian host LPSE berdasarkan nama instansi dan validasi host
+- [mcp] Tool kategori pengadaan (tanpa panggilan jaringan)
+- [mcp] Tool opsi pencarian (penjelasan strategi direct keyword vs local FTS index)
+- [mcp] Local SQLite full-text search index (create, search, list, delete)
+- [cli] Subcommand `masterklpd` untuk query referensi K/L/PD dari ISB API
+- [cli] Subcommand `masterlpse` untuk browser LPSE interaktif dan unduh data tender
+- [cli] Subcommand `tenderumum` untuk unduh data tender dari ISB API
+- [library] Method `Lpse.get_master_klpd()`, `Lpse.get_master_lpse()`, `Lpse.get_tender_umum_publik()`
+- [library] Dukungan context manager untuk class `Lpse`
+
+## 0.2
+
+### Added
+- [cli] Subcommand `daftarhost` untuk unduh daftar host LPSE dalam format JSON dari GitHub Gist
+- [cli] Argument `--instansi-id` untuk filter pencarian berdasarkan kode K/L/PD
+- [cli] Argument `--tipe-swakelola-id` untuk filter tipe swakelola
+- [library] Module `cache.py` untuk SQLite cache store dengan context manager
+- [library] Parameter `verify` pada constructor `Lpse` untuk kontrol verifikasi SSL
+- [test] Unit test framework dengan 102 test case (cache, CLI, API layer)
+
+### Changed
+- [cli] Restruktur CLI dengan subcommand dispatch system, backward compatible
+- [cli] Ganti flag `--non-tender` menjadi `--jenis-paket` dengan dukungan 5 tipe paket
+- [build] Migrasi ke `pyproject.toml`, upgrade GitHub Actions
+
+### Fixed
+- [downloader] Filter tahun anggaran pada infinite scrolling
+- [downloader] Detil referer header
+- [library] Method `get_paket` request method
+
 ## 0.1.7
 
 ### Fix

{pyproc-0.3.1 → pyproc-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyproc
-Version: 0.3.1
+Version: 0.3.3
 Summary: MCP tools for real-time Indonesian public procurement data from SPSE/Inaproc
 Project-URL: Homepage, https://github.com/wakataw/pyproc
 Project-URL: Issues, https://github.com/wakataw/pyproc/issues
@@ -36,7 +36,7 @@ Description-Content-Type: text/markdown
 # PyProc MCP
 
 <p align="center">
-  <img src="docs/assets/logo.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/logo.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="200">
 </p>
@@ -46,7 +46,7 @@ Description-Content-Type: text/markdown
 </p>
 
 <p align="center">
-  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.1-blue" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.3-blue" alt="PyPI version"></a>
   <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/python-≥3.9-yellow" alt="Python ≥3.9"></a>
   <a href="https://github.com/wakataw/pyproc/actions/workflows/test.yml?query=branch%3Amaster"><img src="https://github.com/wakataw/pyproc/actions/workflows/test.yml/badge.svg?branch=master" alt="Test Status"></a>
   <a href="https://github.com/wakataw/pyproc/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT"></a>
@@ -60,7 +60,7 @@ PyProc MCP turns public **SPSE/Inaproc** procurement data into **MCP tools** tha
 ## Why PyProc MCP?
 
 <p align="center">
-  <img src="docs/assets/pyproc-mcp-banner.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/pyproc-mcp-banner.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="800">
 </p>
@@ -160,18 +160,20 @@ Download procurement data in bulk from the command line.
 
 ```bash
 # Download tender data from Kemenkeu LPSE, export as JSON
-pyproc kemenkeu --keyword "mobil dinas" --tahun-anggaran 2025 --output-format json
+pyproc spse kemenkeu --keyword "mobil dinas" --tahun-anggaran 2025 --output-format json
 
 # Download non-tender data from multiple LPSE hosts
-pyproc jakarta,sumbarprov --jenis-paket non_tender --tahun-anggaran 2025
+pyproc spse jakarta,sumbarprov --jenis-paket non_tender --tahun-anggaran 2025
 
 # Download pencatatan non-tender data. This is distinct from non_tender.
-pyproc nasional --jenis-paket pencatatan_non_tender --tahun-anggaran 2026
+pyproc spse nasional --jenis-paket pencatatan_non_tender --tahun-anggaran 2026
 
 # Download with custom output filename
-pyproc "kemenkeu;output_kemenkeu" --output-format csv --separator ";"
+pyproc spse "kemenkeu;output_kemenkeu" --output-format csv --separator ";"
 ```
 
+The `spse` subcommand is the default, so `pyproc kemenkeu` is shorthand for `pyproc spse kemenkeu`.
+
 See [CLI Usage](#cli-usage) for the full argument reference.
 
 ---
@@ -201,6 +203,11 @@ The MCP server exposes the following tools:
 | `get_swakelola_details_bulk` | Get details for multiple swakelola packages |
 | `get_pengadaan_darurat_details_bulk` | Get details for multiple pengadaan darurat packages |
 | `get_procurement_categories` | List supported procurement categories (no network call) |
+| `get_master_klpd` | Get LKPP Satu Data master K/L/PD references (alternative data source) |
+| `get_master_lpse` | Get LKPP Satu Data master LPSE references (alternative data source) |
+| `get_tender_umum_publik` | Get tender data from LKPP ISB Satu Data (alternative data source / fallback) |
+| `set_ssl_verify` | Enable/disable TLS/SSL certificate verification for all SPSE requests |
+| `get_ssl_verify` | Return current SSL verification setting |
 | `validate_lpse_host` | Check if an LPSE host is accessible |
 | `create_procurement_search_index` | Download a bounded package set into a local SQLite full-text index |
 | `search_procurement_index` | Search a local SQLite full-text index |
@@ -253,6 +260,24 @@ The LLM should start with direct keyword search. If results are weak or the user
 
 ---
 
+## ISB Satu Data API (Alternative Data Source)
+
+In addition to direct realtime SPSE/Inaproc queries, PyProc also integrates with the **LKPP ISB (Indonesia Satu Data) API**. This is a curated, government-maintained dataset that provides:
+
+- **Master LPSE** — reference list of all LPSE units with their codes and names
+- **Master K/L/PD** — reference list of all government institutions (Kementerian, Lembaga, Perangkat Daerah)
+- **Tender Umum Publik** — public tender data indexed by LPSE code and budget year
+
+The ISB Satu Data API serves as an **alternative data source** when:
+
+- Direct SPSE/Inaproc queries fail or return errors (e.g., server timeouts, blocked hosts)
+- You need reference data (LPSE codes, institution codes) before making SPSE queries
+- The user explicitly wants the curated ISB dataset rather than realtime SPSE data
+
+Use the `satudata` CLI subcommand or the `get_master_lpse`, `get_master_klpd`, and `get_tender_umum_publik` MCP tools to access ISB data.
+
+---
+
 ## Example LLM Workflows
 
 Here are examples of what you can ask an LLM when PyProc MCP tools are connected:
@@ -317,6 +342,7 @@ The MCP server is configured via environment variables:
 | `PYPROC_TIMEOUT` | `30` | HTTP request timeout in seconds |
 | `PYPROC_RATE_LIMIT_DELAY` | `1.0` | Minimum seconds between requests |
 | `PYPROC_LOG_LEVEL` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) |
+| `PYPROC_SSL_VERIFY` | `0` | Enable SSL/TLS certificate verification (`1`, `true`, `yes`, `on` to enable) |
 
 ---
 
@@ -468,7 +494,9 @@ JenisPengadaan.JASA_LAINNYA               # Other Services
 
 ## CLI Usage
 
-### Basic Download
+### SPSE (Direct Realtime Data)
+
+Download procurement data directly from SPSE/Inaproc servers. The `spse` subcommand is the default.
 
 ```bash
 pyproc kemenkeu
@@ -480,7 +508,7 @@ This downloads tender data from the Kemenkeu LPSE and exports it as `kemenkeu.cs
 
 | Argument | Example | Default | Description |
 |---|---|---|---|
-| `lpse_host` | `pyproc kemenkeu` | Required | LPSE host or text file with host list |
+| `lpse_host` | `pyproc spse kemenkeu` | Required | LPSE host or text file with host list |
 | `-k, --keyword` | `--keyword "mobil dinas"` | `""` | Search keyword filter |
 | `-t, --tahun-anggaran` | `--tahun-anggaran 2025` | Current year | Budget year (single, comma-separated, or range) |
 | `--kategori` | `--kategori PEKERJAAN_KONSTRUKSI` | None | Procurement category |
@@ -501,10 +529,10 @@ This downloads tender data from the Kemenkeu LPSE and exports it as `kemenkeu.cs
 
 ```bash
 # Download from multiple LPSE hosts
-pyproc jakarta,kemenkeu,sumbarprov
+pyproc spse jakarta,kemenkeu,sumbarprov
 
 # With custom output filenames
-pyproc "jakarta;file_jakarta,kemenkeu;file_kemenkeu"
+pyproc spse "jakarta;file_jakarta,kemenkeu;file_kemenkeu"
 ```
 
 ### Download Host List
@@ -517,6 +545,71 @@ pyproc daftarlpse
 pyproc daftarhost
 ```
 
+### Master KLPD
+
+Query K/L/PD reference data from the LKPP ISB Satu Data API:
+
+```bash
+# Show all K/L/PD references
+pyproc masterklpd
+
+# Search by institution name or code
+pyproc masterklpd --query kemenkeu
+
+# Filter by jenis (K, L, PD)
+pyproc masterklpd --jenis K
+
+# Get specific K/L/PD by code
+pyproc masterklpd --kd-klpd K68
+
+# Limit results
+pyproc masterklpd --limit 5
+```
+
+### Satu Data (Alternative Source)
+
+Access tender data from the LKPP ISB Satu Data API as an alternative to direct realtime SPSE queries.
+
+#### Master LPSE (Interactive Browser)
+
+```bash
+# Launch interactive LPSE browser
+pyproc satudata masterlpse
+
+# With custom timeout
+pyproc satudata masterlpse --timeout 60
+
+# Specify output format and file
+pyproc satudata masterlpse --output csv --output-file hasil.csv
+```
+
+#### Tender Umum Publik (Direct Download)
+
+Download tender data for a specific LPSE code and budget year:
+
+```bash
+# Download tender data for a specific LPSE and year
+pyproc satudata tenderumum --kode-lpse 119 --tahun-anggaran 2026
+
+# With custom options
+pyproc satudata tenderumum --kode-lpse 119 --tahun-anggaran 2026 --timeout 60 --output json
+```
+
+**Required arguments for `tenderumum`**:
+
+| Argument | Example | Description |
+|---|---|---|
+| `--tahun-anggaran` | `2026` | Budget year |
+| `--kode-lpse` | `119` | LPSE code from master LPSE |
+
+**Optional arguments**:
+
+| Argument | Default | Description |
+|---|---|---|
+| `--timeout` | `30` | Request timeout in seconds |
+| `--output` | `json` | Output format: `json` or `csv` |
+| `--output-file` | Auto-generated | Output file path |
+
 ---
 
 ## Responsible Use and Disclaimer
@@ -582,11 +675,3 @@ pyproc/
 ## License
 
 MIT License. See [LICENSE](LICENSE) for details.
-
----
-
-## Donatur ☕️
-
-Orang-orang yang berjasa menyediakan kopi sehingga pengembangan paket tetap berjalan:
-
-- Angga Rinaldi Rizal (50 cangkir ☕️)

{pyproc-0.3.1 → pyproc-0.3.3}/README.md

@@ -1,7 +1,7 @@
 # PyProc MCP
 
 <p align="center">
-  <img src="docs/assets/logo.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/logo.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="200">
 </p>
@@ -11,7 +11,7 @@
 </p>
 
 <p align="center">
-  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.1-blue" alt="PyPI version"></a>
+  <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/version-v0.3.3-blue" alt="PyPI version"></a>
   <a href="https://pypi.org/project/pyproc/"><img src="https://img.shields.io/badge/python-≥3.9-yellow" alt="Python ≥3.9"></a>
   <a href="https://github.com/wakataw/pyproc/actions/workflows/test.yml?query=branch%3Amaster"><img src="https://github.com/wakataw/pyproc/actions/workflows/test.yml/badge.svg?branch=master" alt="Test Status"></a>
   <a href="https://github.com/wakataw/pyproc/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT"></a>
@@ -25,7 +25,7 @@ PyProc MCP turns public **SPSE/Inaproc** procurement data into **MCP tools** tha
 ## Why PyProc MCP?
 
 <p align="center">
-  <img src="docs/assets/pyproc-mcp-banner.png"
+  <img src="https://raw.githubusercontent.com/wakataw/pyproc/remove-index-max-packages-cap/docs/assets/pyproc-mcp-banner.png"
        alt="PyProc MCP — Real-time Indonesian procurement data for LLM agents"
        width="800">
 </p>
@@ -125,18 +125,20 @@ Download procurement data in bulk from the command line.
 
 ```bash
 # Download tender data from Kemenkeu LPSE, export as JSON
-pyproc kemenkeu --keyword "mobil dinas" --tahun-anggaran 2025 --output-format json
+pyproc spse kemenkeu --keyword "mobil dinas" --tahun-anggaran 2025 --output-format json
 
 # Download non-tender data from multiple LPSE hosts
-pyproc jakarta,sumbarprov --jenis-paket non_tender --tahun-anggaran 2025
+pyproc spse jakarta,sumbarprov --jenis-paket non_tender --tahun-anggaran 2025
 
 # Download pencatatan non-tender data. This is distinct from non_tender.
-pyproc nasional --jenis-paket pencatatan_non_tender --tahun-anggaran 2026
+pyproc spse nasional --jenis-paket pencatatan_non_tender --tahun-anggaran 2026
 
 # Download with custom output filename
-pyproc "kemenkeu;output_kemenkeu" --output-format csv --separator ";"
+pyproc spse "kemenkeu;output_kemenkeu" --output-format csv --separator ";"
 ```
 
+The `spse` subcommand is the default, so `pyproc kemenkeu` is shorthand for `pyproc spse kemenkeu`.
+
 See [CLI Usage](#cli-usage) for the full argument reference.
 
 ---
@@ -166,6 +168,11 @@ The MCP server exposes the following tools:
 | `get_swakelola_details_bulk` | Get details for multiple swakelola packages |
 | `get_pengadaan_darurat_details_bulk` | Get details for multiple pengadaan darurat packages |
 | `get_procurement_categories` | List supported procurement categories (no network call) |
+| `get_master_klpd` | Get LKPP Satu Data master K/L/PD references (alternative data source) |
+| `get_master_lpse` | Get LKPP Satu Data master LPSE references (alternative data source) |
+| `get_tender_umum_publik` | Get tender data from LKPP ISB Satu Data (alternative data source / fallback) |
+| `set_ssl_verify` | Enable/disable TLS/SSL certificate verification for all SPSE requests |
+| `get_ssl_verify` | Return current SSL verification setting |
 | `validate_lpse_host` | Check if an LPSE host is accessible |
 | `create_procurement_search_index` | Download a bounded package set into a local SQLite full-text index |
 | `search_procurement_index` | Search a local SQLite full-text index |
@@ -218,6 +225,24 @@ The LLM should start with direct keyword search. If results are weak or the user
 
 ---
 
+## ISB Satu Data API (Alternative Data Source)
+
+In addition to direct realtime SPSE/Inaproc queries, PyProc also integrates with the **LKPP ISB (Indonesia Satu Data) API**. This is a curated, government-maintained dataset that provides:
+
+- **Master LPSE** — reference list of all LPSE units with their codes and names
+- **Master K/L/PD** — reference list of all government institutions (Kementerian, Lembaga, Perangkat Daerah)
+- **Tender Umum Publik** — public tender data indexed by LPSE code and budget year
+
+The ISB Satu Data API serves as an **alternative data source** when:
+
+- Direct SPSE/Inaproc queries fail or return errors (e.g., server timeouts, blocked hosts)
+- You need reference data (LPSE codes, institution codes) before making SPSE queries
+- The user explicitly wants the curated ISB dataset rather than realtime SPSE data
+
+Use the `satudata` CLI subcommand or the `get_master_lpse`, `get_master_klpd`, and `get_tender_umum_publik` MCP tools to access ISB data.
+
+---
+
 ## Example LLM Workflows
 
 Here are examples of what you can ask an LLM when PyProc MCP tools are connected:
@@ -282,6 +307,7 @@ The MCP server is configured via environment variables:
 | `PYPROC_TIMEOUT` | `30` | HTTP request timeout in seconds |
 | `PYPROC_RATE_LIMIT_DELAY` | `1.0` | Minimum seconds between requests |
 | `PYPROC_LOG_LEVEL` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) |
+| `PYPROC_SSL_VERIFY` | `0` | Enable SSL/TLS certificate verification (`1`, `true`, `yes`, `on` to enable) |
 
 ---
 
@@ -433,7 +459,9 @@ JenisPengadaan.JASA_LAINNYA               # Other Services
 
 ## CLI Usage
 
-### Basic Download
+### SPSE (Direct Realtime Data)
+
+Download procurement data directly from SPSE/Inaproc servers. The `spse` subcommand is the default.
 
 ```bash
 pyproc kemenkeu
@@ -445,7 +473,7 @@ This downloads tender data from the Kemenkeu LPSE and exports it as `kemenkeu.cs
 
 | Argument | Example | Default | Description |
 |---|---|---|---|
-| `lpse_host` | `pyproc kemenkeu` | Required | LPSE host or text file with host list |
+| `lpse_host` | `pyproc spse kemenkeu` | Required | LPSE host or text file with host list |
 | `-k, --keyword` | `--keyword "mobil dinas"` | `""` | Search keyword filter |
 | `-t, --tahun-anggaran` | `--tahun-anggaran 2025` | Current year | Budget year (single, comma-separated, or range) |
 | `--kategori` | `--kategori PEKERJAAN_KONSTRUKSI` | None | Procurement category |
@@ -466,10 +494,10 @@ This downloads tender data from the Kemenkeu LPSE and exports it as `kemenkeu.cs
 
 ```bash
 # Download from multiple LPSE hosts
-pyproc jakarta,kemenkeu,sumbarprov
+pyproc spse jakarta,kemenkeu,sumbarprov
 
 # With custom output filenames
-pyproc "jakarta;file_jakarta,kemenkeu;file_kemenkeu"
+pyproc spse "jakarta;file_jakarta,kemenkeu;file_kemenkeu"
 ```
 
 ### Download Host List
@@ -482,6 +510,71 @@ pyproc daftarlpse
 pyproc daftarhost
 ```
 
+### Master KLPD
+
+Query K/L/PD reference data from the LKPP ISB Satu Data API:
+
+```bash
+# Show all K/L/PD references
+pyproc masterklpd
+
+# Search by institution name or code
+pyproc masterklpd --query kemenkeu
+
+# Filter by jenis (K, L, PD)
+pyproc masterklpd --jenis K
+
+# Get specific K/L/PD by code
+pyproc masterklpd --kd-klpd K68
+
+# Limit results
+pyproc masterklpd --limit 5
+```
+
+### Satu Data (Alternative Source)
+
+Access tender data from the LKPP ISB Satu Data API as an alternative to direct realtime SPSE queries.
+
+#### Master LPSE (Interactive Browser)
+
+```bash
+# Launch interactive LPSE browser
+pyproc satudata masterlpse
+
+# With custom timeout
+pyproc satudata masterlpse --timeout 60
+
+# Specify output format and file
+pyproc satudata masterlpse --output csv --output-file hasil.csv
+```
+
+#### Tender Umum Publik (Direct Download)
+
+Download tender data for a specific LPSE code and budget year:
+
+```bash
+# Download tender data for a specific LPSE and year
+pyproc satudata tenderumum --kode-lpse 119 --tahun-anggaran 2026
+
+# With custom options
+pyproc satudata tenderumum --kode-lpse 119 --tahun-anggaran 2026 --timeout 60 --output json
+```
+
+**Required arguments for `tenderumum`**:
+
+| Argument | Example | Description |
+|---|---|---|
+| `--tahun-anggaran` | `2026` | Budget year |
+| `--kode-lpse` | `119` | LPSE code from master LPSE |
+
+**Optional arguments**:
+
+| Argument | Default | Description |
+|---|---|---|
+| `--timeout` | `30` | Request timeout in seconds |
+| `--output` | `json` | Output format: `json` or `csv` |
+| `--output-file` | Auto-generated | Output file path |
+
 ---
 
 ## Responsible Use and Disclaimer
@@ -547,11 +640,3 @@ pyproc/
 ## License
 
 MIT License. See [LICENSE](LICENSE) for details.
-
----
-
-## Donatur ☕️
-
-Orang-orang yang berjasa menyediakan kopi sehingga pengembangan paket tetap berjalan:
-
-- Angga Rinaldi Rizal (50 cangkir ☕️)

pyproc-0.3.3/plan/SEARCH_INDEX_LIMIT_REMOVAL_AND_PROGRESS.md

@@ -0,0 +1,87 @@
+# Plan: Remove max_packages cap, default to all data, paginate at 100/req, add progress
+
+## Context
+
+The `create_procurement_search_index` MCP tool has `MAX_INDEX_PACKAGES = 500` capping the **total** number of packages that can be downloaded. Remove this cap so users can download all available data for a host/year. Default behavior becomes "download all available data." Per-request batch size stays fixed at 100 (internal, not exposed). Progress is reported as a running count via stderr logging — no percentage since we don't rely on `recordsTotal`/`recordsFiltered`.
+
+## Parameter design
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `max_packages` | int | `0` | `0` = download all (paginate until exhausted). Positive value caps the total. No upper bound. |
+| Batch size | int (internal) | `100` | Fixed `CHUNK_SIZE`, not a user-facing parameter. Matches CLI `--chunk-size` default. |
+
+## Changes
+
+### 1. `pyproc/mcp/schemas.py`
+
+- Delete `MAX_INDEX_PACKAGES = 500` (line 33)
+- Change `DEFAULT_INDEX_MAX_PACKAGES = 100` → `DEFAULT_INDEX_MAX_PACKAGES = 0` (0 = all)
+- Line 410: `max(1, min(max_packages, MAX_INDEX_PACKAGES))` → `max(0, max_packages)` (0 means unlimited)
+- `CREATE_SEARCH_INDEX_SCHEMA` (lines 952-960):
+  - Remove `"maximum": MAX_INDEX_PACKAGES`
+  - Update description to `"Maximum packages to download. 0 = all available. Default 0."`
+
+### 2. `pyproc/mcp/search_index.py`
+
+- Add `import logging` → `logger = logging.getLogger(__name__)`
+- Add `CHUNK_SIZE = 100`
+
+Replace the single-request block (lines 127-139) with a pagination loop:
+
+```python
+CHUNK_SIZE = 100
+all_rows = []
+start = 0
+limit = max_packages if max_packages > 0 else float('inf')
+
+while len(all_rows) < limit:
+    remaining = limit - len(all_rows)
+    req_length = min(CHUNK_SIZE, int(remaining))
+    chunk_kwargs = {**kwargs, "start": start, "length": req_length}
+    chunk = search_method(**chunk_kwargs)
+    if not chunk:
+        break
+    all_rows.extend(chunk)
+    logger.info("Scrolled: %d rows from %s (TA %s)", len(all_rows), lpse_host, tahun_anggaran)
+    start += len(chunk)
+    if len(chunk) < req_length:
+        break  # partial page → end of data
+
+to_process = len(all_rows)
+logger.info("Will index %d packages from %s (%s, %s)", to_process, lpse_host, package_type, tahun_anggaran)
+```
+
+Progress logging in the detail loop:
+
+```python
+for idx, row in enumerate(all_rows, start=1):
+    title = _package_title(row)
+    logger.info("[%d/%d] Indexing: %s", idx, to_process, title)
+    # ... existing fetch + index + rate_limit ...
+```
+
+### 3. `pyproc/mcp/tools.py`
+
+- Update tool description string to mention "0 = all" default
+- No code changes needed — progress is handled in `search_index.py`'s logger
+
+### 4. Tests
+
+**`tests/test_mcp_schemas.py`** (`TestValidateSearchIndexParams`):
+- Default `max_packages` → 0 (all)
+- `max_packages=5000` → 5000 (not clamped)
+- `max_packages=-1` → 0 (floor at 0)
+
+**`tests/test_mcp_search_index.py`** (`TestSearchIndex`):
+- Pagination: mock returns [100, 100, 50] → all 250 collected
+- Stops on empty chunk
+- Stops on partial chunk (< requested length)
+- Respects max_packages limit when positive
+
+## Verification
+
+```bash
+python -m pytest tests/test_mcp_schemas.py::TestValidateSearchIndexParams tests/test_mcp_search_index.py -v
+python -m pytest tests/ --ignore=tests/test_lpse.py --ignore=tests/test_downloader.py -v
+```

{pyproc-0.3.1 → pyproc-0.3.3}/pyproc/__init__.py

@@ -1,6 +1,6 @@
 from .lpse import Lpse, JenisPengadaan, TipeSwakelola, By, KontrakStatus
 
-__version__ = '0.3.1'
+__version__ = '0.3.3'
 __author__ = 'Agung Pratama'
 __all__ = [
     'Lpse',