pi-web-kit 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+This project follows the spirit of [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and uses semantic versioning for releases.
+
+## [Unreleased]
+
+## [0.1.0] - 2026-05-09
+
+### Added
+
+- Initial `pi-web-kit` package with `web_search` and `web_fetch` Pi tools.
+- Search providers: `exa_mcp`, `exa`, `tinyfish`, `brave`, and `firecrawl`.
+- Fetch providers: `exa_mcp`, `exa`, `tinyfish`, `markdown_new`, and `firecrawl`.
+- Provider-specific tool schemas, startup-provider mismatch guidance, and CLI provider override flags.
+- In-memory fetch cache with TTL, LRU eviction, max entry count, max byte count, and fetch-affecting cache keys.
+- URL validation and normalization for HTTP(S)-only URLs, credential rejection, fragment stripping, duplicate removal, and URL count/length caps.
+- Request limits, bounded provider concurrency, fetch timeouts, and robust provider result mapping for canonicalized or redirected URLs.
+- Tests for config precedence, provider behavior, cache safety, URL validation, concurrency, timeouts, and tool behavior.
+- GitHub-ready project docs, issue templates, CI, and npm provenance publishing workflow.

package/CODE_OF_CONDUCT.md

@@ -0,0 +1,41 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment include:
+
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes
+- Focusing on what is best not just for us as individuals, but for the overall community
+
+Examples of unacceptable behavior include:
+
+- The use of sexualized language or imagery, and sexual attention or advances
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information without explicit permission
+- Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Enforcement Responsibilities
+
+Project maintainers are responsible for clarifying and enforcing standards of acceptable behavior and may remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned with this Code of Conduct.
+
+## Scope
+
+This Code of Conduct applies within all project spaces and also applies when an individual is officially representing the project in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the project maintainers through GitHub. All complaints will be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1.

package/CONTRIBUTING.md

@@ -0,0 +1,55 @@
+# Contributing
+
+Thanks for your interest in contributing to `pi-web-kit`.
+
+## Development setup
+
+```bash
+npm install
+npm run check
+npm test
+```
+
+This package is source-distributed: Pi loads the TypeScript extension files directly. There is no build step for runtime use.
+
+## Local testing
+
+Install this checkout into a temporary Pi project:
+
+```bash
+mkdir -p <test-project>
+cd <test-project>
+pi install -l /path/to/pi-web-kit
+pi
+```
+
+For a one-off run without changing settings:
+
+```bash
+pi -e /path/to/pi-web-kit --web-provider-fetch markdown_new --print "Fetch https://example.com"
+```
+
+## Pull request checklist
+
+Before opening a pull request:
+
+- Run `npm run check`.
+- Run `npm test`.
+- Run `npm audit --omit=dev`.
+- Run `npm run pack:dry-run` and confirm the package contents are intentional.
+- Update `README.md` if user-facing behavior changes.
+- Update `CHANGELOG.md` for notable changes.
+- Keep examples and paths generic; do not commit machine-specific paths, API keys, tokens, or provider config containing secrets.
+
+## Coding guidelines
+
+- Keep `extensions/pi-web-kit/index.ts` focused on Pi tool registration and move reusable implementation details into `src/`.
+- When changing tool parameters, update the Typebox schema, runtime validation, README parameter docs, and tests together.
+- Treat provider names, config keys, env vars, and cache semantics as public interface; changes to defaults or precedence are breaking changes.
+- Validate and normalize external URLs before provider calls.
+- Bound network work: use shared limits, timeouts, and concurrency helpers for new provider integrations.
+- Keep tool `details` compact; full result text belongs in tool content and should remain bounded by `truncateText()`.
+
+## Code of conduct
+
+This project follows the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md).

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jose Mocito
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,232 @@
+# pi-web-kit
+
+Context-efficient web search and fetch tools for [Pi](https://github.com/badlogic/pi-mono): `web_search` and `web_fetch`.
+
+`pi-web-kit` provides provider-backed search and page fetching with bounded output, chunked reads, URL validation, and an in-memory fetch cache designed for agent workflows.
+
+## Features
+
+- `web_search` for current/external web information, including multi-query searches.
+- `web_fetch` for reading one or more URLs, with `offset` / `limit` chunk reads for long pages.
+- Multiple provider backends: Exa MCP, Exa API, TinyFish, Brave Search, Firecrawl, and markdown.new.
+- Provider-tailored tool schemas at Pi startup/reload.
+- URL validation: HTTP(S)-only, no embedded credentials, fragment stripping, duplicate removal, and length/count limits.
+- In-memory fetch cache with TTL, LRU eviction, max entry count, max byte count, and cache keys based on provider/config/fetch-affecting options.
+- Bounded provider concurrency and network timeouts.
+
+## Installation
+
+Install from npm:
+
+```bash
+pi install npm:pi-web-kit
+```
+
+Install from GitHub:
+
+```bash
+pi install git:github.com/jvm/pi-web-kit
+```
+
+Install project-locally with Pi's `-l` flag:
+
+```bash
+pi install -l git:github.com/jvm/pi-web-kit
+```
+
+During local development from this repository:
+
+```bash
+pi install /path/to/pi-web-kit
+```
+
+For a one-off test run without installing:
+
+```bash
+pi -e /path/to/pi-web-kit --web-provider-fetch markdown_new --print "Fetch https://example.com"
+```
+
+This is an npm-compatible TypeScript Pi package. Bun is not required.
+
+## Quick usage
+
+Search:
+
+```text
+Find recent documentation for the Pi extension API.
+```
+
+Multi-query search:
+
+```text
+Search for recent docs on Pi extensions and Pi tool schemas.
+```
+
+Fetch a page:
+
+```text
+Read https://example.com and summarize it.
+```
+
+Fetch a long page in chunks:
+
+```text
+Fetch https://example.com/long-doc with limit 8000, then continue with offset 8000.
+```
+
+Pi chooses `web_search` or `web_fetch` automatically when the request calls for it. You can also mention provider settings explicitly in prompts, but provider changes usually require config or CLI flags.
+
+## Providers
+
+Defaults: `provider_search = "exa_mcp"`, `provider_fetch = "exa_mcp"`.
+
+| Provider | Search | Fetch | Key |
+|---|---:|---:|---|
+| `exa_mcp` | yes | yes | optional `EXA_API_KEY` |
+| `exa` | yes | yes | `EXA_API_KEY` |
+| `tinyfish` | yes | yes | `TINYFISH_API_KEY` |
+| `brave` | yes | no | `BRAVE_SEARCH_API_KEY` |
+| `firecrawl` | yes | yes | `FIRECRAWL_API_KEY` |
+| `markdown_new` | no | yes | none |
+
+Tool schemas are tailored to the configured providers at startup/reload, so only supported provider-specific fields are exposed. Restart/reload Pi after changing provider config.
+
+## Configuration
+
+Resolution order: defaults < environment variables < global config < project config < CLI flags.
+
+### Environment variables
+
+```bash
+PI_WEB_KIT_PROVIDER_SEARCH=exa_mcp|exa|tinyfish|brave|firecrawl
+PI_WEB_KIT_PROVIDER_FETCH=exa_mcp|exa|tinyfish|markdown_new|firecrawl
+EXA_API_KEY=...
+TINYFISH_API_KEY=...
+BRAVE_SEARCH_API_KEY=...
+FIRECRAWL_API_KEY=...
+```
+
+### Config files
+
+Config files, in increasing precedence:
+
+| Scope | Path |
+|---|---|
+| Global | `~/.pi/agent/pi-web-kit.json` |
+| Project | `.pi-web-kit.json` |
+
+Example:
+
+```json
+{
+  "provider_search": "firecrawl",
+  "provider_fetch": "markdown_new",
+  "apiKeys": {
+    "firecrawl": "..."
+  },
+  "markdownNew": {
+    "method": "auto",
+    "retainImages": false
+  }
+}
+```
+
+Do not commit config files containing secrets. Project `.pi-web-kit.json` is ignored by this repo's `.gitignore`, but other repositories may need their own ignore rule.
+
+### CLI overrides
+
+```bash
+pi -e . --web-provider-search firecrawl --web-provider-fetch markdown_new --print "Search and fetch docs"
+```
+
+Provider CLI flags are temporary for the Pi process. Restart/reload Pi after changing provider config so registered tool schemas match the active provider.
+
+## Tools
+
+### `web_search`
+
+Searches with the active search provider and returns compact results grouped by query.
+
+| Parameter | Type | Description |
+|---|---|---|
+| `query` | string | Single search query. |
+| `queries` | string[] | Multiple related search queries. Max 5 after de-duplication. |
+| `numResults` | integer | Results per query. Range: 1-20. Default: 10. |
+
+Provider-specific parameters are exposed only for the configured provider, such as Exa date/domain filters, TinyFish `page`, Brave locale/freshness options, or Firecrawl scrape/search options.
+
+### `web_fetch`
+
+Fetches page content with the active fetch provider. Results are cached in memory by canonical URL plus provider/config/fetch-affecting options.
+
+| Parameter | Type | Description |
+|---|---|---|
+| `url` | string | Single URL. Must be `http:` or `https:`. |
+| `urls` | string[] | Multiple URLs. Max 10 after de-duplication. |
+| `offset` | integer | Character offset for cached/ranged reads. Single URL only. |
+| `limit` | integer | Maximum characters to return. Default: 30,000 for one URL, 8,000 for multiple URLs. |
+| `refresh` | boolean | Refetch even if cached. |
+
+Provider-specific parameters are exposed only for the configured provider, such as TinyFish `format`, markdown.new `method` / `retainImages`, or Firecrawl `format`, `waitFor`, `mobile`, `location`, and `maxAge`.
+
+## Cache and limits
+
+`web_fetch` uses an in-memory cache for the current Pi process.
+
+| Limit | Value |
+|---|---:|
+| Cache TTL | 30 minutes |
+| Max cached entries | 100 |
+| Max cached bytes | 20 MiB |
+| Max URLs per call | 10 |
+| Max queries per call | 5 |
+| Max `numResults` | 20 |
+| Max URL length | 2048 characters |
+
+Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and a short API-key/account scope marker. `refresh: true` bypasses and replaces the cached entry.
+
+## Privacy and security
+
+`pi-web-kit` sends search queries and fetched URLs to the configured provider. Fetch providers may also receive provider-specific options. API keys are read from environment variables or local config files and are used only for provider requests.
+
+The extension rejects non-HTTP(S) URLs and URLs with embedded username/password credentials. Provider responses are not sandboxed; they are returned to Pi as tool output.
+
+Report security issues privately. See [SECURITY.md](SECURITY.md).
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `provider requires ... API_KEY` | Selected provider needs an API key. | Set the provider's env var or `apiKeys` config entry. |
+| Provider mismatch / schema error after config change | Pi registered tools for the previous startup provider. | Restart/reload Pi after provider changes. |
+| Invalid URL / scheme / credentials error | URL validation rejected the input. | Use an absolute `http:` or `https:` URL without username/password credentials. |
+| Timeout error | Provider request exceeded its timeout. | Retry, reduce URL count, or switch provider. |
+| No content returned | Provider returned no matching content or a redirected/canonicalized response could not be mapped. | Retry with `refresh: true`, fetch a single URL, or switch provider. |
+| Large page is truncated | Tool output is bounded for context efficiency. | Use `offset` and `limit` to continue reading chunks. |
+
+## Development
+
+Requirements:
+
+- Node.js >= 20.6.0
+- npm
+
+Common commands:
+
+```bash
+npm install
+npm run check
+npm test
+npm audit --omit=dev
+npm run pack:dry-run
+```
+
+This package is source-distributed. Pi loads the TypeScript extension files directly via its extension loader.
+
+## Contributing
+
+Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow and pull request guidelines.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

package/SECURITY.md

@@ -0,0 +1,26 @@
+# Security Policy
+
+## Supported versions
+
+Security fixes are provided for the latest released version of `pi-web-kit`.
+
+## Reporting a vulnerability
+
+Please do not open a public issue for suspected security vulnerabilities.
+
+Report privately by contacting the repository maintainer through GitHub. Include:
+
+- a description of the issue;
+- steps to reproduce;
+- affected versions or commits, if known;
+- any suggested mitigation.
+
+The maintainer will acknowledge reports as soon as practical and coordinate disclosure once a fix or mitigation is available.
+
+## Security model
+
+`pi-web-kit` is a Pi package. Pi extensions execute with the same permissions as the local user running Pi. Users should review installed Pi packages and only install packages from sources they trust.
+
+`pi-web-kit` sends search queries and fetched URLs to the configured third-party provider. Page content returned by providers is cached in memory for the lifetime of the Pi process, subject to TTL and size limits. API keys are read from environment variables or local config files and are used only for provider requests. Do not commit API keys, tokens, or config files containing secrets.
+
+The extension validates URLs before fetch calls and only accepts `http:` and `https:` URLs without embedded credentials. This validation reduces accidental misuse but does not sandbox provider responses or the local Pi process.