pi-unsloth-webtools 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -13
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -4,6 +4,20 @@ A [pi](https://github.com/earendil-works/pi-coding-agent) extension providing `w
|
|
|
4
4
|
`web_fetch` tools, ported from the Unsloth Studio codebase
|
|
5
5
|
([`unslothai/unsloth`](https://github.com/unslothai/unsloth), `studio/backend/core/inference/`).
|
|
6
6
|
|
|
7
|
+
## Install
|
|
8
|
+
|
|
9
|
+
```sh
|
|
10
|
+
pi install npm:pi-unsloth-webtools
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
or add `npm:pi-unsloth-webtools` to the `packages` array in `~/.pi/agent/settings.json`.
|
|
14
|
+
|
|
15
|
+
To install from source instead:
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
pi install /path/to/pi-unsloth-webtools
|
|
19
|
+
```
|
|
20
|
+
|
|
7
21
|
## What it does
|
|
8
22
|
|
|
9
23
|
### web_search
|
|
@@ -38,9 +52,9 @@ Port of Studio's `_fetch_page_text` / `_fetch_url_raw` pipeline:
|
|
|
38
52
|
object streams, all filters, ToUnicode fonts, encryption detection, and a
|
|
39
53
|
pymupdf4llm-style markdown layer (headings, bold/italic, code fences, links, tables)
|
|
40
54
|
with Studio's corrupted/incomplete fallback to plain text.
|
|
41
|
-
- Content sniffing: MIME allow/deny, binary magic signatures, PDF magic
|
|
42
|
-
(
|
|
43
|
-
|
|
55
|
+
- Content sniffing: MIME allow/deny, binary magic signatures, PDF magic detection, and charset
|
|
56
|
+
decoding (declared charset, BOM sniffing for UTF-8/16/32, cp1252 rescue for mislabeled
|
|
57
|
+
single-byte pages).
|
|
44
58
|
- HTML → Markdown conversion ported from Studio's dependency-free `_html_to_md.py`: headings,
|
|
45
59
|
links, emphasis, lists, tables, blockquotes, code fences, entity decoding; hidden-element
|
|
46
60
|
stripping (`hidden`, `aria-hidden`, inline styles); `<article>`/`<main>` main-content scoping
|
|
@@ -51,16 +65,8 @@ Port of Studio's `_fetch_page_text` / `_fetch_url_raw` pipeline:
|
|
|
51
65
|
- HTML entity decoding replicates CPython's `html.unescape` (full 2,231-entry HTML5 table,
|
|
52
66
|
longest-prefix rule, Windows-1252 numeric mappings), matching Studio byte-for-byte.
|
|
53
67
|
|
|
54
|
-
See `ROADMAP.md` for the remaining gaps (
|
|
55
|
-
|
|
56
|
-
## Install
|
|
57
|
-
|
|
58
|
-
```sh
|
|
59
|
-
pi install /path/to/pi-unsloth-webtools
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
or add `npm:pi-unsloth-webtools` (once published) to the `packages` array in
|
|
63
|
-
`~/.pi/agent/settings.json`.
|
|
68
|
+
See `ROADMAP.md` for the remaining gaps (per-line styling, table detection, TLS
|
|
69
|
+
fingerprinting, proxies).
|
|
64
70
|
|
|
65
71
|
## Development
|
|
66
72
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-unsloth-webtools",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Pi extension: web_search and web_fetch tools ported from the Unsloth Studio codebase (DuckDuckGo search, SSRF-safe direct fetching, HTML-to-Markdown extraction)",
|
|
6
6
|
"main": "index.ts",
|