pi-lean-portal 0.2.1 β 0.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +29 -19
- package/backends/playwright-base/playwright-plugin.ts +237 -181
- package/backends/python-adapter.ts +7 -2
- package/core/fetch-backend.ts +2 -8
- package/core/guides.ts +1 -2
- package/core/plugin-config.ts +27 -3
- package/core/router.ts +2 -1
- package/core/shared/storage-state.ts +6 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -6,9 +6,7 @@
|
|
|
6
6
|
> domain. A `/web` toggle removes the tools from the agent's context when
|
|
7
7
|
> switched off, so web browsing doesn't consume tokens on sessions that aren't
|
|
8
8
|
> doing web work. If a site blocks the shipped browsers, drop in your own
|
|
9
|
-
> backend (e.g. [Camoufox](https://github.com/daijro/camoufox))
|
|
10
|
-
> we're aware, no other Pi web plugin lets you run a browser backend you wrote
|
|
11
|
-
> yourself.
|
|
9
|
+
> backend (e.g. [Camoufox](https://github.com/daijro/camoufox)).
|
|
12
10
|
>
|
|
13
11
|
> Part of the [pi-lean-dimension](https://github.com/coreyryanhanson/pi-lean-dimension)
|
|
14
12
|
> web-tools suite. For SearXNG search support, install
|
|
@@ -42,7 +40,7 @@ npx playwright install chromium firefox
|
|
|
42
40
|
|
|
43
41
|
Once loaded, you'll see a notification like:
|
|
44
42
|
|
|
45
|
-
> π Browser extension loaded (plugins: chromium, firefox)
|
|
43
|
+
> π Browser extension loaded (plugins: chromium, firefox). Try: web-fetch for static pages or browser-navigate for interactive browsing.
|
|
46
44
|
|
|
47
45
|
The browser tools are **enabled by default**. You can:
|
|
48
46
|
|
|
@@ -122,7 +120,8 @@ which defaults to `true`).
|
|
|
122
120
|
**Auto-captured screenshots:** `browser-navigate` and `browser-snapshot` automatically
|
|
123
121
|
capture a screenshot to a temp file (`/tmp/pi-lean-portal/screenshot-<taskId>.jpg`).
|
|
124
122
|
Use the `read` tool to visually inspect the page when the accessibility tree isn't enough.
|
|
125
|
-
|
|
123
|
+
Screenshots capture the 1280Γ720 viewport (not full-page) β the same viewport the
|
|
124
|
+
browser uses for navigation.
|
|
126
125
|
|
|
127
126
|
### 1. `browser-navigate` β Visit a Page
|
|
128
127
|
|
|
@@ -201,7 +200,7 @@ Navigates back in browser history. Returns the previous page's snapshot.
|
|
|
201
200
|
browser-press key="Enter"
|
|
202
201
|
```
|
|
203
202
|
|
|
204
|
-
Useful keys: `Enter`, `Tab`, `Escape`, `ArrowDown`, `ArrowUp`,
|
|
203
|
+
Useful keys: `Enter`, `Tab`, `Escape`, `ArrowDown`, `ArrowUp`, `Backspace`.
|
|
205
204
|
|
|
206
205
|
### 8. `browser-console` β Read Console / Run JS
|
|
207
206
|
|
|
@@ -285,7 +284,7 @@ navigation guidance for specific sites using `web-learn`.
|
|
|
285
284
|
web-learn domain="reddit.com" content="β¦guidance textβ¦"
|
|
286
285
|
```
|
|
287
286
|
|
|
288
|
-
This creates a `.md` file with YAML frontmatter in
|
|
287
|
+
This creates a `.md` file with YAML frontmatter in `~/.pi/agent/pi-lean-portal/web-guides/`.
|
|
289
288
|
The guide becomes available immediately via `web-guide` and appears in the guide footer on
|
|
290
289
|
future navigations to that domain.
|
|
291
290
|
|
|
@@ -306,12 +305,13 @@ Shows everything about the browser runtime in one notification:
|
|
|
306
305
|
π Browser tools: β
on | π Learn mode: β off
|
|
307
306
|
ββββββββββββββββββββββββββββββββββββββββ
|
|
308
307
|
Status: idle
|
|
309
|
-
Plugins: chromium
|
|
308
|
+
Plugins: chromium, firefox, chromium-py (disabled), firefox-py (disabled)
|
|
309
|
+
Use web-fetch for stateless HTTP fetches.
|
|
310
310
|
Active sessions: 1
|
|
311
311
|
PW [chromium] https://example.com β Example Domain [profile: session]
|
|
312
|
-
Profiles:
|
|
313
|
-
|
|
314
|
-
|
|
312
|
+
Profiles: 1 on disk (named)
|
|
313
|
+
shopping (0.3 KB) β active
|
|
314
|
+
Session profiles: 1 (manage with /web profile)
|
|
315
315
|
```
|
|
316
316
|
|
|
317
317
|
Covers:
|
|
@@ -320,7 +320,9 @@ Covers:
|
|
|
320
320
|
- **Backend health** β idle, busy, or error state
|
|
321
321
|
- **All registered plugins** β enabled/disabled status
|
|
322
322
|
- **Active sessions** β current URL, title, profile name per session
|
|
323
|
-
- **Profiles on disk** β state size and which
|
|
323
|
+
- **Profiles on disk** β named profiles with state size and which is
|
|
324
|
+
currently active; session profiles are collapsed into a single count line
|
|
325
|
+
(inspect individually with `/web profile list`)
|
|
324
326
|
|
|
325
327
|
When `pi-lean-search` is also installed, the status bar shows two independent
|
|
326
328
|
glyphs: `β idle` (browser state) and `β searxng` (search health/state).
|
|
@@ -593,7 +595,7 @@ Size threshold for profile state warnings (default: 10 MB):
|
|
|
593
595
|
```jsonc
|
|
594
596
|
{
|
|
595
597
|
"browser": {
|
|
596
|
-
"maxStorageStateSize": 10485760
|
|
598
|
+
"maxStorageStateSize": 10485760
|
|
597
599
|
}
|
|
598
600
|
}
|
|
599
601
|
```
|
|
@@ -614,8 +616,9 @@ Size threshold for profile state warnings (default: 10 MB):
|
|
|
614
616
|
### Working with `@e` Element References
|
|
615
617
|
|
|
616
618
|
- `@e1`, `@e2`, etc. are assigned based on the accessibility tree order
|
|
617
|
-
-
|
|
618
|
-
|
|
619
|
+
- `browser-click`/`type`/`scroll` already return a fresh snapshot and cache
|
|
620
|
+
the full tree to disk β no separate `browser-snapshot` needed unless you
|
|
621
|
+
want the uncompacted tree (`full=true`) or a screenshot
|
|
619
622
|
- `browser-inspect` is cheaper than `browser-snapshot full=true` for finding
|
|
620
623
|
specific elements
|
|
621
624
|
|
|
@@ -623,8 +626,9 @@ Size threshold for profile state warnings (default: 10 MB):
|
|
|
623
626
|
|
|
624
627
|
- Snapshots are automatically compacted to ~2500 characters
|
|
625
628
|
- Very large pages (>8000 chars) preserve the top ~2000 chars
|
|
626
|
-
- The **full tree is cached to disk** at `/tmp/pi-lean-portal/snapshot-*.txt`
|
|
627
|
-
|
|
629
|
+
- The **full tree is cached to disk** at `/tmp/pi-lean-portal/snapshot-*.txt`
|
|
630
|
+
when it would otherwise be truncated β use `read` on the cache file with
|
|
631
|
+
offset/limit to retrieve the complete tree
|
|
628
632
|
- `browser-inspect text=true query="keyword"` finds specific content without
|
|
629
633
|
loading the full tree
|
|
630
634
|
|
|
@@ -636,8 +640,14 @@ When a page triggers anti-automation:
|
|
|
636
640
|
2. The **bot-detection guide** footer appears with strategies available via `web-guide`
|
|
637
641
|
3. If very few elements are detected (<5), the navigation is treated as
|
|
638
642
|
a hard failure β the agent won't try to interact with a challenge page
|
|
639
|
-
4.
|
|
640
|
-
|
|
643
|
+
4. Retry with a stealth backend β the `browser-navigate` `strategy`
|
|
644
|
+
parameter lists registered backend names; a stealth backend (e.g.
|
|
645
|
+
`strategy="camoufox"`) can pass challenges the default `chromium`/
|
|
646
|
+
`firefox` triggers. Only names listed in the `strategy` description
|
|
647
|
+
are valid β there is no `"stealth"` alias
|
|
648
|
+
5. Try `web-fetch` on the same URL β it skips JS execution, so it can
|
|
649
|
+
retrieve raw HTML on pages that block the interactive browser via
|
|
650
|
+
client-side fingerprinting (it won't help against server-side WAFs)
|
|
641
651
|
|
|
642
652
|
### Guide Creation Discipline
|
|
643
653
|
|
|
@@ -31,6 +31,7 @@ import { NAV_SETTLE } from "../../core/shared/browser-data.js";
|
|
|
31
31
|
import { join } from "node:path";
|
|
32
32
|
import { mkdirSync } from "node:fs";
|
|
33
33
|
import { persistSessionState } from "../../core/shared/storage-state.js";
|
|
34
|
+
import { loadFullConfig } from "../../core/plugin-config.js";
|
|
34
35
|
import type {
|
|
35
36
|
BrowserPlugin,
|
|
36
37
|
PluginCapabilities,
|
|
@@ -422,7 +423,12 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
422
423
|
context: BrowserContext,
|
|
423
424
|
): Promise<{ cookies: unknown[]; origins: unknown[] } | undefined> {
|
|
424
425
|
const session = sessionManager.getSession(taskId);
|
|
425
|
-
return persistSessionState(
|
|
426
|
+
return persistSessionState(
|
|
427
|
+
session,
|
|
428
|
+
() => context.storageState(),
|
|
429
|
+
"",
|
|
430
|
+
loadFullConfig().browser.maxStorageStateSize,
|
|
431
|
+
);
|
|
426
432
|
}
|
|
427
433
|
|
|
428
434
|
private async takeSnapshot(
|
|
@@ -656,19 +662,28 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
656
662
|
};
|
|
657
663
|
}
|
|
658
664
|
|
|
659
|
-
return this._logOp(
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
|
|
665
|
+
return this._logOp(
|
|
666
|
+
"snapshot",
|
|
667
|
+
{ taskId },
|
|
668
|
+
async () => {
|
|
669
|
+
const {
|
|
670
|
+
snapshot: snapText,
|
|
671
|
+
elementCount,
|
|
672
|
+
dialogEvents,
|
|
673
|
+
} = await this.takeSnapshot(taskId, page);
|
|
674
|
+
return {
|
|
675
|
+
success: true,
|
|
676
|
+
snapshot: snapText,
|
|
677
|
+
elementCount,
|
|
678
|
+
dialogEvents,
|
|
679
|
+
};
|
|
680
|
+
},
|
|
681
|
+
(r) => ({
|
|
682
|
+
elementCount: r.elementCount,
|
|
683
|
+
dialogEvents: r.dialogEvents.length,
|
|
684
|
+
fingerprint: r.snapshot.slice(0, 16),
|
|
685
|
+
}),
|
|
686
|
+
);
|
|
672
687
|
}
|
|
673
688
|
|
|
674
689
|
// ββ Interaction ββββββββββββββββββββββββββββββββββββββββββββ
|
|
@@ -713,39 +728,46 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
713
728
|
};
|
|
714
729
|
}
|
|
715
730
|
|
|
716
|
-
return this._logOp(
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
|
|
728
|
-
|
|
729
|
-
|
|
730
|
-
|
|
731
|
-
|
|
732
|
-
|
|
731
|
+
return this._logOp(
|
|
732
|
+
"click",
|
|
733
|
+
{ taskId, ref, role: node.role, name: node.name },
|
|
734
|
+
async () => {
|
|
735
|
+
try {
|
|
736
|
+
const urlBefore = page.url();
|
|
737
|
+
await locator.click({ timeout: 5000 });
|
|
738
|
+
|
|
739
|
+
// Wait for potential navigation to settle (replaces fixed sleep)
|
|
740
|
+
const { navigated: _navigated } = await waitForNavigationSettle(
|
|
741
|
+
page,
|
|
742
|
+
urlBefore,
|
|
743
|
+
);
|
|
744
|
+
|
|
745
|
+
const newUrl = page.url();
|
|
746
|
+
const newTitle = await page.title();
|
|
747
|
+
sessionManager.updateSession(taskId, {
|
|
748
|
+
currentUrl: newUrl,
|
|
749
|
+
currentTitle: newTitle,
|
|
750
|
+
});
|
|
733
751
|
|
|
734
|
-
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
752
|
+
// Auto-snapshot
|
|
753
|
+
const snapResult = await this.takeSnapshot(taskId, page);
|
|
754
|
+
|
|
755
|
+
return {
|
|
756
|
+
success: true,
|
|
757
|
+
newUrl,
|
|
758
|
+
newTitle,
|
|
759
|
+
snapshot: snapResult.snapshot,
|
|
760
|
+
elementCount: snapResult.elementCount,
|
|
761
|
+
dialogEvents: snapResult.dialogEvents,
|
|
762
|
+
};
|
|
763
|
+
} catch (err: unknown) {
|
|
764
|
+
return {
|
|
765
|
+
success: false,
|
|
766
|
+
error: `Click failed: ${err instanceof Error ? err.message : String(err)}`,
|
|
767
|
+
};
|
|
768
|
+
}
|
|
769
|
+
},
|
|
770
|
+
);
|
|
749
771
|
}
|
|
750
772
|
|
|
751
773
|
async type(
|
|
@@ -792,27 +814,32 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
792
814
|
};
|
|
793
815
|
}
|
|
794
816
|
|
|
795
|
-
return this._logOp(
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
803
|
-
|
|
804
|
-
|
|
805
|
-
|
|
806
|
-
|
|
807
|
-
|
|
808
|
-
|
|
809
|
-
|
|
810
|
-
|
|
811
|
-
|
|
812
|
-
|
|
813
|
-
|
|
814
|
-
|
|
815
|
-
|
|
817
|
+
return this._logOp(
|
|
818
|
+
"type",
|
|
819
|
+
{ taskId, ref, role: node.role, name: node.name },
|
|
820
|
+
async () => {
|
|
821
|
+
try {
|
|
822
|
+
await locator.click({ timeout: 5000 }); // Focus first
|
|
823
|
+
await locator.fill(text);
|
|
824
|
+
|
|
825
|
+
// Auto-snapshot
|
|
826
|
+
const snapResult = await this.takeSnapshot(taskId, page);
|
|
827
|
+
|
|
828
|
+
return {
|
|
829
|
+
success: true,
|
|
830
|
+
snapshot: snapResult.snapshot,
|
|
831
|
+
elementCount: snapResult.elementCount,
|
|
832
|
+
dialogEvents: snapResult.dialogEvents,
|
|
833
|
+
};
|
|
834
|
+
} catch (err: unknown) {
|
|
835
|
+
return {
|
|
836
|
+
success: false,
|
|
837
|
+
error: `Type failed: ${err instanceof Error ? err.message : String(err)}`,
|
|
838
|
+
};
|
|
839
|
+
}
|
|
840
|
+
},
|
|
841
|
+
(r) => ({ elementCount: r.elementCount }),
|
|
842
|
+
);
|
|
816
843
|
}
|
|
817
844
|
|
|
818
845
|
async scroll(
|
|
@@ -824,29 +851,34 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
824
851
|
return { success: false, error: "No active session" };
|
|
825
852
|
}
|
|
826
853
|
|
|
827
|
-
return this._logOp(
|
|
828
|
-
|
|
829
|
-
|
|
830
|
-
|
|
831
|
-
|
|
832
|
-
|
|
833
|
-
|
|
834
|
-
|
|
835
|
-
|
|
836
|
-
|
|
837
|
-
|
|
838
|
-
|
|
839
|
-
|
|
840
|
-
|
|
841
|
-
|
|
842
|
-
|
|
843
|
-
|
|
844
|
-
|
|
845
|
-
|
|
846
|
-
|
|
847
|
-
|
|
848
|
-
|
|
849
|
-
|
|
854
|
+
return this._logOp(
|
|
855
|
+
"scroll",
|
|
856
|
+
{ taskId, direction },
|
|
857
|
+
async () => {
|
|
858
|
+
try {
|
|
859
|
+
const delta = direction === "down" ? 800 : -800;
|
|
860
|
+
await page.evaluate(
|
|
861
|
+
`window.scrollBy({ top: ${delta}, behavior: "smooth" })`,
|
|
862
|
+
);
|
|
863
|
+
await page.waitForTimeout(200);
|
|
864
|
+
|
|
865
|
+
const snapResult = await this.takeSnapshot(taskId, page);
|
|
866
|
+
|
|
867
|
+
return {
|
|
868
|
+
success: true,
|
|
869
|
+
snapshot: snapResult.snapshot,
|
|
870
|
+
elementCount: snapResult.elementCount,
|
|
871
|
+
dialogEvents: snapResult.dialogEvents,
|
|
872
|
+
};
|
|
873
|
+
} catch (err: unknown) {
|
|
874
|
+
return {
|
|
875
|
+
success: false,
|
|
876
|
+
error: `Scroll failed: ${err instanceof Error ? err.message : String(err)}`,
|
|
877
|
+
};
|
|
878
|
+
}
|
|
879
|
+
},
|
|
880
|
+
(r) => ({ elementCount: r.elementCount }),
|
|
881
|
+
);
|
|
850
882
|
}
|
|
851
883
|
|
|
852
884
|
async goBack(taskId: string): Promise<InteractionResult> {
|
|
@@ -855,35 +887,40 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
855
887
|
return { success: false, error: "No active session" };
|
|
856
888
|
}
|
|
857
889
|
|
|
858
|
-
return this._logOp(
|
|
859
|
-
|
|
860
|
-
|
|
861
|
-
|
|
862
|
-
|
|
863
|
-
|
|
864
|
-
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
|
|
890
|
+
return this._logOp(
|
|
891
|
+
"goBack",
|
|
892
|
+
{ taskId },
|
|
893
|
+
async () => {
|
|
894
|
+
try {
|
|
895
|
+
await page.goBack({ waitUntil: "networkidle" });
|
|
896
|
+
await page.waitForTimeout(300);
|
|
897
|
+
|
|
898
|
+
const newUrl = page.url();
|
|
899
|
+
const newTitle = await page.title();
|
|
900
|
+
sessionManager.updateSession(taskId, {
|
|
901
|
+
currentUrl: newUrl,
|
|
902
|
+
currentTitle: newTitle,
|
|
903
|
+
});
|
|
871
904
|
|
|
872
|
-
|
|
873
|
-
|
|
874
|
-
|
|
875
|
-
|
|
876
|
-
|
|
877
|
-
|
|
878
|
-
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
884
|
-
|
|
885
|
-
|
|
886
|
-
|
|
905
|
+
const snapResult = await this.takeSnapshot(taskId, page);
|
|
906
|
+
|
|
907
|
+
return {
|
|
908
|
+
success: true,
|
|
909
|
+
newUrl,
|
|
910
|
+
newTitle,
|
|
911
|
+
snapshot: snapResult.snapshot,
|
|
912
|
+
elementCount: snapResult.elementCount,
|
|
913
|
+
dialogEvents: snapResult.dialogEvents,
|
|
914
|
+
};
|
|
915
|
+
} catch (err: unknown) {
|
|
916
|
+
return {
|
|
917
|
+
success: false,
|
|
918
|
+
error: `GoBack failed: ${err instanceof Error ? err.message : String(err)}`,
|
|
919
|
+
};
|
|
920
|
+
}
|
|
921
|
+
},
|
|
922
|
+
(r) => ({ elementCount: r.elementCount }),
|
|
923
|
+
);
|
|
887
924
|
}
|
|
888
925
|
|
|
889
926
|
async press(taskId: string, key: string): Promise<InteractionResult> {
|
|
@@ -892,41 +929,50 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
892
929
|
return { success: false, error: "No active session" };
|
|
893
930
|
}
|
|
894
931
|
|
|
895
|
-
return this._logOp(
|
|
896
|
-
|
|
897
|
-
|
|
898
|
-
|
|
899
|
-
|
|
900
|
-
|
|
901
|
-
|
|
902
|
-
|
|
903
|
-
|
|
904
|
-
|
|
905
|
-
|
|
906
|
-
|
|
907
|
-
|
|
908
|
-
|
|
909
|
-
|
|
910
|
-
|
|
911
|
-
|
|
912
|
-
|
|
913
|
-
|
|
932
|
+
return this._logOp(
|
|
933
|
+
"press",
|
|
934
|
+
{ taskId, key },
|
|
935
|
+
async () => {
|
|
936
|
+
try {
|
|
937
|
+
const urlBefore = page.url();
|
|
938
|
+
await page.keyboard.press(key);
|
|
939
|
+
|
|
940
|
+
// Wait for potential navigation to settle (replaces fixed sleep).
|
|
941
|
+
// Shorter nav timeout since Enter-on-link nav is typically fast.
|
|
942
|
+
const { navigated: _navigated } = await waitForNavigationSettle(
|
|
943
|
+
page,
|
|
944
|
+
urlBefore,
|
|
945
|
+
{
|
|
946
|
+
navTimeoutMs: 3000,
|
|
947
|
+
},
|
|
948
|
+
);
|
|
949
|
+
|
|
950
|
+
const newUrl = page.url();
|
|
951
|
+
const newTitle = await page.title();
|
|
952
|
+
sessionManager.updateSession(taskId, {
|
|
953
|
+
currentUrl: newUrl,
|
|
954
|
+
currentTitle: newTitle,
|
|
955
|
+
});
|
|
914
956
|
|
|
915
|
-
|
|
916
|
-
|
|
917
|
-
|
|
918
|
-
|
|
919
|
-
|
|
920
|
-
|
|
921
|
-
|
|
922
|
-
|
|
923
|
-
|
|
924
|
-
|
|
925
|
-
|
|
926
|
-
|
|
927
|
-
|
|
928
|
-
|
|
929
|
-
|
|
957
|
+
const snapResult = await this.takeSnapshot(taskId, page);
|
|
958
|
+
|
|
959
|
+
return {
|
|
960
|
+
success: true,
|
|
961
|
+
newUrl,
|
|
962
|
+
newTitle,
|
|
963
|
+
snapshot: snapResult.snapshot,
|
|
964
|
+
elementCount: snapResult.elementCount,
|
|
965
|
+
dialogEvents: snapResult.dialogEvents,
|
|
966
|
+
};
|
|
967
|
+
} catch (err: unknown) {
|
|
968
|
+
return {
|
|
969
|
+
success: false,
|
|
970
|
+
error: `Press failed: ${err instanceof Error ? err.message : String(err)}`,
|
|
971
|
+
};
|
|
972
|
+
}
|
|
973
|
+
},
|
|
974
|
+
(r) => ({ elementCount: r.elementCount }),
|
|
975
|
+
);
|
|
930
976
|
}
|
|
931
977
|
|
|
932
978
|
// ββ Media ββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
@@ -1002,18 +1048,23 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
1002
1048
|
return { success: false, cookies: [], error: "No active session" };
|
|
1003
1049
|
}
|
|
1004
1050
|
|
|
1005
|
-
return this._logOp(
|
|
1006
|
-
|
|
1007
|
-
|
|
1008
|
-
|
|
1009
|
-
|
|
1010
|
-
|
|
1011
|
-
success:
|
|
1012
|
-
|
|
1013
|
-
|
|
1014
|
-
|
|
1015
|
-
|
|
1016
|
-
|
|
1051
|
+
return this._logOp(
|
|
1052
|
+
"getCookies",
|
|
1053
|
+
{ taskId },
|
|
1054
|
+
async () => {
|
|
1055
|
+
try {
|
|
1056
|
+
const cookies = await entry.context.cookies(urls);
|
|
1057
|
+
return { success: true, cookies };
|
|
1058
|
+
} catch (err: unknown) {
|
|
1059
|
+
return {
|
|
1060
|
+
success: false,
|
|
1061
|
+
cookies: [],
|
|
1062
|
+
error: err instanceof Error ? err.message : String(err),
|
|
1063
|
+
};
|
|
1064
|
+
}
|
|
1065
|
+
},
|
|
1066
|
+
(r) => ({ count: r.cookies.length }),
|
|
1067
|
+
);
|
|
1017
1068
|
}
|
|
1018
1069
|
|
|
1019
1070
|
async addCookies(taskId: string, cookies: Cookie[]): Promise<ResultBase> {
|
|
@@ -1072,23 +1123,28 @@ export abstract class PlaywrightPluginBase implements BrowserPlugin {
|
|
|
1072
1123
|
};
|
|
1073
1124
|
}
|
|
1074
1125
|
|
|
1075
|
-
return this._logOp(
|
|
1076
|
-
|
|
1077
|
-
|
|
1078
|
-
|
|
1079
|
-
|
|
1080
|
-
|
|
1081
|
-
|
|
1082
|
-
|
|
1083
|
-
|
|
1084
|
-
|
|
1085
|
-
|
|
1086
|
-
|
|
1087
|
-
|
|
1088
|
-
|
|
1089
|
-
|
|
1090
|
-
|
|
1091
|
-
|
|
1126
|
+
return this._logOp(
|
|
1127
|
+
"getStorageState",
|
|
1128
|
+
{ taskId },
|
|
1129
|
+
async () => {
|
|
1130
|
+
try {
|
|
1131
|
+
const state = await entry.context.storageState();
|
|
1132
|
+
return {
|
|
1133
|
+
success: true,
|
|
1134
|
+
cookies: state.cookies,
|
|
1135
|
+
origins: state.origins,
|
|
1136
|
+
};
|
|
1137
|
+
} catch (err: unknown) {
|
|
1138
|
+
return {
|
|
1139
|
+
success: false,
|
|
1140
|
+
cookies: [],
|
|
1141
|
+
origins: [],
|
|
1142
|
+
error: err instanceof Error ? err.message : String(err),
|
|
1143
|
+
};
|
|
1144
|
+
}
|
|
1145
|
+
},
|
|
1146
|
+
(r) => ({ cookies: r.cookies.length, origins: r.origins.length }),
|
|
1147
|
+
);
|
|
1092
1148
|
}
|
|
1093
1149
|
|
|
1094
1150
|
// ββ Per-task cleanup βββββββββββββββββββββββββββββββββββββββ
|
|
@@ -10,7 +10,10 @@ import { join, delimiter as pathDelimiter } from "node:path";
|
|
|
10
10
|
|
|
11
11
|
import { sessionManager } from "../core/shared/session-manager.js";
|
|
12
12
|
import { persistSessionState } from "../core/shared/storage-state.js";
|
|
13
|
-
import {
|
|
13
|
+
import {
|
|
14
|
+
DEFAULT_BACKENDS_ROOT,
|
|
15
|
+
loadFullConfig,
|
|
16
|
+
} from "../core/plugin-config.js";
|
|
14
17
|
|
|
15
18
|
import {
|
|
16
19
|
DEFAULT_CAPABILITIES,
|
|
@@ -957,7 +960,8 @@ export class PythonPluginAdapter implements BrowserPlugin {
|
|
|
957
960
|
skip_default_viewport: !!raw.skip_default_viewport,
|
|
958
961
|
skip_networkidle: !!raw.skip_networkidle,
|
|
959
962
|
wrap_mw_eval_in_eval: !!raw.wrap_mw_eval_in_eval,
|
|
960
|
-
csp_safe_readonly_via_init_script:
|
|
963
|
+
csp_safe_readonly_via_init_script:
|
|
964
|
+
!!raw.csp_safe_readonly_via_init_script,
|
|
961
965
|
}),
|
|
962
966
|
(error): QuirksDescriptor => ({
|
|
963
967
|
success: false,
|
|
@@ -1116,6 +1120,7 @@ export class PythonPluginAdapter implements BrowserPlugin {
|
|
|
1116
1120
|
};
|
|
1117
1121
|
},
|
|
1118
1122
|
"via Python bridge",
|
|
1123
|
+
loadFullConfig().browser.maxStorageStateSize,
|
|
1119
1124
|
);
|
|
1120
1125
|
}
|
|
1121
1126
|
|
package/core/fetch-backend.ts
CHANGED
|
@@ -250,14 +250,8 @@ function writeFetchTempFile(content: string, taskId: string): string {
|
|
|
250
250
|
|
|
251
251
|
function trackFetchFile(taskId: string, filePath: string): void {
|
|
252
252
|
const existing = activeFetchFiles.get(taskId) ?? [];
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
rmSync(oldPath, { force: true });
|
|
256
|
-
} catch {
|
|
257
|
-
/* best-effort */
|
|
258
|
-
}
|
|
259
|
-
}
|
|
260
|
-
activeFetchFiles.set(taskId, [filePath]);
|
|
253
|
+
if (!existing.includes(filePath)) existing.push(filePath);
|
|
254
|
+
activeFetchFiles.set(taskId, existing);
|
|
261
255
|
}
|
|
262
256
|
|
|
263
257
|
interface CappedFetchContent {
|
package/core/guides.ts
CHANGED
|
@@ -90,8 +90,7 @@ export const BUILTIN_GUIDES: Record<string, Guide> = {
|
|
|
90
90
|
"",
|
|
91
91
|
"### Backend Strategy",
|
|
92
92
|
"- The default `chromium` / `firefox` backends are detected by many anti-automation systems",
|
|
93
|
-
'- A stealth browser backend may be available β
|
|
94
|
-
"- If no stealth backend is configured, this will fail with a clear error β no harm in trying",
|
|
93
|
+
'- A stealth browser backend may be available β the `browser-navigate` `strategy` parameter lists the registered backend names; retry with a stealth backend name listed there (e.g. `strategy="camoufox"`) if the default is blocked',
|
|
95
94
|
"",
|
|
96
95
|
"### Verifying the Page After a Challenge",
|
|
97
96
|
'- Use `browser-inspect role="dialog"` to check if a challenge dialog is still present β cheaper than a full snapshot',
|
package/core/plugin-config.ts
CHANGED
|
@@ -8,6 +8,11 @@
|
|
|
8
8
|
import { existsSync } from "node:fs";
|
|
9
9
|
import { join, isAbsolute } from "node:path";
|
|
10
10
|
import { USER_BACKENDS_DIR } from "./shared/paths.js";
|
|
11
|
+
import {
|
|
12
|
+
sanitizeProfileName,
|
|
13
|
+
DEFAULT_MAX_STORAGE_STATE_SIZE,
|
|
14
|
+
} from "./shared/storage-state.js";
|
|
15
|
+
import { readMergedSettings } from "./shared/settings-reader.js";
|
|
11
16
|
// βββ Plugin Config types ββββββββββββββββββββββββββββββββββββββββββ
|
|
12
17
|
|
|
13
18
|
/** A single plugin entry from the user's settings.json */
|
|
@@ -34,9 +39,6 @@ export interface PluginDetection {
|
|
|
34
39
|
/** Absolute or relative path to the entry point */
|
|
35
40
|
entryPoint: string;
|
|
36
41
|
}
|
|
37
|
-
import { sanitizeProfileName } from "./shared/storage-state.js";
|
|
38
|
-
import { readMergedSettings } from "./shared/settings-reader.js";
|
|
39
|
-
|
|
40
42
|
// βββ Config types & loading ββββββββββββββββββββββββββββββββββββββββ
|
|
41
43
|
|
|
42
44
|
/**
|
|
@@ -51,6 +53,12 @@ export interface BrowserConfig {
|
|
|
51
53
|
* - A named profile string (e.g. "shopping", "work")
|
|
52
54
|
*/
|
|
53
55
|
defaultProfile: "none" | "session" | string;
|
|
56
|
+
/**
|
|
57
|
+
* Byte threshold before a profile's saved storage state logs a size
|
|
58
|
+
* warning (still saved). Defaults to 10 MB (`DEFAULT_MAX_STORAGE_STATE_SIZE`).
|
|
59
|
+
* Set via `browser.maxStorageStateSize` in settings.json.
|
|
60
|
+
*/
|
|
61
|
+
maxStorageStateSize: number;
|
|
54
62
|
}
|
|
55
63
|
|
|
56
64
|
/** Raw plugin entry from settings.json (before validation) */
|
|
@@ -87,6 +95,7 @@ function parseBrowserConfig(
|
|
|
87
95
|
// Defaults
|
|
88
96
|
const config: BrowserConfig = {
|
|
89
97
|
defaultProfile: "session",
|
|
98
|
+
maxStorageStateSize: DEFAULT_MAX_STORAGE_STATE_SIZE,
|
|
90
99
|
};
|
|
91
100
|
|
|
92
101
|
if (!raw) return config;
|
|
@@ -114,6 +123,21 @@ function parseBrowserConfig(
|
|
|
114
123
|
}
|
|
115
124
|
}
|
|
116
125
|
|
|
126
|
+
// ββ maxStorageStateSize βββββββββββββββββββββββββββββββββββββ
|
|
127
|
+
if (raw.maxStorageStateSize !== undefined) {
|
|
128
|
+
if (
|
|
129
|
+
typeof raw.maxStorageStateSize === "number" &&
|
|
130
|
+
Number.isFinite(raw.maxStorageStateSize) &&
|
|
131
|
+
raw.maxStorageStateSize > 0
|
|
132
|
+
) {
|
|
133
|
+
config.maxStorageStateSize = Math.floor(raw.maxStorageStateSize);
|
|
134
|
+
} else {
|
|
135
|
+
errors.push(
|
|
136
|
+
`browser.maxStorageStateSize: expected a positive finite number, got ${JSON.stringify(raw.maxStorageStateSize)}`,
|
|
137
|
+
);
|
|
138
|
+
}
|
|
139
|
+
}
|
|
140
|
+
|
|
117
141
|
// Log validation errors (non-fatal, but surfaced in return)
|
|
118
142
|
for (const err of errors) {
|
|
119
143
|
console.warn(`[pi-lean-portal] Config warning: ${err}`);
|
package/core/router.ts
CHANGED
|
@@ -580,7 +580,8 @@ export async function navigate(
|
|
|
580
580
|
elementCount: 0,
|
|
581
581
|
error:
|
|
582
582
|
"Page appears to be blocked by anti-automation protection. " +
|
|
583
|
-
"
|
|
583
|
+
"Retry `browser-navigate` with a stealth backend name from the `strategy` parameter's listed backends, " +
|
|
584
|
+
"or try `web-fetch` for the raw HTML (it skips JS fingerprinting but loses interactivity).",
|
|
584
585
|
backendUsed: plugin.name,
|
|
585
586
|
botDetectionWarning: true,
|
|
586
587
|
} as NavigateResult & {
|
|
@@ -32,7 +32,7 @@ export const PROFILE_DIR = join(PORTAL_DATA_DIR, "browser-state");
|
|
|
32
32
|
const STORAGE_STATE_VERSION = 1;
|
|
33
33
|
|
|
34
34
|
/** Default size limit (10 MB) before a warning is logged on save. */
|
|
35
|
-
const DEFAULT_MAX_STORAGE_STATE_SIZE = 10 * 1024 * 1024;
|
|
35
|
+
export const DEFAULT_MAX_STORAGE_STATE_SIZE = 10 * 1024 * 1024;
|
|
36
36
|
|
|
37
37
|
/** Profile name validation regex. */
|
|
38
38
|
const PROFILE_NAME_RE = /^[a-zA-Z0-9_-]{1,64}$/;
|
|
@@ -468,6 +468,9 @@ export function deleteStorageState(profileName: string): void {
|
|
|
468
468
|
* @param getStorageState - Callback that produces the raw storage state.
|
|
469
469
|
* @param viaLabel - Optional backend label inserted into the warning
|
|
470
470
|
* (e.g. `"via Python bridge"`) for diagnostic output.
|
|
471
|
+
* @param maxSizeBytes - Optional byte threshold for the size warning
|
|
472
|
+
* (defaults to `DEFAULT_MAX_STORAGE_STATE_SIZE` when
|
|
473
|
+
* omitted).
|
|
471
474
|
* @returns The raw state object, or `undefined` if the session is
|
|
472
475
|
* non-persistent or the save failed.
|
|
473
476
|
*/
|
|
@@ -475,12 +478,13 @@ export async function persistSessionState(
|
|
|
475
478
|
session: { persistState?: boolean; profileName?: string } | null | undefined,
|
|
476
479
|
getStorageState: () => Promise<{ cookies: unknown[]; origins: unknown[] }>,
|
|
477
480
|
viaLabel: string = "",
|
|
481
|
+
maxSizeBytes?: number,
|
|
478
482
|
): Promise<{ cookies: unknown[]; origins: unknown[] } | undefined> {
|
|
479
483
|
if (!session?.persistState) return undefined;
|
|
480
484
|
const name = session.profileName ?? "default";
|
|
481
485
|
try {
|
|
482
486
|
const state = await getStorageState();
|
|
483
|
-
saveStorageState(name, state);
|
|
487
|
+
saveStorageState(name, state, maxSizeBytes);
|
|
484
488
|
return state;
|
|
485
489
|
} catch (err) {
|
|
486
490
|
console.warn(
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-lean-portal",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.3",
|
|
4
4
|
"description": "Pi extension. Interactive web browsing for Pi β a /web toggle removes the tools from context when switched off, Playwright Chromium/Firefox deliver accessibility-tree snapshots, persistent profiles, cookies, and domain-matched guides; custom/stealth backends (Camoufox) plug in when a site blocks the shipped browsers.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi-package",
|