smart-web-mcp 0.8.3 → 0.8.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -4,6 +4,61 @@ All notable changes to this project will be documented in this file.
4
4
 
5
5
  The format is based on Keep a Changelog[^1] and this project uses SemVer.
6
6
 
7
+ ## Unreleased
8
+
9
+ ### Changed
10
+
11
+ - removed the unused Claude marketplace plugin wrapper files and repo references so the project only documents the direct MCP setup paths it still supports
12
+ - release-please now bumps the MCP registry versions inside `server.json`, and the tag workflow remains the single owner of npm, MCP registry, and GitHub release publication
13
+ - release-please stays on the existing `vX.Y.Z` tag format instead of switching to component-prefixed tags when manifest mode is enabled
14
+ - release PRs now stay open for batching, while the push-to-main workflow creates the release tag only after an operator merges that release PR
15
+ - release PR tag pushes now use the GitHub App token instead of `github.token`, so the downstream Release workflow actually fires on the created tag
16
+ - MCP registry metadata now points at the current jojo-labs GitHub namespace so release publication no longer targets the retired rich-jojo server slug
17
+
18
+ ## [0.8.8](https://github.com/jojo-labs/smart-web/compare/v0.8.7...v0.8.8) (2026-04-13)
19
+
20
+
21
+ ### Bug Fixes
22
+
23
+ * **ci:** trigger smart-web releases from tags ([#85](https://github.com/jojo-labs/smart-web/issues/85)) ([ee46946](https://github.com/jojo-labs/smart-web/commit/ee469464a71a757e2dec83be0eadcf1710002ebe))
24
+
25
+ ## [0.8.7](https://github.com/jojo-labs/smart-web/compare/v0.8.6...v0.8.7) (2026-04-13)
26
+
27
+ ### Bug Fixes
28
+
29
+ - **ci:** batch smart-web releases behind tags ([#82](https://github.com/jojo-labs/smart-web/issues/82)) ([b7d706f](https://github.com/jojo-labs/smart-web/commit/b7d706fa3a39ff96651dc02c9922f16328ad6c50))
30
+ - **ci:** dispatch release PR checks ([#74](https://github.com/jojo-labs/smart-web/issues/74)) ([4d8ac1d](https://github.com/jojo-labs/smart-web/commit/4d8ac1d41a44800128dce1b7d3914bc6a58e306e))
31
+ - **ci:** keep smart-web release tags stable ([#81](https://github.com/jojo-labs/smart-web/issues/81)) ([3ad4d06](https://github.com/jojo-labs/smart-web/commit/3ad4d0637b86eeda3db6c14ae8b3f865ed861ff4))
32
+ - **ci:** let tags own smart-web releases ([#77](https://github.com/jojo-labs/smart-web/issues/77)) ([16d619e](https://github.com/jojo-labs/smart-web/commit/16d619e1ea32f67b17ae6053ebdfd136c028c7ca))
33
+ - **ci:** publish smart-web under jojo-labs ([#84](https://github.com/jojo-labs/smart-web/issues/84)) ([602ca0f](https://github.com/jojo-labs/smart-web/commit/602ca0f0d3ca04fe8bb503e8f692727d077b22cc))
34
+ - **ci:** sync smart-web release metadata ([#80](https://github.com/jojo-labs/smart-web/issues/80)) ([38f7e40](https://github.com/jojo-labs/smart-web/commit/38f7e4039813775ffc49235141677a0b89864c81))
35
+
36
+ ## [0.8.6](https://github.com/jojo-labs/smart-web/compare/v0.8.5...v0.8.6) (2026-04-12)
37
+
38
+ ### Bug Fixes
39
+
40
+ - **ci:** grant dependabot caller permissions ([#72](https://github.com/jojo-labs/smart-web/issues/72)) ([e57fd32](https://github.com/jojo-labs/smart-web/commit/e57fd329de150e6700bc7e62c6bba78f4ac6046a))
41
+ - **ci:** grant release-please caller permissions ([#71](https://github.com/jojo-labs/smart-web/issues/71)) ([290ed1a](https://github.com/jojo-labs/smart-web/commit/290ed1afdb5962e9b6b14a4808509b9c44c331ba))
42
+ - **ci:** remove unsupported github-actions Dependabot cooldown ([#69](https://github.com/jojo-labs/smart-web/issues/69)) ([360c9e9](https://github.com/jojo-labs/smart-web/commit/360c9e983cdd3c958a12b473b3236d9c543e0c7f))
43
+
44
+ ## [0.8.5](https://github.com/jojo-labs/smart-web/compare/v0.8.4...v0.8.5) (2026-04-11)
45
+
46
+ ### Bug Fixes
47
+
48
+ - **ci:** tighten PR policy and automerge cleanup ([#49](https://github.com/jojo-labs/smart-web/issues/49)) ([c30353e](https://github.com/jojo-labs/smart-web/commit/c30353e21472b5e3999f3691020e0b9394f5a0a0))
49
+
50
+ ## [0.8.4](https://github.com/rich-jojo/smart-web/compare/v0.8.3...v0.8.4) (2026-04-09)
51
+
52
+ ### Bug Fixes
53
+
54
+ - add public fallbacks for linkedin authwalls ([#43](https://github.com/rich-jojo/smart-web/issues/43)) ([31eb2f1](https://github.com/rich-jojo/smart-web/commit/31eb2f112fe7607c1e50bbd8e66be7037f30d542))
55
+
56
+ ## [0.8.4](https://github.com/rich-jojo/smart-web/compare/v0.8.3...v0.8.4) (2026-04-09)
57
+
58
+ ### Fixed
59
+
60
+ - LinkedIn post authwall fetches now run a narrow public fallback ladder: optional Jina Reader reference extraction when enabled, public archive snapshot recovery when available, and search-indexed reference metadata before returning a blocked result
61
+
7
62
  ## [0.8.3](https://github.com/rich-jojo/smart-web/compare/v0.8.2...v0.8.3) (2026-04-09)
8
63
 
9
64
  ### Fixed
package/README.md CHANGED
@@ -12,6 +12,8 @@ Local MCP for agentic web retrieval.
12
12
 
13
13
  It is designed for **agents and MCP hosts**, not for manual browser automation. The job is to return structured retrieval output first, then signal when a browser-task runtime would be a better next step.
14
14
 
15
+ If a page is login-gated, keep the login UI, session persistence, and capture flow in a companion runtime instead of moving that stateful behavior into the `smart-web` core.
16
+
15
17
  - npm: `https://www.npmjs.com/package/smart-web-mcp`
16
18
  - MCP Registry: `io.github.rich-jojo/smart-web`
17
19
  - GitHub: `https://github.com/rich-jojo/smart-web`
@@ -24,6 +26,7 @@ If `smart-web` returns reproducible incorrect or misleading output, open a GitHu
24
26
  - explicit routing: URL -> `smartfetch`, query -> `smartsearch`, known site -> `smartcrawl`
25
27
  - up-front smartfetch acquisition lanes instead of broad direct -> browser retry chains
26
28
  - optional Jina Reader relay for weak generic/article fetches when you explicitly enable it
29
+ - LinkedIn authwall handling now prefers compliant public fallbacks such as archives and search-indexed reference metadata before giving up
27
30
  - structured output with `assessment` and `pipeline` fields for host-side decision making
28
31
  - local-first defaults with privacy-aware search/fetch behavior
29
32
  - useful normalization for common public surfaces instead of raw HTML dumps
@@ -56,11 +59,11 @@ Common high-signal fields include:
56
59
  - `post`, `thread`, `comments`: normalized content surfaces
57
60
  - `assets` and `download`: file-like outputs when present
58
61
 
59
- This lets hosts decide whether deterministic retrieval was good enough or whether they should escalate to a browser-task runtime.[^2]
62
+ This lets hosts decide whether deterministic retrieval was good enough or whether they should escalate to an auth-aware companion flow or a broader browser-task runtime.[^2]
60
63
 
61
64
  ## Supported high-signal surfaces
62
65
 
63
- - communities: Reddit, DCInside, LinkedIn public posts, Algumon
66
+ - communities: Reddit, DCInside, LinkedIn public posts with public fallback recovery, Algumon
64
67
  - reference/article pages: NamuWiki, Wikipedia, Naver Blog, Tistory, Velog
65
68
  - media/social: YouTube, X, Threads, Instagram, Telegram
66
69
  - commerce pages: Amazon, Coupang, Danawa, Aladin, AliExpress
@@ -86,7 +89,7 @@ When `smartfetch` or `smartcrawl` needs Playwright-backed page loading, `smart-w
86
89
  claude mcp add smart-web -- npx -y smart-web-mcp
87
90
  ```
88
91
 
89
- The repo also ships a small Claude plugin wrapper for marketplace-style distribution.
92
+ For a checked-out local repo, use [examples/claude.mcp.json](examples/claude.mcp.json).
90
93
 
91
94
  ### Codex
92
95
 
@@ -74,8 +74,8 @@ export function assessSmartfetchOutput(input) {
74
74
  ];
75
75
  const needsHandoff = confidence === "low" || authRequiredLikely || interactiveFlowLikely;
76
76
  const suppressRecommendedHandoff = input.provider.id === "linkedin"
77
- && /^authwall_/.test(postStatus)
78
- && /^(playwright_|undetected_chromedriver)/.test(input.active.method);
77
+ && ((/^authwall_/.test(postStatus) && /^(playwright_|undetected_chromedriver)/.test(input.active.method))
78
+ || postStatus === "reference_only");
79
79
  const recommendedHandoff = needsHandoff && !suppressRecommendedHandoff
80
80
  ? createHandoffSuggestion({
81
81
  tool: "web_task",
@@ -1 +1 @@
1
- {"version":3,"file":"pipeline.js","sourceRoot":"","sources":["../../src/smartfetch/pipeline.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,uBAAuB,EAAE,kBAAkB,EAAE,yBAAyB,EAAE,4BAA4B,EAAE,MAAM,kBAAkB,CAAA;AACzJ,OAAO,EAAE,iBAAiB,EAA+F,MAAM,cAAc,CAAA;AAG7I,SAAS,QAAQ,CAAC,KAAc;IAC9B,OAAO,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAA;AACnC,CAAC;AAED,SAAS,eAAe,CAAC,QAAoC;IAC3D,OAAO,QAAQ,CAAC,YAAY,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,YAAY,IAAI,EAAE,CAAC,CAAA;AAChE,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,KAOjC;IACC,OAAO;QACL,MAAM,EAAE,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,IAAa,CAAC,CAAC,CAAC,QAAiB;QAC3D,gBAAgB,EAAE,KAAK,CAAC,eAAe;QACvC,YAAY,EAAE,KAAK,CAAC,WAAW;QAC/B,aAAa,EAAE,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE;QAC/C,SAAS,EAAE,KAAK,CAAC,QAAQ,IAAI,KAAK,CAAC,GAAG;QACtC,GAAG,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,cAAc,EAAE,KAAK,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACxE,CAAA;AACH,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,KASjC;IACC,MAAM,OAAO,GAAG,KAAK,CAAC,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAA;IAC9C,OAAO;QACL,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,IAAa,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,SAAkB,CAAC,CAAC,CAAC,QAAiB;QACrG,QAAQ,EAAE,OAAO,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,MAAM,IAAI,MAAM;QACrD,QAAQ,EAAE,OAAO;QACjB,YAAY,EAAE,KAAK,CAAC,MAAM,CAAC,MAAM;QACjC,SAAS,EAAE,KAAK,CAAC,MAAM,CAAC,SAAS,IAAI,EAAE;QACvC,GAAG,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,CAAC,EAAE,oBAAoB,EAAE,KAAK,CAAC,kBAAkB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACvF,GAAG,CAAC,KAAK,CAAC,WAAW,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACjE,GAAG,CAAC,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,KAAK,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC9D,GAAG,CAAC,KAAK,CAAC,eAAe,CAAC,CAAC,CAAC,EAAE,gBAAgB,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC5D,YAAY,EAAE,KAAK,CAAC,WAAW;QAC/B,YAAY,EAAE,KAAK,CAAC,WAAW;KAChC,CAAA;AACH,CAAC;AAED,MAAM,UAAU,mBAAmB,CAAC,KAGnC;IACC,OAAO;QACL,MAAM,EAAE,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,SAAkB,CAAC,CAAC,CAAC,IAAa;QAC1D,WAAW,EAAE,KAAK,CAAC,QAAQ,CAAC,EAAE;QAC9B,eAAe,EAAE,KAAK,CAAC,QAAQ,CAAC,YAAY;QAC5C,YAAY,EAAE,KAAK,CAAC,QAAQ,CAAC,YAAY;KAC1C,CAAA;AACH,CAAC;AAED,MAAM,UAAU,sBAAsB,CAAC,KAOtC;IACC,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,CAAC,MAAqB,CAAA;IACjD,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,IAAI,EAAE,CAA4B,CAAA;IACjE,MAAM,UAAU,GAAG,QAAQ,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA;IACxC,MAAM,QAAQ,GAAG,QAAQ,CAAC,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,WAAW,IAAI,IAAI,CAAC,KAAK,CAAC,CAAA;IACtE,MAAM,aAAa,GAAG,4BAA4B,CAAC;QACjD,GAAG,MAAM;QACT,UAAU,IAAI,UAAU,KAAK,IAAI,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS;QAC1D,KAAK,CAAC,MAAM,CAAC,KAAK;KACnB,CAAC,CAAA;IACF,MAAM,oBAAoB,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,cAAc,CAAC,MAAM,GAAG,CAAC,CAAA;IAC9J,MAAM,kBAAkB,GAAG,OAAO,CAAC,KAAK,CAAC,QAAQ,CAAC,MAAM,CAAC,cAAc,IAAI,CAAC,OAAO,CAAC,aAAa,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAA;IACrJ,MAAM,kBAAkB,GAAG,OAAO,CAChC,kBAAkB;WACf,kBAAkB,CAAC,GAAG,aAAa,IAAI,UAAU,IAAI,QAAQ,EAAE,CAAC;WAChE,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,2BAA2B,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC,CAC3F,CAAA;IACD,MAAM,qBAAqB,GAAG,OAAO,CACnC,CAAC,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,QAAQ,CAAC,qBAAqB,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,CAAC,oBAAoB,IAAI,OAAO,CAAC,aAAa,CAAC,CAAC,CAAC;WACvI,yBAAyB,CAAC,GAAG,aAAa,IAAI,QAAQ,IAAI,KAAK,CAAC,MAAM,CAAC,OAAO,EAAE,CAAC;WACjF,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,IAAI,CAAC,kBAAkB,CAAC;WACzC,CAAC,iBAAiB,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,WAAW,KAAK,SAAS,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAC3H,CAAA;IACD,MAAM,UAAU,GAAG,kBAAkB,IAAI,CAAC,CAAC,oBAAoB,IAAI,KAAK,CAAC,MAAM,CAAC,OAAO,CAAC;QACtF,CAAC,CAAC,KAAK;QACP,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC,IAAI,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,QAAQ,CAAC,kBAAkB,CAAC;YACrG,CAAC,CAAC,QAAQ;YACV,CAAC,CAAC,MAAM,CAAA;IAEZ,MAAM,OAAO,GAAG;QACd,GAAG,eAAe,CAAC,KAAK,CAAC,QAAQ,CAAC;QAClC,KAAK,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,gBAAgB,CAAC,CAAC,CAAC,iBAAiB;QAC3D,oBAAoB,CAAC,CAAC,CAAC,iBAAiB,CAAC,CAAC,CAAC,gBAAgB;QAC3D,kBAAkB,CAAC,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,eAAe;QACpD,qBAAqB,CAAC,CAAC,CAAC,oBAAoB,CAAC,CAAC,CAAC,sBAAsB;QACrE,aAAa,CAAC,CAAC,CAAC,WAAW,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE;KAChD,CAAA;IAED,MAAM,YAAY,GAAG,UAAU,KAAK,KAAK,IAAI,kBAAkB,IAAI,qBAAqB,CAAA;IACxF,MAAM,0BAA0B,GAAG,KAAK,CAAC,QAAQ,CAAC,EAAE,KAAK,UAAU;WAC9D,YAAY,CAAC,IAAI,CAAC,UAAU,CAAC;WAC7B,wCAAwC,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,CAAA;IACvE,MAAM,kBAAkB,GAAG,YAAY,IAAI,CAAC,0BAA0B;QACpE,CAAC,CAAC,uBAAuB,CAAC;YACtB,IAAI,EAAE,UAAU;YAChB,IAAI,EAAE,MAAM;YACZ,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,MAAM,EAAE,kBAAkB;gBACxB,CAAC,CAAC,8DAA8D;gBAChE,CAAC,CAAC,qBAAqB;oBACrB,CAAC,CAAC,gEAAgE;oBAClE,CAAC,CAAC,wDAAwD;YAC9D,IAAI,EAAE,QAAQ,KAAK,CAAC,GAAG,8GAA8G;SACtI,CAAC;QACJ,CAAC,CAAC,IAAI,CAAA;IAER,OAAO,gBAAgB,CAAC;QACtB,UAAU;QACV,cAAc,EAAE,aAAa;QAC7B,oBAAoB,EAAE,kBAAkB;QACxC,uBAAuB,EAAE,qBAAqB;QAC9C,OAAO;QACP,mBAAmB,EAAE,kBAAkB;KACxC,CAAC,CAAA;AACJ,CAAC"}
1
+ {"version":3,"file":"pipeline.js","sourceRoot":"","sources":["../../src/smartfetch/pipeline.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,uBAAuB,EAAE,kBAAkB,EAAE,yBAAyB,EAAE,4BAA4B,EAAE,MAAM,kBAAkB,CAAA;AACzJ,OAAO,EAAE,iBAAiB,EAA+F,MAAM,cAAc,CAAA;AAG7I,SAAS,QAAQ,CAAC,KAAc;IAC9B,OAAO,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAA;AACnC,CAAC;AAED,SAAS,eAAe,CAAC,QAAoC;IAC3D,OAAO,QAAQ,CAAC,YAAY,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,YAAY,IAAI,EAAE,CAAC,CAAA;AAChE,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,KAOjC;IACC,OAAO;QACL,MAAM,EAAE,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,IAAa,CAAC,CAAC,CAAC,QAAiB;QAC3D,gBAAgB,EAAE,KAAK,CAAC,eAAe;QACvC,YAAY,EAAE,KAAK,CAAC,WAAW;QAC/B,aAAa,EAAE,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE;QAC/C,SAAS,EAAE,KAAK,CAAC,QAAQ,IAAI,KAAK,CAAC,GAAG;QACtC,GAAG,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,cAAc,EAAE,KAAK,CAAC,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACxE,CAAA;AACH,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,KASjC;IACC,MAAM,OAAO,GAAG,KAAK,CAAC,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAA;IAC9C,OAAO;QACL,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,IAAa,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,SAAkB,CAAC,CAAC,CAAC,QAAiB;QACrG,QAAQ,EAAE,OAAO,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,MAAM,IAAI,MAAM;QACrD,QAAQ,EAAE,OAAO;QACjB,YAAY,EAAE,KAAK,CAAC,MAAM,CAAC,MAAM;QACjC,SAAS,EAAE,KAAK,CAAC,MAAM,CAAC,SAAS,IAAI,EAAE;QACvC,GAAG,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,CAAC,EAAE,oBAAoB,EAAE,KAAK,CAAC,kBAAkB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACvF,GAAG,CAAC,KAAK,CAAC,WAAW,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,KAAK,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACjE,GAAG,CAAC,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,KAAK,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC9D,GAAG,CAAC,KAAK,CAAC,eAAe,CAAC,CAAC,CAAC,EAAE,gBAAgB,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC5D,YAAY,EAAE,KAAK,CAAC,WAAW;QAC/B,YAAY,EAAE,KAAK,CAAC,WAAW;KAChC,CAAA;AACH,CAAC;AAED,MAAM,UAAU,mBAAmB,CAAC,KAGnC;IACC,OAAO;QACL,MAAM,EAAE,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,SAAkB,CAAC,CAAC,CAAC,IAAa;QAC1D,WAAW,EAAE,KAAK,CAAC,QAAQ,CAAC,EAAE;QAC9B,eAAe,EAAE,KAAK,CAAC,QAAQ,CAAC,YAAY;QAC5C,YAAY,EAAE,KAAK,CAAC,QAAQ,CAAC,YAAY;KAC1C,CAAA;AACH,CAAC;AAED,MAAM,UAAU,sBAAsB,CAAC,KAOtC;IACC,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,CAAC,MAAqB,CAAA;IACjD,MAAM,IAAI,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,IAAI,EAAE,CAA4B,CAAA;IACjE,MAAM,UAAU,GAAG,QAAQ,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA;IACxC,MAAM,QAAQ,GAAG,QAAQ,CAAC,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,WAAW,IAAI,IAAI,CAAC,KAAK,CAAC,CAAA;IACtE,MAAM,aAAa,GAAG,4BAA4B,CAAC;QACjD,GAAG,MAAM;QACT,UAAU,IAAI,UAAU,KAAK,IAAI,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS;QAC1D,KAAK,CAAC,MAAM,CAAC,KAAK;KACnB,CAAC,CAAA;IACF,MAAM,oBAAoB,GAAG,OAAO,CAAC,QAAQ,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,cAAc,CAAC,MAAM,GAAG,CAAC,CAAA;IAC9J,MAAM,kBAAkB,GAAG,OAAO,CAAC,KAAK,CAAC,QAAQ,CAAC,MAAM,CAAC,cAAc,IAAI,CAAC,OAAO,CAAC,aAAa,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAA;IACrJ,MAAM,kBAAkB,GAAG,OAAO,CAChC,kBAAkB;WACf,kBAAkB,CAAC,GAAG,aAAa,IAAI,UAAU,IAAI,QAAQ,EAAE,CAAC;WAChE,MAAM,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,2BAA2B,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC,CAC3F,CAAA;IACD,MAAM,qBAAqB,GAAG,OAAO,CACnC,CAAC,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,QAAQ,CAAC,qBAAqB,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,CAAC,oBAAoB,IAAI,OAAO,CAAC,aAAa,CAAC,CAAC,CAAC;WACvI,yBAAyB,CAAC,GAAG,aAAa,IAAI,QAAQ,IAAI,KAAK,CAAC,MAAM,CAAC,OAAO,EAAE,CAAC;WACjF,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,IAAI,CAAC,kBAAkB,CAAC;WACzC,CAAC,iBAAiB,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,WAAW,KAAK,SAAS,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAC3H,CAAA;IACD,MAAM,UAAU,GAAG,kBAAkB,IAAI,CAAC,CAAC,oBAAoB,IAAI,KAAK,CAAC,MAAM,CAAC,OAAO,CAAC;QACtF,CAAC,CAAC,KAAK;QACP,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC,IAAI,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,QAAQ,CAAC,kBAAkB,CAAC;YACrG,CAAC,CAAC,QAAQ;YACV,CAAC,CAAC,MAAM,CAAA;IAEZ,MAAM,OAAO,GAAG;QACd,GAAG,eAAe,CAAC,KAAK,CAAC,QAAQ,CAAC;QAClC,KAAK,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,gBAAgB,CAAC,CAAC,CAAC,iBAAiB;QAC3D,oBAAoB,CAAC,CAAC,CAAC,iBAAiB,CAAC,CAAC,CAAC,gBAAgB;QAC3D,kBAAkB,CAAC,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,eAAe;QACpD,qBAAqB,CAAC,CAAC,CAAC,oBAAoB,CAAC,CAAC,CAAC,sBAAsB;QACrE,aAAa,CAAC,CAAC,CAAC,WAAW,aAAa,EAAE,CAAC,CAAC,CAAC,EAAE;KAChD,CAAA;IAED,MAAM,YAAY,GAAG,UAAU,KAAK,KAAK,IAAI,kBAAkB,IAAI,qBAAqB,CAAA;IACxF,MAAM,0BAA0B,GAAG,KAAK,CAAC,QAAQ,CAAC,EAAE,KAAK,UAAU;WAC9D,CACD,CAAC,YAAY,CAAC,IAAI,CAAC,UAAU,CAAC,IAAI,wCAAwC,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;eAClG,UAAU,KAAK,gBAAgB,CACnC,CAAA;IACH,MAAM,kBAAkB,GAAG,YAAY,IAAI,CAAC,0BAA0B;QACpE,CAAC,CAAC,uBAAuB,CAAC;YACtB,IAAI,EAAE,UAAU;YAChB,IAAI,EAAE,MAAM;YACZ,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,MAAM,EAAE,kBAAkB;gBACxB,CAAC,CAAC,8DAA8D;gBAChE,CAAC,CAAC,qBAAqB;oBACrB,CAAC,CAAC,gEAAgE;oBAClE,CAAC,CAAC,wDAAwD;YAC9D,IAAI,EAAE,QAAQ,KAAK,CAAC,GAAG,8GAA8G;SACtI,CAAC;QACJ,CAAC,CAAC,IAAI,CAAA;IAER,OAAO,gBAAgB,CAAC;QACtB,UAAU;QACV,cAAc,EAAE,aAAa;QAC7B,oBAAoB,EAAE,kBAAkB;QACxC,uBAAuB,EAAE,qBAAqB;QAC9C,OAAO;QACP,mBAAmB,EAAE,kBAAkB;KACxC,CAAC,CAAA;AACJ,CAAC"}
@@ -1,11 +1,18 @@
1
- import { dedupeUrls } from "../../shared.js";
2
- import { extractLinkedInPostData, isLinkedInPostUrl, isLinkedInProfileUrl, looksLikeLinkedInAuthwallPage } from "../../linkedin.js";
1
+ import { extractLinkedInActivityId, extractLinkedInPostData, isLinkedInPostUrl, isLinkedInProfileUrl, looksLikeLinkedInAuthwallPage, } from "../../linkedin.js";
2
+ import { dedupeUrls, envEnabled, extractUrls } from "../../shared.js";
3
+ import { runSmartsearch } from "../../smartsearch.js";
4
+ import { fetchProviderJson, fetchProviderText } from "../provider-policy.js";
5
+ const USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36";
6
+ const ARCHIVE_MD_HOSTS = ["archive.md", "archive.ph", "archive.today", "archive.is"];
3
7
  function truncate(value, maxLength) {
4
8
  const text = String(value || "").trim();
5
9
  if (text.length <= maxLength)
6
10
  return text;
7
11
  return `${text.slice(0, maxLength - 1).trimEnd()}…`;
8
12
  }
13
+ function normalizeSpace(value) {
14
+ return String(value || "").replace(/\s+/g, " ").trim();
15
+ }
9
16
  function profileLabel(url) {
10
17
  try {
11
18
  const slug = new URL(url).pathname.split("/").filter(Boolean).at(-1) || "profile";
@@ -15,6 +22,33 @@ function profileLabel(url) {
15
22
  return "LinkedIn profile";
16
23
  }
17
24
  }
25
+ function archiveFallbackEnabled() {
26
+ return envEnabled("SMARTFETCH_ENABLE_ARCHIVE_FALLBACK", true);
27
+ }
28
+ function waybackEnabled() {
29
+ return envEnabled("SMARTFETCH_ENABLE_WAYBACK", true);
30
+ }
31
+ function archiveMdEnabled() {
32
+ return envEnabled("SMARTFETCH_ENABLE_ARCHIVE_MD", true);
33
+ }
34
+ function jinaReaderEnabled() {
35
+ return envEnabled("SMARTFETCH_ENABLE_JINA_READER", false, { private: false });
36
+ }
37
+ function cleanMarkdownText(value) {
38
+ return String(value || "")
39
+ .replace(/\r\n/g, "\n")
40
+ .replace(/!\[[^\]]*\]\(([^)]+)\)/g, " ")
41
+ .replace(/\[([^\]]+)\]\((https?:\/\/[^)]+)\)/g, "$1 $2")
42
+ .replace(/^>\s*/gm, "")
43
+ .replace(/^#{1,6}\s*/gm, "")
44
+ .replace(/[`*_~]+/g, "")
45
+ .replace(/\n{3,}/g, "\n\n")
46
+ .trim();
47
+ }
48
+ function markdownTitle(value) {
49
+ const lines = String(value || "").split(/\n+/).map((line) => normalizeSpace(line)).filter(Boolean);
50
+ return lines.find((line) => line.length <= 180) || lines[0] || "";
51
+ }
18
52
  function linkedinBlockedResult(input) {
19
53
  return {
20
54
  post: {
@@ -31,6 +65,302 @@ function linkedinBlockedResult(input) {
31
65
  method: input.context.active.method,
32
66
  };
33
67
  }
68
+ function buildLinkedInPostResult(input) {
69
+ const hasBody = Boolean(input.post.text);
70
+ const comments = input.post.comments.slice(0, 5).map((item) => ({
71
+ ...item,
72
+ text: truncate(item.text, 180),
73
+ }));
74
+ const note = normalizeSpace(input.sourceNote || "");
75
+ return {
76
+ post: {
77
+ url: input.context.url,
78
+ canonical_url: input.post.canonical_url,
79
+ title: input.post.title,
80
+ description: input.post.description,
81
+ text: truncate([note, input.post.text].filter(Boolean).join("\n\n"), 2500),
82
+ author: input.post.author,
83
+ author_profile_url: input.post.author_profile_url,
84
+ author_image: input.post.author_image,
85
+ published: input.post.published,
86
+ comment_count: input.post.comment_count,
87
+ reaction_count: input.post.reaction_count,
88
+ images: input.post.images,
89
+ attachment_links: input.post.outbound_links,
90
+ extractor: input.post.found ? "linkedin_json_ld" : "fallback",
91
+ status: hasBody ? "ok" : "blocked_or_unavailable",
92
+ ...(input.sourceUrl ? { source_snapshot_url: input.sourceUrl } : {}),
93
+ },
94
+ thread: [],
95
+ comments,
96
+ outbound_links: dedupeUrls([input.sourceUrl || "", ...input.post.outbound_links].filter(Boolean)),
97
+ partial: !input.post.found || !hasBody,
98
+ errors: [],
99
+ method: input.method,
100
+ };
101
+ }
102
+ function buildLinkedInReferenceResult(input) {
103
+ const title = normalizeSpace(input.title) || "LinkedIn post — reference metadata";
104
+ const text = truncate(normalizeSpace(input.text), 1800);
105
+ const links = dedupeUrls([input.context.url, input.sourceUrl, ...(input.links || [])].filter(Boolean));
106
+ return {
107
+ post: {
108
+ url: input.context.url,
109
+ canonical_url: input.sourceUrl,
110
+ title,
111
+ description: `${input.sourceLabel} fallback because direct LinkedIn extraction hit an authwall.`,
112
+ text: [
113
+ "Direct LinkedIn extraction hit an authwall for this post.",
114
+ `Reference source: ${input.sourceLabel}`,
115
+ text,
116
+ "This fallback came from public relays, archives, or search-indexed metadata and is not a directly verified LinkedIn render.",
117
+ ].filter(Boolean).join("\n\n"),
118
+ status: "reference_only",
119
+ extractor: input.method,
120
+ },
121
+ thread: (input.thread || []).slice(0, 5).map((item, index) => ({
122
+ index: index + 1,
123
+ title: item.title,
124
+ url: item.url,
125
+ snippet: item.snippet,
126
+ ...(item.source ? { source: item.source } : {}),
127
+ })),
128
+ comments: [],
129
+ outbound_links: links,
130
+ partial: true,
131
+ errors: [],
132
+ method: input.method,
133
+ };
134
+ }
135
+ function parseArchiveTimemap(value) {
136
+ const snapshots = [];
137
+ const regex = /<(https?:\/\/archive\.(?:md|ph|today|is)\/[0-9]{8,14}\/(?:https?:\/\/)?[^>]+)>;\s*rel="(?:first\s+|last\s+)?memento"/gi;
138
+ for (const match of value.matchAll(regex)) {
139
+ const snapshotUrl = normalizeSpace(match[1] || "");
140
+ const timestamp = normalizeSpace(snapshotUrl.match(/archive\.(?:md|ph|today|is)\/([0-9]{8,14})\//i)?.[1] || "");
141
+ if (!snapshotUrl || !timestamp)
142
+ continue;
143
+ snapshots.push({ url: snapshotUrl, timestamp });
144
+ }
145
+ return snapshots.sort((left, right) => right.timestamp.localeCompare(left.timestamp));
146
+ }
147
+ function parseLinkedInRelayCandidate(content, pageUrl) {
148
+ const text = cleanMarkdownText(content);
149
+ const title = markdownTitle(content);
150
+ const links = dedupeUrls(extractUrls(content));
151
+ if (!title || text.length < 120)
152
+ return null;
153
+ if (looksLikeLinkedInAuthwallPage({ html: content, resolvedUrl: pageUrl }))
154
+ return null;
155
+ return buildLinkedInReferenceResult({
156
+ context: {
157
+ url: pageUrl,
158
+ resolvedUrl: pageUrl,
159
+ target: "linkedin_post",
160
+ timeoutMs: 0,
161
+ active: { ok: true, method: "jina_reader", content, links, final_url: pageUrl },
162
+ errors: [],
163
+ },
164
+ method: "linkedin_jina_reference",
165
+ sourceLabel: "Jina Reader relay",
166
+ sourceUrl: pageUrl,
167
+ title,
168
+ text,
169
+ links,
170
+ });
171
+ }
172
+ function snapshotNote(label, sourceUrl) {
173
+ return label === "Wayback"
174
+ ? `Archived snapshot source: ${sourceUrl}`
175
+ : `Archive snapshot source: ${sourceUrl}`;
176
+ }
177
+ async function fetchLinkedInSnapshot(input) {
178
+ const snapshot = await fetchProviderText(input.snapshotUrl, input.context.timeoutMs, {
179
+ headers: {
180
+ accept: "text/html,text/plain,*/*",
181
+ "user-agent": USER_AGENT,
182
+ },
183
+ }, {
184
+ mode: "relay",
185
+ sourceUrl: input.context.url,
186
+ });
187
+ if (!snapshot.ok)
188
+ return null;
189
+ if (looksLikeLinkedInAuthwallPage({ html: snapshot.text, resolvedUrl: input.snapshotUrl }))
190
+ return null;
191
+ const post = extractLinkedInPostData({
192
+ html: snapshot.text,
193
+ pageUrl: input.context.url,
194
+ pageLinks: snapshot.text ? extractUrls(snapshot.text) : [],
195
+ });
196
+ if (post.text && (post.found || post.text.length >= 160)) {
197
+ return buildLinkedInPostResult({
198
+ context: input.context,
199
+ method: input.method,
200
+ post,
201
+ sourceNote: snapshotNote(input.sourceLabel, input.snapshotUrl),
202
+ sourceUrl: input.snapshotUrl,
203
+ });
204
+ }
205
+ return null;
206
+ }
207
+ async function tryLinkedInJinaReference(context) {
208
+ if (!jinaReaderEnabled())
209
+ return null;
210
+ const relay = await fetchProviderText(`https://r.jina.ai/${context.url}`, context.timeoutMs, {
211
+ headers: {
212
+ accept: "text/plain,text/markdown,*/*",
213
+ "user-agent": USER_AGENT,
214
+ "x-respond-with": "markdown",
215
+ },
216
+ }, {
217
+ mode: "relay",
218
+ sourceUrl: context.url,
219
+ });
220
+ if (!relay.ok)
221
+ return null;
222
+ const reference = parseLinkedInRelayCandidate(relay.text, context.url);
223
+ if (!reference)
224
+ return null;
225
+ return { ...reference, method: "linkedin_jina_reference" };
226
+ }
227
+ async function tryLinkedInWayback(context) {
228
+ if (!archiveFallbackEnabled() || !waybackEnabled())
229
+ return null;
230
+ const availability = await fetchProviderJson(`https://archive.org/wayback/available?url=${encodeURIComponent(context.url)}`, context.timeoutMs, {
231
+ headers: {
232
+ accept: "application/json,text/plain,*/*",
233
+ "user-agent": USER_AGENT,
234
+ },
235
+ }, {
236
+ mode: "relay",
237
+ sourceUrl: context.url,
238
+ });
239
+ if (!availability.ok)
240
+ return null;
241
+ const snapshotUrl = normalizeSpace(String(availability.data?.archived_snapshots?.closest?.url || ""));
242
+ if (!snapshotUrl)
243
+ return null;
244
+ return fetchLinkedInSnapshot({ context, snapshotUrl, method: "linkedin_wayback_archive", sourceLabel: "Wayback" });
245
+ }
246
+ async function tryLinkedInArchiveMd(context) {
247
+ if (!archiveFallbackEnabled() || !archiveMdEnabled())
248
+ return null;
249
+ for (const host of ARCHIVE_MD_HOSTS) {
250
+ const timemap = await fetchProviderText(`https://${host}/timemap/${context.url}`, context.timeoutMs, {
251
+ headers: {
252
+ accept: "application/link-format,text/plain,*/*",
253
+ "user-agent": USER_AGENT,
254
+ },
255
+ }, {
256
+ mode: "relay",
257
+ sourceUrl: context.url,
258
+ });
259
+ if (!timemap.ok)
260
+ continue;
261
+ const snapshotUrl = parseArchiveTimemap(timemap.text)[0]?.url || "";
262
+ if (!snapshotUrl)
263
+ continue;
264
+ const result = await fetchLinkedInSnapshot({ context, snapshotUrl, method: "linkedin_archive_md", sourceLabel: host });
265
+ if (result)
266
+ return result;
267
+ }
268
+ return null;
269
+ }
270
+ function linkedInPostSlugWords(url) {
271
+ try {
272
+ const tail = decodeURIComponent(new URL(url).pathname.split("/").filter(Boolean).at(-1) || "");
273
+ return tail
274
+ .replace(/activity-\d+/i, " ")
275
+ .replace(/share-\d+/i, " ")
276
+ .replace(/[0-9]+/g, " ")
277
+ .replace(/[-_]+/g, " ")
278
+ .replace(/\bshare\b/gi, " ")
279
+ .replace(/\s+/g, " ")
280
+ .trim();
281
+ }
282
+ catch {
283
+ return "";
284
+ }
285
+ }
286
+ function linkedInPostSearchQueries(url) {
287
+ const activityId = extractLinkedInActivityId(url);
288
+ const slugWords = linkedInPostSlugWords(url);
289
+ return Array.from(new Set([
290
+ activityId ? `site:linkedin.com/posts ${activityId} ${slugWords}`.trim() : "",
291
+ activityId ? `site:linkedin.com/feed/update ${activityId} ${slugWords}`.trim() : "",
292
+ slugWords ? `site:linkedin.com/posts ${slugWords}`.trim() : "",
293
+ ].filter(Boolean)));
294
+ }
295
+ function scoreLinkedInSearchResult(url, item) {
296
+ const activityId = extractLinkedInActivityId(url);
297
+ const resultActivityId = extractLinkedInActivityId(item.url);
298
+ const title = normalizeSpace(item.title.replace(/\s*\|\s*LinkedIn.*$/i, ""));
299
+ const snippet = normalizeSpace(item.snippet);
300
+ let score = 0;
301
+ if (resultActivityId && activityId && resultActivityId === activityId)
302
+ score += 80;
303
+ if (isLinkedInPostUrl(item.url))
304
+ score += 20;
305
+ if (snippet.length >= 120)
306
+ score += 20;
307
+ if (snippet.length >= 240)
308
+ score += 10;
309
+ if (/agree & join linkedin|sign in to view more content|this post is unavailable/i.test(`${title} ${snippet}`))
310
+ score -= 120;
311
+ return { score, title, snippet };
312
+ }
313
+ async function tryLinkedInSearchReference(context) {
314
+ const queries = linkedInPostSearchQueries(context.url);
315
+ if (queries.length === 0)
316
+ return null;
317
+ const searchRunner = context.runSmartsearch || runSmartsearch;
318
+ const gathered = [];
319
+ for (const query of queries) {
320
+ try {
321
+ const response = await searchRunner({ query, numResults: 8, timeoutMs: Math.min(context.timeoutMs, 12000), searchType: "fast" });
322
+ if (Array.isArray(response.results))
323
+ gathered.push(...response.results);
324
+ }
325
+ catch {
326
+ // Ignore individual search failure and continue the ladder.
327
+ }
328
+ }
329
+ const ranked = gathered
330
+ .map((item) => {
331
+ const scored = scoreLinkedInSearchResult(context.url, item);
332
+ return {
333
+ item,
334
+ ...scored,
335
+ };
336
+ })
337
+ .filter((item) => item.score >= 60 && item.title && item.snippet)
338
+ .sort((left, right) => right.score - left.score);
339
+ if (ranked.length === 0)
340
+ return null;
341
+ const best = ranked[0];
342
+ return buildLinkedInReferenceResult({
343
+ context,
344
+ method: "linkedin_post_reference_search",
345
+ sourceLabel: `search index (${best.item.engine})`,
346
+ sourceUrl: best.item.url,
347
+ title: best.title,
348
+ text: best.snippet,
349
+ links: [best.item.url],
350
+ thread: ranked.slice(0, 5).map((entry) => ({
351
+ title: entry.title,
352
+ url: entry.item.url,
353
+ snippet: entry.snippet,
354
+ source: `reference:${entry.item.engine}`,
355
+ })),
356
+ });
357
+ }
358
+ async function tryLinkedInPublicFallback(context) {
359
+ return await tryLinkedInJinaReference(context)
360
+ || await tryLinkedInWayback(context)
361
+ || await tryLinkedInArchiveMd(context)
362
+ || await tryLinkedInSearchReference(context);
363
+ }
34
364
  export const linkedinProvider = {
35
365
  id: "linkedin",
36
366
  matches: (url, target) => target === "linkedin_post" || (target === "generic" && (isLinkedInPostUrl(url) || isLinkedInProfileUrl(url))),
@@ -47,6 +377,9 @@ export const linkedinProvider = {
47
377
  });
48
378
  }
49
379
  if (looksLikeLinkedInAuthwallPage({ html: context.active.content, resolvedUrl: context.resolvedUrl })) {
380
+ const fallback = await tryLinkedInPublicFallback(context);
381
+ if (fallback)
382
+ return fallback;
50
383
  return linkedinBlockedResult({
51
384
  url: context.url,
52
385
  title: "LinkedIn post unavailable",
@@ -62,36 +395,11 @@ export const linkedinProvider = {
62
395
  pageUrl: context.resolvedUrl || context.url,
63
396
  pageLinks: context.active.links,
64
397
  });
65
- const hasBody = Boolean(post.text);
66
- const comments = post.comments.slice(0, 5).map((item) => ({
67
- ...item,
68
- text: truncate(item.text, 180),
69
- }));
70
- return {
71
- post: {
72
- url: context.url,
73
- canonical_url: post.canonical_url,
74
- title: post.title,
75
- description: post.description,
76
- text: truncate(post.text, 2500),
77
- author: post.author,
78
- author_profile_url: post.author_profile_url,
79
- author_image: post.author_image,
80
- published: post.published,
81
- comment_count: post.comment_count,
82
- reaction_count: post.reaction_count,
83
- images: post.images,
84
- attachment_links: post.outbound_links,
85
- extractor: post.found ? "linkedin_json_ld" : "fallback",
86
- status: hasBody ? "ok" : "blocked_or_unavailable",
87
- },
88
- thread: [],
89
- comments,
90
- outbound_links: dedupeUrls(post.outbound_links),
91
- partial: !post.found || !hasBody,
92
- errors: context.errors,
398
+ return buildLinkedInPostResult({
399
+ context,
93
400
  method: context.active.method,
94
- };
401
+ post,
402
+ });
95
403
  },
96
404
  };
97
405
  //# sourceMappingURL=linkedin.js.map
@@ -1 +1 @@
1
- {"version":3,"file":"linkedin.js","sourceRoot":"","sources":["../../../src/smartfetch/providers/linkedin.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAA;AAC5C,OAAO,EAAE,uBAAuB,EAAE,iBAAiB,EAAE,oBAAoB,EAAE,6BAA6B,EAAE,MAAM,mBAAmB,CAAA;AAGnI,SAAS,QAAQ,CAAC,KAAa,EAAE,SAAiB;IAChD,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAA;IACvC,IAAI,IAAI,CAAC,MAAM,IAAI,SAAS;QAAE,OAAO,IAAI,CAAA;IACzC,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,SAAS,GAAG,CAAC,CAAC,CAAC,OAAO,EAAE,GAAG,CAAA;AACrD,CAAC;AAED,SAAS,YAAY,CAAC,GAAW;IAC/B,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,SAAS,CAAA;QACjF,OAAO,IAAI,CAAC,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,IAAI,IAAI,CAAA;IACnD,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,kBAAkB,CAAA;IAC3B,CAAC;AACH,CAAC;AAED,SAAS,qBAAqB,CAAC,KAQ9B;IACC,OAAO;QACL,IAAI,EAAE;YACJ,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,KAAK,EAAE,KAAK,CAAC,KAAK;YAClB,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,MAAM,EAAE,KAAK,CAAC,MAAM;SACrB;QACD,MAAM,EAAE,EAAE;QACV,QAAQ,EAAE,EAAE;QACZ,cAAc,EAAE,EAAE;QAClB,OAAO,EAAE,IAAI;QACb,MAAM,EAAE,CAAC,GAAG,KAAK,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,QAAQ,EAAE,aAAsB,EAAE,IAAI,EAAE,KAAK,CAAC,SAAS,EAAE,OAAO,EAAE,KAAK,CAAC,YAAY,EAAE,CAAC;QAC3H,MAAM,EAAE,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,MAAM;KACpC,CAAA;AACH,CAAC;AAED,MAAM,CAAC,MAAM,gBAAgB,GAAuB;IAClD,EAAE,EAAE,UAAU;IACd,OAAO,EAAE,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,CAAC,MAAM,KAAK,eAAe,IAAI,CAAC,MAAM,KAAK,SAAS,IAAI,CAAC,iBAAiB,CAAC,GAAG,CAAC,IAAI,oBAAoB,CAAC,GAAG,CAAC,CAAC,CAAC;IACvI,KAAK,CAAC,SAAS,CAAC,OAA0B;QACxC,IAAI,oBAAoB,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YACtC,OAAO,qBAAqB,CAAC;gBAC3B,GAAG,EAAE,OAAO,CAAC,GAAG;gBAChB,KAAK,EAAE,qBAAqB,YAAY,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE;gBACvD,IAAI,EAAE,mMAAmM;gBACzM,MAAM,EAAE,kBAAkB;gBAC1B,SAAS,EAAE,2BAA2B;gBACtC,YAAY,EAAE,wDAAwD;gBACtE,OAAO;aACR,CAAC,CAAA;QACJ,CAAC;QAED,IAAI,6BAA6B,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,OAAO,EAAE,WAAW,EAAE,OAAO,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;YACtG,OAAO,qBAAqB,CAAC;gBAC3B,GAAG,EAAE,OAAO,CAAC,GAAG;gBAChB,KAAK,EAAE,2BAA2B;gBAClC,IAAI,EAAE,sHAAsH;gBAC5H,MAAM,EAAE,eAAe;gBACvB,SAAS,EAAE,wBAAwB;gBACnC,YAAY,EAAE,6DAA6D;gBAC3E,OAAO;aACR,CAAC,CAAA;QACJ,CAAC;QAED,MAAM,IAAI,GAAG,uBAAuB,CAAC;YACnC,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,OAAO;YAC5B,OAAO,EAAE,OAAO,CAAC,WAAW,IAAI,OAAO,CAAC,GAAG;YAC3C,SAAS,EAAE,OAAO,CAAC,MAAM,CAAC,KAAK;SAChC,CAAC,CAAA;QAEF,MAAM,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;QAClC,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;YACxD,GAAG,IAAI;YACP,IAAI,EAAE,QAAQ,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,CAAC;SAC/B,CAAC,CAAC,CAAA;QACH,OAAO;YACL,IAAI,EAAE;gBACJ,GAAG,EAAE,OAAO,CAAC,GAAG;gBAChB,aAAa,EAAE,IAAI,CAAC,aAAa;gBACjC,KAAK,EAAE,IAAI,CAAC,KAAK;gBACjB,WAAW,EAAE,IAAI,CAAC,WAAW;gBAC7B,IAAI,EAAE,QAAQ,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC;gBAC/B,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,kBAAkB,EAAE,IAAI,CAAC,kBAAkB;gBAC3C,YAAY,EAAE,IAAI,CAAC,YAAY;gBAC/B,SAAS,EAAE,IAAI,CAAC,SAAS;gBACzB,aAAa,EAAE,IAAI,CAAC,aAAa;gBACjC,cAAc,EAAE,IAAI,CAAC,cAAc;gBACnC,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,gBAAgB,EAAE,IAAI,CAAC,cAAc;gBACrC,SAAS,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,kBAAkB,CAAC,CAAC,CAAC,UAAU;gBACvD,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,wBAAwB;aAClD;YACD,MAAM,EAAE,EAAE;YACV,QAAQ;YACR,cAAc,EAAE,UAAU,CAAC,IAAI,CAAC,cAAc,CAAC;YAC/C,OAAO,EAAE,CAAC,IAAI,CAAC,KAAK,IAAI,CAAC,OAAO;YAChC,MAAM,EAAE,OAAO,CAAC,MAAM;YACtB,MAAM,EAAE,OAAO,CAAC,MAAM,CAAC,MAAM;SAC9B,CAAA;IACH,CAAC;CACF,CAAA"}
1
+ {"version":3,"file":"linkedin.js","sourceRoot":"","sources":["../../../src/smartfetch/providers/linkedin.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,yBAAyB,EACzB,uBAAuB,EACvB,iBAAiB,EACjB,oBAAoB,EACpB,6BAA6B,GAC9B,MAAM,mBAAmB,CAAA;AAE1B,OAAO,EAAE,UAAU,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,iBAAiB,CAAA;AACrE,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAA;AACrD,OAAO,EAAE,iBAAiB,EAAE,iBAAiB,EAAE,MAAM,uBAAuB,CAAA;AAG5E,MAAM,UAAU,GAAG,iHAAiH,CAAA;AACpI,MAAM,gBAAgB,GAAG,CAAC,YAAY,EAAE,YAAY,EAAE,eAAe,EAAE,YAAY,CAAC,CAAA;AAEpF,SAAS,QAAQ,CAAC,KAAa,EAAE,SAAiB;IAChD,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAA;IACvC,IAAI,IAAI,CAAC,MAAM,IAAI,SAAS;QAAE,OAAO,IAAI,CAAA;IACzC,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,SAAS,GAAG,CAAC,CAAC,CAAC,OAAO,EAAE,GAAG,CAAA;AACrD,CAAC;AAED,SAAS,cAAc,CAAC,KAAa;IACnC,OAAO,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,CAAA;AACxD,CAAC;AAED,SAAS,YAAY,CAAC,GAAW;IAC/B,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,SAAS,CAAA;QACjF,OAAO,IAAI,CAAC,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC,IAAI,EAAE,IAAI,IAAI,CAAA;IACnD,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,kBAAkB,CAAA;IAC3B,CAAC;AACH,CAAC;AAED,SAAS,sBAAsB;IAC7B,OAAO,UAAU,CAAC,oCAAoC,EAAE,IAAI,CAAC,CAAA;AAC/D,CAAC;AAED,SAAS,cAAc;IACrB,OAAO,UAAU,CAAC,2BAA2B,EAAE,IAAI,CAAC,CAAA;AACtD,CAAC;AAED,SAAS,gBAAgB;IACvB,OAAO,UAAU,CAAC,8BAA8B,EAAE,IAAI,CAAC,CAAA;AACzD,CAAC;AAED,SAAS,iBAAiB;IACxB,OAAO,UAAU,CAAC,+BAA+B,EAAE,KAAK,EAAE,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC,CAAA;AAC/E,CAAC;AAED,SAAS,iBAAiB,CAAC,KAAa;IACtC,OAAO,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC;SACvB,OAAO,CAAC,OAAO,EAAE,IAAI,CAAC;SACtB,OAAO,CAAC,yBAAyB,EAAE,GAAG,CAAC;SACvC,OAAO,CAAC,qCAAqC,EAAE,OAAO,CAAC;SACvD,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC;SACtB,OAAO,CAAC,cAAc,EAAE,EAAE,CAAC;SAC3B,OAAO,CAAC,UAAU,EAAE,EAAE,CAAC;SACvB,OAAO,CAAC,SAAS,EAAE,MAAM,CAAC;SAC1B,IAAI,EAAE,CAAA;AACX,CAAC;AAED,SAAS,aAAa,CAAC,KAAa;IAClC,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,cAAc,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAA;IAClG,OAAO,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,IAAI,EAAE,CAAA;AACnE,CAAC;AAED,SAAS,qBAAqB,CAAC,KAQ9B;IACC,OAAO;QACL,IAAI,EAAE;YACJ,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,KAAK,EAAE,KAAK,CAAC,KAAK;YAClB,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,MAAM,EAAE,KAAK,CAAC,MAAM;SACrB;QACD,MAAM,EAAE,EAAE;QACV,QAAQ,EAAE,EAAE;QACZ,cAAc,EAAE,EAAE;QAClB,OAAO,EAAE,IAAI;QACb,MAAM,EAAE,CAAC,GAAG,KAAK,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,QAAQ,EAAE,aAAsB,EAAE,IAAI,EAAE,KAAK,CAAC,SAAS,EAAE,OAAO,EAAE,KAAK,CAAC,YAAY,EAAE,CAAC;QAC3H,MAAM,EAAE,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,MAAM;KACF,CAAA;AACrC,CAAC;AAED,SAAS,uBAAuB,CAAC,KAMhC;IACC,MAAM,OAAO,GAAG,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;IACxC,MAAM,QAAQ,GAAG,KAAK,CAAC,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;QAC9D,GAAG,IAAI;QACP,IAAI,EAAE,QAAQ,CAAC,IAAI,CAAC,IAAI,EAAE,GAAG,CAAC;KAC/B,CAAC,CAAC,CAAA;IACH,MAAM,IAAI,GAAG,cAAc,CAAC,KAAK,CAAC,UAAU,IAAI,EAAE,CAAC,CAAA;IACnD,OAAO;QACL,IAAI,EAAE;YACJ,GAAG,EAAE,KAAK,CAAC,OAAO,CAAC,GAAG;YACtB,aAAa,EAAE,KAAK,CAAC,IAAI,CAAC,aAAa;YACvC,KAAK,EAAE,KAAK,CAAC,IAAI,CAAC,KAAK;YACvB,WAAW,EAAE,KAAK,CAAC,IAAI,CAAC,WAAW;YACnC,IAAI,EAAE,QAAQ,CAAC,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,IAAI,CAAC;YAC1E,MAAM,EAAE,KAAK,CAAC,IAAI,CAAC,MAAM;YACzB,kBAAkB,EAAE,KAAK,CAAC,IAAI,CAAC,kBAAkB;YACjD,YAAY,EAAE,KAAK,CAAC,IAAI,CAAC,YAAY;YACrC,SAAS,EAAE,KAAK,CAAC,IAAI,CAAC,SAAS;YAC/B,aAAa,EAAE,KAAK,CAAC,IAAI,CAAC,aAAa;YACvC,cAAc,EAAE,KAAK,CAAC,IAAI,CAAC,cAAc;YACzC,MAAM,EAAE,KAAK,CAAC,IAAI,CAAC,MAAM;YACzB,gBAAgB,EAAE,KAAK,CAAC,IAAI,CAAC,cAAc;YAC3C,SAAS,EAAE,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,kBAAkB,CAAC,CAAC,CAAC,UAAU;YAC7D,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,wBAAwB;YACjD,GAAG,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC,CAAC,EAAE,mBAAmB,EAAE,KAAK,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SACrE;QACD,MAAM,EAAE,EAAE;QACV,QAAQ;QACR,cAAc,EAAE,UAAU,CAAC,CAAC,KAAK,CAAC,SAAS,IAAI,EAAE,EAAE,GAAG,KAAK,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACjG,OAAO,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,IAAI,CAAC,OAAO;QACtC,MAAM,EAAE,EAAE;QACV,MAAM,EAAE,KAAK,CAAC,MAAM;KACa,CAAA;AACrC,CAAC;AAED,SAAS,4BAA4B,CAAC,KASrC;IACC,MAAM,KAAK,GAAG,cAAc,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,oCAAoC,CAAA;IACjF,MAAM,IAAI,GAAG,QAAQ,CAAC,cAAc,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,IAAI,CAAC,CAAA;IACvD,MAAM,KAAK,GAAG,UAAU,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,EAAE,KAAK,CAAC,SAAS,EAAE,GAAG,CAAC,KAAK,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,CAAA;IACtG,OAAO;QACL,IAAI,EAAE;YACJ,GAAG,EAAE,KAAK,CAAC,OAAO,CAAC,GAAG;YACtB,aAAa,EAAE,KAAK,CAAC,SAAS;YAC9B,KAAK;YACL,WAAW,EAAE,GAAG,KAAK,CAAC,WAAW,+DAA+D;YAChG,IAAI,EAAE;gBACJ,2DAA2D;gBAC3D,qBAAqB,KAAK,CAAC,WAAW,EAAE;gBACxC,IAAI;gBACJ,6HAA6H;aAC9H,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;YAC9B,MAAM,EAAE,gBAAgB;YACxB,SAAS,EAAE,KAAK,CAAC,MAAM;SACxB;QACD,MAAM,EAAE,CAAC,KAAK,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,CAAC;YAC7D,KAAK,EAAE,KAAK,GAAG,CAAC;YAChB,KAAK,EAAE,IAAI,CAAC,KAAK;YACjB,GAAG,EAAE,IAAI,CAAC,GAAG;YACb,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SAChD,CAAC,CAAC;QACH,QAAQ,EAAE,EAAE;QACZ,cAAc,EAAE,KAAK;QACrB,OAAO,EAAE,IAAI;QACb,MAAM,EAAE,EAAE;QACV,MAAM,EAAE,KAAK,CAAC,MAAM;KACa,CAAA;AACrC,CAAC;AAED,SAAS,mBAAmB,CAAC,KAAa;IACxC,MAAM,SAAS,GAA8C,EAAE,CAAA;IAC/D,MAAM,KAAK,GAAG,wHAAwH,CAAA;IACtI,KAAK,MAAM,KAAK,IAAI,KAAK,CAAC,QAAQ,CAAC,KAAK,CAAC,EAAE,CAAC;QAC1C,MAAM,WAAW,GAAG,cAAc,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAA;QAClD,MAAM,SAAS,GAAG,cAAc,CAAC,WAAW,CAAC,KAAK,CAAC,+CAA+C,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAA;QAC/G,IAAI,CAAC,WAAW,IAAI,CAAC,SAAS;YAAE,SAAQ;QACxC,SAAS,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,WAAW,EAAE,SAAS,EAAE,CAAC,CAAA;IACjD,CAAC;IACD,OAAO,SAAS,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,SAAS,CAAC,aAAa,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,CAAA;AACvF,CAAC;AAED,SAAS,2BAA2B,CAAC,OAAe,EAAE,OAAe;IACnE,MAAM,IAAI,GAAG,iBAAiB,CAAC,OAAO,CAAC,CAAA;IACvC,MAAM,KAAK,GAAG,aAAa,CAAC,OAAO,CAAC,CAAA;IACpC,MAAM,KAAK,GAAG,UAAU,CAAC,WAAW,CAAC,OAAO,CAAC,CAAC,CAAA;IAC9C,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,MAAM,GAAG,GAAG;QAAE,OAAO,IAAI,CAAA;IAC5C,IAAI,6BAA6B,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,WAAW,EAAE,OAAO,EAAE,CAAC;QAAE,OAAO,IAAI,CAAA;IACvF,OAAO,4BAA4B,CAAC;QAClC,OAAO,EAAE;YACP,GAAG,EAAE,OAAO;YACZ,WAAW,EAAE,OAAO;YACpB,MAAM,EAAE,eAAe;YACvB,SAAS,EAAE,CAAC;YACZ,MAAM,EAAE,EAAE,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,aAAa,EAAE,OAAO,EAAE,KAAK,EAAE,SAAS,EAAE,OAAO,EAAE;YAC/E,MAAM,EAAE,EAAE;SACX;QACD,MAAM,EAAE,yBAAyB;QACjC,WAAW,EAAE,mBAAmB;QAChC,SAAS,EAAE,OAAO;QAClB,KAAK;QACL,IAAI;QACJ,KAAK;KACN,CAAC,CAAA;AACJ,CAAC;AAED,SAAS,YAAY,CAAC,KAAa,EAAE,SAAiB;IACpD,OAAO,KAAK,KAAK,SAAS;QACxB,CAAC,CAAC,6BAA6B,SAAS,EAAE;QAC1C,CAAC,CAAC,4BAA4B,SAAS,EAAE,CAAA;AAC7C,CAAC;AAED,KAAK,UAAU,qBAAqB,CAAC,KAKpC;IACC,MAAM,QAAQ,GAAG,MAAM,iBAAiB,CACtC,KAAK,CAAC,WAAW,EACjB,KAAK,CAAC,OAAO,CAAC,SAAS,EACvB;QACE,OAAO,EAAE;YACP,MAAM,EAAE,0BAA0B;YAClC,YAAY,EAAE,UAAU;SACzB;KACF,EACD;QACE,IAAI,EAAE,OAAO;QACb,SAAS,EAAE,KAAK,CAAC,OAAO,CAAC,GAAG;KAC7B,CACF,CAAA;IAED,IAAI,CAAC,QAAQ,CAAC,EAAE;QAAE,OAAO,IAAI,CAAA;IAC7B,IAAI,6BAA6B,CAAC,EAAE,IAAI,EAAE,QAAQ,CAAC,IAAI,EAAE,WAAW,EAAE,KAAK,CAAC,WAAW,EAAE,CAAC;QAAE,OAAO,IAAI,CAAA;IAEvG,MAAM,IAAI,GAAG,uBAAuB,CAAC;QACnC,IAAI,EAAE,QAAQ,CAAC,IAAI;QACnB,OAAO,EAAE,KAAK,CAAC,OAAO,CAAC,GAAG;QAC1B,SAAS,EAAE,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,WAAW,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE;KAC3D,CAAC,CAAA;IACF,IAAI,IAAI,CAAC,IAAI,IAAI,CAAC,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,GAAG,CAAC,EAAE,CAAC;QACzD,OAAO,uBAAuB,CAAC;YAC7B,OAAO,EAAE,KAAK,CAAC,OAAO;YACtB,MAAM,EAAE,KAAK,CAAC,MAAM;YACpB,IAAI;YACJ,UAAU,EAAE,YAAY,CAAC,KAAK,CAAC,WAAW,EAAE,KAAK,CAAC,WAAW,CAAC;YAC9D,SAAS,EAAE,KAAK,CAAC,WAAW;SAC7B,CAAC,CAAA;IACJ,CAAC;IAED,OAAO,IAAI,CAAA;AACb,CAAC;AAED,KAAK,UAAU,wBAAwB,CAAC,OAA0B;IAChE,IAAI,CAAC,iBAAiB,EAAE;QAAE,OAAO,IAAI,CAAA;IACrC,MAAM,KAAK,GAAG,MAAM,iBAAiB,CACnC,qBAAqB,OAAO,CAAC,GAAG,EAAE,EAClC,OAAO,CAAC,SAAS,EACjB;QACE,OAAO,EAAE;YACP,MAAM,EAAE,8BAA8B;YACtC,YAAY,EAAE,UAAU;YACxB,gBAAgB,EAAE,UAAU;SAC7B;KACF,EACD;QACE,IAAI,EAAE,OAAO;QACb,SAAS,EAAE,OAAO,CAAC,GAAG;KACvB,CACF,CAAA;IACD,IAAI,CAAC,KAAK,CAAC,EAAE;QAAE,OAAO,IAAI,CAAA;IAC1B,MAAM,SAAS,GAAG,2BAA2B,CAAC,KAAK,CAAC,IAAI,EAAE,OAAO,CAAC,GAAG,CAAC,CAAA;IACtE,IAAI,CAAC,SAAS;QAAE,OAAO,IAAI,CAAA;IAC3B,OAAO,EAAE,GAAG,SAAS,EAAE,MAAM,EAAE,yBAAyB,EAAoC,CAAA;AAC9F,CAAC;AAED,KAAK,UAAU,kBAAkB,CAAC,OAA0B;IAC1D,IAAI,CAAC,sBAAsB,EAAE,IAAI,CAAC,cAAc,EAAE;QAAE,OAAO,IAAI,CAAA;IAC/D,MAAM,YAAY,GAAG,MAAM,iBAAiB,CAC1C,6CAA6C,kBAAkB,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,EAC9E,OAAO,CAAC,SAAS,EACjB;QACE,OAAO,EAAE;YACP,MAAM,EAAE,iCAAiC;YACzC,YAAY,EAAE,UAAU;SACzB;KACF,EACD;QACE,IAAI,EAAE,OAAO;QACb,SAAS,EAAE,OAAO,CAAC,GAAG;KACvB,CACF,CAAA;IACD,IAAI,CAAC,YAAY,CAAC,EAAE;QAAE,OAAO,IAAI,CAAA;IACjC,MAAM,WAAW,GAAG,cAAc,CAAC,MAAM,CAAE,YAAY,CAAC,IAAY,EAAE,kBAAkB,EAAE,OAAO,EAAE,GAAG,IAAI,EAAE,CAAC,CAAC,CAAA;IAC9G,IAAI,CAAC,WAAW;QAAE,OAAO,IAAI,CAAA;IAC7B,OAAO,qBAAqB,CAAC,EAAE,OAAO,EAAE,WAAW,EAAE,MAAM,EAAE,0BAA0B,EAAE,WAAW,EAAE,SAAS,EAAE,CAAC,CAAA;AACpH,CAAC;AAED,KAAK,UAAU,oBAAoB,CAAC,OAA0B;IAC5D,IAAI,CAAC,sBAAsB,EAAE,IAAI,CAAC,gBAAgB,EAAE;QAAE,OAAO,IAAI,CAAA;IACjE,KAAK,MAAM,IAAI,IAAI,gBAAgB,EAAE,CAAC;QACpC,MAAM,OAAO,GAAG,MAAM,iBAAiB,CACrC,WAAW,IAAI,YAAY,OAAO,CAAC,GAAG,EAAE,EACxC,OAAO,CAAC,SAAS,EACjB;YACE,OAAO,EAAE;gBACP,MAAM,EAAE,wCAAwC;gBAChD,YAAY,EAAE,UAAU;aACzB;SACF,EACD;YACE,IAAI,EAAE,OAAO;YACb,SAAS,EAAE,OAAO,CAAC,GAAG;SACvB,CACF,CAAA;QACD,IAAI,CAAC,OAAO,CAAC,EAAE;YAAE,SAAQ;QACzB,MAAM,WAAW,GAAG,mBAAmB,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,GAAG,IAAI,EAAE,CAAA;QACnE,IAAI,CAAC,WAAW;YAAE,SAAQ;QAC1B,MAAM,MAAM,GAAG,MAAM,qBAAqB,CAAC,EAAE,OAAO,EAAE,WAAW,EAAE,MAAM,EAAE,qBAAqB,EAAE,WAAW,EAAE,IAAI,EAAE,CAAC,CAAA;QACtH,IAAI,MAAM;YAAE,OAAO,MAAM,CAAA;IAC3B,CAAC;IACD,OAAO,IAAI,CAAA;AACb,CAAC;AAED,SAAS,qBAAqB,CAAC,GAAW;IACxC,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,kBAAkB,CAAC,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAA;QAC9F,OAAO,IAAI;aACR,OAAO,CAAC,eAAe,EAAE,GAAG,CAAC;aAC7B,OAAO,CAAC,YAAY,EAAE,GAAG,CAAC;aAC1B,OAAO,CAAC,SAAS,EAAE,GAAG,CAAC;aACvB,OAAO,CAAC,QAAQ,EAAE,GAAG,CAAC;aACtB,OAAO,CAAC,aAAa,EAAE,GAAG,CAAC;aAC3B,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC;aACpB,IAAI,EAAE,CAAA;IACX,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,EAAE,CAAA;IACX,CAAC;AACH,CAAC;AAED,SAAS,yBAAyB,CAAC,GAAW;IAC5C,MAAM,UAAU,GAAG,yBAAyB,CAAC,GAAG,CAAC,CAAA;IACjD,MAAM,SAAS,GAAG,qBAAqB,CAAC,GAAG,CAAC,CAAA;IAC5C,OAAO,KAAK,CAAC,IAAI,CAAC,IAAI,GAAG,CAAC;QACxB,UAAU,CAAC,CAAC,CAAC,2BAA2B,UAAU,IAAI,SAAS,EAAE,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE;QAC7E,UAAU,CAAC,CAAC,CAAC,iCAAiC,UAAU,IAAI,SAAS,EAAE,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE;QACnF,SAAS,CAAC,CAAC,CAAC,2BAA2B,SAAS,EAAE,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE;KAC/D,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,CAAA;AACrB,CAAC;AAED,SAAS,yBAAyB,CAAC,GAAW,EAAE,IAAkB;IAChE,MAAM,UAAU,GAAG,yBAAyB,CAAC,GAAG,CAAC,CAAA;IACjD,MAAM,gBAAgB,GAAG,yBAAyB,CAAC,IAAI,CAAC,GAAG,CAAC,CAAA;IAC5D,MAAM,KAAK,GAAG,cAAc,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,sBAAsB,EAAE,EAAE,CAAC,CAAC,CAAA;IAC5E,MAAM,OAAO,GAAG,cAAc,CAAC,IAAI,CAAC,OAAO,CAAC,CAAA;IAC5C,IAAI,KAAK,GAAG,CAAC,CAAA;IACb,IAAI,gBAAgB,IAAI,UAAU,IAAI,gBAAgB,KAAK,UAAU;QAAE,KAAK,IAAI,EAAE,CAAA;IAClF,IAAI,iBAAiB,CAAC,IAAI,CAAC,GAAG,CAAC;QAAE,KAAK,IAAI,EAAE,CAAA;IAC5C,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG;QAAE,KAAK,IAAI,EAAE,CAAA;IACtC,IAAI,OAAO,CAAC,MAAM,IAAI,GAAG;QAAE,KAAK,IAAI,EAAE,CAAA;IACtC,IAAI,8EAA8E,CAAC,IAAI,CAAC,GAAG,KAAK,IAAI,OAAO,EAAE,CAAC;QAAE,KAAK,IAAI,GAAG,CAAA;IAC5H,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,OAAO,EAAE,CAAA;AAClC,CAAC;AAED,KAAK,UAAU,0BAA0B,CAAC,OAA0B;IAClE,MAAM,OAAO,GAAG,yBAAyB,CAAC,OAAO,CAAC,GAAG,CAAC,CAAA;IACtD,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,IAAI,CAAA;IACrC,MAAM,YAAY,GAAG,OAAO,CAAC,cAAc,IAAI,cAAc,CAAA;IAC7D,MAAM,QAAQ,GAAmB,EAAE,CAAA;IAEnC,KAAK,MAAM,KAAK,IAAI,OAAO,EAAE,CAAC;QAC5B,IAAI,CAAC;YACH,MAAM,QAAQ,GAAG,MAAM,YAAY,CAAC,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,SAAS,EAAE,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,SAAS,EAAE,KAAK,CAAC,EAAE,UAAU,EAAE,MAAM,EAAE,CAAC,CAAA;YAChI,IAAI,KAAK,CAAC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC;gBAAE,QAAQ,CAAC,IAAI,CAAC,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAA;QACzE,CAAC;QAAC,MAAM,CAAC;YACP,4DAA4D;QAC9D,CAAC;IACH,CAAC;IAED,MAAM,MAAM,GAAG,QAAQ;SACpB,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;QACZ,MAAM,MAAM,GAAG,yBAAyB,CAAC,OAAO,CAAC,GAAG,EAAE,IAAI,CAAC,CAAA;QAC3D,OAAO;YACL,IAAI;YACJ,GAAG,MAAM;SACV,CAAA;IACH,CAAC,CAAC;SACD,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,IAAI,EAAE,IAAI,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,OAAO,CAAC;SAChE,IAAI,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAA;IAElD,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,IAAI,CAAA;IACpC,MAAM,IAAI,GAAG,MAAM,CAAC,CAAC,CAAE,CAAA;IACvB,OAAO,4BAA4B,CAAC;QAClC,OAAO;QACP,MAAM,EAAE,gCAAgC;QACxC,WAAW,EAAE,iBAAiB,IAAI,CAAC,IAAI,CAAC,MAAM,GAAG;QACjD,SAAS,EAAE,IAAI,CAAC,IAAI,CAAC,GAAG;QACxB,KAAK,EAAE,IAAI,CAAC,KAAK;QACjB,IAAI,EAAE,IAAI,CAAC,OAAO;QAClB,KAAK,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC;QACtB,MAAM,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;YACzC,KAAK,EAAE,KAAK,CAAC,KAAK;YAClB,GAAG,EAAE,KAAK,CAAC,IAAI,CAAC,GAAG;YACnB,OAAO,EAAE,KAAK,CAAC,OAAO;YACtB,MAAM,EAAE,aAAa,KAAK,CAAC,IAAI,CAAC,MAAM,EAAE;SACzC,CAAC,CAAC;KACJ,CAAC,CAAA;AACJ,CAAC;AAED,KAAK,UAAU,yBAAyB,CAAC,OAA0B;IACjE,OAAO,MAAM,wBAAwB,CAAC,OAAO,CAAC;WACzC,MAAM,kBAAkB,CAAC,OAAO,CAAC;WACjC,MAAM,oBAAoB,CAAC,OAAO,CAAC;WACnC,MAAM,0BAA0B,CAAC,OAAO,CAAC,CAAA;AAChD,CAAC;AAED,MAAM,CAAC,MAAM,gBAAgB,GAAuB;IAClD,EAAE,EAAE,UAAU;IACd,OAAO,EAAE,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,CAAC,MAAM,KAAK,eAAe,IAAI,CAAC,MAAM,KAAK,SAAS,IAAI,CAAC,iBAAiB,CAAC,GAAG,CAAC,IAAI,oBAAoB,CAAC,GAAG,CAAC,CAAC,CAAC;IACvI,KAAK,CAAC,SAAS,CAAC,OAAO;QACrB,IAAI,oBAAoB,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YACtC,OAAO,qBAAqB,CAAC;gBAC3B,GAAG,EAAE,OAAO,CAAC,GAAG;gBAChB,KAAK,EAAE,qBAAqB,YAAY,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE;gBACvD,IAAI,EAAE,mMAAmM;gBACzM,MAAM,EAAE,kBAAkB;gBAC1B,SAAS,EAAE,2BAA2B;gBACtC,YAAY,EAAE,wDAAwD;gBACtE,OAAO;aACR,CAAC,CAAA;QACJ,CAAC;QAED,IAAI,6BAA6B,CAAC,EAAE,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,OAAO,EAAE,WAAW,EAAE,OAAO,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;YACtG,MAAM,QAAQ,GAAG,MAAM,yBAAyB,CAAC,OAAO,CAAC,CAAA;YACzD,IAAI,QAAQ;gBAAE,OAAO,QAAQ,CAAA;YAC7B,OAAO,qBAAqB,CAAC;gBAC3B,GAAG,EAAE,OAAO,CAAC,GAAG;gBAChB,KAAK,EAAE,2BAA2B;gBAClC,IAAI,EAAE,sHAAsH;gBAC5H,MAAM,EAAE,eAAe;gBACvB,SAAS,EAAE,wBAAwB;gBACnC,YAAY,EAAE,6DAA6D;gBAC3E,OAAO;aACR,CAAC,CAAA;QACJ,CAAC;QAED,MAAM,IAAI,GAAG,uBAAuB,CAAC;YACnC,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,OAAO;YAC5B,OAAO,EAAE,OAAO,CAAC,WAAW,IAAI,OAAO,CAAC,GAAG;YAC3C,SAAS,EAAE,OAAO,CAAC,MAAM,CAAC,KAAK;SAChC,CAAC,CAAA;QAEF,OAAO,uBAAuB,CAAC;YAC7B,OAAO;YACP,MAAM,EAAE,OAAO,CAAC,MAAM,CAAC,MAAM;YAC7B,IAAI;SACL,CAAC,CAAA;IACJ,CAAC;CACF,CAAA"}
package/package.json CHANGED
@@ -1,16 +1,16 @@
1
1
  {
2
2
  "name": "smart-web-mcp",
3
- "mcpName": "io.github.rich-jojo/smart-web",
4
- "version": "0.8.3",
3
+ "mcpName": "io.github.jojo-labs/smart-web",
4
+ "version": "0.8.8",
5
5
  "packageManager": "npm@11.9.0",
6
6
  "description": "Portable MCP server for web search, direct-URL fetch, site crawling, and docs export.",
7
- "homepage": "https://github.com/rich-jojo/smart-web",
7
+ "homepage": "https://github.com/jojo-labs/smart-web",
8
8
  "bugs": {
9
- "url": "https://github.com/rich-jojo/smart-web/issues"
9
+ "url": "https://github.com/jojo-labs/smart-web/issues"
10
10
  },
11
11
  "repository": {
12
12
  "type": "git",
13
- "url": "git+https://github.com/rich-jojo/smart-web.git"
13
+ "url": "git+https://github.com/jojo-labs/smart-web.git"
14
14
  },
15
15
  "type": "module",
16
16
  "exports": {
@@ -1,3 +1,3 @@
1
1
  undetected-chromedriver==3.5.5
2
- selenium==4.29.0
3
- setuptools==75.8.0
2
+ selenium==4.43.0
3
+ setuptools==82.0.1