@dreamteamer/connectors-linkedin 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +40 -2
- package/command-bindings/scrape-contact--contacts.command-binding.yaml +11 -0
- package/commands/scrape-contact.command.md +35 -0
- package/package.json +45 -5
- package/skills/scraping-contacts/SKILL.md +312 -0
- package/skills/scraping-contacts/bin/import-avatars.mjs +90 -0
- package/skills/scraping-contacts/bin/li.mjs +285 -0
- package/ui-views/contacts-needs-linkedin.ui-view.yaml +29 -0
- package/ui-views/contacts-review-linkedin.ui-view.yaml +39 -0
package/README.md
CHANGED
|
@@ -1,3 +1,41 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @dreamteamer/connectors-linkedin
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Fills your `contacts` in from LinkedIn and the web. A command enriches a person from their LinkedIn
|
|
4
|
+
profile; two views list who still needs a LinkedIn link and who needs your review. Not part of
|
|
5
|
+
`@dreamteamer/core`: install it when you want a workspace to reach LinkedIn.
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
dt install @dreamteamer/connectors-linkedin
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
or, with npm:
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
npm install @dreamteamer/connectors-linkedin @dreamteamer/contacts @dreamteamer/companies @dreamteamer/assets
|
|
15
|
+
npx dreamteamer compile
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
| alone | with others |
|
|
19
|
+
|---|---|
|
|
20
|
+
| `/scrape-contact` on any `draft` contact, the `scraping-contacts` skill, the views `/contacts/needs-linkedin` and `/contacts/review-linkedin` | **meetings** — a meeting's attendees become the `draft` contacts `/scrape-contact` queues (the calendar sync in `@dreamteamer/connectors-google-workspace` writes them) |
|
|
21
|
+
|
|
22
|
+
| needs | for |
|
|
23
|
+
|---|---|
|
|
24
|
+
| `@dreamteamer/contacts` 0.2.0 or later | the records it fills in |
|
|
25
|
+
| `@dreamteamer/companies` | where a contact works |
|
|
26
|
+
| `@dreamteamer/assets` | avatars |
|
|
27
|
+
| Docker and one LinkedIn sign-in | `/scrape-contact`; the views work without them |
|
|
28
|
+
|
|
29
|
+
The module has no setup unit. `/scrape-contact` is its one command, and the `scraping-contacts` skill
|
|
30
|
+
carries the procedure, including the one-time LinkedIn sign-in and the optional `LINKEDIN_USER_AGENT`
|
|
31
|
+
in `.env`. Every call is a request from your own LinkedIn account: the skill scrapes in small batches
|
|
32
|
+
and stops at the first captcha.
|
|
33
|
+
|
|
34
|
+
## 0.1.0 — moved out of contacts (cr-modules-018)
|
|
35
|
+
|
|
36
|
+
The scraper, `/scrape-contact`, its binding and the two LinkedIn views, as `@dreamteamer/contacts` 0.1.3
|
|
37
|
+
shipped them, moved here unchanged so a workspace on `@dreamteamer/core` reaches no LinkedIn. It needs
|
|
38
|
+
contacts 0.2.0 or later (the older contacts ships the same skill and command, and is refused). On a
|
|
39
|
+
workspace that had them through contacts: `dt update @dreamteamer/core` (or `@dreamteamer/contacts`)
|
|
40
|
+
first, then `dt install @dreamteamer/connectors-linkedin`. `LINKEDIN_USER_AGENT` in `.env` is read
|
|
41
|
+
again as it was; no record changes.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
command: commands/scrape-contact
|
|
2
|
+
collection: collections/contacts
|
|
3
|
+
scope: record
|
|
4
|
+
# the enrichment queue, derived rather than stored: placeholders await a scrape
|
|
5
|
+
available_when:
|
|
6
|
+
status:
|
|
7
|
+
_eq: draft
|
|
8
|
+
done_when:
|
|
9
|
+
status:
|
|
10
|
+
_in: [enriched, verified]
|
|
11
|
+
description: enrich this placeholder contact from LinkedIn + web search
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: scrape-contact
|
|
3
|
+
description: build or complete a contact record from LinkedIn + web search (uses your own LinkedIn session)
|
|
4
|
+
argument-hint: <contacts/id | "name + hint"> [more…]
|
|
5
|
+
---
|
|
6
|
+
load the `scraping-contacts` skill. for each target in: $ARGUMENTS — a `contacts/<id>` ref, an
|
|
7
|
+
identity hint (name, company, LinkedIn URL), or both — produce a complete contact record.
|
|
8
|
+
|
|
9
|
+
profile data comes from the **`linkedin-mcp-server` v4 MCP tools**, run through the skill's `li.mjs` —
|
|
10
|
+
`get_person_profile(slug, sections?)`. to resolve a name to a slug, use `WebSearch` first, then
|
|
11
|
+
`get_company_employees`, then the `references` of a profile you already scraped; `search_people` does
|
|
12
|
+
not work. confirm the right person on **two signals** before scraping; a wrong slug writes wrong data,
|
|
13
|
+
which is worse than a wasted call.
|
|
14
|
+
|
|
15
|
+
**never call `connect_with_person` or `send_message`** — they act publicly and irreversibly as you, and
|
|
16
|
+
`connect_with_person` has no confirmation parameter. same for the inbox tools: reading private DMs is
|
|
17
|
+
not enrichment. if a task seems to want one, stop and ask.
|
|
18
|
+
|
|
19
|
+
**every call is an authenticated request from your own LinkedIn account** — that, not money, is the
|
|
20
|
+
constraint. so: **one batch at a time**, small batches budgeted in page loads rather than the whole
|
|
21
|
+
`draft` queue. if you hit a captcha, a "confirm in the mobile app" prompt, or the
|
|
22
|
+
`No valid LinkedIn session` error, **stop and say so** — do not retry in a loop. fall back to web
|
|
23
|
+
search instead. an **all-null profile with no error** is a stale server image, not a bad session —
|
|
24
|
+
pull the image, don't re-auth.
|
|
25
|
+
|
|
26
|
+
then complement from web search: current role, company, location, public writing. set `source` to what
|
|
27
|
+
you read — `linkedin:<slug>` for a scrape, `url:<…>` for a page — and name any other source in the
|
|
28
|
+
body. when two sources disagree, keep both in the body with the discrepancy named.
|
|
29
|
+
|
|
30
|
+
set `status: enriched` on a pass that produced something; a pass that found nothing stays `draft` with
|
|
31
|
+
the reason in `creation_context`. **never `verified`** — that is a person's to give. `author` is
|
|
32
|
+
`agents/<harness>` with the model in `creation_context`; `confidence: low` plus a reason in
|
|
33
|
+
`creation_context` for anything inferred rather than read.
|
|
34
|
+
|
|
35
|
+
report per contact: what you found, what you could not, and which source each part came from.
|
package/package.json
CHANGED
|
@@ -1,6 +1,46 @@
|
|
|
1
1
|
{
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
2
|
+
"name": "@dreamteamer/connectors-linkedin",
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Fill in contacts from LinkedIn and the web: /scrape-contact, the scraping-contacts skill and two LinkedIn views.",
|
|
5
|
+
"license": "Apache-2.0",
|
|
6
|
+
"repository": {
|
|
7
|
+
"type": "git",
|
|
8
|
+
"url": "git+https://github.com/dreamteamer/dreamteamer-modules.git",
|
|
9
|
+
"directory": "modules/connectors-linkedin"
|
|
10
|
+
},
|
|
11
|
+
"keywords": [
|
|
12
|
+
"dreamteamer",
|
|
13
|
+
"dreamteamer-module",
|
|
14
|
+
"connectors",
|
|
15
|
+
"linkedin",
|
|
16
|
+
"contacts"
|
|
17
|
+
],
|
|
18
|
+
"files": [
|
|
19
|
+
"README.md",
|
|
20
|
+
"skills",
|
|
21
|
+
"commands",
|
|
22
|
+
"command-bindings",
|
|
23
|
+
"ui-views"
|
|
24
|
+
],
|
|
25
|
+
"dreamteamer": {
|
|
26
|
+
"title": "LinkedIn connector",
|
|
27
|
+
"description": "Reaches LinkedIn for `contacts`: the `scraping-contacts` skill and `/scrape-contact` fill a person in from their LinkedIn profile, one batched container per run, and from web search, with their `companies` record and an avatar in `assets`. Every `draft` contact is queued; two views list who still needs a LinkedIn link and who needs review. Optional key: `LINKEDIN_USER_AGENT`.",
|
|
28
|
+
"engine": ">=0.33.0",
|
|
29
|
+
"dependencies": [
|
|
30
|
+
"@dreamteamer/contacts",
|
|
31
|
+
"@dreamteamer/companies",
|
|
32
|
+
"@dreamteamer/assets"
|
|
33
|
+
],
|
|
34
|
+
"peer_collections": [
|
|
35
|
+
"meetings"
|
|
36
|
+
],
|
|
37
|
+
"env": [
|
|
38
|
+
"LINKEDIN_USER_AGENT"
|
|
39
|
+
]
|
|
40
|
+
},
|
|
41
|
+
"peerDependencies": {
|
|
42
|
+
"@dreamteamer/contacts": ">=0.2.0",
|
|
43
|
+
"@dreamteamer/companies": ">=0.1.1",
|
|
44
|
+
"@dreamteamer/assets": ">=0.1.0"
|
|
45
|
+
}
|
|
46
|
+
}
|
|
@@ -0,0 +1,312 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: scraping-contacts
|
|
3
|
+
description: Use when you need LinkedIn profile or company data (headline, current role, experience, education, skills) for one or more people — to build or enrich contact records — given their profile URLs, or just names to search for, and WebFetch/a logged-in browser can't get past LinkedIn's auth wall.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Scraping Contacts
|
|
7
|
+
|
|
8
|
+
## Overview
|
|
9
|
+
|
|
10
|
+
Pull LinkedIn data through **`linkedin-mcp-server` v4** (stickerdaniel), a container driving a headless
|
|
11
|
+
Chromium that holds **your own LinkedIn session**. LinkedIn blocks direct `WebFetch` with a
|
|
12
|
+
domain-verification wall, so this is the way in.
|
|
13
|
+
|
|
14
|
+
**The session is a singleton, and the whole design follows from that.** One browser, one profile
|
|
15
|
+
directory, one cookie. Used from two places at once, LinkedIn invalidates it, and that costs a host
|
|
16
|
+
re-login rather than a retry. So there is **no `linkedin` entry in `.mcp.json`**: a registered stdio
|
|
17
|
+
server spawns one container per agent session, which is exactly that collision. Instead you run **one
|
|
18
|
+
ephemeral, lock-guarded container per batch** through `bin/li.mjs`, and it exits when the batch does.
|
|
19
|
+
Nothing idles, and the lock makes "exactly one" true.
|
|
20
|
+
|
|
21
|
+
**Every call is an authenticated request from your real account.** That, not money, is the constraint
|
|
22
|
+
that shapes everything below.
|
|
23
|
+
|
|
24
|
+
## NEVER use these tools
|
|
25
|
+
|
|
26
|
+
The server ships write and inbox tools. They are not part of contact enrichment, and each one acts as
|
|
27
|
+
you, irreversibly and in public:
|
|
28
|
+
|
|
29
|
+
| tool | why not |
|
|
30
|
+
|---|---|
|
|
31
|
+
| `connect_with_person` | sends a real connection request. **No confirmation parameter** — the call *is* the action |
|
|
32
|
+
| `send_message` | sends a real DM. `confirm_send` is a guard, not permission |
|
|
33
|
+
| `get_inbox` / `get_conversation` / `search_conversations` | reads your private DMs. Enrichment never needs this |
|
|
34
|
+
| `get_feed` / `search_posts` / `get_saved_jobs` / `get_recommended_jobs` | a different job |
|
|
35
|
+
|
|
36
|
+
`li.mjs` refuses `connect_with_person` and `send_message` with exit code 3 before any container
|
|
37
|
+
starts. If a task seems to want one of these, **stop and ask** — that is a person's decision, not a
|
|
38
|
+
scraper's.
|
|
39
|
+
|
|
40
|
+
## The tools you do use
|
|
41
|
+
|
|
42
|
+
| tool | signature | use |
|
|
43
|
+
|---|---|---|
|
|
44
|
+
| `get_person_profile` | `(linkedin_username, sections?, max_scrolls?)` | one person. **`linkedin_username` is the slug**, not the URL: `https://www.linkedin.com/in/dana-oren/` → `dana-oren` |
|
|
45
|
+
| `get_company_profile` | `(company_name, sections?)` | a company page (slug from `linkedin.com/company/<slug>`) |
|
|
46
|
+
| `search_companies` | `(keywords)` | company name → slug |
|
|
47
|
+
| `get_company_employees` | `(company_name, keywords?)` | find a person through their employer — identifies a colleague from an email convention alone |
|
|
48
|
+
| `close_session` | `()` | end the browser session after a batch (`li.mjs` calls it for you) |
|
|
49
|
+
|
|
50
|
+
`search_people` fails even on a healthy session (upstream #526). Do not spend a call on it.
|
|
51
|
+
|
|
52
|
+
**`sections` is a comma-separated STRING, not an array** — `"experience,education"`. A list is a
|
|
53
|
+
pydantic validation error. Available: `experience`, `education`, `interests`, `honors`, `languages`,
|
|
54
|
+
`certifications`, `skills`, `projects`, `contact_info`, `posts`. **Omitting `sections` returns only the
|
|
55
|
+
top card** — name, headline, location, current company, school, follower count. Each section is another
|
|
56
|
+
page load, so ask only for what you will write down.
|
|
57
|
+
|
|
58
|
+
`max_scrolls` (1–50) caps "Show more" clicks per section, default 5 (10 for posts). Raise it for a
|
|
59
|
+
profile with 30+ items in one section, and request heavy sections in a separate call.
|
|
60
|
+
|
|
61
|
+
A section can fail on its own while the rest succeed: its value is LinkedIn's `"Something went wrong"`,
|
|
62
|
+
and `li.mjs` lists it under `failed_sections`. Fetching by URN fails this way more often than by slug,
|
|
63
|
+
so prefer the slug once you have it.
|
|
64
|
+
|
|
65
|
+
## First-time setup
|
|
66
|
+
|
|
67
|
+
**Order matters** — the login (step 2) must exist before any container runs. Docker creates a missing
|
|
68
|
+
bind-mount source **as root**, so a container started first leaves `~/.linkedin-mcp` root-owned and the
|
|
69
|
+
login cannot write to it (recoverable with `sudo rm -rf ~/.linkedin-mcp`).
|
|
70
|
+
|
|
71
|
+
**1. Two things installed.**
|
|
72
|
+
- **Docker Desktop**, running. Add nothing to its MCP Toolkit and nothing to `.mcp.json` (see
|
|
73
|
+
*Why there is no MCP entry*).
|
|
74
|
+
- **`uv`** (which provides `uvx`): `brew install uv`, or the installer at astral.sh/uv. Check with
|
|
75
|
+
`uvx --version`.
|
|
76
|
+
|
|
77
|
+
**2. The LinkedIn session, on the HOST.** Prefer the import — it reuses the session already in
|
|
78
|
+
Chrome/Brave/Edge/Arc/Vivaldi, and repeated fresh logins are what trigger a captcha:
|
|
79
|
+
|
|
80
|
+
```sh
|
|
81
|
+
uvx mcp-server-linkedin@latest --import-from-browser
|
|
82
|
+
# if the import fails or finds nothing: --login (opens a browser; 2FA/captcha fine, ~5 min allowed)
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
This creates **`~/.linkedin-mcp/`** — outside the repo, never committed — holding `cookies.json`,
|
|
86
|
+
`source-state.json`, a Patchright browser and `trace-runs/` diagnostics, about 220 MB in all. Each
|
|
87
|
+
container derives a fresh browser profile from it at startup, so there is nothing to restart afterwards.
|
|
88
|
+
|
|
89
|
+
**3. Optionally, `LINKEDIN_USER_AGENT` in the workspace `.env`.** `li.mjs` passes it to the container
|
|
90
|
+
as `USER_AGENT` and warns on stderr when it is unset. The session token is minted on the host and then
|
|
91
|
+
used from a Linux container; a matching user agent removes that one discrepancy. Use the exact string
|
|
92
|
+
of the browser the session came from, never an invented one. Whether it raises the page-load ceiling is
|
|
93
|
+
unproven, and it does not revive a dead cookie.
|
|
94
|
+
|
|
95
|
+
**4. Confirm it worked**, spending no LinkedIn request:
|
|
96
|
+
|
|
97
|
+
```sh
|
|
98
|
+
node .dreamteamer/skills/scraping-contacts/bin/li.mjs --doctor
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Want `cookies: true`, a recent `login_at`, `batch_lock: free`, `containers_running: 0`,
|
|
102
|
+
`host_clients: none`. Right after a login the verdict is *"login is NEWER than any scraping evidence —
|
|
103
|
+
session untested"*: the doctor never reuses evidence older than the login. The **first** batch also
|
|
104
|
+
pulls the image (about 0.5 GB), so it runs slow once.
|
|
105
|
+
|
|
106
|
+
## Running a batch — `bin/li.mjs`
|
|
107
|
+
|
|
108
|
+
Run from the workspace root, after a compile:
|
|
109
|
+
|
|
110
|
+
```sh
|
|
111
|
+
D=.dreamteamer/skills/scraping-contacts/bin
|
|
112
|
+
|
|
113
|
+
# is the session alive? costs NO LinkedIn request — always start here
|
|
114
|
+
node $D/li.mjs --doctor
|
|
115
|
+
|
|
116
|
+
# one call
|
|
117
|
+
node $D/li.mjs get_person_profile '{"linkedin_username":"dana-oren","sections":"experience,education"}'
|
|
118
|
+
|
|
119
|
+
# a BATCH: one container, one browser, N calls, then gone
|
|
120
|
+
echo '[
|
|
121
|
+
{"tool":"get_person_profile","args":{"linkedin_username":"dana-oren"}},
|
|
122
|
+
{"tool":"get_person_profile","args":{"linkedin_username":"sam-diaz"}},
|
|
123
|
+
{"tool":"get_company_employees","args":{"company_name":"northwind-group","keywords":"Diaz"}}
|
|
124
|
+
]' | node $D/li.mjs --stdin
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
Batch rather than looping single calls — a container per person pays startup and session derivation
|
|
128
|
+
every time. Output is a JSON array of `{tool, args, ok, result|error}`, plus `failed_sections` when a
|
|
129
|
+
call returned `ok` with some section missing.
|
|
130
|
+
|
|
131
|
+
Exit codes: `0` all good · `1` at least one call failed · `2` bad usage · `3` refused a banned tool ·
|
|
132
|
+
**`4` another batch holds the lock** — not an error to retry past. Wait, or find out why the other
|
|
133
|
+
batch is stuck; two containers on one profile is what invalidates the session.
|
|
134
|
+
|
|
135
|
+
`--doctor` reads the newest trace run's post-import screenshot from disk: a real page is about 36 KB, a
|
|
136
|
+
bounce to the sign-in wall leaves it blank at about 4 KB. It also lists `host_clients`, any
|
|
137
|
+
`mcp-server-linkedin` process on the host still holding the browser lease. It finds them with `ps`,
|
|
138
|
+
because on macOS `lsof` shows no holder for a held flock and Docker's VM always has the lock file open.
|
|
139
|
+
|
|
140
|
+
## The session is the scarce resource
|
|
141
|
+
|
|
142
|
+
There is no per-call charge. What breaks:
|
|
143
|
+
|
|
144
|
+
1. **Two containers on one profile.** LinkedIn invalidates the `li_at` cookie for being used from two
|
|
145
|
+
places, and stopping the containers does not undo it — the session needs a fresh login. `li.mjs`'s
|
|
146
|
+
lock prevents it. `docker ps --filter ancestor=stickerdaniel/linkedin-mcp-server` is **0 between
|
|
147
|
+
batches**.
|
|
148
|
+
2. **Expiry.** The cookie lasts weeks, and a password change ends it early. `--doctor` tells you before
|
|
149
|
+
you spend a request.
|
|
150
|
+
3. **About six PAGE LOADS per session, and a page load is not a tool call.** Each requested section is
|
|
151
|
+
a separate navigation:
|
|
152
|
+
|
|
153
|
+
| call | page loads |
|
|
154
|
+
|---|---|
|
|
155
|
+
| `get_person_profile(slug)` | **1** |
|
|
156
|
+
| `get_person_profile(slug, sections: "experience,education")` | **3** |
|
|
157
|
+
|
|
158
|
+
Two profiles at 3 loads each = 6, and load 7 was refused. So `sections` is the expensive parameter,
|
|
159
|
+
not the number of people. **Budget in loads**: take the top card alone on a first pass (4–5 people),
|
|
160
|
+
come back for sections only on people who matter, and order the batch most-important-first — once
|
|
161
|
+
the wall hits, every later call returns nothing.
|
|
162
|
+
4. **Volume looks like automation.** LinkedIn restricts accounts for automated access. A captcha or a
|
|
163
|
+
"confirm in the mobile app" prompt means **stop** and tell the person; it needs a human on a phone,
|
|
164
|
+
and retrying escalates toward a restriction. Never sweep the whole `draft` queue.
|
|
165
|
+
|
|
166
|
+
If any of these fire, **fall back to web search** rather than retrying. A web-only pass is a legitimate
|
|
167
|
+
result as long as `creation_context` says so.
|
|
168
|
+
|
|
169
|
+
**When the session dies mid-batch, write up what you already got.** Record the profiles that arrived,
|
|
170
|
+
leave the rest `draft`, and say which is which.
|
|
171
|
+
|
|
172
|
+
## Reading failures
|
|
173
|
+
|
|
174
|
+
| what you see | what it means | do |
|
|
175
|
+
|---|---|---|
|
|
176
|
+
| `No valid LinkedIn session is available in Docker` (`isError: true`) | the cookie is gone | re-run `--import-from-browser` (or `--login`) on the host; the next batch picks it up |
|
|
177
|
+
| every navigation fails with `net::ERR_TOO_MANY_REDIRECTS` | LinkedIn is bouncing between its authwall and the target: the session is dead | check `<newest trace run>/screens/003-bridge-after-cookie-import.png` (`ls -td ~/.linkedin-mcp/trace-runs/*/ \| head -1`); a Sign in page confirms it. `auth_barrier=None` in the server log is not evidence — the navigation dies before the page is classified |
|
|
178
|
+
| a profile with every field `null` and no error | a stale server image, not a bad session | read the version in the startup banner; `docker pull stickerdaniel/linkedin-mcp-server:latest` (`:latest` is whatever was last pulled) |
|
|
179
|
+
| `BrowserBusyError: Another LinkedIn MCP client is using the browser` on `--login` or `--import-from-browser` | a host `mcp-server-linkedin` process never exited and holds `~/.linkedin-mcp/profile.lock` | `--doctor` names the pid under `host_clients`; `kill` it — the kernel releases the flock |
|
|
180
|
+
| `Feed auth check failed: HTTP ERROR 400` then `Imported cookies did not produce a valid session` | the import failed; the cookies may still be good | run `--login` instead of debugging the import |
|
|
181
|
+
| a section's value is `"Something went wrong"` | that section failed alone | re-fetch it by slug, in its own call |
|
|
182
|
+
| host CLI and container report different versions | the image drifted from the host tool | `docker pull stickerdaniel/linkedin-mcp-server` |
|
|
183
|
+
|
|
184
|
+
`--no-headless` does not reach the import's validation; the log says `headless=True` whatever you pass.
|
|
185
|
+
|
|
186
|
+
## Name → slug
|
|
187
|
+
|
|
188
|
+
`WebSearch` is the first angle — Google indexes the public shell of most profiles, so the slug is
|
|
189
|
+
usually in the result URL:
|
|
190
|
+
|
|
191
|
+
```
|
|
192
|
+
WebSearch: "Dana Oren" northwind linkedin.com/in
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
`get_company_employees(company_name, keywords)` is the second, and the strongest when you know the
|
|
196
|
+
employer but not the person. **`company_name` is the URL slug**, not the display name
|
|
197
|
+
(`northwind-group`, not `northwind`); `search_companies` returns it.
|
|
198
|
+
|
|
199
|
+
The third is free: **a scraped profile's `references` list carries other people's slugs.** Harvest them
|
|
200
|
+
from every scrape before spending a call on resolution.
|
|
201
|
+
|
|
202
|
+
**Confirm the person before scraping** on two matching signals (company, city, headline, role). A wrong
|
|
203
|
+
slug writes wrong data into a contact record, which is worse than a wasted call.
|
|
204
|
+
|
|
205
|
+
## What the profile says about your relationship
|
|
206
|
+
|
|
207
|
+
Parsed out of the `main_profile` text; capture it while you have the page, because nothing queries by
|
|
208
|
+
degree:
|
|
209
|
+
|
|
210
|
+
| in the text | means | record as |
|
|
211
|
+
|---|---|---|
|
|
212
|
+
| `· 1st` / `· 2nd` / `· 3rd` | network distance from the signed-in account | `linkedin_degree` |
|
|
213
|
+
| `Priya, Tom and 67 other mutual connections` | 69 mutuals — **a floor, not exact** | `linkedin_mutuals` |
|
|
214
|
+
| the action button: `Connect` / `Message` / `Pending` | `Pending` = an invitation is already out | note it in the body |
|
|
215
|
+
|
|
216
|
+
`linkedin_degree` and `linkedin_mutuals` are relative to the **signed-in account**. Write them only from
|
|
217
|
+
your own account; photos, names, roles and companies are account-neutral.
|
|
218
|
+
|
|
219
|
+
## Photos
|
|
220
|
+
|
|
221
|
+
`get_person_profile` returns text sections only — no profile-photo URL in any response — so
|
|
222
|
+
`contacts.avatar` cannot be filled from the scraper. The route that works is a browser session that is
|
|
223
|
+
already logged in, plus `bin/import-avatars.mjs`.
|
|
224
|
+
|
|
225
|
+
**The browser session** works the connections list (`/mynetwork/invite-connect/connections/`), where
|
|
226
|
+
every row already carries the avatar `<img>`, rather than one profile at a time. Tell it: no Connect,
|
|
227
|
+
Message or `…` clicks and no Messaging; stop on the first captcha and return what it has; and return one
|
|
228
|
+
JSON array keyed by contact id:
|
|
229
|
+
|
|
230
|
+
```json
|
|
231
|
+
[
|
|
232
|
+
{ "id": "dana-oren", "photo": "https://media.licdn.com/dms/image/…" },
|
|
233
|
+
{ "id": "sam-diaz", "photo": null, "note": "ghost avatar" }
|
|
234
|
+
]
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
**The import** fetches each URL, writes `assets/avatars/<id>` and sets `contacts.avatar`:
|
|
238
|
+
|
|
239
|
+
```sh
|
|
240
|
+
node .dreamteamer/skills/scraping-contacts/bin/import-avatars.mjs harvested.json --dry # then without --dry
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
It accepts only `media.licdn.com` URLs (a ghost avatar on `static.licdn.com` is rejected), sniffs the
|
|
244
|
+
**magic bytes** rather than trusting the HTTP status or the URL — an error page can arrive as `200 OK`
|
|
245
|
+
and a licdn URL has no extension — and refuses anything over the `assets` collection's 200 KB cap.
|
|
246
|
+
|
|
247
|
+
## Web search — after the scrape, or instead of it
|
|
248
|
+
|
|
249
|
+
LinkedIn is one source and often not the best one. After the scrape — or instead of it when the person
|
|
250
|
+
has no profile or the session is dead — fill the gaps from web search. It is free and risks nothing.
|
|
251
|
+
|
|
252
|
+
Web search reliably adds a **current** role when the profile is stale, company news that explains why
|
|
253
|
+
they are in a meeting, talks, podcasts, published writing, and a personal or company site. It is bad at
|
|
254
|
+
employment dates and at telling two people with the same name apart — cross-check against something
|
|
255
|
+
you already know (the company domain from their meeting, a city, a mutual contact).
|
|
256
|
+
|
|
257
|
+
## Writing the record
|
|
258
|
+
|
|
259
|
+
**`source` names what you read**, prefixed:
|
|
260
|
+
|
|
261
|
+
| you read | `source` |
|
|
262
|
+
|---|---|
|
|
263
|
+
| the LinkedIn scrape | `linkedin:<slug>` |
|
|
264
|
+
| a web page | `url:<the page>` |
|
|
265
|
+
| the calendar attendee list (a placeholder's starting point) | `calendar:<id>` |
|
|
266
|
+
|
|
267
|
+
`source` holds one value. When a pass read several, set the one most fields came from and name the
|
|
268
|
+
others in the body. When two sources **disagree** (LinkedIn says one company, a recent article another),
|
|
269
|
+
put the more current value in the field and name the discrepancy in the body with both sources.
|
|
270
|
+
Anything you inferred rather than read gets `confidence: low` and the reason in `creation_context`.
|
|
271
|
+
|
|
272
|
+
`author` is `agents/<harness>` (for example `agents/claude-code`); the model goes in `creation_context`.
|
|
273
|
+
|
|
274
|
+
**`status` is what makes the queue derivable** — `/scrape-contact` enters on `draft`:
|
|
275
|
+
|
|
276
|
+
- **`draft`** — a placeholder, usually created from a meeting's attendee list (`fetching-meetings`, in
|
|
277
|
+
`@dreamteamer/connectors-google-workspace`): a name, maybe an email.
|
|
278
|
+
- **`enriched`** — set it after a pass that actually produced something, from LinkedIn, web search or
|
|
279
|
+
both. A pass that found nothing leaves the record `draft` and says why in `creation_context`; an empty
|
|
280
|
+
record marked `enriched` drops out of the queue for good.
|
|
281
|
+
- **`verified`** — **never yours to set.** A person confirms a contact; a scraper does not.
|
|
282
|
+
|
|
283
|
+
Write scraped JSON dumps to scratch space; commit only the distilled contact records.
|
|
284
|
+
|
|
285
|
+
## Why there is no MCP entry
|
|
286
|
+
|
|
287
|
+
- **Docker Desktop's MCP Toolkit** pins an old v1 image whose config (a `cookie` secret mapped to
|
|
288
|
+
`LINKEDIN_COOKIE`) v4 no longer reads: the old image returns all-null profiles, and a retagged one
|
|
289
|
+
starts with no session at all.
|
|
290
|
+
- **A stdio `linkedin` entry in `.mcp.json`** serves exactly one client, so every agent session spawns
|
|
291
|
+
its own container — and two containers on one profile invalidate the cookie.
|
|
292
|
+
- **A long-lived HTTP server** would make the singleton explicit, at the price of a daemon holding a
|
|
293
|
+
live LinkedIn session.
|
|
294
|
+
|
|
295
|
+
Native `mcp__linkedin__*` tools need something reachable when a session starts, so you cannot have
|
|
296
|
+
native tools and zero idle footprint. `li.mjs` chooses zero idle footprint.
|
|
297
|
+
|
|
298
|
+
## Common mistakes
|
|
299
|
+
|
|
300
|
+
- **Calling `connect_with_person` or `send_message`** — irreversible, public, and as you.
|
|
301
|
+
- **Reading an all-null profile as an auth problem** — it is a stale image.
|
|
302
|
+
- **Trusting `auth_barrier=None`** on a redirect loop — look at the screenshot.
|
|
303
|
+
- **Adding the server to the MCP Toolkit or `.mcp.json`** — batches go through `bin/li.mjs`.
|
|
304
|
+
- **Passing the full URL where the slug is wanted** — `linkedin_username` is `dana-oren`, not
|
|
305
|
+
`https://www.linkedin.com/in/dana-oren/`.
|
|
306
|
+
- **Letting Docker create `~/.linkedin-mcp`** — it lands root-owned and the host login then fails.
|
|
307
|
+
- **Trusting a web-search hit blindly** — it is a candidate. Confirm on two signals, especially for
|
|
308
|
+
common names.
|
|
309
|
+
- **Asking for sections you will not write down** — each one is a page load against a budget of six.
|
|
310
|
+
- **Retrying through a captcha or a mobile-app confirmation** — stop and report.
|
|
311
|
+
- **Skipping web search because the scrape succeeded** — it usually has the current role.
|
|
312
|
+
- **Marking a record `enriched` when the pass found nothing, or setting `verified` at all.**
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Import the photo URLs a browser session harvested → `assets/avatars/<id>` + `contacts.avatar`.
|
|
3
|
+
//
|
|
4
|
+
// node import-avatars.mjs harvested.json [--dry]
|
|
5
|
+
//
|
|
6
|
+
// harvested.json: [{ "id": "<contact id>", "photo": "https://media.licdn.com/…" | null, "note"?: "…" }]
|
|
7
|
+
//
|
|
8
|
+
// Checks the bytes, not the status: an error page can arrive as HTTP 200 and a URL can name the wrong
|
|
9
|
+
// type, so the image type is sniffed from the magic bytes and the extension comes from that.
|
|
10
|
+
import { execFileSync } from 'node:child_process';
|
|
11
|
+
import fs from 'node:fs';
|
|
12
|
+
import os from 'node:os';
|
|
13
|
+
import path from 'node:path';
|
|
14
|
+
|
|
15
|
+
// The workspace root: the nearest folder carrying package-lock.json. `package.json` alone is not
|
|
16
|
+
// enough, because a module folder has one too.
|
|
17
|
+
function workspaceRoot(from = process.cwd()) {
|
|
18
|
+
let dir = path.resolve(from);
|
|
19
|
+
for (;;) {
|
|
20
|
+
// a folder sitting directly inside `git_modules/` is a module clone, never the workspace, even when `npm install` left it a lockfile
|
|
21
|
+
const inGitModules = path.basename(path.dirname(dir)) === 'git_modules';
|
|
22
|
+
if (!inGitModules && fs.existsSync(path.join(dir, 'package-lock.json'))) return dir;
|
|
23
|
+
const up = path.dirname(dir);
|
|
24
|
+
if (up === dir) throw new Error('not inside a dreamteamer workspace (no package-lock.json above ' + from + ')');
|
|
25
|
+
dir = up;
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
const WORKSPACE = workspaceRoot(path.dirname(new URL(import.meta.url).pathname));
|
|
29
|
+
const MAX = 204800; // the assets collection's own max_bytes
|
|
30
|
+
const dry = process.argv.includes('--dry');
|
|
31
|
+
const file = process.argv[2];
|
|
32
|
+
if (!file) { console.error('usage: import-avatars.mjs <harvested.json> [--dry]\n harvested.json: [{"id": "<contact id>", "photo": "https://media.licdn.com/…" | null, "note"?: "…"}]'); process.exit(2); }
|
|
33
|
+
|
|
34
|
+
/** Magic-byte sniff → extension, or null when these bytes are not an image we accept. */
|
|
35
|
+
function extOf(buf) {
|
|
36
|
+
if (buf.length < 12) return null;
|
|
37
|
+
const hex = buf.subarray(0, 12).toString('hex');
|
|
38
|
+
if (hex.startsWith('89504e47')) return 'png';
|
|
39
|
+
if (hex.startsWith('ffd8ff')) return 'jpg';
|
|
40
|
+
if (hex.startsWith('47494638')) return 'gif';
|
|
41
|
+
if (buf.subarray(0, 4).toString('ascii') === 'RIFF' && buf.subarray(8, 12).toString('ascii') === 'WEBP') return 'webp';
|
|
42
|
+
const head = buf.subarray(0, 400).toString('utf8').trimStart().toLowerCase();
|
|
43
|
+
if (head.startsWith('<svg') || head.startsWith('<?xml')) return 'svg';
|
|
44
|
+
return null; // an HTML error page lands here, which is the whole point
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
// The workspace's own installed engine, run directly: a workspace need not define an npm `dt` script.
|
|
48
|
+
function engineBin(root) {
|
|
49
|
+
const pkgPath = path.join(root, 'node_modules', 'dreamteamer', 'package.json');
|
|
50
|
+
const pkg = JSON.parse(fs.readFileSync(pkgPath, 'utf8'));
|
|
51
|
+
const bin = typeof pkg.bin === 'string' ? pkg.bin : (pkg.bin.dreamteamer ?? Object.values(pkg.bin)[0]);
|
|
52
|
+
return path.join(path.dirname(pkgPath), bin);
|
|
53
|
+
}
|
|
54
|
+
const dt = (...args) => execFileSync(process.execPath, [engineBin(WORKSPACE), ...args], { cwd: WORKSPACE, encoding: 'utf8' });
|
|
55
|
+
|
|
56
|
+
const rows = JSON.parse(fs.readFileSync(file, 'utf8'));
|
|
57
|
+
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'avatars-'));
|
|
58
|
+
let ok = 0, skipped = 0, failed = 0;
|
|
59
|
+
|
|
60
|
+
for (const row of rows) {
|
|
61
|
+
const { id, photo } = row ?? {};
|
|
62
|
+
if (!id) { console.log(`✖ entry with no id: ${JSON.stringify(row)}`); failed++; continue; }
|
|
63
|
+
if (!photo) { console.log(`· ${id}: no photo${row.note ? ` (${row.note})` : ''}`); skipped++; continue; }
|
|
64
|
+
if (!/^https:\/\/media\.licdn\.com\//.test(photo)) {
|
|
65
|
+
console.log(`✖ ${id}: not a media.licdn.com URL — a ghost avatar or the wrong link`); failed++; continue;
|
|
66
|
+
}
|
|
67
|
+
let buf;
|
|
68
|
+
try {
|
|
69
|
+
const res = await fetch(photo);
|
|
70
|
+
if (!res.ok) { console.log(`✖ ${id}: HTTP ${res.status}`); failed++; continue; }
|
|
71
|
+
buf = Buffer.from(await res.arrayBuffer());
|
|
72
|
+
} catch (e) { console.log(`✖ ${id}: ${e.message}`); failed++; continue; }
|
|
73
|
+
|
|
74
|
+
const ext = extOf(buf);
|
|
75
|
+
if (!ext) { console.log(`✖ ${id}: ${buf.length} bytes that are not an image (an error page?)`); failed++; continue; }
|
|
76
|
+
if (buf.length > MAX) { console.log(`✖ ${id}: ${buf.length} bytes, over max_bytes ${MAX}`); failed++; continue; }
|
|
77
|
+
|
|
78
|
+
const out = path.join(tmp, `${id.replace(/\//g, '_')}.${ext}`);
|
|
79
|
+
fs.writeFileSync(out, buf);
|
|
80
|
+
if (dry) { console.log(`· ${id}: would import ${buf.length} bytes (.${ext})`); ok++; continue; }
|
|
81
|
+
try {
|
|
82
|
+
dt('add', 'assets', `avatars/${id}`, '--from', out, '--force');
|
|
83
|
+
dt('set', `contacts/${id}`, `avatar=assets/avatars/${id}`);
|
|
84
|
+
console.log(`✔ ${id}: ${buf.length} bytes (.${ext})`);
|
|
85
|
+
ok++;
|
|
86
|
+
} catch (e) { console.log(`✖ ${id}: ${String(e.stderr ?? e.message).trim().split('\n')[0]}`); failed++; }
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
console.log(`\n${ok} imported · ${skipped} had no photo · ${failed} failed`);
|
|
90
|
+
if (!dry && ok) console.log(`then: npx dt check, and commit with\n npx dt commit ${rows.filter((r) => r.photo).map((r) => `assets/avatars/${r.id} contacts/${r.id}`).slice(0, 3).join(' ')} …`);
|
|
@@ -0,0 +1,285 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* li.mjs — drive linkedin-mcp-server for ONE batch, in ONE ephemeral container, then exit.
|
|
4
|
+
*
|
|
5
|
+
* Why this exists instead of an `.mcp.json` entry: the LinkedIn session is a SINGLETON. A stdio MCP
|
|
6
|
+
* entry spawns one container per agent session, and two containers on the one ~/.linkedin-mcp
|
|
7
|
+
* profile make LinkedIn invalidate the cookie — which costs a host re-login, not a retry. A stdio
|
|
8
|
+
* server cannot be shared between clients (one stdin/stdout pair), and a long-lived HTTP server would
|
|
9
|
+
* be a daemon holding a live session. So: no registered server, no idle container. The lock below is
|
|
10
|
+
* what makes "exactly one" true.
|
|
11
|
+
*
|
|
12
|
+
* Usage
|
|
13
|
+
* node li.mjs <tool> '<jsonArgs>' one call
|
|
14
|
+
* echo '[{"tool":..,"args":{..}}, ...]' | node li.mjs --stdin a batch, one container
|
|
15
|
+
* node li.mjs --doctor is the session alive? (no LinkedIn request)
|
|
16
|
+
*
|
|
17
|
+
* Always prints a JSON array of {tool, args, ok, result|error} to stdout; logs go to stderr.
|
|
18
|
+
* close_session is called for you at the end of every batch.
|
|
19
|
+
*/
|
|
20
|
+
import { spawn, execFileSync } from 'node:child_process';
|
|
21
|
+
import fs from 'node:fs';
|
|
22
|
+
import os from 'node:os';
|
|
23
|
+
import path from 'node:path';
|
|
24
|
+
|
|
25
|
+
const IMAGE = 'stickerdaniel/linkedin-mcp-server:latest';
|
|
26
|
+
const PROFILE = path.join(os.homedir(), '.linkedin-mcp');
|
|
27
|
+
const LOCK = path.join(PROFILE, 'linkedin-batch.lock'); // ours; the server owns profile.lock
|
|
28
|
+
const STALE_MS = 30 * 60 * 1000;
|
|
29
|
+
const CALL_TIMEOUT_MS = 300_000;
|
|
30
|
+
|
|
31
|
+
// ---------- the lock: a directory, because mkdir is atomic on every filesystem -------------------
|
|
32
|
+
function acquire() {
|
|
33
|
+
try {
|
|
34
|
+
fs.mkdirSync(LOCK);
|
|
35
|
+
} catch (e) {
|
|
36
|
+
if (e.code !== 'EEXIST') throw e;
|
|
37
|
+
const info = readLock();
|
|
38
|
+
if (info && Date.now() - info.at < STALE_MS && alive(info.pid)) {
|
|
39
|
+
throw new Error(
|
|
40
|
+
`another LinkedIn batch is running (pid ${info.pid}, started ${new Date(info.at).toISOString()}).\n`
|
|
41
|
+
+ 'Wait for it. Two containers on one profile invalidate the LinkedIn session — that costs a\n'
|
|
42
|
+
+ `host re-login, not a retry. If you are certain it is dead: rm -rf ${LOCK}`,
|
|
43
|
+
);
|
|
44
|
+
}
|
|
45
|
+
console.error(`[li] clearing a stale lock (${info ? `pid ${info.pid}` : 'unreadable'})`);
|
|
46
|
+
fs.rmSync(LOCK, { recursive: true, force: true });
|
|
47
|
+
fs.mkdirSync(LOCK);
|
|
48
|
+
}
|
|
49
|
+
fs.writeFileSync(path.join(LOCK, 'owner.json'), JSON.stringify({ pid: process.pid, at: Date.now() }));
|
|
50
|
+
}
|
|
51
|
+
const readLock = () => {
|
|
52
|
+
try { return JSON.parse(fs.readFileSync(path.join(LOCK, 'owner.json'), 'utf8')); } catch { return null; }
|
|
53
|
+
};
|
|
54
|
+
const alive = (pid) => { try { process.kill(pid, 0); return true; } catch { return false; } };
|
|
55
|
+
const release = () => { try { fs.rmSync(LOCK, { recursive: true, force: true }); } catch { /* gone */ } };
|
|
56
|
+
|
|
57
|
+
/** read one key from the nearest ancestor .env — the workspace convention for machine config */
|
|
58
|
+
function readEnvFile(key) {
|
|
59
|
+
let dir = process.cwd();
|
|
60
|
+
for (let i = 0; i < 6; i++) {
|
|
61
|
+
const f = path.join(dir, '.env');
|
|
62
|
+
if (fs.existsSync(f)) {
|
|
63
|
+
const m = fs.readFileSync(f, 'utf8').match(new RegExp(`^${key}=(.*)$`, 'm'));
|
|
64
|
+
if (m) return m[1].trim().replace(/^["']|["']$/g, '');
|
|
65
|
+
}
|
|
66
|
+
const up = path.dirname(dir);
|
|
67
|
+
if (up === dir) break;
|
|
68
|
+
dir = up;
|
|
69
|
+
}
|
|
70
|
+
return null;
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
// ---------- doctor: answer "is the session alive" from disk, spending no LinkedIn request --------
|
|
74
|
+
function doctor() {
|
|
75
|
+
const out = { profile_dir: PROFILE, exists: fs.existsSync(PROFILE) };
|
|
76
|
+
if (!out.exists) {
|
|
77
|
+
out.verdict = 'no profile — run: uvx mcp-server-linkedin@latest --import-from-browser';
|
|
78
|
+
return out;
|
|
79
|
+
}
|
|
80
|
+
out.cookies = fs.existsSync(path.join(PROFILE, 'cookies.json'));
|
|
81
|
+
try {
|
|
82
|
+
const st = JSON.parse(fs.readFileSync(path.join(PROFILE, 'source-state.json'), 'utf8'));
|
|
83
|
+
out.login_at = st.created_at;
|
|
84
|
+
out.age_days = +((Date.now() - Date.parse(st.created_at)) / 864e5).toFixed(1);
|
|
85
|
+
} catch { out.login_at = null; }
|
|
86
|
+
|
|
87
|
+
// the newest trace run's post-import screenshot is the ground truth: a real page is ~36KB,
|
|
88
|
+
// a bounce to the sign-in wall leaves it blank at ~4KB.
|
|
89
|
+
const loginMs = out.login_at ? Date.parse(out.login_at) : 0;
|
|
90
|
+
const runs = path.join(PROFILE, 'trace-runs');
|
|
91
|
+
if (fs.existsSync(runs)) {
|
|
92
|
+
// newest FIRST, but skip runs with no screenshot — a container killed before it scraped
|
|
93
|
+
// leaves an empty trace dir, and letting that mask the real evidence is how you conclude
|
|
94
|
+
// "no information" while the answer sits one directory down.
|
|
95
|
+
const ordered = fs.readdirSync(runs)
|
|
96
|
+
.map((d) => ({ d, m: fs.statSync(path.join(runs, d)).mtimeMs }))
|
|
97
|
+
.sort((a, b) => b.m - a.m);
|
|
98
|
+
out.newest_trace = ordered[0]?.d ?? null;
|
|
99
|
+
const withShot = ordered.find((r) => fs.existsSync(path.join(runs, r.d, 'screens', '004-feed-after-goto.png')));
|
|
100
|
+
if (withShot) {
|
|
101
|
+
const dir = path.join(runs, withShot.d, 'screens');
|
|
102
|
+
const bytes = fs.statSync(path.join(dir, '004-feed-after-goto.png')).size;
|
|
103
|
+
out.evidence_from = withShot.d;
|
|
104
|
+
out.last_feed_screenshot_bytes = bytes;
|
|
105
|
+
// A verdict from a run that PREDATES the current login is about the previous session, so it
|
|
106
|
+
// must not be reported as the current one.
|
|
107
|
+
if (loginMs && withShot.m < loginMs) {
|
|
108
|
+
out.evidence_is_stale = `trace is ${Math.round((loginMs - withShot.m) / 1000)}s older than the login`;
|
|
109
|
+
out.verdict = 'login is NEWER than any scraping evidence — session untested. Just run the batch.';
|
|
110
|
+
} else {
|
|
111
|
+
out.verdict = bytes < 10_000
|
|
112
|
+
? 'session looks DEAD (blank feed after cookie import). Fix: uvx mcp-server-linkedin@latest '
|
|
113
|
+
+ `--import-from-browser. Confirm by eye: ${path.join(dir, '003-bridge-after-cookie-import.png')}`
|
|
114
|
+
: 'last scraping run reached a real feed page — session was healthy then';
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
out.verdict ??= 'no trace evidence yet; the first call will tell you';
|
|
119
|
+
const held = fs.existsSync(LOCK) ? readLock() : null;
|
|
120
|
+
out.batch_lock = !held ? 'free'
|
|
121
|
+
: alive(held.pid) && Date.now() - held.at < STALE_MS ? `HELD by pid ${held.pid}`
|
|
122
|
+
: `stale (pid ${held.pid} is gone) — the next batch clears it automatically`;
|
|
123
|
+
out.containers_running = execFileSync('docker',
|
|
124
|
+
['ps', '--filter', `ancestor=${IMAGE}`, '--format', '{{.ID}}']).toString().trim().split('\n').filter(Boolean).length;
|
|
125
|
+
Object.assign(out, strandedClients());
|
|
126
|
+
return out;
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
/**
|
|
130
|
+
* Stranded HOST-side clients. A `mcp-server-linkedin` process that never exited keeps the flock on
|
|
131
|
+
* `~/.linkedin-mcp/profile.lock`, and while it lives every `--import-from-browser` and `--login`
|
|
132
|
+
* refuses with `BrowserBusyError` — so a dead-session verdict must not send you to re-login.
|
|
133
|
+
*
|
|
134
|
+
* This greps `ps` rather than reading the lock: macOS `lsof` shows an empty lock field even for a
|
|
135
|
+
* held flock, and Docker Desktop's VM always has the file open because `~/.linkedin-mcp` is
|
|
136
|
+
* bind-mounted into the container.
|
|
137
|
+
*
|
|
138
|
+
* flock is released by the KERNEL when its holder dies, so anything found here is alive and `kill`
|
|
139
|
+
* is the whole fix.
|
|
140
|
+
*/
|
|
141
|
+
function strandedClients() {
|
|
142
|
+
let lines = [];
|
|
143
|
+
try {
|
|
144
|
+
lines = execFileSync('ps', ['-axo', 'pid=,lstart=,command='], { stdio: ['ignore', 'pipe', 'ignore'] })
|
|
145
|
+
.toString().split('\n')
|
|
146
|
+
.filter((l) => /mcp-server-linkedin|patchright-browsers/.test(l) && !/\bgrep\b/.test(l));
|
|
147
|
+
} catch { return { host_clients: 'unknown (ps failed)' }; }
|
|
148
|
+
if (!lines.length) return { host_clients: 'none' };
|
|
149
|
+
// The chromium tree is noise once its parent is named; report the SERVER processes, and count
|
|
150
|
+
// the browsers so a half-dead tree is still visible.
|
|
151
|
+
const servers = lines.filter((l) => /mcp-server-linkedin/.test(l)).map((l) => {
|
|
152
|
+
const m = /^\s*(\d+)\s+(.{24})\s+(.*)$/.exec(l);
|
|
153
|
+
return m ? { pid: Number(m[1]), started: m[2].trim() } : null;
|
|
154
|
+
}).filter(Boolean);
|
|
155
|
+
const browsers = lines.filter((l) => /patchright-browsers/.test(l)).length;
|
|
156
|
+
if (!servers.length) return { host_clients: `${browsers} orphaned patchright browser process(es), no server — kill them before logging in` };
|
|
157
|
+
return {
|
|
158
|
+
host_clients: `HELD by ${servers.map((sv) => `pid ${sv.pid} (started ${sv.started})`).join(', ')}, ${browsers} browser process(es)`,
|
|
159
|
+
verdict_override: `a LinkedIn MCP process still holds the browser lease, so --login and --import-from-browser BOTH refuse with BrowserBusyError. Fix: kill ${servers.map((sv) => sv.pid).join(' ')}`,
|
|
160
|
+
};
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
// ---------- one container, N calls ---------------------------------------------------------------
|
|
164
|
+
async function runBatch(calls) {
|
|
165
|
+
// USER_AGENT exists to reduce anti-scraping detection: the session is minted on the host and used
|
|
166
|
+
// from a Linux container. Set LINKEDIN_USER_AGENT in .env to the browser the session came from.
|
|
167
|
+
const ua = process.env.LINKEDIN_USER_AGENT || readEnvFile('LINKEDIN_USER_AGENT');
|
|
168
|
+
if (!ua) console.error('[li] no LINKEDIN_USER_AGENT set — see the skill; this may raise detection');
|
|
169
|
+
const args = [
|
|
170
|
+
'run', '--rm', '-i', '--init', '--security-opt', 'no-new-privileges',
|
|
171
|
+
'-v', `${PROFILE}:/home/pwuser/.linkedin-mcp`,
|
|
172
|
+
'-e', 'TIMEOUT=10000', '-e', 'TOOL_TIMEOUT=300',
|
|
173
|
+
...(ua ? ['-e', `USER_AGENT=${ua}`] : []),
|
|
174
|
+
IMAGE,
|
|
175
|
+
];
|
|
176
|
+
const p = spawn('docker', args, { stdio: ['pipe', 'pipe', 'pipe'] });
|
|
177
|
+
const pending = new Map();
|
|
178
|
+
let buf = '';
|
|
179
|
+
let nextId = 1;
|
|
180
|
+
|
|
181
|
+
p.stdout.on('data', (d) => {
|
|
182
|
+
buf += d;
|
|
183
|
+
let i;
|
|
184
|
+
while ((i = buf.indexOf('\n')) !== -1) {
|
|
185
|
+
const line = buf.slice(0, i).trim(); buf = buf.slice(i + 1);
|
|
186
|
+
if (!line) continue;
|
|
187
|
+
let m; try { m = JSON.parse(line); } catch { continue; }
|
|
188
|
+
const r = pending.get(m.id);
|
|
189
|
+
if (r) { pending.delete(m.id); r(m); }
|
|
190
|
+
}
|
|
191
|
+
});
|
|
192
|
+
p.stderr.on('data', (d) => {
|
|
193
|
+
const s = String(d);
|
|
194
|
+
if (/ERROR|WARNING|Traceback|session/i.test(s)) process.stderr.write('[server] ' + s);
|
|
195
|
+
});
|
|
196
|
+
|
|
197
|
+
const rpc = (method, params) => new Promise((resolve, reject) => {
|
|
198
|
+
const id = nextId++;
|
|
199
|
+
pending.set(id, resolve);
|
|
200
|
+
const t = setTimeout(() => { pending.delete(id); reject(new Error(`timeout after ${CALL_TIMEOUT_MS}ms`)); }, CALL_TIMEOUT_MS);
|
|
201
|
+
const done = (v) => { clearTimeout(t); resolve(v); };
|
|
202
|
+
pending.set(id, done);
|
|
203
|
+
p.stdin.write(JSON.stringify({ jsonrpc: '2.0', id, method, params }) + '\n');
|
|
204
|
+
});
|
|
205
|
+
|
|
206
|
+
const init = await rpc('initialize', {
|
|
207
|
+
protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'dt-li', version: '1' },
|
|
208
|
+
});
|
|
209
|
+
console.error(`[li] ${init.result?.serverInfo?.name} ${init.result?.serverInfo?.version} — ${calls.length} call(s)`);
|
|
210
|
+
|
|
211
|
+
const out = [];
|
|
212
|
+
for (const [n, c] of calls.entries()) {
|
|
213
|
+
console.error(`[li] ${n + 1}/${calls.length} ${c.tool} ${JSON.stringify(c.args ?? {})}`);
|
|
214
|
+
let row = { tool: c.tool, args: c.args ?? {} };
|
|
215
|
+
try {
|
|
216
|
+
const m = await rpc('tools/call', { name: c.tool, arguments: c.args ?? {} });
|
|
217
|
+
if (m.error) row = { ...row, ok: false, error: m.error };
|
|
218
|
+
else {
|
|
219
|
+
const text = (m.result?.content ?? []).filter((x) => x.type === 'text').map((x) => x.text).join('\n');
|
|
220
|
+
let parsed; try { parsed = JSON.parse(text); } catch { parsed = text; }
|
|
221
|
+
// a non-error response can still carry per-section failures — surface them
|
|
222
|
+
const partial = parsed?.section_errors ? Object.keys(parsed.section_errors) : [];
|
|
223
|
+
row = { ...row, ok: !m.result?.isError, result: parsed, ...(partial.length ? { failed_sections: partial } : {}) };
|
|
224
|
+
}
|
|
225
|
+
} catch (e) {
|
|
226
|
+
row = { ...row, ok: false, error: e.message };
|
|
227
|
+
}
|
|
228
|
+
out.push(row);
|
|
229
|
+
}
|
|
230
|
+
|
|
231
|
+
try { await rpc('tools/call', { name: 'close_session', arguments: {} }); console.error('[li] session closed'); }
|
|
232
|
+
catch { console.error('[li] close_session did not answer (container is exiting anyway)'); }
|
|
233
|
+
p.kill();
|
|
234
|
+
return out;
|
|
235
|
+
}
|
|
236
|
+
|
|
237
|
+
// ---------- main --------------------------------------------------------------------------------
|
|
238
|
+
const argv = process.argv.slice(2);
|
|
239
|
+
if (argv[0] === '--doctor') { console.log(JSON.stringify(doctor(), null, 2)); process.exit(0); }
|
|
240
|
+
|
|
241
|
+
let calls;
|
|
242
|
+
if (argv[0] === '--stdin') {
|
|
243
|
+
calls = JSON.parse(fs.readFileSync(0, 'utf8'));
|
|
244
|
+
if (!Array.isArray(calls)) { console.error('--stdin wants a JSON array of {tool, args}'); process.exit(2); }
|
|
245
|
+
} else if (argv[0]) {
|
|
246
|
+
calls = [{ tool: argv[0], args: argv[1] ? JSON.parse(argv[1]) : {} }];
|
|
247
|
+
} else {
|
|
248
|
+
console.error(`usage:
|
|
249
|
+
li.mjs <tool> '<jsonArgs>'
|
|
250
|
+
echo '[{"tool":"get_person_profile","args":{"linkedin_username":"dana-oren"}}]' | li.mjs --stdin
|
|
251
|
+
li.mjs --doctor
|
|
252
|
+
|
|
253
|
+
NEVER call connect_with_person or send_message — they act publicly as you.`);
|
|
254
|
+
process.exit(2);
|
|
255
|
+
}
|
|
256
|
+
|
|
257
|
+
const BANNED = new Set(['connect_with_person', 'send_message']);
|
|
258
|
+
const bad = calls.filter((c) => BANNED.has(c.tool));
|
|
259
|
+
if (bad.length) {
|
|
260
|
+
console.error(`refusing ${bad.map((b) => b.tool).join(', ')} — these act publicly and irreversibly as`
|
|
261
|
+
+ ' you. If it is genuinely wanted, do it by hand, not through a scraper.');
|
|
262
|
+
process.exit(3);
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
try {
|
|
266
|
+
acquire();
|
|
267
|
+
} catch (e) {
|
|
268
|
+
// a contended lock is an expected outcome, not a crash — say so without a stack trace
|
|
269
|
+
console.error(`[li] ${e.message}`);
|
|
270
|
+
process.exit(4);
|
|
271
|
+
}
|
|
272
|
+
let code = 0;
|
|
273
|
+
try {
|
|
274
|
+
const results = await runBatch(calls);
|
|
275
|
+
console.log(JSON.stringify(results, null, 2));
|
|
276
|
+
code = results.every((r) => r.ok) ? 0 : 1;
|
|
277
|
+
} catch (e) {
|
|
278
|
+
console.error(`[li] batch failed: ${e.message}`);
|
|
279
|
+
code = 1;
|
|
280
|
+
} finally {
|
|
281
|
+
// MUST release before exiting: process.exit() does NOT run finally blocks, so an exit inside
|
|
282
|
+
// the try would leave the lock held after the batch is gone.
|
|
283
|
+
release();
|
|
284
|
+
}
|
|
285
|
+
process.exit(code);
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Every contact who is not both on LinkedIn and a 1st-degree connection; a row leaves by gaining a
|
|
2
|
+
# `linkedin` URL and `linkedin_degree: 1st`.
|
|
3
|
+
# `_neq` does not match a missing field, so the `_null` clauses are what list contacts with no degree.
|
|
4
|
+
# The filter is top-level: `options` passes to the layout untouched, so a filter there is read by nothing.
|
|
5
|
+
route: /contacts/needs-linkedin
|
|
6
|
+
scope: collection
|
|
7
|
+
collection: collections/contacts
|
|
8
|
+
filter:
|
|
9
|
+
_or:
|
|
10
|
+
- linkedin:
|
|
11
|
+
_null: true
|
|
12
|
+
- linkedin_degree:
|
|
13
|
+
_null: true
|
|
14
|
+
- linkedin_degree:
|
|
15
|
+
_neq: '1st'
|
|
16
|
+
display:
|
|
17
|
+
nav:
|
|
18
|
+
title: contacts — needs LinkedIn
|
|
19
|
+
icon: person-add
|
|
20
|
+
list:
|
|
21
|
+
layout: table
|
|
22
|
+
columns:
|
|
23
|
+
- avatar
|
|
24
|
+
- name
|
|
25
|
+
- role
|
|
26
|
+
- company
|
|
27
|
+
- linkedin_degree
|
|
28
|
+
- linkedin
|
|
29
|
+
sort: name
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# Contacts whose LinkedIn identity needs a person's look: no `linkedin` at all, or a slug tagged
|
|
2
|
+
# `linkedin:unsure` — a URL the record carries but nobody has confirmed. Remove the tag by hand once the
|
|
3
|
+
# profile is confirmed or corrected.
|
|
4
|
+
# Cards rather than a table, because `review_note` is a paragraph a table cell would clip; it is its own
|
|
5
|
+
# field because a template can render a field but cannot slice the record body. `columns` and `sort`
|
|
6
|
+
# still apply when the list is switched to a table.
|
|
7
|
+
route: /contacts/review-linkedin
|
|
8
|
+
scope: collection
|
|
9
|
+
collection: collections/contacts
|
|
10
|
+
# The filter is top-level: `options` passes to the layout untouched, so a filter there is read by nothing.
|
|
11
|
+
filter:
|
|
12
|
+
_or:
|
|
13
|
+
- linkedin:
|
|
14
|
+
_null: true
|
|
15
|
+
- tags:
|
|
16
|
+
_contains: 'linkedin:unsure'
|
|
17
|
+
display:
|
|
18
|
+
nav:
|
|
19
|
+
title: contacts — review LinkedIn
|
|
20
|
+
icon: question
|
|
21
|
+
list:
|
|
22
|
+
layout: cards
|
|
23
|
+
columns:
|
|
24
|
+
- avatar
|
|
25
|
+
- name
|
|
26
|
+
- company
|
|
27
|
+
- role
|
|
28
|
+
- linkedin
|
|
29
|
+
- tags
|
|
30
|
+
sort: name
|
|
31
|
+
options:
|
|
32
|
+
# No `image_field`: most people here have no photo, and an unset image still reserves a full-width box.
|
|
33
|
+
# `review_note` renders as plain text on the card (`format: markdown` sets the editor, not the view),
|
|
34
|
+
# so write it without markdown emphasis.
|
|
35
|
+
card_title: |-
|
|
36
|
+
### {{ name }}
|
|
37
|
+
{{ company }}
|
|
38
|
+
|
|
39
|
+
{{ review_note }}
|