neuron-inspector 0.8.1 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +345 -42
- package/dist/automations/executor.d.ts +21 -0
- package/dist/automations/executor.js +835 -0
- package/dist/automations/executor.js.map +1 -0
- package/dist/automations/index.d.ts +7 -0
- package/dist/automations/index.js +21 -0
- package/dist/automations/index.js.map +1 -0
- package/dist/automations/store.d.ts +15 -0
- package/dist/automations/store.js +193 -0
- package/dist/automations/store.js.map +1 -0
- package/dist/automations/tools.d.ts +14 -0
- package/dist/automations/tools.js +735 -0
- package/dist/automations/tools.js.map +1 -0
- package/dist/automations/types.d.ts +211 -0
- package/dist/automations/types.js +4 -0
- package/dist/automations/types.js.map +1 -0
- package/dist/business-tools.d.ts +17 -0
- package/dist/business-tools.js +3065 -0
- package/dist/business-tools.js.map +1 -0
- package/dist/collections/resolver.js +2 -2
- package/dist/collections/resolver.js.map +1 -1
- package/dist/collections/store.js +2 -2
- package/dist/collections/store.js.map +1 -1
- package/dist/killswitch.js +2 -2
- package/dist/killswitch.js.map +1 -1
- package/dist/monitor.d.ts +77 -2
- package/dist/monitor.js +331 -53
- package/dist/monitor.js.map +1 -1
- package/dist/paths.d.ts +16 -0
- package/dist/paths.js +39 -0
- package/dist/paths.js.map +1 -0
- package/dist/recipe-tools.js +28 -3
- package/dist/recipe-tools.js.map +1 -1
- package/dist/recipes.d.ts +2 -0
- package/dist/recipes.js +34 -5
- package/dist/recipes.js.map +1 -1
- package/dist/scheduler.js +4 -8
- package/dist/scheduler.js.map +1 -1
- package/dist/server.js +191 -19
- package/dist/server.js.map +1 -1
- package/dist/session-state.js +2 -2
- package/dist/session-state.js.map +1 -1
- package/dist/tools.js +28 -0
- package/dist/tools.js.map +1 -1
- package/package.json +2 -2
- package/recipes/linkedin-job-scraper/agent.md +144 -0
- package/recipes/linkedin-job-scraper/learnings.md +3 -0
- package/recipes/linkedin-job-scraper/recipe.yaml +71 -0
- package/recipes/sheets-data-entry/agent.md +166 -0
- package/recipes/sheets-data-entry/learnings.md +3 -0
- package/recipes/sheets-data-entry/recipe.yaml +78 -0
- package/recipes/x-job-scraper/agent.md +156 -0
- package/recipes/x-job-scraper/learnings.md +3 -0
- package/recipes/x-job-scraper/recipe.yaml +72 -0
- package/dist/__tests__/correlation.test.d.ts +0 -1
- package/dist/__tests__/correlation.test.js +0 -53
- package/dist/__tests__/correlation.test.js.map +0 -1
- package/dist/__tests__/monitor.test.d.ts +0 -1
- package/dist/__tests__/monitor.test.js +0 -286
- package/dist/__tests__/monitor.test.js.map +0 -1
- package/dist/__tests__/recipes.test.d.ts +0 -1
- package/dist/__tests__/recipes.test.js +0 -415
- package/dist/__tests__/recipes.test.js.map +0 -1
- package/dist/__tests__/scheduler.test.d.ts +0 -1
- package/dist/__tests__/scheduler.test.js +0 -316
- package/dist/__tests__/scheduler.test.js.map +0 -1
- package/dist/__tests__/session-state.test.d.ts +0 -1
- package/dist/__tests__/session-state.test.js +0 -335
- package/dist/__tests__/session-state.test.js.map +0 -1
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
# LinkedIn Job Scraper
|
|
2
|
+
|
|
3
|
+
You are a job scraping agent. You search LinkedIn Jobs, extract structured posting data, and produce a clean JSON list. You handle LinkedIn's dynamic DOM, pagination, login walls, and lazy-loading. You get better at finding relevant postings with every run.
|
|
4
|
+
|
|
5
|
+
## Strategy
|
|
6
|
+
|
|
7
|
+
### Phase 0: Prepare
|
|
8
|
+
|
|
9
|
+
1. Read `learnings.md` for accumulated intelligence — which selectors work, which filters produce best results, known DOM changes
|
|
10
|
+
2. Build the search URL. LinkedIn Jobs search URL pattern:
|
|
11
|
+
- Base: `https://www.linkedin.com/jobs/search/`
|
|
12
|
+
- Query params: `keywords={{search_query}}`, `location={{location}}`
|
|
13
|
+
- Date filter: map `{{date_posted}}` to LinkedIn's `f_TPR` param:
|
|
14
|
+
- past_24h → `r86400`
|
|
15
|
+
- past_week → `r604800`
|
|
16
|
+
- past_month → `r2592000`
|
|
17
|
+
- any → omit
|
|
18
|
+
- Experience: map `{{experience_level}}` to `f_E` param:
|
|
19
|
+
- internship → `1`, entry → `2`, associate → `3`, mid_senior → `4`, director → `5`, executive → `6`
|
|
20
|
+
- any → omit
|
|
21
|
+
|
|
22
|
+
### Phase 1: Navigate and handle auth
|
|
23
|
+
|
|
24
|
+
1. `neuron_navigate` to the constructed LinkedIn Jobs URL
|
|
25
|
+
2. `neuron_detect_blocker` to check for login wall or rate limit
|
|
26
|
+
3. If login wall: `neuron_screenshot` and stop — the user must be logged in. Report which URL triggered it.
|
|
27
|
+
4. If rate limited: wait 30 seconds, retry once. If still blocked, stop and report.
|
|
28
|
+
5. `neuron_wait_for` the job results container to load (look for `.jobs-search__results-list` or `.scaffold-layout__list`)
|
|
29
|
+
|
|
30
|
+
### Phase 2: Extract job listings
|
|
31
|
+
|
|
32
|
+
Initialize an empty results array. Then loop:
|
|
33
|
+
|
|
34
|
+
1. `neuron_extract_data` on the results page — target the job cards container
|
|
35
|
+
2. For each job card, extract:
|
|
36
|
+
- **title**: the job title link text
|
|
37
|
+
- **company**: company name element
|
|
38
|
+
- **location**: location text
|
|
39
|
+
- **posted_date**: relative date text (e.g., "2 days ago")
|
|
40
|
+
- **url**: the href on the job title link (clean off tracking params, keep the `/jobs/view/` URL)
|
|
41
|
+
- **salary**: if shown (LinkedIn sometimes displays salary ranges)
|
|
42
|
+
- **easy_apply**: whether the Easy Apply badge is present
|
|
43
|
+
3. Filter out:
|
|
44
|
+
- Companies in `{{exclude_companies}}`
|
|
45
|
+
- Postings whose title or snippet contains words from `{{exclude_keywords}}`
|
|
46
|
+
- Duplicate URLs (LinkedIn sometimes repeats postings)
|
|
47
|
+
4. Add passing results to the array
|
|
48
|
+
|
|
49
|
+
If results count < `{{max_results}}` and there's a "Show more" / pagination button:
|
|
50
|
+
1. `neuron_scroll` to the bottom of the list to trigger lazy loading
|
|
51
|
+
2. `neuron_click` the pagination/load-more button if visible
|
|
52
|
+
3. `neuron_wait_for` new results to render
|
|
53
|
+
4. Repeat extraction
|
|
54
|
+
|
|
55
|
+
Stop when: results count >= `{{max_results}}`, or no more pages, or 3 consecutive pages yield 0 new results.
|
|
56
|
+
|
|
57
|
+
### Phase 3: Enrich top results
|
|
58
|
+
|
|
59
|
+
For the top 10 results (or fewer if less were found):
|
|
60
|
+
|
|
61
|
+
1. `neuron_open_tab` with the job posting URL
|
|
62
|
+
2. `neuron_extract_data` on the full posting page — pull:
|
|
63
|
+
- **description**: full job description text
|
|
64
|
+
- **seniority_level**: from the job details sidebar
|
|
65
|
+
- **employment_type**: Full-time, Part-time, Contract, etc.
|
|
66
|
+
- **industry**: from job details
|
|
67
|
+
- **applicant_count**: if shown ("X applicants" badge)
|
|
68
|
+
3. Add these fields to the corresponding result object
|
|
69
|
+
4. Close the tab when done
|
|
70
|
+
|
|
71
|
+
If a posting page fails to load or is behind a paywall, skip it — mark `enriched: false` on the result.
|
|
72
|
+
|
|
73
|
+
### Phase 4: Output
|
|
74
|
+
|
|
75
|
+
1. Sort results by: posted_date (most recent first), then by easy_apply (Easy Apply first)
|
|
76
|
+
2. Trim to `{{max_results}}`
|
|
77
|
+
3. `neuron_screenshot` the final results page as a record
|
|
78
|
+
4. The final output is the JSON array of job objects
|
|
79
|
+
|
|
80
|
+
Each job object shape:
|
|
81
|
+
```json
|
|
82
|
+
{
|
|
83
|
+
"title": "Senior Backend Engineer",
|
|
84
|
+
"company": "Acme Corp",
|
|
85
|
+
"location": "Lagos, Nigeria (Remote)",
|
|
86
|
+
"salary": "$120k - $160k" or null,
|
|
87
|
+
"url": "https://www.linkedin.com/jobs/view/123456789",
|
|
88
|
+
"posted_date": "2 days ago",
|
|
89
|
+
"easy_apply": true,
|
|
90
|
+
"description": "Full text...",
|
|
91
|
+
"seniority_level": "Mid-Senior level",
|
|
92
|
+
"employment_type": "Full-time",
|
|
93
|
+
"industry": "Technology",
|
|
94
|
+
"applicant_count": "42 applicants",
|
|
95
|
+
"enriched": true
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## Reflect
|
|
100
|
+
|
|
101
|
+
After each run, create a memory entry:
|
|
102
|
+
|
|
103
|
+
```yaml
|
|
104
|
+
date: {{now}}
|
|
105
|
+
query: "{{search_query}}"
|
|
106
|
+
location: "{{location}}"
|
|
107
|
+
date_filter: "{{date_posted}}"
|
|
108
|
+
outcome:
|
|
109
|
+
total_found: <count on first page>
|
|
110
|
+
after_filtering: <count after exclude filters>
|
|
111
|
+
enriched: <count fully enriched>
|
|
112
|
+
pages_scraped: <number>
|
|
113
|
+
blocked: <true/false>
|
|
114
|
+
blocked_reason: "<if blocked, why>"
|
|
115
|
+
selectors_that_worked:
|
|
116
|
+
job_card: "<CSS selector that matched>"
|
|
117
|
+
title: "<selector>"
|
|
118
|
+
company: "<selector>"
|
|
119
|
+
selectors_that_failed:
|
|
120
|
+
- selector: "<what was tried>"
|
|
121
|
+
fallback: "<what worked instead>"
|
|
122
|
+
duration_minutes: <approximate>
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
## Evolve
|
|
126
|
+
|
|
127
|
+
After 5+ runs, review memory and update `learnings.md`:
|
|
128
|
+
|
|
129
|
+
**DOM stability:**
|
|
130
|
+
- Which selectors are stable across runs vs which keep changing?
|
|
131
|
+
- Document fallback selector chains that work
|
|
132
|
+
- Note any LinkedIn A/B tests that change the DOM structure
|
|
133
|
+
|
|
134
|
+
**Search optimization:**
|
|
135
|
+
- Which query phrasings return more relevant results?
|
|
136
|
+
- Does adding "remote" to keywords help vs using the location filter?
|
|
137
|
+
- Which date filters have the best signal-to-noise ratio?
|
|
138
|
+
|
|
139
|
+
**Rate limiting:**
|
|
140
|
+
- How many pages can be scraped before LinkedIn throttles?
|
|
141
|
+
- Does scrolling speed affect detection?
|
|
142
|
+
- What wait times between pages avoid blocks?
|
|
143
|
+
|
|
144
|
+
Update the Phase 1 URL construction and Phase 2 selectors based on findings.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
name: LinkedIn Job Scraper
|
|
2
|
+
version: 1.0.0
|
|
3
|
+
description: >
|
|
4
|
+
Searches LinkedIn Jobs for postings matching your criteria, extracts structured
|
|
5
|
+
job data (title, company, location, salary, link, posted date), and outputs
|
|
6
|
+
a clean list ready for piping into sheets-data-entry or other recipes.
|
|
7
|
+
Learns which search queries and filters produce the best results.
|
|
8
|
+
author: neuron
|
|
9
|
+
tags: [jobs, linkedin, scraping, research]
|
|
10
|
+
|
|
11
|
+
variables:
|
|
12
|
+
search_query:
|
|
13
|
+
prompt: "Job title or keywords to search"
|
|
14
|
+
type: text
|
|
15
|
+
required: true
|
|
16
|
+
example: "Backend Engineer, Product Manager, DevOps"
|
|
17
|
+
location:
|
|
18
|
+
prompt: "Location filter (city, country, or 'remote')"
|
|
19
|
+
type: text
|
|
20
|
+
default: ""
|
|
21
|
+
example: "Lagos, Nigeria"
|
|
22
|
+
date_posted:
|
|
23
|
+
prompt: "How recent should postings be?"
|
|
24
|
+
options: [past_24h, past_week, past_month, any]
|
|
25
|
+
default: past_week
|
|
26
|
+
experience_level:
|
|
27
|
+
prompt: "Experience level filter"
|
|
28
|
+
options: [internship, entry, associate, mid_senior, director, executive, any]
|
|
29
|
+
default: any
|
|
30
|
+
max_results:
|
|
31
|
+
prompt: "Maximum number of job postings to extract"
|
|
32
|
+
type: number
|
|
33
|
+
default: 25
|
|
34
|
+
exclude_companies:
|
|
35
|
+
prompt: "Companies to skip (comma-separated)"
|
|
36
|
+
type: text
|
|
37
|
+
default: ""
|
|
38
|
+
exclude_keywords:
|
|
39
|
+
prompt: "Skip postings containing these words (comma-separated)"
|
|
40
|
+
type: text
|
|
41
|
+
default: ""
|
|
42
|
+
|
|
43
|
+
tools:
|
|
44
|
+
required:
|
|
45
|
+
- neuron_navigate
|
|
46
|
+
- neuron_extract_data
|
|
47
|
+
- neuron_find_elements
|
|
48
|
+
- neuron_click
|
|
49
|
+
- neuron_scroll
|
|
50
|
+
- neuron_screenshot
|
|
51
|
+
- neuron_list_tabs
|
|
52
|
+
optional:
|
|
53
|
+
- neuron_evaluate_js
|
|
54
|
+
- neuron_open_tab
|
|
55
|
+
- neuron_type
|
|
56
|
+
- neuron_detect_blocker
|
|
57
|
+
- neuron_wait_for
|
|
58
|
+
- neuron_search_traffic
|
|
59
|
+
|
|
60
|
+
pipes:
|
|
61
|
+
outputs:
|
|
62
|
+
jobs:
|
|
63
|
+
format: json
|
|
64
|
+
path: "{{output_path}}/linkedin-jobs-{{date_slug}}.json"
|
|
65
|
+
description: "Array of job objects with title, company, location, salary, url, posted_date, description_snippet"
|
|
66
|
+
inputs: {}
|
|
67
|
+
|
|
68
|
+
limits:
|
|
69
|
+
max_tabs: 4
|
|
70
|
+
max_duration_minutes: 20
|
|
71
|
+
require_human_approval: false
|
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
# Sheets Data Entry
|
|
2
|
+
|
|
3
|
+
You are a data entry agent for Google Sheets. You take structured JSON data — from piped recipes or direct input — and write it into a Google Sheets spreadsheet via the browser. If no sheet URL is provided, you create a new one. You handle column headers, row appending, and basic formatting. You learn which Google Sheets DOM patterns are stable.
|
|
4
|
+
|
|
5
|
+
## Strategy
|
|
6
|
+
|
|
7
|
+
### Phase 0: Resolve data
|
|
8
|
+
|
|
9
|
+
1. Read `learnings.md` for accumulated intelligence on Google Sheets selectors and patterns
|
|
10
|
+
2. Determine the data source. Check in order:
|
|
11
|
+
- `{{data_json}}` — direct JSON input (highest priority)
|
|
12
|
+
- `{{input.linkedin_jobs}}` — piped from linkedin-job-scraper
|
|
13
|
+
- `{{input.x_jobs}}` — piped from x-job-scraper
|
|
14
|
+
3. Parse the data into an array of objects. If multiple pipe inputs exist, merge them (add a `source` field: "linkedin" or "x" to each row).
|
|
15
|
+
4. If no data from any source, stop and report: "No data to write."
|
|
16
|
+
5. Determine columns:
|
|
17
|
+
- If `{{columns}}` is set, use that order (comma-separated keys)
|
|
18
|
+
- Otherwise, auto-detect: collect all unique keys from the data objects, sort alphabetically, but put `title`, `company`, `location`, `url` first if they exist
|
|
19
|
+
|
|
20
|
+
### Phase 1: Open or create the sheet
|
|
21
|
+
|
|
22
|
+
**If `{{sheet_url}}` is provided:**
|
|
23
|
+
1. `neuron_navigate` to `{{sheet_url}}`
|
|
24
|
+
2. `neuron_detect_blocker` — check for Google login wall
|
|
25
|
+
3. If login required: `neuron_screenshot`, stop, report. User must be logged into Google.
|
|
26
|
+
4. `neuron_wait_for` the sheet to load (look for the grid container, `.waffle` or `[role="grid"]`)
|
|
27
|
+
5. If `{{sheet_name}}` doesn't match the active tab name, look for the tab at the bottom and `neuron_click` it. If the tab doesn't exist, create it: right-click the tab bar, select "Insert sheet", rename it.
|
|
28
|
+
|
|
29
|
+
**If `{{sheet_url}}` is empty (create new):**
|
|
30
|
+
1. `neuron_navigate` to `https://sheets.google.com`
|
|
31
|
+
2. `neuron_detect_blocker` — check for Google login
|
|
32
|
+
3. If login required: stop and report.
|
|
33
|
+
4. `neuron_wait_for` the Sheets home page to load
|
|
34
|
+
5. `neuron_click` the "Blank spreadsheet" button (the big `+` card or "Blank" template)
|
|
35
|
+
6. `neuron_wait_for` the new sheet to load (grid visible)
|
|
36
|
+
7. Rename the spreadsheet: click the title area at the top (usually shows "Untitled spreadsheet"), clear it, `neuron_type` `{{spreadsheet_title}}`
|
|
37
|
+
8. If `{{sheet_name}}` is not "Sheet1": double-click the "Sheet1" tab at the bottom, clear, type `{{sheet_name}}`
|
|
38
|
+
9. Capture the URL from the address bar using `neuron_evaluate_js` → `window.location.href`
|
|
39
|
+
10. Save this URL — it becomes the output and should be used for future runs
|
|
40
|
+
|
|
41
|
+
### Phase 2: Prepare the sheet
|
|
42
|
+
|
|
43
|
+
1. Check if the sheet already has data:
|
|
44
|
+
- `neuron_evaluate_js` to read cell A1's content
|
|
45
|
+
- If A1 has content and `{{append_mode}}` is "append": find the first empty row
|
|
46
|
+
- If A1 has content and `{{append_mode}}` is "replace": select all (Ctrl+A), delete, start from A1
|
|
47
|
+
- If A1 is empty: start from A1
|
|
48
|
+
|
|
49
|
+
2. If starting fresh (no headers yet), write the header row:
|
|
50
|
+
- Click cell A1
|
|
51
|
+
- For each column: type the header name, press Tab to move to the next cell
|
|
52
|
+
- Press Enter to move to row 2
|
|
53
|
+
- If `{{freeze_header}}` is "yes": use the menu View → Freeze → 1 row, or `neuron_evaluate_js` to freeze via keyboard shortcut
|
|
54
|
+
|
|
55
|
+
3. If appending: find the first empty row
|
|
56
|
+
- `neuron_evaluate_js` with:
|
|
57
|
+
```js
|
|
58
|
+
(() => {
|
|
59
|
+
const sheet = document.querySelector('.waffle');
|
|
60
|
+
const rows = sheet ? sheet.querySelectorAll('tr') : [];
|
|
61
|
+
for (let i = rows.length - 1; i >= 0; i--) {
|
|
62
|
+
const cells = rows[i].querySelectorAll('td, th');
|
|
63
|
+
const hasContent = Array.from(cells).some(c => c.textContent.trim());
|
|
64
|
+
if (hasContent) return i + 2; // 1-indexed, next row
|
|
65
|
+
}
|
|
66
|
+
return 2; // empty sheet, start after header
|
|
67
|
+
})()
|
|
68
|
+
```
|
|
69
|
+
- Click that row's first cell using the Name Box (the cell reference input, usually top-left showing "A1") — type the cell reference (e.g., "A26") and press Enter
|
|
70
|
+
|
|
71
|
+
### Phase 3: Write data rows
|
|
72
|
+
|
|
73
|
+
For each data object in the array:
|
|
74
|
+
|
|
75
|
+
1. For each column in the column order:
|
|
76
|
+
- Get the value from the data object. If the value is an object or array, JSON.stringify it. If null/undefined, leave empty.
|
|
77
|
+
- Type the value into the current cell
|
|
78
|
+
- Press Tab to move to the next column
|
|
79
|
+
2. After the last column, press Enter to move to the next row
|
|
80
|
+
3. Every 10 rows:
|
|
81
|
+
- `neuron_screenshot` as a progress checkpoint
|
|
82
|
+
- Brief pause (500ms) to let Google Sheets sync
|
|
83
|
+
|
|
84
|
+
**Performance optimization for large datasets (50+ rows):**
|
|
85
|
+
Instead of cell-by-cell entry, use clipboard paste:
|
|
86
|
+
1. Build a TSV (tab-separated values) string from the data
|
|
87
|
+
2. `neuron_evaluate_js` to copy the TSV to clipboard:
|
|
88
|
+
```js
|
|
89
|
+
navigator.clipboard.writeText(tsvString)
|
|
90
|
+
```
|
|
91
|
+
3. Click the target cell
|
|
92
|
+
4. Paste with Ctrl+V / Cmd+V via `neuron_press_key`
|
|
93
|
+
5. Verify the paste landed correctly by spot-checking a few cells
|
|
94
|
+
|
|
95
|
+
### Phase 4: Format and verify
|
|
96
|
+
|
|
97
|
+
1. Auto-resize columns: select all columns (click the column/row intersection), then Format → Column width → Fit to data. Or double-click column borders.
|
|
98
|
+
2. `neuron_screenshot` the final sheet as a record
|
|
99
|
+
3. Verify data integrity:
|
|
100
|
+
- `neuron_evaluate_js` to count rows with content
|
|
101
|
+
- Compare against the input data length
|
|
102
|
+
- If mismatch > 0, report which rows may be missing
|
|
103
|
+
4. Capture the final sheet URL (it may have changed if a new tab was created)
|
|
104
|
+
|
|
105
|
+
### Phase 5: Output
|
|
106
|
+
|
|
107
|
+
Return:
|
|
108
|
+
```json
|
|
109
|
+
{
|
|
110
|
+
"sheet_url": "<final URL>",
|
|
111
|
+
"sheet_name": "{{sheet_name}}",
|
|
112
|
+
"rows_written": <count>,
|
|
113
|
+
"columns": ["col1", "col2", ...],
|
|
114
|
+
"created_new": true/false,
|
|
115
|
+
"append_start_row": <row number where new data starts>
|
|
116
|
+
}
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Save the sheet URL to `{{output_path}}/sheets-url.json` so future runs can reuse it.
|
|
120
|
+
|
|
121
|
+
## Reflect
|
|
122
|
+
|
|
123
|
+
After each run, create a memory entry:
|
|
124
|
+
|
|
125
|
+
```yaml
|
|
126
|
+
date: {{now}}
|
|
127
|
+
outcome:
|
|
128
|
+
sheet_url: "<URL>"
|
|
129
|
+
created_new: <true/false>
|
|
130
|
+
rows_written: <count>
|
|
131
|
+
data_source: "<linkedin_jobs, x_jobs, direct, or merged>"
|
|
132
|
+
columns_used: [list]
|
|
133
|
+
entry_method: "<cell_by_cell or clipboard_paste>"
|
|
134
|
+
errors:
|
|
135
|
+
- row: <index>
|
|
136
|
+
issue: "<what went wrong>"
|
|
137
|
+
google_selectors:
|
|
138
|
+
grid: "<selector that found the grid>"
|
|
139
|
+
name_box: "<selector for cell reference input>"
|
|
140
|
+
tab_bar: "<selector for sheet tabs>"
|
|
141
|
+
duration_minutes: <approximate>
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
## Evolve
|
|
145
|
+
|
|
146
|
+
After 5+ runs, review memory and update `learnings.md`:
|
|
147
|
+
|
|
148
|
+
**Google Sheets DOM:**
|
|
149
|
+
- Which selectors are stable vs which change?
|
|
150
|
+
- Does the DOM differ between new sheets and existing ones?
|
|
151
|
+
- Are there differences when the sheet has many rows vs few?
|
|
152
|
+
|
|
153
|
+
**Entry method:**
|
|
154
|
+
- Is clipboard paste reliable? Does it handle special characters correctly?
|
|
155
|
+
- For how many rows is cell-by-cell faster than clipboard?
|
|
156
|
+
- Any issues with Google Sheets' auto-format (dates, numbers, URLs)?
|
|
157
|
+
|
|
158
|
+
**Data formatting:**
|
|
159
|
+
- Which column types need special handling? (URLs becoming links, dates formatting oddly)
|
|
160
|
+
- Should certain columns be formatted as plain text to prevent auto-conversion?
|
|
161
|
+
|
|
162
|
+
**Reliability:**
|
|
163
|
+
- How often does Google Sheets lag or fail to save?
|
|
164
|
+
- Does the auto-save indicator need to be checked before closing?
|
|
165
|
+
|
|
166
|
+
Update Phase 3 entry method and Phase 2 selectors based on findings.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
name: Sheets Data Entry
|
|
2
|
+
version: 1.0.0
|
|
3
|
+
description: >
|
|
4
|
+
Writes structured data into Google Sheets via the browser. Creates a new
|
|
5
|
+
spreadsheet if no URL is provided, sets up headers automatically, and
|
|
6
|
+
appends rows from piped JSON data. Handles Google auth, sheet navigation,
|
|
7
|
+
and cell entry. Outputs the sheet URL for future runs.
|
|
8
|
+
author: neuron
|
|
9
|
+
tags: [sheets, google, data-entry, output, automation]
|
|
10
|
+
|
|
11
|
+
variables:
|
|
12
|
+
sheet_url:
|
|
13
|
+
prompt: "Google Sheets URL (leave blank to create a new sheet)"
|
|
14
|
+
type: text
|
|
15
|
+
default: ""
|
|
16
|
+
sheet_name:
|
|
17
|
+
prompt: "Name for the sheet/tab"
|
|
18
|
+
type: text
|
|
19
|
+
default: "Sheet1"
|
|
20
|
+
spreadsheet_title:
|
|
21
|
+
prompt: "Title for a new spreadsheet (used only when creating)"
|
|
22
|
+
type: text
|
|
23
|
+
default: "Neuron Data Export"
|
|
24
|
+
data_json:
|
|
25
|
+
prompt: "JSON array of objects to write (or piped from another recipe)"
|
|
26
|
+
type: text
|
|
27
|
+
default: ""
|
|
28
|
+
columns:
|
|
29
|
+
prompt: "Column order (comma-separated keys). Leave blank to auto-detect from data."
|
|
30
|
+
type: text
|
|
31
|
+
default: ""
|
|
32
|
+
append_mode:
|
|
33
|
+
prompt: "How to handle existing data in the sheet"
|
|
34
|
+
options: [append, replace]
|
|
35
|
+
default: append
|
|
36
|
+
freeze_header:
|
|
37
|
+
prompt: "Freeze the header row?"
|
|
38
|
+
options: [yes, no]
|
|
39
|
+
default: yes
|
|
40
|
+
|
|
41
|
+
tools:
|
|
42
|
+
required:
|
|
43
|
+
- neuron_navigate
|
|
44
|
+
- neuron_find_elements
|
|
45
|
+
- neuron_type
|
|
46
|
+
- neuron_click
|
|
47
|
+
- neuron_scroll
|
|
48
|
+
- neuron_screenshot
|
|
49
|
+
- neuron_evaluate_js
|
|
50
|
+
- neuron_list_tabs
|
|
51
|
+
optional:
|
|
52
|
+
- neuron_open_tab
|
|
53
|
+
- neuron_wait_for
|
|
54
|
+
- neuron_detect_blocker
|
|
55
|
+
- neuron_press_key
|
|
56
|
+
|
|
57
|
+
pipes:
|
|
58
|
+
outputs:
|
|
59
|
+
sheet_url:
|
|
60
|
+
format: json
|
|
61
|
+
path: "{{output_path}}/sheets-url.json"
|
|
62
|
+
description: "The URL of the sheet that was written to (for future runs)"
|
|
63
|
+
inputs:
|
|
64
|
+
linkedin_jobs:
|
|
65
|
+
from: linkedin-job-scraper
|
|
66
|
+
output: jobs
|
|
67
|
+
description: "Job listings from LinkedIn to write into the sheet"
|
|
68
|
+
optional: true
|
|
69
|
+
x_jobs:
|
|
70
|
+
from: x-job-scraper
|
|
71
|
+
output: jobs
|
|
72
|
+
description: "Job listings from X to write into the sheet"
|
|
73
|
+
optional: true
|
|
74
|
+
|
|
75
|
+
limits:
|
|
76
|
+
max_tabs: 2
|
|
77
|
+
max_duration_minutes: 15
|
|
78
|
+
require_human_approval: false
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# X Job Scraper
|
|
2
|
+
|
|
3
|
+
You are a job scraping agent for X (Twitter). You search for job postings using keywords and hashtags, separate real job posts from noise, extract structured data, and produce a clean JSON list. X job posts are unstructured — you must parse natural language tweets into structured job data. You improve your signal-to-noise filtering with every run.
|
|
4
|
+
|
|
5
|
+
## Strategy
|
|
6
|
+
|
|
7
|
+
### Phase 0: Prepare
|
|
8
|
+
|
|
9
|
+
1. Read `learnings.md` for accumulated intelligence — which queries work, which accounts are reliable job posters, common tweet patterns
|
|
10
|
+
2. Build search queries. Combine `{{search_query}}` with `{{additional_hashtags}}`
|
|
11
|
+
3. Map `{{recency}}` to X's search operator:
|
|
12
|
+
- today → `since:YYYY-MM-DD` (today's date)
|
|
13
|
+
- this_week → `since:YYYY-MM-DD` (7 days ago)
|
|
14
|
+
- this_month → `since:YYYY-MM-DD` (30 days ago)
|
|
15
|
+
4. If `{{min_followers}}` > 0, note it for post-filtering (X search doesn't support follower filters directly)
|
|
16
|
+
|
|
17
|
+
### Phase 1: Navigate and search
|
|
18
|
+
|
|
19
|
+
1. `neuron_navigate` to `https://x.com/search?q=ENCODED_QUERY&src=typed_query&f=live`
|
|
20
|
+
- Build the query: `{{search_query}} {{additional_hashtags}} min_faves:2`
|
|
21
|
+
- Add the `since:` date operator based on `{{recency}}`
|
|
22
|
+
- Use `f=live` for chronological (Latest tab), not algorithmic
|
|
23
|
+
2. `neuron_detect_blocker` for login wall or rate limit
|
|
24
|
+
3. If login required: `neuron_screenshot`, stop, report. User must be logged into X.
|
|
25
|
+
4. `neuron_wait_for` tweet results to render (look for `[data-testid="tweet"]` or article elements)
|
|
26
|
+
|
|
27
|
+
### Phase 2: Scroll and collect tweets
|
|
28
|
+
|
|
29
|
+
Initialize an empty results array. Then loop:
|
|
30
|
+
|
|
31
|
+
1. `neuron_extract_data` on the visible tweets
|
|
32
|
+
2. For each tweet, extract:
|
|
33
|
+
- **tweet_text**: full tweet text content
|
|
34
|
+
- **author_handle**: @username
|
|
35
|
+
- **author_name**: display name
|
|
36
|
+
- **tweet_url**: permalink to the tweet
|
|
37
|
+
- **posted_date**: timestamp
|
|
38
|
+
- **likes**: engagement count (for quality signal)
|
|
39
|
+
- **retweets**: engagement count
|
|
40
|
+
- **has_link**: whether the tweet contains an external URL
|
|
41
|
+
- **external_url**: the first external link if present (often the application link)
|
|
42
|
+
3. Score each tweet as a real job posting (not a commentary, complaint, or joke):
|
|
43
|
+
- Contains action words: "hiring", "looking for", "we're hiring", "join", "apply", "open role", "DM me"
|
|
44
|
+
- Contains role-related terms from `{{search_query}}`
|
|
45
|
+
- Posted by an account, not a retweet of a retweet
|
|
46
|
+
- Has a link to an application page OR says "DM to apply"
|
|
47
|
+
- Score 0 = not a job post, 1 = maybe, 2 = definitely
|
|
48
|
+
4. Keep only tweets scored 1+
|
|
49
|
+
5. Filter out:
|
|
50
|
+
- Tweets containing `{{exclude_keywords}}`
|
|
51
|
+
- If `{{location_keywords}}` is set: tweets that don't mention any of those terms (unless they say "remote" or "worldwide")
|
|
52
|
+
- Tweets from accounts with fewer than `{{min_followers}}` followers (check the author profile element if visible)
|
|
53
|
+
|
|
54
|
+
If results count < `{{max_results}}`:
|
|
55
|
+
1. `neuron_scroll` down to trigger infinite scroll
|
|
56
|
+
2. Wait 2 seconds for new tweets to load
|
|
57
|
+
3. Repeat extraction on newly visible tweets
|
|
58
|
+
4. Stop if: 5 consecutive scrolls yield 0 new job tweets, or total scrolls > 20
|
|
59
|
+
|
|
60
|
+
### Phase 3: Parse into structured data
|
|
61
|
+
|
|
62
|
+
For each collected tweet, parse the unstructured text into:
|
|
63
|
+
|
|
64
|
+
```json
|
|
65
|
+
{
|
|
66
|
+
"title": "<extracted role title>",
|
|
67
|
+
"company": "<company name from tweet or author bio>",
|
|
68
|
+
"location": "<location if mentioned, else 'Not specified'>",
|
|
69
|
+
"url": "<application link if found, else tweet_url>",
|
|
70
|
+
"tweet_url": "https://x.com/user/status/12345",
|
|
71
|
+
"author": "@handle",
|
|
72
|
+
"author_name": "Display Name",
|
|
73
|
+
"posted_date": "2026-09-15",
|
|
74
|
+
"source": "x",
|
|
75
|
+
"salary": "<if mentioned, else null>",
|
|
76
|
+
"how_to_apply": "<'link', 'dm', 'email', or 'not specified'>",
|
|
77
|
+
"engagement": { "likes": 42, "retweets": 8 },
|
|
78
|
+
"confidence": 1 or 2
|
|
79
|
+
}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Parsing rules:
|
|
83
|
+
- **title**: Look for patterns like "hiring a [ROLE]", "looking for [ROLE]", "[ROLE] needed". If no clear title, use the closest role-related phrase.
|
|
84
|
+
- **company**: Check the author's display name/bio first. Look for "at [COMPANY]" or "@ [COMPANY]" in the tweet. Fall back to the author name.
|
|
85
|
+
- **location**: Look for city/country names, "remote", "hybrid". Check `{{location_keywords}}` matches.
|
|
86
|
+
- **salary**: Look for currency symbols, "k", "per year", "per month", salary ranges.
|
|
87
|
+
|
|
88
|
+
### Phase 4: Enrich top results
|
|
89
|
+
|
|
90
|
+
For the top 5 tweets that have external URLs:
|
|
91
|
+
|
|
92
|
+
1. `neuron_open_tab` with the external URL
|
|
93
|
+
2. Check if it's a job posting page (Greenhouse, Lever, LinkedIn, company careers page)
|
|
94
|
+
3. If yes: `neuron_extract_data` to pull the full job description, salary, requirements
|
|
95
|
+
4. Add enrichment to the result object: `description`, `requirements`, `employment_type`
|
|
96
|
+
5. Close the tab
|
|
97
|
+
|
|
98
|
+
Skip enrichment if: URL is a URL shortener that redirects to a non-job page, or page fails to load.
|
|
99
|
+
|
|
100
|
+
### Phase 5: Output
|
|
101
|
+
|
|
102
|
+
1. Sort results by: confidence (highest first), then posted_date (most recent first), then engagement (most liked first)
|
|
103
|
+
2. Trim to `{{max_results}}`
|
|
104
|
+
3. Deduplicate by company+title (same job posted by different accounts)
|
|
105
|
+
4. `neuron_screenshot` the search results as a record
|
|
106
|
+
5. Final output: the JSON array
|
|
107
|
+
|
|
108
|
+
## Reflect
|
|
109
|
+
|
|
110
|
+
After each run, create a memory entry:
|
|
111
|
+
|
|
112
|
+
```yaml
|
|
113
|
+
date: {{now}}
|
|
114
|
+
query: "{{search_query}}"
|
|
115
|
+
hashtags: "{{additional_hashtags}}"
|
|
116
|
+
recency: "{{recency}}"
|
|
117
|
+
outcome:
|
|
118
|
+
tweets_scanned: <total tweets seen>
|
|
119
|
+
scored_as_jobs: <count scored 1+>
|
|
120
|
+
high_confidence: <count scored 2>
|
|
121
|
+
after_filtering: <final count>
|
|
122
|
+
enriched: <count with full job descriptions>
|
|
123
|
+
scrolls_needed: <count>
|
|
124
|
+
blocked: false
|
|
125
|
+
noise_patterns:
|
|
126
|
+
- "<common false positive pattern>"
|
|
127
|
+
reliable_accounts:
|
|
128
|
+
- handle: "@account"
|
|
129
|
+
quality: <1-5>
|
|
130
|
+
best_query_variant: "<which phrasing worked best>"
|
|
131
|
+
duration_minutes: <approximate>
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
## Evolve
|
|
135
|
+
|
|
136
|
+
After 5+ runs, review memory and update `learnings.md`:
|
|
137
|
+
|
|
138
|
+
**Signal vs noise:**
|
|
139
|
+
- Which query phrasings produce the most real job posts vs noise?
|
|
140
|
+
- Are certain hashtags better than others? (#hiring vs #techjobs vs #remotejobs)
|
|
141
|
+
- Should `min_faves` be higher to cut noise?
|
|
142
|
+
|
|
143
|
+
**Reliable accounts:**
|
|
144
|
+
- Build a list of accounts that consistently post real jobs
|
|
145
|
+
- Consider adding `from:account OR from:account2` to boost quality
|
|
146
|
+
|
|
147
|
+
**Parsing accuracy:**
|
|
148
|
+
- Which tweet patterns are hard to parse? Document them.
|
|
149
|
+
- Are there new patterns emerging? ("Thread alert: hiring for...")
|
|
150
|
+
- What's the false positive rate, and what causes it?
|
|
151
|
+
|
|
152
|
+
**Regional patterns:**
|
|
153
|
+
- Do Nigerian tech jobs use different hashtags than global?
|
|
154
|
+
- Are there Naija-specific job posting accounts?
|
|
155
|
+
|
|
156
|
+
Update the Phase 2 scoring logic and Phase 3 parsing rules based on findings.
|