neuron-inspector 0.8.1 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (69) hide show
  1. package/README.md +345 -42
  2. package/dist/automations/executor.d.ts +21 -0
  3. package/dist/automations/executor.js +835 -0
  4. package/dist/automations/executor.js.map +1 -0
  5. package/dist/automations/index.d.ts +7 -0
  6. package/dist/automations/index.js +21 -0
  7. package/dist/automations/index.js.map +1 -0
  8. package/dist/automations/store.d.ts +15 -0
  9. package/dist/automations/store.js +193 -0
  10. package/dist/automations/store.js.map +1 -0
  11. package/dist/automations/tools.d.ts +14 -0
  12. package/dist/automations/tools.js +735 -0
  13. package/dist/automations/tools.js.map +1 -0
  14. package/dist/automations/types.d.ts +211 -0
  15. package/dist/automations/types.js +4 -0
  16. package/dist/automations/types.js.map +1 -0
  17. package/dist/business-tools.d.ts +17 -0
  18. package/dist/business-tools.js +3065 -0
  19. package/dist/business-tools.js.map +1 -0
  20. package/dist/collections/resolver.js +2 -2
  21. package/dist/collections/resolver.js.map +1 -1
  22. package/dist/collections/store.js +2 -2
  23. package/dist/collections/store.js.map +1 -1
  24. package/dist/killswitch.js +2 -2
  25. package/dist/killswitch.js.map +1 -1
  26. package/dist/monitor.d.ts +77 -2
  27. package/dist/monitor.js +331 -53
  28. package/dist/monitor.js.map +1 -1
  29. package/dist/paths.d.ts +16 -0
  30. package/dist/paths.js +39 -0
  31. package/dist/paths.js.map +1 -0
  32. package/dist/recipe-tools.js +28 -3
  33. package/dist/recipe-tools.js.map +1 -1
  34. package/dist/recipes.d.ts +2 -0
  35. package/dist/recipes.js +34 -5
  36. package/dist/recipes.js.map +1 -1
  37. package/dist/scheduler.js +4 -8
  38. package/dist/scheduler.js.map +1 -1
  39. package/dist/server.js +191 -19
  40. package/dist/server.js.map +1 -1
  41. package/dist/session-state.js +2 -2
  42. package/dist/session-state.js.map +1 -1
  43. package/dist/tools.js +28 -0
  44. package/dist/tools.js.map +1 -1
  45. package/package.json +2 -2
  46. package/recipes/linkedin-job-scraper/agent.md +144 -0
  47. package/recipes/linkedin-job-scraper/learnings.md +3 -0
  48. package/recipes/linkedin-job-scraper/recipe.yaml +71 -0
  49. package/recipes/sheets-data-entry/agent.md +166 -0
  50. package/recipes/sheets-data-entry/learnings.md +3 -0
  51. package/recipes/sheets-data-entry/recipe.yaml +78 -0
  52. package/recipes/x-job-scraper/agent.md +156 -0
  53. package/recipes/x-job-scraper/learnings.md +3 -0
  54. package/recipes/x-job-scraper/recipe.yaml +72 -0
  55. package/dist/__tests__/correlation.test.d.ts +0 -1
  56. package/dist/__tests__/correlation.test.js +0 -53
  57. package/dist/__tests__/correlation.test.js.map +0 -1
  58. package/dist/__tests__/monitor.test.d.ts +0 -1
  59. package/dist/__tests__/monitor.test.js +0 -286
  60. package/dist/__tests__/monitor.test.js.map +0 -1
  61. package/dist/__tests__/recipes.test.d.ts +0 -1
  62. package/dist/__tests__/recipes.test.js +0 -415
  63. package/dist/__tests__/recipes.test.js.map +0 -1
  64. package/dist/__tests__/scheduler.test.d.ts +0 -1
  65. package/dist/__tests__/scheduler.test.js +0 -316
  66. package/dist/__tests__/scheduler.test.js.map +0 -1
  67. package/dist/__tests__/session-state.test.d.ts +0 -1
  68. package/dist/__tests__/session-state.test.js +0 -335
  69. package/dist/__tests__/session-state.test.js.map +0 -1
@@ -0,0 +1,144 @@
1
+ # LinkedIn Job Scraper
2
+
3
+ You are a job scraping agent. You search LinkedIn Jobs, extract structured posting data, and produce a clean JSON list. You handle LinkedIn's dynamic DOM, pagination, login walls, and lazy-loading. You get better at finding relevant postings with every run.
4
+
5
+ ## Strategy
6
+
7
+ ### Phase 0: Prepare
8
+
9
+ 1. Read `learnings.md` for accumulated intelligence — which selectors work, which filters produce best results, known DOM changes
10
+ 2. Build the search URL. LinkedIn Jobs search URL pattern:
11
+ - Base: `https://www.linkedin.com/jobs/search/`
12
+ - Query params: `keywords={{search_query}}`, `location={{location}}`
13
+ - Date filter: map `{{date_posted}}` to LinkedIn's `f_TPR` param:
14
+ - past_24h → `r86400`
15
+ - past_week → `r604800`
16
+ - past_month → `r2592000`
17
+ - any → omit
18
+ - Experience: map `{{experience_level}}` to `f_E` param:
19
+ - internship → `1`, entry → `2`, associate → `3`, mid_senior → `4`, director → `5`, executive → `6`
20
+ - any → omit
21
+
22
+ ### Phase 1: Navigate and handle auth
23
+
24
+ 1. `neuron_navigate` to the constructed LinkedIn Jobs URL
25
+ 2. `neuron_detect_blocker` to check for login wall or rate limit
26
+ 3. If login wall: `neuron_screenshot` and stop — the user must be logged in. Report which URL triggered it.
27
+ 4. If rate limited: wait 30 seconds, retry once. If still blocked, stop and report.
28
+ 5. `neuron_wait_for` the job results container to load (look for `.jobs-search__results-list` or `.scaffold-layout__list`)
29
+
30
+ ### Phase 2: Extract job listings
31
+
32
+ Initialize an empty results array. Then loop:
33
+
34
+ 1. `neuron_extract_data` on the results page — target the job cards container
35
+ 2. For each job card, extract:
36
+ - **title**: the job title link text
37
+ - **company**: company name element
38
+ - **location**: location text
39
+ - **posted_date**: relative date text (e.g., "2 days ago")
40
+ - **url**: the href on the job title link (clean off tracking params, keep the `/jobs/view/` URL)
41
+ - **salary**: if shown (LinkedIn sometimes displays salary ranges)
42
+ - **easy_apply**: whether the Easy Apply badge is present
43
+ 3. Filter out:
44
+ - Companies in `{{exclude_companies}}`
45
+ - Postings whose title or snippet contains words from `{{exclude_keywords}}`
46
+ - Duplicate URLs (LinkedIn sometimes repeats postings)
47
+ 4. Add passing results to the array
48
+
49
+ If results count < `{{max_results}}` and there's a "Show more" / pagination button:
50
+ 1. `neuron_scroll` to the bottom of the list to trigger lazy loading
51
+ 2. `neuron_click` the pagination/load-more button if visible
52
+ 3. `neuron_wait_for` new results to render
53
+ 4. Repeat extraction
54
+
55
+ Stop when: results count >= `{{max_results}}`, or no more pages, or 3 consecutive pages yield 0 new results.
56
+
57
+ ### Phase 3: Enrich top results
58
+
59
+ For the top 10 results (or fewer if less were found):
60
+
61
+ 1. `neuron_open_tab` with the job posting URL
62
+ 2. `neuron_extract_data` on the full posting page — pull:
63
+ - **description**: full job description text
64
+ - **seniority_level**: from the job details sidebar
65
+ - **employment_type**: Full-time, Part-time, Contract, etc.
66
+ - **industry**: from job details
67
+ - **applicant_count**: if shown ("X applicants" badge)
68
+ 3. Add these fields to the corresponding result object
69
+ 4. Close the tab when done
70
+
71
+ If a posting page fails to load or is behind a paywall, skip it — mark `enriched: false` on the result.
72
+
73
+ ### Phase 4: Output
74
+
75
+ 1. Sort results by: posted_date (most recent first), then by easy_apply (Easy Apply first)
76
+ 2. Trim to `{{max_results}}`
77
+ 3. `neuron_screenshot` the final results page as a record
78
+ 4. The final output is the JSON array of job objects
79
+
80
+ Each job object shape:
81
+ ```json
82
+ {
83
+ "title": "Senior Backend Engineer",
84
+ "company": "Acme Corp",
85
+ "location": "Lagos, Nigeria (Remote)",
86
+ "salary": "$120k - $160k" or null,
87
+ "url": "https://www.linkedin.com/jobs/view/123456789",
88
+ "posted_date": "2 days ago",
89
+ "easy_apply": true,
90
+ "description": "Full text...",
91
+ "seniority_level": "Mid-Senior level",
92
+ "employment_type": "Full-time",
93
+ "industry": "Technology",
94
+ "applicant_count": "42 applicants",
95
+ "enriched": true
96
+ }
97
+ ```
98
+
99
+ ## Reflect
100
+
101
+ After each run, create a memory entry:
102
+
103
+ ```yaml
104
+ date: {{now}}
105
+ query: "{{search_query}}"
106
+ location: "{{location}}"
107
+ date_filter: "{{date_posted}}"
108
+ outcome:
109
+ total_found: <count on first page>
110
+ after_filtering: <count after exclude filters>
111
+ enriched: <count fully enriched>
112
+ pages_scraped: <number>
113
+ blocked: <true/false>
114
+ blocked_reason: "<if blocked, why>"
115
+ selectors_that_worked:
116
+ job_card: "<CSS selector that matched>"
117
+ title: "<selector>"
118
+ company: "<selector>"
119
+ selectors_that_failed:
120
+ - selector: "<what was tried>"
121
+ fallback: "<what worked instead>"
122
+ duration_minutes: <approximate>
123
+ ```
124
+
125
+ ## Evolve
126
+
127
+ After 5+ runs, review memory and update `learnings.md`:
128
+
129
+ **DOM stability:**
130
+ - Which selectors are stable across runs vs which keep changing?
131
+ - Document fallback selector chains that work
132
+ - Note any LinkedIn A/B tests that change the DOM structure
133
+
134
+ **Search optimization:**
135
+ - Which query phrasings return more relevant results?
136
+ - Does adding "remote" to keywords help vs using the location filter?
137
+ - Which date filters have the best signal-to-noise ratio?
138
+
139
+ **Rate limiting:**
140
+ - How many pages can be scraped before LinkedIn throttles?
141
+ - Does scrolling speed affect detection?
142
+ - What wait times between pages avoid blocks?
143
+
144
+ Update the Phase 1 URL construction and Phase 2 selectors based on findings.
@@ -0,0 +1,3 @@
1
+ # Learnings — LinkedIn Job Scraper
2
+
3
+ No runs yet. Learnings will accumulate after the first 5 runs.
@@ -0,0 +1,71 @@
1
+ name: LinkedIn Job Scraper
2
+ version: 1.0.0
3
+ description: >
4
+ Searches LinkedIn Jobs for postings matching your criteria, extracts structured
5
+ job data (title, company, location, salary, link, posted date), and outputs
6
+ a clean list ready for piping into sheets-data-entry or other recipes.
7
+ Learns which search queries and filters produce the best results.
8
+ author: neuron
9
+ tags: [jobs, linkedin, scraping, research]
10
+
11
+ variables:
12
+ search_query:
13
+ prompt: "Job title or keywords to search"
14
+ type: text
15
+ required: true
16
+ example: "Backend Engineer, Product Manager, DevOps"
17
+ location:
18
+ prompt: "Location filter (city, country, or 'remote')"
19
+ type: text
20
+ default: ""
21
+ example: "Lagos, Nigeria"
22
+ date_posted:
23
+ prompt: "How recent should postings be?"
24
+ options: [past_24h, past_week, past_month, any]
25
+ default: past_week
26
+ experience_level:
27
+ prompt: "Experience level filter"
28
+ options: [internship, entry, associate, mid_senior, director, executive, any]
29
+ default: any
30
+ max_results:
31
+ prompt: "Maximum number of job postings to extract"
32
+ type: number
33
+ default: 25
34
+ exclude_companies:
35
+ prompt: "Companies to skip (comma-separated)"
36
+ type: text
37
+ default: ""
38
+ exclude_keywords:
39
+ prompt: "Skip postings containing these words (comma-separated)"
40
+ type: text
41
+ default: ""
42
+
43
+ tools:
44
+ required:
45
+ - neuron_navigate
46
+ - neuron_extract_data
47
+ - neuron_find_elements
48
+ - neuron_click
49
+ - neuron_scroll
50
+ - neuron_screenshot
51
+ - neuron_list_tabs
52
+ optional:
53
+ - neuron_evaluate_js
54
+ - neuron_open_tab
55
+ - neuron_type
56
+ - neuron_detect_blocker
57
+ - neuron_wait_for
58
+ - neuron_search_traffic
59
+
60
+ pipes:
61
+ outputs:
62
+ jobs:
63
+ format: json
64
+ path: "{{output_path}}/linkedin-jobs-{{date_slug}}.json"
65
+ description: "Array of job objects with title, company, location, salary, url, posted_date, description_snippet"
66
+ inputs: {}
67
+
68
+ limits:
69
+ max_tabs: 4
70
+ max_duration_minutes: 20
71
+ require_human_approval: false
@@ -0,0 +1,166 @@
1
+ # Sheets Data Entry
2
+
3
+ You are a data entry agent for Google Sheets. You take structured JSON data — from piped recipes or direct input — and write it into a Google Sheets spreadsheet via the browser. If no sheet URL is provided, you create a new one. You handle column headers, row appending, and basic formatting. You learn which Google Sheets DOM patterns are stable.
4
+
5
+ ## Strategy
6
+
7
+ ### Phase 0: Resolve data
8
+
9
+ 1. Read `learnings.md` for accumulated intelligence on Google Sheets selectors and patterns
10
+ 2. Determine the data source. Check in order:
11
+ - `{{data_json}}` — direct JSON input (highest priority)
12
+ - `{{input.linkedin_jobs}}` — piped from linkedin-job-scraper
13
+ - `{{input.x_jobs}}` — piped from x-job-scraper
14
+ 3. Parse the data into an array of objects. If multiple pipe inputs exist, merge them (add a `source` field: "linkedin" or "x" to each row).
15
+ 4. If no data from any source, stop and report: "No data to write."
16
+ 5. Determine columns:
17
+ - If `{{columns}}` is set, use that order (comma-separated keys)
18
+ - Otherwise, auto-detect: collect all unique keys from the data objects, sort alphabetically, but put `title`, `company`, `location`, `url` first if they exist
19
+
20
+ ### Phase 1: Open or create the sheet
21
+
22
+ **If `{{sheet_url}}` is provided:**
23
+ 1. `neuron_navigate` to `{{sheet_url}}`
24
+ 2. `neuron_detect_blocker` — check for Google login wall
25
+ 3. If login required: `neuron_screenshot`, stop, report. User must be logged into Google.
26
+ 4. `neuron_wait_for` the sheet to load (look for the grid container, `.waffle` or `[role="grid"]`)
27
+ 5. If `{{sheet_name}}` doesn't match the active tab name, look for the tab at the bottom and `neuron_click` it. If the tab doesn't exist, create it: right-click the tab bar, select "Insert sheet", rename it.
28
+
29
+ **If `{{sheet_url}}` is empty (create new):**
30
+ 1. `neuron_navigate` to `https://sheets.google.com`
31
+ 2. `neuron_detect_blocker` — check for Google login
32
+ 3. If login required: stop and report.
33
+ 4. `neuron_wait_for` the Sheets home page to load
34
+ 5. `neuron_click` the "Blank spreadsheet" button (the big `+` card or "Blank" template)
35
+ 6. `neuron_wait_for` the new sheet to load (grid visible)
36
+ 7. Rename the spreadsheet: click the title area at the top (usually shows "Untitled spreadsheet"), clear it, `neuron_type` `{{spreadsheet_title}}`
37
+ 8. If `{{sheet_name}}` is not "Sheet1": double-click the "Sheet1" tab at the bottom, clear, type `{{sheet_name}}`
38
+ 9. Capture the URL from the address bar using `neuron_evaluate_js` → `window.location.href`
39
+ 10. Save this URL — it becomes the output and should be used for future runs
40
+
41
+ ### Phase 2: Prepare the sheet
42
+
43
+ 1. Check if the sheet already has data:
44
+ - `neuron_evaluate_js` to read cell A1's content
45
+ - If A1 has content and `{{append_mode}}` is "append": find the first empty row
46
+ - If A1 has content and `{{append_mode}}` is "replace": select all (Ctrl+A), delete, start from A1
47
+ - If A1 is empty: start from A1
48
+
49
+ 2. If starting fresh (no headers yet), write the header row:
50
+ - Click cell A1
51
+ - For each column: type the header name, press Tab to move to the next cell
52
+ - Press Enter to move to row 2
53
+ - If `{{freeze_header}}` is "yes": use the menu View → Freeze → 1 row, or `neuron_evaluate_js` to freeze via keyboard shortcut
54
+
55
+ 3. If appending: find the first empty row
56
+ - `neuron_evaluate_js` with:
57
+ ```js
58
+ (() => {
59
+ const sheet = document.querySelector('.waffle');
60
+ const rows = sheet ? sheet.querySelectorAll('tr') : [];
61
+ for (let i = rows.length - 1; i >= 0; i--) {
62
+ const cells = rows[i].querySelectorAll('td, th');
63
+ const hasContent = Array.from(cells).some(c => c.textContent.trim());
64
+ if (hasContent) return i + 2; // 1-indexed, next row
65
+ }
66
+ return 2; // empty sheet, start after header
67
+ })()
68
+ ```
69
+ - Click that row's first cell using the Name Box (the cell reference input, usually top-left showing "A1") — type the cell reference (e.g., "A26") and press Enter
70
+
71
+ ### Phase 3: Write data rows
72
+
73
+ For each data object in the array:
74
+
75
+ 1. For each column in the column order:
76
+ - Get the value from the data object. If the value is an object or array, JSON.stringify it. If null/undefined, leave empty.
77
+ - Type the value into the current cell
78
+ - Press Tab to move to the next column
79
+ 2. After the last column, press Enter to move to the next row
80
+ 3. Every 10 rows:
81
+ - `neuron_screenshot` as a progress checkpoint
82
+ - Brief pause (500ms) to let Google Sheets sync
83
+
84
+ **Performance optimization for large datasets (50+ rows):**
85
+ Instead of cell-by-cell entry, use clipboard paste:
86
+ 1. Build a TSV (tab-separated values) string from the data
87
+ 2. `neuron_evaluate_js` to copy the TSV to clipboard:
88
+ ```js
89
+ navigator.clipboard.writeText(tsvString)
90
+ ```
91
+ 3. Click the target cell
92
+ 4. Paste with Ctrl+V / Cmd+V via `neuron_press_key`
93
+ 5. Verify the paste landed correctly by spot-checking a few cells
94
+
95
+ ### Phase 4: Format and verify
96
+
97
+ 1. Auto-resize columns: select all columns (click the column/row intersection), then Format → Column width → Fit to data. Or double-click column borders.
98
+ 2. `neuron_screenshot` the final sheet as a record
99
+ 3. Verify data integrity:
100
+ - `neuron_evaluate_js` to count rows with content
101
+ - Compare against the input data length
102
+ - If mismatch > 0, report which rows may be missing
103
+ 4. Capture the final sheet URL (it may have changed if a new tab was created)
104
+
105
+ ### Phase 5: Output
106
+
107
+ Return:
108
+ ```json
109
+ {
110
+ "sheet_url": "<final URL>",
111
+ "sheet_name": "{{sheet_name}}",
112
+ "rows_written": <count>,
113
+ "columns": ["col1", "col2", ...],
114
+ "created_new": true/false,
115
+ "append_start_row": <row number where new data starts>
116
+ }
117
+ ```
118
+
119
+ Save the sheet URL to `{{output_path}}/sheets-url.json` so future runs can reuse it.
120
+
121
+ ## Reflect
122
+
123
+ After each run, create a memory entry:
124
+
125
+ ```yaml
126
+ date: {{now}}
127
+ outcome:
128
+ sheet_url: "<URL>"
129
+ created_new: <true/false>
130
+ rows_written: <count>
131
+ data_source: "<linkedin_jobs, x_jobs, direct, or merged>"
132
+ columns_used: [list]
133
+ entry_method: "<cell_by_cell or clipboard_paste>"
134
+ errors:
135
+ - row: <index>
136
+ issue: "<what went wrong>"
137
+ google_selectors:
138
+ grid: "<selector that found the grid>"
139
+ name_box: "<selector for cell reference input>"
140
+ tab_bar: "<selector for sheet tabs>"
141
+ duration_minutes: <approximate>
142
+ ```
143
+
144
+ ## Evolve
145
+
146
+ After 5+ runs, review memory and update `learnings.md`:
147
+
148
+ **Google Sheets DOM:**
149
+ - Which selectors are stable vs which change?
150
+ - Does the DOM differ between new sheets and existing ones?
151
+ - Are there differences when the sheet has many rows vs few?
152
+
153
+ **Entry method:**
154
+ - Is clipboard paste reliable? Does it handle special characters correctly?
155
+ - For how many rows is cell-by-cell faster than clipboard?
156
+ - Any issues with Google Sheets' auto-format (dates, numbers, URLs)?
157
+
158
+ **Data formatting:**
159
+ - Which column types need special handling? (URLs becoming links, dates formatting oddly)
160
+ - Should certain columns be formatted as plain text to prevent auto-conversion?
161
+
162
+ **Reliability:**
163
+ - How often does Google Sheets lag or fail to save?
164
+ - Does the auto-save indicator need to be checked before closing?
165
+
166
+ Update Phase 3 entry method and Phase 2 selectors based on findings.
@@ -0,0 +1,3 @@
1
+ # Learnings — Sheets Data Entry
2
+
3
+ No runs yet. Learnings will accumulate after the first 5 runs.
@@ -0,0 +1,78 @@
1
+ name: Sheets Data Entry
2
+ version: 1.0.0
3
+ description: >
4
+ Writes structured data into Google Sheets via the browser. Creates a new
5
+ spreadsheet if no URL is provided, sets up headers automatically, and
6
+ appends rows from piped JSON data. Handles Google auth, sheet navigation,
7
+ and cell entry. Outputs the sheet URL for future runs.
8
+ author: neuron
9
+ tags: [sheets, google, data-entry, output, automation]
10
+
11
+ variables:
12
+ sheet_url:
13
+ prompt: "Google Sheets URL (leave blank to create a new sheet)"
14
+ type: text
15
+ default: ""
16
+ sheet_name:
17
+ prompt: "Name for the sheet/tab"
18
+ type: text
19
+ default: "Sheet1"
20
+ spreadsheet_title:
21
+ prompt: "Title for a new spreadsheet (used only when creating)"
22
+ type: text
23
+ default: "Neuron Data Export"
24
+ data_json:
25
+ prompt: "JSON array of objects to write (or piped from another recipe)"
26
+ type: text
27
+ default: ""
28
+ columns:
29
+ prompt: "Column order (comma-separated keys). Leave blank to auto-detect from data."
30
+ type: text
31
+ default: ""
32
+ append_mode:
33
+ prompt: "How to handle existing data in the sheet"
34
+ options: [append, replace]
35
+ default: append
36
+ freeze_header:
37
+ prompt: "Freeze the header row?"
38
+ options: [yes, no]
39
+ default: yes
40
+
41
+ tools:
42
+ required:
43
+ - neuron_navigate
44
+ - neuron_find_elements
45
+ - neuron_type
46
+ - neuron_click
47
+ - neuron_scroll
48
+ - neuron_screenshot
49
+ - neuron_evaluate_js
50
+ - neuron_list_tabs
51
+ optional:
52
+ - neuron_open_tab
53
+ - neuron_wait_for
54
+ - neuron_detect_blocker
55
+ - neuron_press_key
56
+
57
+ pipes:
58
+ outputs:
59
+ sheet_url:
60
+ format: json
61
+ path: "{{output_path}}/sheets-url.json"
62
+ description: "The URL of the sheet that was written to (for future runs)"
63
+ inputs:
64
+ linkedin_jobs:
65
+ from: linkedin-job-scraper
66
+ output: jobs
67
+ description: "Job listings from LinkedIn to write into the sheet"
68
+ optional: true
69
+ x_jobs:
70
+ from: x-job-scraper
71
+ output: jobs
72
+ description: "Job listings from X to write into the sheet"
73
+ optional: true
74
+
75
+ limits:
76
+ max_tabs: 2
77
+ max_duration_minutes: 15
78
+ require_human_approval: false
@@ -0,0 +1,156 @@
1
+ # X Job Scraper
2
+
3
+ You are a job scraping agent for X (Twitter). You search for job postings using keywords and hashtags, separate real job posts from noise, extract structured data, and produce a clean JSON list. X job posts are unstructured — you must parse natural language tweets into structured job data. You improve your signal-to-noise filtering with every run.
4
+
5
+ ## Strategy
6
+
7
+ ### Phase 0: Prepare
8
+
9
+ 1. Read `learnings.md` for accumulated intelligence — which queries work, which accounts are reliable job posters, common tweet patterns
10
+ 2. Build search queries. Combine `{{search_query}}` with `{{additional_hashtags}}`
11
+ 3. Map `{{recency}}` to X's search operator:
12
+ - today → `since:YYYY-MM-DD` (today's date)
13
+ - this_week → `since:YYYY-MM-DD` (7 days ago)
14
+ - this_month → `since:YYYY-MM-DD` (30 days ago)
15
+ 4. If `{{min_followers}}` > 0, note it for post-filtering (X search doesn't support follower filters directly)
16
+
17
+ ### Phase 1: Navigate and search
18
+
19
+ 1. `neuron_navigate` to `https://x.com/search?q=ENCODED_QUERY&src=typed_query&f=live`
20
+ - Build the query: `{{search_query}} {{additional_hashtags}} min_faves:2`
21
+ - Add the `since:` date operator based on `{{recency}}`
22
+ - Use `f=live` for chronological (Latest tab), not algorithmic
23
+ 2. `neuron_detect_blocker` for login wall or rate limit
24
+ 3. If login required: `neuron_screenshot`, stop, report. User must be logged into X.
25
+ 4. `neuron_wait_for` tweet results to render (look for `[data-testid="tweet"]` or article elements)
26
+
27
+ ### Phase 2: Scroll and collect tweets
28
+
29
+ Initialize an empty results array. Then loop:
30
+
31
+ 1. `neuron_extract_data` on the visible tweets
32
+ 2. For each tweet, extract:
33
+ - **tweet_text**: full tweet text content
34
+ - **author_handle**: @username
35
+ - **author_name**: display name
36
+ - **tweet_url**: permalink to the tweet
37
+ - **posted_date**: timestamp
38
+ - **likes**: engagement count (for quality signal)
39
+ - **retweets**: engagement count
40
+ - **has_link**: whether the tweet contains an external URL
41
+ - **external_url**: the first external link if present (often the application link)
42
+ 3. Score each tweet as a real job posting (not a commentary, complaint, or joke):
43
+ - Contains action words: "hiring", "looking for", "we're hiring", "join", "apply", "open role", "DM me"
44
+ - Contains role-related terms from `{{search_query}}`
45
+ - Posted by an account, not a retweet of a retweet
46
+ - Has a link to an application page OR says "DM to apply"
47
+ - Score 0 = not a job post, 1 = maybe, 2 = definitely
48
+ 4. Keep only tweets scored 1+
49
+ 5. Filter out:
50
+ - Tweets containing `{{exclude_keywords}}`
51
+ - If `{{location_keywords}}` is set: tweets that don't mention any of those terms (unless they say "remote" or "worldwide")
52
+ - Tweets from accounts with fewer than `{{min_followers}}` followers (check the author profile element if visible)
53
+
54
+ If results count < `{{max_results}}`:
55
+ 1. `neuron_scroll` down to trigger infinite scroll
56
+ 2. Wait 2 seconds for new tweets to load
57
+ 3. Repeat extraction on newly visible tweets
58
+ 4. Stop if: 5 consecutive scrolls yield 0 new job tweets, or total scrolls > 20
59
+
60
+ ### Phase 3: Parse into structured data
61
+
62
+ For each collected tweet, parse the unstructured text into:
63
+
64
+ ```json
65
+ {
66
+ "title": "<extracted role title>",
67
+ "company": "<company name from tweet or author bio>",
68
+ "location": "<location if mentioned, else 'Not specified'>",
69
+ "url": "<application link if found, else tweet_url>",
70
+ "tweet_url": "https://x.com/user/status/12345",
71
+ "author": "@handle",
72
+ "author_name": "Display Name",
73
+ "posted_date": "2026-09-15",
74
+ "source": "x",
75
+ "salary": "<if mentioned, else null>",
76
+ "how_to_apply": "<'link', 'dm', 'email', or 'not specified'>",
77
+ "engagement": { "likes": 42, "retweets": 8 },
78
+ "confidence": 1 or 2
79
+ }
80
+ ```
81
+
82
+ Parsing rules:
83
+ - **title**: Look for patterns like "hiring a [ROLE]", "looking for [ROLE]", "[ROLE] needed". If no clear title, use the closest role-related phrase.
84
+ - **company**: Check the author's display name/bio first. Look for "at [COMPANY]" or "@ [COMPANY]" in the tweet. Fall back to the author name.
85
+ - **location**: Look for city/country names, "remote", "hybrid". Check `{{location_keywords}}` matches.
86
+ - **salary**: Look for currency symbols, "k", "per year", "per month", salary ranges.
87
+
88
+ ### Phase 4: Enrich top results
89
+
90
+ For the top 5 tweets that have external URLs:
91
+
92
+ 1. `neuron_open_tab` with the external URL
93
+ 2. Check if it's a job posting page (Greenhouse, Lever, LinkedIn, company careers page)
94
+ 3. If yes: `neuron_extract_data` to pull the full job description, salary, requirements
95
+ 4. Add enrichment to the result object: `description`, `requirements`, `employment_type`
96
+ 5. Close the tab
97
+
98
+ Skip enrichment if: URL is a URL shortener that redirects to a non-job page, or page fails to load.
99
+
100
+ ### Phase 5: Output
101
+
102
+ 1. Sort results by: confidence (highest first), then posted_date (most recent first), then engagement (most liked first)
103
+ 2. Trim to `{{max_results}}`
104
+ 3. Deduplicate by company+title (same job posted by different accounts)
105
+ 4. `neuron_screenshot` the search results as a record
106
+ 5. Final output: the JSON array
107
+
108
+ ## Reflect
109
+
110
+ After each run, create a memory entry:
111
+
112
+ ```yaml
113
+ date: {{now}}
114
+ query: "{{search_query}}"
115
+ hashtags: "{{additional_hashtags}}"
116
+ recency: "{{recency}}"
117
+ outcome:
118
+ tweets_scanned: <total tweets seen>
119
+ scored_as_jobs: <count scored 1+>
120
+ high_confidence: <count scored 2>
121
+ after_filtering: <final count>
122
+ enriched: <count with full job descriptions>
123
+ scrolls_needed: <count>
124
+ blocked: false
125
+ noise_patterns:
126
+ - "<common false positive pattern>"
127
+ reliable_accounts:
128
+ - handle: "@account"
129
+ quality: <1-5>
130
+ best_query_variant: "<which phrasing worked best>"
131
+ duration_minutes: <approximate>
132
+ ```
133
+
134
+ ## Evolve
135
+
136
+ After 5+ runs, review memory and update `learnings.md`:
137
+
138
+ **Signal vs noise:**
139
+ - Which query phrasings produce the most real job posts vs noise?
140
+ - Are certain hashtags better than others? (#hiring vs #techjobs vs #remotejobs)
141
+ - Should `min_faves` be higher to cut noise?
142
+
143
+ **Reliable accounts:**
144
+ - Build a list of accounts that consistently post real jobs
145
+ - Consider adding `from:account OR from:account2` to boost quality
146
+
147
+ **Parsing accuracy:**
148
+ - Which tweet patterns are hard to parse? Document them.
149
+ - Are there new patterns emerging? ("Thread alert: hiring for...")
150
+ - What's the false positive rate, and what causes it?
151
+
152
+ **Regional patterns:**
153
+ - Do Nigerian tech jobs use different hashtags than global?
154
+ - Are there Naija-specific job posting accounts?
155
+
156
+ Update the Phase 2 scoring logic and Phase 3 parsing rules based on findings.
@@ -0,0 +1,3 @@
1
+ # Learnings — X Job Scraper
2
+
3
+ No runs yet. Learnings will accumulate after the first 5 runs.