python-jobspy-damarowen 1.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- python_jobspy_damarowen-1.2.0/LICENSE +21 -0
- python_jobspy_damarowen-1.2.0/PKG-INFO +288 -0
- python_jobspy_damarowen-1.2.0/README.md +260 -0
- python_jobspy_damarowen-1.2.0/jobspy/__init__.py +229 -0
- python_jobspy_damarowen-1.2.0/jobspy/bayt/__init__.py +145 -0
- python_jobspy_damarowen-1.2.0/jobspy/bdjobs/__init__.py +353 -0
- python_jobspy_damarowen-1.2.0/jobspy/bdjobs/constant.py +32 -0
- python_jobspy_damarowen-1.2.0/jobspy/bdjobs/util.py +100 -0
- python_jobspy_damarowen-1.2.0/jobspy/exception.py +50 -0
- python_jobspy_damarowen-1.2.0/jobspy/glassdoor/__init__.py +322 -0
- python_jobspy_damarowen-1.2.0/jobspy/glassdoor/constant.py +184 -0
- python_jobspy_damarowen-1.2.0/jobspy/glassdoor/util.py +42 -0
- python_jobspy_damarowen-1.2.0/jobspy/google/__init__.py +202 -0
- python_jobspy_damarowen-1.2.0/jobspy/google/constant.py +52 -0
- python_jobspy_damarowen-1.2.0/jobspy/google/util.py +41 -0
- python_jobspy_damarowen-1.2.0/jobspy/indeed/__init__.py +260 -0
- python_jobspy_damarowen-1.2.0/jobspy/indeed/constant.py +109 -0
- python_jobspy_damarowen-1.2.0/jobspy/indeed/util.py +83 -0
- python_jobspy_damarowen-1.2.0/jobspy/jobstreet/__init__.py +254 -0
- python_jobspy_damarowen-1.2.0/jobspy/jobstreet/constant.py +16 -0
- python_jobspy_damarowen-1.2.0/jobspy/jobstreet/util.py +50 -0
- python_jobspy_damarowen-1.2.0/jobspy/linkedin/__init__.py +345 -0
- python_jobspy_damarowen-1.2.0/jobspy/linkedin/constant.py +8 -0
- python_jobspy_damarowen-1.2.0/jobspy/linkedin/util.py +96 -0
- python_jobspy_damarowen-1.2.0/jobspy/model.py +336 -0
- python_jobspy_damarowen-1.2.0/jobspy/naukri/__init__.py +304 -0
- python_jobspy_damarowen-1.2.0/jobspy/naukri/constant.py +11 -0
- python_jobspy_damarowen-1.2.0/jobspy/naukri/util.py +38 -0
- python_jobspy_damarowen-1.2.0/jobspy/util.py +363 -0
- python_jobspy_damarowen-1.2.0/jobspy/ziprecruiter/__init__.py +219 -0
- python_jobspy_damarowen-1.2.0/jobspy/ziprecruiter/constant.py +29 -0
- python_jobspy_damarowen-1.2.0/jobspy/ziprecruiter/util.py +31 -0
- python_jobspy_damarowen-1.2.0/pyproject.toml +34 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2023 Cullen Watson
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,288 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: python-jobspy-damarowen
|
|
3
|
+
Version: 1.2.0
|
|
4
|
+
Summary: Job scraper for LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, and JobStreet (Indonesia fork)
|
|
5
|
+
License-File: LICENSE
|
|
6
|
+
Keywords: jobs-scraper,jobstreet,linkedin,indeed,glassdoor,ziprecruiter,bayt,naukri
|
|
7
|
+
Author: Cullen Watson
|
|
8
|
+
Author-email: cullen@cullenwatson.com
|
|
9
|
+
Requires-Python: >=3.10,<4.0
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
16
|
+
Requires-Dist: beautifulsoup4 (>=4.12.2,<5.0.0)
|
|
17
|
+
Requires-Dist: markdownify (>=1.1.0,<2.0.0)
|
|
18
|
+
Requires-Dist: numpy (>=1.26.0)
|
|
19
|
+
Requires-Dist: pandas (>=2.1.0,<3.0.0)
|
|
20
|
+
Requires-Dist: pydantic (>=2.3.0,<3.0.0)
|
|
21
|
+
Requires-Dist: regex (>=2024.4.28,<2025.0.0)
|
|
22
|
+
Requires-Dist: requests (>=2.31.0,<3.0.0)
|
|
23
|
+
Requires-Dist: tls-client (>=1.0.1,<2.0.0)
|
|
24
|
+
Project-URL: Homepage, https://github.com/damarowen/JobSpy
|
|
25
|
+
Project-URL: Repository, https://github.com/damarowen/JobSpy
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
|
|
28
|
+
<img src="https://github.com/cullenwatson/JobSpy/assets/78247585/ae185b7e-e444-4712-8bb9-fa97f53e896b" width="400">
|
|
29
|
+
|
|
30
|
+
**JobSpy** is a job scraping library with the goal of aggregating all the jobs from popular job boards with one tool.
|
|
31
|
+
|
|
32
|
+
## Features
|
|
33
|
+
|
|
34
|
+
- Scrapes job postings from **LinkedIn**, **Indeed**, **Glassdoor**, **Google**, **ZipRecruiter**, & other job boards concurrently
|
|
35
|
+
- Aggregates the job postings in a dataframe
|
|
36
|
+
- Proxies support to bypass blocking
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
### Installation
|
|
41
|
+
|
|
42
|
+
```
|
|
43
|
+
pip install -U python-jobspy-damarowen
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
_Python version >= [3.10](https://www.python.org/downloads/release/python-3100/) required_
|
|
47
|
+
|
|
48
|
+
### Usage
|
|
49
|
+
|
|
50
|
+
```python
|
|
51
|
+
import csv
|
|
52
|
+
from jobspy import scrape_jobs
|
|
53
|
+
|
|
54
|
+
jobs = scrape_jobs(
|
|
55
|
+
site_name=["indeed", "linkedin", "zip_recruiter", "google"], # "glassdoor", "bayt", "naukri", "bdjobs"
|
|
56
|
+
search_term="software engineer",
|
|
57
|
+
google_search_term="software engineer jobs near San Francisco, CA since yesterday",
|
|
58
|
+
location="San Francisco, CA",
|
|
59
|
+
results_wanted=20,
|
|
60
|
+
hours_old=72,
|
|
61
|
+
country_indeed='USA',
|
|
62
|
+
|
|
63
|
+
# linkedin_fetch_description=True # gets more info such as description, direct job url (slower)
|
|
64
|
+
# proxies=["208.195.175.46:65095", "208.195.175.45:65095", "localhost"],
|
|
65
|
+
)
|
|
66
|
+
print(f"Found {len(jobs)} jobs")
|
|
67
|
+
print(jobs.head())
|
|
68
|
+
jobs.to_csv("jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\\", index=False) # to_excel
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
### Output
|
|
72
|
+
|
|
73
|
+
```
|
|
74
|
+
SITE TITLE COMPANY CITY STATE JOB_TYPE INTERVAL MIN_AMOUNT MAX_AMOUNT JOB_URL DESCRIPTION
|
|
75
|
+
indeed Software Engineer AMERICAN SYSTEMS Arlington VA None yearly 200000 150000 https://www.indeed.com/viewjob?jk=5e409e577046... THIS POSITION COMES WITH A 10K SIGNING BONUS!...
|
|
76
|
+
indeed Senior Software Engineer TherapyNotes.com Philadelphia PA fulltime yearly 135000 110000 https://www.indeed.com/viewjob?jk=da39574a40cb... About Us TherapyNotes is the national leader i...
|
|
77
|
+
linkedin Software Engineer - Early Career Lockheed Martin Sunnyvale CA fulltime yearly None None https://www.linkedin.com/jobs/view/3693012711 Description:By bringing together people that u...
|
|
78
|
+
linkedin Full-Stack Software Engineer Rain New York NY fulltime yearly None None https://www.linkedin.com/jobs/view/3696158877 Rain’s mission is to create the fastest and ea...
|
|
79
|
+
zip_recruiter Software Engineer - New Grad ZipRecruiter Santa Monica CA fulltime yearly 130000 150000 https://www.ziprecruiter.com/jobs/ziprecruiter... We offer a hybrid work environment. Most US-ba...
|
|
80
|
+
zip_recruiter Software Developer TEKsystems Phoenix AZ fulltime hourly 65 75 https://www.ziprecruiter.com/jobs/teksystems-0... Top Skills' Details• 6 years of Java developme...
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### Parameters for `scrape_jobs()`
|
|
85
|
+
|
|
86
|
+
```plaintext
|
|
87
|
+
Optional
|
|
88
|
+
├── site_name (list|str):
|
|
89
|
+
| linkedin, zip_recruiter, indeed, glassdoor, google, bayt, bdjobs, jobstreet
|
|
90
|
+
| (default is all)
|
|
91
|
+
│
|
|
92
|
+
├── search_term (str)
|
|
93
|
+
|
|
|
94
|
+
├── google_search_term (str)
|
|
95
|
+
| search term for google jobs. This is the only param for filtering google jobs.
|
|
96
|
+
│
|
|
97
|
+
├── location (str)
|
|
98
|
+
│
|
|
99
|
+
├── distance (int):
|
|
100
|
+
| in miles, default 50
|
|
101
|
+
│
|
|
102
|
+
├── job_type (str):
|
|
103
|
+
| fulltime, parttime, internship, contract
|
|
104
|
+
│
|
|
105
|
+
├── proxies (list):
|
|
106
|
+
| in format ['user:pass@host:port', 'localhost']
|
|
107
|
+
| each job board scraper will round robin through the proxies
|
|
108
|
+
|
|
|
109
|
+
├── is_remote (bool)
|
|
110
|
+
│
|
|
111
|
+
├── results_wanted (int):
|
|
112
|
+
| number of job results to retrieve for each site specified in 'site_name'
|
|
113
|
+
│
|
|
114
|
+
├── easy_apply (bool):
|
|
115
|
+
| filters for jobs that are hosted on the job board site (LinkedIn easy apply filter no longer works)
|
|
116
|
+
|
|
|
117
|
+
├── user_agent (str):
|
|
118
|
+
| override the default user agent which may be outdated
|
|
119
|
+
│
|
|
120
|
+
├── description_format (str):
|
|
121
|
+
| markdown, html (Format type of the job descriptions. Default is markdown.)
|
|
122
|
+
│
|
|
123
|
+
├── offset (int):
|
|
124
|
+
| starts the search from an offset (e.g. 25 will start the search from the 25th result)
|
|
125
|
+
│
|
|
126
|
+
├── hours_old (int):
|
|
127
|
+
| filters jobs by the number of hours since the job was posted
|
|
128
|
+
| (ZipRecruiter and Glassdoor round up to next day.)
|
|
129
|
+
│
|
|
130
|
+
├── verbose (int) {0, 1, 2}:
|
|
131
|
+
| Controls the verbosity of the runtime printouts
|
|
132
|
+
| (0 prints only errors, 1 is errors+warnings, 2 is all logs. Default is 2.)
|
|
133
|
+
|
|
134
|
+
├── linkedin_fetch_description (bool):
|
|
135
|
+
| fetches full description and direct job url for LinkedIn (Increases requests by O(n))
|
|
136
|
+
│
|
|
137
|
+
├── linkedin_company_ids (list[int]):
|
|
138
|
+
| searches for linkedin jobs with specific company ids
|
|
139
|
+
|
|
|
140
|
+
├── country_indeed (str):
|
|
141
|
+
| filters the country on Indeed & Glassdoor (see below for correct spelling)
|
|
142
|
+
|
|
|
143
|
+
├── enforce_annual_salary (bool):
|
|
144
|
+
| converts wages to annual salary
|
|
145
|
+
|
|
|
146
|
+
├── ca_cert (str)
|
|
147
|
+
| path to CA Certificate file for proxies
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
```
|
|
151
|
+
├── Indeed limitations:
|
|
152
|
+
| Only one from this list can be used in a search:
|
|
153
|
+
| - hours_old
|
|
154
|
+
| - job_type & is_remote
|
|
155
|
+
| - easy_apply
|
|
156
|
+
│
|
|
157
|
+
└── LinkedIn limitations:
|
|
158
|
+
| Only one from this list can be used in a search:
|
|
159
|
+
| - hours_old
|
|
160
|
+
| - easy_apply
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
## Supported Countries for Job Searching
|
|
164
|
+
|
|
165
|
+
### **LinkedIn**
|
|
166
|
+
|
|
167
|
+
LinkedIn searches globally & uses only the `location` parameter.
|
|
168
|
+
|
|
169
|
+
### **ZipRecruiter**
|
|
170
|
+
|
|
171
|
+
ZipRecruiter searches for jobs in **US/Canada** & uses only the `location` parameter.
|
|
172
|
+
|
|
173
|
+
### **Indeed / Glassdoor**
|
|
174
|
+
|
|
175
|
+
Indeed & Glassdoor supports most countries, but the `country_indeed` parameter is required. Additionally, use the `location`
|
|
176
|
+
parameter to narrow down the location, e.g. city & state if necessary.
|
|
177
|
+
|
|
178
|
+
You can specify the following countries when searching on Indeed (use the exact name, * indicates support for Glassdoor):
|
|
179
|
+
|
|
180
|
+
| | | | |
|
|
181
|
+
|----------------------|--------------|------------|----------------|
|
|
182
|
+
| Argentina | Australia* | Austria* | Bahrain |
|
|
183
|
+
| Belgium* | Brazil* | Canada* | Chile |
|
|
184
|
+
| China | Colombia | Costa Rica | Czech Republic |
|
|
185
|
+
| Denmark | Ecuador | Egypt | Finland |
|
|
186
|
+
| France* | Germany* | Greece | Hong Kong* |
|
|
187
|
+
| Hungary | India* | Indonesia | Ireland* |
|
|
188
|
+
| Israel | Italy* | Japan | Kuwait |
|
|
189
|
+
| Luxembourg | Malaysia | Mexico* | Morocco |
|
|
190
|
+
| Netherlands* | New Zealand* | Nigeria | Norway |
|
|
191
|
+
| Oman | Pakistan | Panama | Peru |
|
|
192
|
+
| Philippines | Poland | Portugal | Qatar |
|
|
193
|
+
| Romania | Saudi Arabia | Singapore* | South Africa |
|
|
194
|
+
| South Korea | Spain* | Sweden | Switzerland* |
|
|
195
|
+
| Taiwan | Thailand | Turkey | Ukraine |
|
|
196
|
+
| United Arab Emirates | UK* | USA* | Uruguay |
|
|
197
|
+
| Venezuela | Vietnam* | | |
|
|
198
|
+
|
|
199
|
+
### **Bayt**
|
|
200
|
+
|
|
201
|
+
Bayt only uses the search_term parameter currently and searches internationally
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
|
|
205
|
+
## Notes
|
|
206
|
+
* Indeed is the best scraper currently with no rate limiting.
|
|
207
|
+
* All the job board endpoints are capped at around 1000 jobs on a given search.
|
|
208
|
+
* LinkedIn is the most restrictive and usually rate limits around the 10th page with one ip. Proxies are a must basically.
|
|
209
|
+
|
|
210
|
+
## Frequently Asked Questions
|
|
211
|
+
|
|
212
|
+
---
|
|
213
|
+
**Q: Why is Indeed giving unrelated roles?**
|
|
214
|
+
**A:** Indeed searches the description too.
|
|
215
|
+
|
|
216
|
+
- use - to remove words
|
|
217
|
+
- "" for exact match
|
|
218
|
+
|
|
219
|
+
Example of a good Indeed query
|
|
220
|
+
|
|
221
|
+
```py
|
|
222
|
+
search_term='"engineering intern" software summer (java OR python OR c++) 2025 -tax -marketing'
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
This searches the description/title and must include software, summer, 2025, one of the languages, engineering intern exactly, no tax, no marketing.
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
|
|
229
|
+
**Q: No results when using "google"?**
|
|
230
|
+
**A:** You have to use super specific syntax. Search for google jobs on your browser and then whatever pops up in the google jobs search box after applying some filters is what you need to copy & paste into the google_search_term.
|
|
231
|
+
|
|
232
|
+
---
|
|
233
|
+
|
|
234
|
+
**Q: Received a response code 429?**
|
|
235
|
+
**A:** This indicates that you have been blocked by the job board site for sending too many requests. All of the job board sites are aggressive with blocking. We recommend:
|
|
236
|
+
|
|
237
|
+
- Wait some time between scrapes (site-dependent).
|
|
238
|
+
- Try using the proxies param to change your IP address.
|
|
239
|
+
|
|
240
|
+
---
|
|
241
|
+
|
|
242
|
+
### JobPost Schema
|
|
243
|
+
|
|
244
|
+
```plaintext
|
|
245
|
+
JobPost
|
|
246
|
+
├── title
|
|
247
|
+
├── company
|
|
248
|
+
├── company_url
|
|
249
|
+
├── job_url
|
|
250
|
+
├── location
|
|
251
|
+
│ ├── country
|
|
252
|
+
│ ├── city
|
|
253
|
+
│ ├── state
|
|
254
|
+
├── is_remote
|
|
255
|
+
├── description
|
|
256
|
+
├── job_type: fulltime, parttime, internship, contract
|
|
257
|
+
├── job_function
|
|
258
|
+
│ ├── interval: yearly, monthly, weekly, daily, hourly
|
|
259
|
+
│ ├── min_amount
|
|
260
|
+
│ ├── max_amount
|
|
261
|
+
│ ├── currency
|
|
262
|
+
│ └── salary_source: direct_data, description (parsed from posting)
|
|
263
|
+
├── date_posted
|
|
264
|
+
└── emails
|
|
265
|
+
|
|
266
|
+
Linkedin specific
|
|
267
|
+
└── job_level
|
|
268
|
+
|
|
269
|
+
Linkedin & Indeed specific
|
|
270
|
+
└── company_industry
|
|
271
|
+
|
|
272
|
+
Indeed specific
|
|
273
|
+
├── company_country
|
|
274
|
+
├── company_addresses
|
|
275
|
+
├── company_employees_label
|
|
276
|
+
├── company_revenue_label
|
|
277
|
+
├── company_description
|
|
278
|
+
└── company_logo
|
|
279
|
+
|
|
280
|
+
Naukri specific
|
|
281
|
+
├── skills
|
|
282
|
+
├── experience_range
|
|
283
|
+
├── company_rating
|
|
284
|
+
├── company_reviews_count
|
|
285
|
+
├── vacancy_count
|
|
286
|
+
└── work_from_home_type
|
|
287
|
+
```
|
|
288
|
+
|
|
@@ -0,0 +1,260 @@
|
|
|
1
|
+
<img src="https://github.com/cullenwatson/JobSpy/assets/78247585/ae185b7e-e444-4712-8bb9-fa97f53e896b" width="400">
|
|
2
|
+
|
|
3
|
+
**JobSpy** is a job scraping library with the goal of aggregating all the jobs from popular job boards with one tool.
|
|
4
|
+
|
|
5
|
+
## Features
|
|
6
|
+
|
|
7
|
+
- Scrapes job postings from **LinkedIn**, **Indeed**, **Glassdoor**, **Google**, **ZipRecruiter**, & other job boards concurrently
|
|
8
|
+
- Aggregates the job postings in a dataframe
|
|
9
|
+
- Proxies support to bypass blocking
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### Installation
|
|
14
|
+
|
|
15
|
+
```
|
|
16
|
+
pip install -U python-jobspy-damarowen
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
_Python version >= [3.10](https://www.python.org/downloads/release/python-3100/) required_
|
|
20
|
+
|
|
21
|
+
### Usage
|
|
22
|
+
|
|
23
|
+
```python
|
|
24
|
+
import csv
|
|
25
|
+
from jobspy import scrape_jobs
|
|
26
|
+
|
|
27
|
+
jobs = scrape_jobs(
|
|
28
|
+
site_name=["indeed", "linkedin", "zip_recruiter", "google"], # "glassdoor", "bayt", "naukri", "bdjobs"
|
|
29
|
+
search_term="software engineer",
|
|
30
|
+
google_search_term="software engineer jobs near San Francisco, CA since yesterday",
|
|
31
|
+
location="San Francisco, CA",
|
|
32
|
+
results_wanted=20,
|
|
33
|
+
hours_old=72,
|
|
34
|
+
country_indeed='USA',
|
|
35
|
+
|
|
36
|
+
# linkedin_fetch_description=True # gets more info such as description, direct job url (slower)
|
|
37
|
+
# proxies=["208.195.175.46:65095", "208.195.175.45:65095", "localhost"],
|
|
38
|
+
)
|
|
39
|
+
print(f"Found {len(jobs)} jobs")
|
|
40
|
+
print(jobs.head())
|
|
41
|
+
jobs.to_csv("jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\\", index=False) # to_excel
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
### Output
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
SITE TITLE COMPANY CITY STATE JOB_TYPE INTERVAL MIN_AMOUNT MAX_AMOUNT JOB_URL DESCRIPTION
|
|
48
|
+
indeed Software Engineer AMERICAN SYSTEMS Arlington VA None yearly 200000 150000 https://www.indeed.com/viewjob?jk=5e409e577046... THIS POSITION COMES WITH A 10K SIGNING BONUS!...
|
|
49
|
+
indeed Senior Software Engineer TherapyNotes.com Philadelphia PA fulltime yearly 135000 110000 https://www.indeed.com/viewjob?jk=da39574a40cb... About Us TherapyNotes is the national leader i...
|
|
50
|
+
linkedin Software Engineer - Early Career Lockheed Martin Sunnyvale CA fulltime yearly None None https://www.linkedin.com/jobs/view/3693012711 Description:By bringing together people that u...
|
|
51
|
+
linkedin Full-Stack Software Engineer Rain New York NY fulltime yearly None None https://www.linkedin.com/jobs/view/3696158877 Rain’s mission is to create the fastest and ea...
|
|
52
|
+
zip_recruiter Software Engineer - New Grad ZipRecruiter Santa Monica CA fulltime yearly 130000 150000 https://www.ziprecruiter.com/jobs/ziprecruiter... We offer a hybrid work environment. Most US-ba...
|
|
53
|
+
zip_recruiter Software Developer TEKsystems Phoenix AZ fulltime hourly 65 75 https://www.ziprecruiter.com/jobs/teksystems-0... Top Skills' Details• 6 years of Java developme...
|
|
54
|
+
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### Parameters for `scrape_jobs()`
|
|
58
|
+
|
|
59
|
+
```plaintext
|
|
60
|
+
Optional
|
|
61
|
+
├── site_name (list|str):
|
|
62
|
+
| linkedin, zip_recruiter, indeed, glassdoor, google, bayt, bdjobs, jobstreet
|
|
63
|
+
| (default is all)
|
|
64
|
+
│
|
|
65
|
+
├── search_term (str)
|
|
66
|
+
|
|
|
67
|
+
├── google_search_term (str)
|
|
68
|
+
| search term for google jobs. This is the only param for filtering google jobs.
|
|
69
|
+
│
|
|
70
|
+
├── location (str)
|
|
71
|
+
│
|
|
72
|
+
├── distance (int):
|
|
73
|
+
| in miles, default 50
|
|
74
|
+
│
|
|
75
|
+
├── job_type (str):
|
|
76
|
+
| fulltime, parttime, internship, contract
|
|
77
|
+
│
|
|
78
|
+
├── proxies (list):
|
|
79
|
+
| in format ['user:pass@host:port', 'localhost']
|
|
80
|
+
| each job board scraper will round robin through the proxies
|
|
81
|
+
|
|
|
82
|
+
├── is_remote (bool)
|
|
83
|
+
│
|
|
84
|
+
├── results_wanted (int):
|
|
85
|
+
| number of job results to retrieve for each site specified in 'site_name'
|
|
86
|
+
│
|
|
87
|
+
├── easy_apply (bool):
|
|
88
|
+
| filters for jobs that are hosted on the job board site (LinkedIn easy apply filter no longer works)
|
|
89
|
+
|
|
|
90
|
+
├── user_agent (str):
|
|
91
|
+
| override the default user agent which may be outdated
|
|
92
|
+
│
|
|
93
|
+
├── description_format (str):
|
|
94
|
+
| markdown, html (Format type of the job descriptions. Default is markdown.)
|
|
95
|
+
│
|
|
96
|
+
├── offset (int):
|
|
97
|
+
| starts the search from an offset (e.g. 25 will start the search from the 25th result)
|
|
98
|
+
│
|
|
99
|
+
├── hours_old (int):
|
|
100
|
+
| filters jobs by the number of hours since the job was posted
|
|
101
|
+
| (ZipRecruiter and Glassdoor round up to next day.)
|
|
102
|
+
│
|
|
103
|
+
├── verbose (int) {0, 1, 2}:
|
|
104
|
+
| Controls the verbosity of the runtime printouts
|
|
105
|
+
| (0 prints only errors, 1 is errors+warnings, 2 is all logs. Default is 2.)
|
|
106
|
+
|
|
107
|
+
├── linkedin_fetch_description (bool):
|
|
108
|
+
| fetches full description and direct job url for LinkedIn (Increases requests by O(n))
|
|
109
|
+
│
|
|
110
|
+
├── linkedin_company_ids (list[int]):
|
|
111
|
+
| searches for linkedin jobs with specific company ids
|
|
112
|
+
|
|
|
113
|
+
├── country_indeed (str):
|
|
114
|
+
| filters the country on Indeed & Glassdoor (see below for correct spelling)
|
|
115
|
+
|
|
|
116
|
+
├── enforce_annual_salary (bool):
|
|
117
|
+
| converts wages to annual salary
|
|
118
|
+
|
|
|
119
|
+
├── ca_cert (str)
|
|
120
|
+
| path to CA Certificate file for proxies
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
├── Indeed limitations:
|
|
125
|
+
| Only one from this list can be used in a search:
|
|
126
|
+
| - hours_old
|
|
127
|
+
| - job_type & is_remote
|
|
128
|
+
| - easy_apply
|
|
129
|
+
│
|
|
130
|
+
└── LinkedIn limitations:
|
|
131
|
+
| Only one from this list can be used in a search:
|
|
132
|
+
| - hours_old
|
|
133
|
+
| - easy_apply
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## Supported Countries for Job Searching
|
|
137
|
+
|
|
138
|
+
### **LinkedIn**
|
|
139
|
+
|
|
140
|
+
LinkedIn searches globally & uses only the `location` parameter.
|
|
141
|
+
|
|
142
|
+
### **ZipRecruiter**
|
|
143
|
+
|
|
144
|
+
ZipRecruiter searches for jobs in **US/Canada** & uses only the `location` parameter.
|
|
145
|
+
|
|
146
|
+
### **Indeed / Glassdoor**
|
|
147
|
+
|
|
148
|
+
Indeed & Glassdoor supports most countries, but the `country_indeed` parameter is required. Additionally, use the `location`
|
|
149
|
+
parameter to narrow down the location, e.g. city & state if necessary.
|
|
150
|
+
|
|
151
|
+
You can specify the following countries when searching on Indeed (use the exact name, * indicates support for Glassdoor):
|
|
152
|
+
|
|
153
|
+
| | | | |
|
|
154
|
+
|----------------------|--------------|------------|----------------|
|
|
155
|
+
| Argentina | Australia* | Austria* | Bahrain |
|
|
156
|
+
| Belgium* | Brazil* | Canada* | Chile |
|
|
157
|
+
| China | Colombia | Costa Rica | Czech Republic |
|
|
158
|
+
| Denmark | Ecuador | Egypt | Finland |
|
|
159
|
+
| France* | Germany* | Greece | Hong Kong* |
|
|
160
|
+
| Hungary | India* | Indonesia | Ireland* |
|
|
161
|
+
| Israel | Italy* | Japan | Kuwait |
|
|
162
|
+
| Luxembourg | Malaysia | Mexico* | Morocco |
|
|
163
|
+
| Netherlands* | New Zealand* | Nigeria | Norway |
|
|
164
|
+
| Oman | Pakistan | Panama | Peru |
|
|
165
|
+
| Philippines | Poland | Portugal | Qatar |
|
|
166
|
+
| Romania | Saudi Arabia | Singapore* | South Africa |
|
|
167
|
+
| South Korea | Spain* | Sweden | Switzerland* |
|
|
168
|
+
| Taiwan | Thailand | Turkey | Ukraine |
|
|
169
|
+
| United Arab Emirates | UK* | USA* | Uruguay |
|
|
170
|
+
| Venezuela | Vietnam* | | |
|
|
171
|
+
|
|
172
|
+
### **Bayt**
|
|
173
|
+
|
|
174
|
+
Bayt only uses the search_term parameter currently and searches internationally
|
|
175
|
+
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
## Notes
|
|
179
|
+
* Indeed is the best scraper currently with no rate limiting.
|
|
180
|
+
* All the job board endpoints are capped at around 1000 jobs on a given search.
|
|
181
|
+
* LinkedIn is the most restrictive and usually rate limits around the 10th page with one ip. Proxies are a must basically.
|
|
182
|
+
|
|
183
|
+
## Frequently Asked Questions
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
**Q: Why is Indeed giving unrelated roles?**
|
|
187
|
+
**A:** Indeed searches the description too.
|
|
188
|
+
|
|
189
|
+
- use - to remove words
|
|
190
|
+
- "" for exact match
|
|
191
|
+
|
|
192
|
+
Example of a good Indeed query
|
|
193
|
+
|
|
194
|
+
```py
|
|
195
|
+
search_term='"engineering intern" software summer (java OR python OR c++) 2025 -tax -marketing'
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
This searches the description/title and must include software, summer, 2025, one of the languages, engineering intern exactly, no tax, no marketing.
|
|
199
|
+
|
|
200
|
+
---
|
|
201
|
+
|
|
202
|
+
**Q: No results when using "google"?**
|
|
203
|
+
**A:** You have to use super specific syntax. Search for google jobs on your browser and then whatever pops up in the google jobs search box after applying some filters is what you need to copy & paste into the google_search_term.
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
**Q: Received a response code 429?**
|
|
208
|
+
**A:** This indicates that you have been blocked by the job board site for sending too many requests. All of the job board sites are aggressive with blocking. We recommend:
|
|
209
|
+
|
|
210
|
+
- Wait some time between scrapes (site-dependent).
|
|
211
|
+
- Try using the proxies param to change your IP address.
|
|
212
|
+
|
|
213
|
+
---
|
|
214
|
+
|
|
215
|
+
### JobPost Schema
|
|
216
|
+
|
|
217
|
+
```plaintext
|
|
218
|
+
JobPost
|
|
219
|
+
├── title
|
|
220
|
+
├── company
|
|
221
|
+
├── company_url
|
|
222
|
+
├── job_url
|
|
223
|
+
├── location
|
|
224
|
+
│ ├── country
|
|
225
|
+
│ ├── city
|
|
226
|
+
│ ├── state
|
|
227
|
+
├── is_remote
|
|
228
|
+
├── description
|
|
229
|
+
├── job_type: fulltime, parttime, internship, contract
|
|
230
|
+
├── job_function
|
|
231
|
+
│ ├── interval: yearly, monthly, weekly, daily, hourly
|
|
232
|
+
│ ├── min_amount
|
|
233
|
+
│ ├── max_amount
|
|
234
|
+
│ ├── currency
|
|
235
|
+
│ └── salary_source: direct_data, description (parsed from posting)
|
|
236
|
+
├── date_posted
|
|
237
|
+
└── emails
|
|
238
|
+
|
|
239
|
+
Linkedin specific
|
|
240
|
+
└── job_level
|
|
241
|
+
|
|
242
|
+
Linkedin & Indeed specific
|
|
243
|
+
└── company_industry
|
|
244
|
+
|
|
245
|
+
Indeed specific
|
|
246
|
+
├── company_country
|
|
247
|
+
├── company_addresses
|
|
248
|
+
├── company_employees_label
|
|
249
|
+
├── company_revenue_label
|
|
250
|
+
├── company_description
|
|
251
|
+
└── company_logo
|
|
252
|
+
|
|
253
|
+
Naukri specific
|
|
254
|
+
├── skills
|
|
255
|
+
├── experience_range
|
|
256
|
+
├── company_rating
|
|
257
|
+
├── company_reviews_count
|
|
258
|
+
├── vacancy_count
|
|
259
|
+
└── work_from_home_type
|
|
260
|
+
```
|