jobdrop 2.0.0__tar.gz

jobdrop-2.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Cullen Watson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

jobdrop-2.0.0/PKG-INFO

@@ -0,0 +1,210 @@
+Metadata-Version: 2.4
+Name: jobdrop
+Version: 2.0.0
+Summary: 20-source job scraper library + MCP server. LinkedIn, Indeed, Glassdoor, Google, ZipRecruiter, Wellfound, Hiring Cafe, Greenhouse, USAJobs, Adzuna, Jooble, Findwork, The Muse, Insight Global, Clearance Jobs, Kforce, CollabWork, Naukri, Bayt, BDJobs.
+License-File: LICENSE
+Keywords: jobs-scraper,mcp,linkedin,indeed,glassdoor,ziprecruiter,wellfound,hiring-cafe,greenhouse,usajobs
+Author: kbwhodat
+Author-email: katobyan@gmail.com
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Provides-Extra: mcp
+Requires-Dist: beautifulsoup4 (>=4.12.2,<5.0.0)
+Requires-Dist: camoufox (>=0.4.11,<0.5.0)
+Requires-Dist: curl-cffi (>=0.7.0,<0.8.0)
+Requires-Dist: fastmcp (>=0.2.0) ; extra == "mcp"
+Requires-Dist: markdownify (>=1.1.0,<2.0.0)
+Requires-Dist: mcp (>=1.1.0) ; extra == "mcp"
+Requires-Dist: numpy (>=1.26.0)
+Requires-Dist: pandas (>=2.1.0,<3.0.0)
+Requires-Dist: pydantic (>=2.3.0,<3.0.0)
+Requires-Dist: regex (>=2024.4.28,<2025.0.0)
+Requires-Dist: requests (>=2.31.0,<3.0.0)
+Requires-Dist: selenium-driverless (>=1.9.4,<2.0.0)
+Requires-Dist: tls-client (>=1.0.1,<2.0.0)
+Project-URL: Homepage, https://github.com/kbwhodat/jobdrop
+Description-Content-Type: text/markdown
+
+# jobdrop
+
+A multi-source job scraper. Hits 20 job boards in one call, normalizes
+the results into a pandas DataFrame, and ships with anti-bot bypasses
+for the boards that block standard scrapers.
+
+## What's in here
+
+### 17 sources
+
+| `site_name` | Source | Mechanism |
+|---|---|---|
+| `linkedin` | LinkedIn | Public listing scrape with optional detail-page enrichment |
+| `indeed` | Indeed | GraphQL with the `Int!` radius fix + per-company cap + paginate-until-quota |
+| `glassdoor` | Glassdoor | selenium-driverless headless to defeat Cloudflare 403; in-page GraphQL fetch |
+| `google` | Google Jobs | selenium-driverless headless against `udm=8`; SERP DOM walk |
+| `zip_recruiter` | ZipRecruiter | `curl_cffi` + `safari17_2_ios` TLS impersonation against the web HTML endpoint |
+| `bayt` | Bayt | Public scrape |
+| `naukri` | Naukri | Public scrape |
+| `bdjobs` | BDJobs | Public scrape |
+| `usajobs` | USAJobs.gov | Federal public API |
+| `adzuna` | Adzuna | Public API |
+| `jooble` | Jooble | Public API |
+| `findwork` | Findwork.dev | Public API |
+| `the_muse` | The Muse | Public API |
+| `insight_global` | Insight Global staffing | Server-rendered HTML scrape with hidden JSON blob per result |
+| `clearance_jobs` | ClearanceJobs (DHI) | Public JSON API + parallel detail-page enrichment for full JD, salary, type, remote bool |
+| `kforce` | Kforce staffing | Direct Azure Cognitive Search calls (bypasses Imperva on the public host) |
+| `greenhouse` | Greenhouse-hosted boards | Google `site:` dorks via selenium-driverless → public Greenhouse API; 3-layer staleness filter |
+
+### Anti-bot solved
+
+- **Google** — selenium-driverless cold-start headless. Defeats the 2026 CAPTCHA wall that takes out Playwright / undetected-chromedriver / nodriver / patchright.
+- **Glassdoor** — selenium-driverless rewrite to bypass Cloudflare 403; URL-encoded location, partial-GraphQL-error tolerance.
+- **ZipRecruiter** — `curl_cffi` + `safari17_2_ios` against the web HTML endpoint. The iOS-app API is dead behind Cloudflare.
+- **Kforce** — bypasses Imperva on the public host by calling the Azure Cognitive Search backend directly.
+- **Greenhouse** — uses the same selenium-driverless infrastructure as Google for `site:` dorks across all greenhouse-hosted boards.
+
+### Other tightening
+
+- **LinkedIn** — salary extraction from description body, optional per-company cap, parallel detail fetches.
+- **Indeed** — fixed `radius=25` default after Indeed promoted the GraphQL field to `Int!`; per-company cap to surface diverse employers; pagination loop fixed.
+- **ClearanceJobs** — search API gives a 200-char preview; this fork parallel-fetches `/api/v1/jobs/{id}` so you get the full JD, salary range, structured `job_type`, and authoritative `remote` bool.
+- **Greenhouse** — three layers of stale-protection (404 drop / past `application_deadline` / `first_published` age with 90-day default that respects `hours_old`).
+
+### Bundled credentials
+
+API keys for USAJobs, Adzuna, Jooble, Findwork, and The Muse are baked
+into a positional resolver (`jobdrop/_defaults.py`) so the new sources
+work without environment setup. User-set env vars still win via
+`setdefault` semantics.
+
+## Installation
+
+```
+pip install -U jobdrop
+```
+
+Python ≥ 3.10 required.
+
+## Usage
+
+```python
+from jobdrop import scrape_jobs
+
+jobs = scrape_jobs(
+    site_name=["insight_global", "clearance_jobs", "kforce", "greenhouse",
+               "linkedin", "indeed", "google"],
+    search_term="site reliability engineer",
+    location="Atlanta, GA",
+    results_wanted=20,
+    hours_old=720,          # 30-day freshness cap
+    country_indeed="usa",
+)
+print(f"Found {len(jobs)} jobs")
+print(jobs[["site", "title", "company", "location", "min_amount", "max_amount", "job_url"]].head())
+```
+
+## Parameters
+
+```
+scrape_jobs(
+  site_name              list[str] | str — any of the 17 sources above (default: all)
+  search_term            str        — keyword query
+  google_search_term     str        — Google Jobs override (only filter for `google`)
+  location               str        — "City, ST" or ZIP. Each scraper geocodes its own way.
+  distance               int        — radius miles, default 50
+  is_remote              bool       — remote-only filter (where supported)
+  job_type               str        — "fulltime" | "parttime" | "contract" | "internship"
+  easy_apply             bool       — direct-board apply only (LinkedIn easy-apply is broken)
+  results_wanted         int        — per-site target
+  offset                 int        — pagination offset
+  hours_old              int        — drop postings older than N hours
+  country_indeed         str        — Indeed/Glassdoor country (see list below)
+  description_format     str        — "markdown" | "html"
+  enforce_annual_salary  bool       — convert hourly/monthly to yearly
+  linkedin_fetch_description  bool  — full JD + direct URL (slower)
+  linkedin_company_ids   list[int]  — filter LinkedIn by company IDs
+  proxies                list[str]  — round-robin proxies, "user:pass@host:port"
+  ca_cert                str        — CA cert path for proxies
+  user_agent             str        — override the default UA
+  verbose                int        — 0 errors / 1 warnings / 2 all
+)
+```
+
+### Per-scraper limitations
+
+- **Indeed** — only one of `hours_old` / (`job_type`+`is_remote`) / `easy_apply` per call.
+- **LinkedIn** — only one of `hours_old` / `easy_apply` per call.
+- **ClearanceJobs** — location/remote filters require facet IDs from the dropdown endpoints (not implemented). Filter client-side or scope by keyword.
+- **InsightGlobal** — does not expose client-company name (it's the staffing firm). `is_remote` is not available in their data.
+- **Greenhouse** — Google indexes some postings after they're filled. Stale 404s are filtered out; the freshness cutoff filters "live but ancient" postings (default 90 days, override with `hours_old`).
+
+## JobPost schema
+
+```
+JobPost
+├── id, title, company_name, company_url, job_url
+├── location { country, city, state }
+├── description
+├── is_remote
+├── date_posted
+├── job_type        fulltime | parttime | contract | internship
+├── compensation
+│   ├── interval   yearly | monthly | weekly | daily | hourly
+│   ├── min_amount, max_amount, currency
+│   └── salary_source
+├── job_level                                  (LinkedIn, ClearanceJobs)
+├── company_industry                           (LinkedIn, Indeed, Greenhouse, Kforce)
+├── company_country, company_addresses,
+│   company_employees_label, company_revenue_label,
+│   company_description, company_logo          (Indeed)
+├── skills, experience_range,
+│   company_rating, company_reviews_count,
+│   vacancy_count, work_from_home_type         (Naukri)
+└── emails
+```
+
+## Indeed / Glassdoor country list
+
+Pass `country_indeed` (use the exact name; `*` = also supported on Glassdoor):
+
+| | | | |
+|---|---|---|---|
+| Argentina | Australia* | Austria* | Bahrain |
+| Belgium* | Brazil* | Canada* | Chile |
+| China | Colombia | Costa Rica | Czech Republic |
+| Denmark | Ecuador | Egypt | Finland |
+| France* | Germany* | Greece | Hong Kong* |
+| Hungary | India* | Indonesia | Ireland* |
+| Israel | Italy* | Japan | Kuwait |
+| Luxembourg | Malaysia | Mexico* | Morocco |
+| Netherlands* | New Zealand* | Nigeria | Norway |
+| Oman | Pakistan | Panama | Peru |
+| Philippines | Poland | Portugal | Qatar |
+| Romania | Saudi Arabia | Singapore* | South Africa |
+| South Korea | Spain* | Sweden | Switzerland* |
+| Taiwan | Thailand | Turkey | Ukraine |
+| United Arab Emirates | UK* | USA* | Uruguay |
+| Venezuela | Vietnam* | | |
+
+LinkedIn searches globally and uses only `location`. ZipRecruiter is US/Canada and uses only `location`. Bayt searches internationally with only `search_term`.
+
+## Notes
+
+- Most boards cap a single search at ~1000 results.
+- LinkedIn rate-limits aggressively around the 10th page of pagination on a single IP. Use `proxies`.
+- For Indeed search-term tuning: it searches the description too. Use `-foo` to exclude, `"exact phrase"` for exact match. Example:
+  ```python
+  search_term='"site reliability engineer" (kubernetes OR terraform) -recruiter'
+  ```
+- For Google: copy the exact filter syntax from a real Google Jobs search and pass it as `google_search_term`.
+- For Greenhouse: keyword + location are passed straight to a Google `site:greenhouse.io` query, so Boolean operators and quotes work. Don't quote the full `"City, ST"` — quote the city alone, leave the state bare.
+
+## License
+
+MIT. See `LICENSE`.
+

jobdrop-2.0.0/README.md

@@ -0,0 +1,177 @@
+# jobdrop
+
+A multi-source job scraper. Hits 20 job boards in one call, normalizes
+the results into a pandas DataFrame, and ships with anti-bot bypasses
+for the boards that block standard scrapers.
+
+## What's in here
+
+### 17 sources
+
+| `site_name` | Source | Mechanism |
+|---|---|---|
+| `linkedin` | LinkedIn | Public listing scrape with optional detail-page enrichment |
+| `indeed` | Indeed | GraphQL with the `Int!` radius fix + per-company cap + paginate-until-quota |
+| `glassdoor` | Glassdoor | selenium-driverless headless to defeat Cloudflare 403; in-page GraphQL fetch |
+| `google` | Google Jobs | selenium-driverless headless against `udm=8`; SERP DOM walk |
+| `zip_recruiter` | ZipRecruiter | `curl_cffi` + `safari17_2_ios` TLS impersonation against the web HTML endpoint |
+| `bayt` | Bayt | Public scrape |
+| `naukri` | Naukri | Public scrape |
+| `bdjobs` | BDJobs | Public scrape |
+| `usajobs` | USAJobs.gov | Federal public API |
+| `adzuna` | Adzuna | Public API |
+| `jooble` | Jooble | Public API |
+| `findwork` | Findwork.dev | Public API |
+| `the_muse` | The Muse | Public API |
+| `insight_global` | Insight Global staffing | Server-rendered HTML scrape with hidden JSON blob per result |
+| `clearance_jobs` | ClearanceJobs (DHI) | Public JSON API + parallel detail-page enrichment for full JD, salary, type, remote bool |
+| `kforce` | Kforce staffing | Direct Azure Cognitive Search calls (bypasses Imperva on the public host) |
+| `greenhouse` | Greenhouse-hosted boards | Google `site:` dorks via selenium-driverless → public Greenhouse API; 3-layer staleness filter |
+
+### Anti-bot solved
+
+- **Google** — selenium-driverless cold-start headless. Defeats the 2026 CAPTCHA wall that takes out Playwright / undetected-chromedriver / nodriver / patchright.
+- **Glassdoor** — selenium-driverless rewrite to bypass Cloudflare 403; URL-encoded location, partial-GraphQL-error tolerance.
+- **ZipRecruiter** — `curl_cffi` + `safari17_2_ios` against the web HTML endpoint. The iOS-app API is dead behind Cloudflare.
+- **Kforce** — bypasses Imperva on the public host by calling the Azure Cognitive Search backend directly.
+- **Greenhouse** — uses the same selenium-driverless infrastructure as Google for `site:` dorks across all greenhouse-hosted boards.
+
+### Other tightening
+
+- **LinkedIn** — salary extraction from description body, optional per-company cap, parallel detail fetches.
+- **Indeed** — fixed `radius=25` default after Indeed promoted the GraphQL field to `Int!`; per-company cap to surface diverse employers; pagination loop fixed.
+- **ClearanceJobs** — search API gives a 200-char preview; this fork parallel-fetches `/api/v1/jobs/{id}` so you get the full JD, salary range, structured `job_type`, and authoritative `remote` bool.
+- **Greenhouse** — three layers of stale-protection (404 drop / past `application_deadline` / `first_published` age with 90-day default that respects `hours_old`).
+
+### Bundled credentials
+
+API keys for USAJobs, Adzuna, Jooble, Findwork, and The Muse are baked
+into a positional resolver (`jobdrop/_defaults.py`) so the new sources
+work without environment setup. User-set env vars still win via
+`setdefault` semantics.
+
+## Installation
+
+```
+pip install -U jobdrop
+```
+
+Python ≥ 3.10 required.
+
+## Usage
+
+```python
+from jobdrop import scrape_jobs
+
+jobs = scrape_jobs(
+    site_name=["insight_global", "clearance_jobs", "kforce", "greenhouse",
+               "linkedin", "indeed", "google"],
+    search_term="site reliability engineer",
+    location="Atlanta, GA",
+    results_wanted=20,
+    hours_old=720,          # 30-day freshness cap
+    country_indeed="usa",
+)
+print(f"Found {len(jobs)} jobs")
+print(jobs[["site", "title", "company", "location", "min_amount", "max_amount", "job_url"]].head())
+```
+
+## Parameters
+
+```
+scrape_jobs(
+  site_name              list[str] | str — any of the 17 sources above (default: all)
+  search_term            str        — keyword query
+  google_search_term     str        — Google Jobs override (only filter for `google`)
+  location               str        — "City, ST" or ZIP. Each scraper geocodes its own way.
+  distance               int        — radius miles, default 50
+  is_remote              bool       — remote-only filter (where supported)
+  job_type               str        — "fulltime" | "parttime" | "contract" | "internship"
+  easy_apply             bool       — direct-board apply only (LinkedIn easy-apply is broken)
+  results_wanted         int        — per-site target
+  offset                 int        — pagination offset
+  hours_old              int        — drop postings older than N hours
+  country_indeed         str        — Indeed/Glassdoor country (see list below)
+  description_format     str        — "markdown" | "html"
+  enforce_annual_salary  bool       — convert hourly/monthly to yearly
+  linkedin_fetch_description  bool  — full JD + direct URL (slower)
+  linkedin_company_ids   list[int]  — filter LinkedIn by company IDs
+  proxies                list[str]  — round-robin proxies, "user:pass@host:port"
+  ca_cert                str        — CA cert path for proxies
+  user_agent             str        — override the default UA
+  verbose                int        — 0 errors / 1 warnings / 2 all
+)
+```
+
+### Per-scraper limitations
+
+- **Indeed** — only one of `hours_old` / (`job_type`+`is_remote`) / `easy_apply` per call.
+- **LinkedIn** — only one of `hours_old` / `easy_apply` per call.
+- **ClearanceJobs** — location/remote filters require facet IDs from the dropdown endpoints (not implemented). Filter client-side or scope by keyword.
+- **InsightGlobal** — does not expose client-company name (it's the staffing firm). `is_remote` is not available in their data.
+- **Greenhouse** — Google indexes some postings after they're filled. Stale 404s are filtered out; the freshness cutoff filters "live but ancient" postings (default 90 days, override with `hours_old`).
+
+## JobPost schema
+
+```
+JobPost
+├── id, title, company_name, company_url, job_url
+├── location { country, city, state }
+├── description
+├── is_remote
+├── date_posted
+├── job_type        fulltime | parttime | contract | internship
+├── compensation
+│   ├── interval   yearly | monthly | weekly | daily | hourly
+│   ├── min_amount, max_amount, currency
+│   └── salary_source
+├── job_level                                  (LinkedIn, ClearanceJobs)
+├── company_industry                           (LinkedIn, Indeed, Greenhouse, Kforce)
+├── company_country, company_addresses,
+│   company_employees_label, company_revenue_label,
+│   company_description, company_logo          (Indeed)
+├── skills, experience_range,
+│   company_rating, company_reviews_count,
+│   vacancy_count, work_from_home_type         (Naukri)
+└── emails
+```
+
+## Indeed / Glassdoor country list
+
+Pass `country_indeed` (use the exact name; `*` = also supported on Glassdoor):
+
+| | | | |
+|---|---|---|---|
+| Argentina | Australia* | Austria* | Bahrain |
+| Belgium* | Brazil* | Canada* | Chile |
+| China | Colombia | Costa Rica | Czech Republic |
+| Denmark | Ecuador | Egypt | Finland |
+| France* | Germany* | Greece | Hong Kong* |
+| Hungary | India* | Indonesia | Ireland* |
+| Israel | Italy* | Japan | Kuwait |
+| Luxembourg | Malaysia | Mexico* | Morocco |
+| Netherlands* | New Zealand* | Nigeria | Norway |
+| Oman | Pakistan | Panama | Peru |
+| Philippines | Poland | Portugal | Qatar |
+| Romania | Saudi Arabia | Singapore* | South Africa |
+| South Korea | Spain* | Sweden | Switzerland* |
+| Taiwan | Thailand | Turkey | Ukraine |
+| United Arab Emirates | UK* | USA* | Uruguay |
+| Venezuela | Vietnam* | | |
+
+LinkedIn searches globally and uses only `location`. ZipRecruiter is US/Canada and uses only `location`. Bayt searches internationally with only `search_term`.
+
+## Notes
+
+- Most boards cap a single search at ~1000 results.
+- LinkedIn rate-limits aggressively around the 10th page of pagination on a single IP. Use `proxies`.
+- For Indeed search-term tuning: it searches the description too. Use `-foo` to exclude, `"exact phrase"` for exact match. Example:
+  ```python
+  search_term='"site reliability engineer" (kubernetes OR terraform) -recruiter'
+  ```
+- For Google: copy the exact filter syntax from a real Google Jobs search and pass it as `google_search_term`.
+- For Greenhouse: keyword + location are passed straight to a Google `site:greenhouse.io` query, so Boolean operators and quotes work. Don't quote the full `"City, ST"` — quote the city alone, leave the state bare.
+
+## License
+
+MIT. See `LICENSE`.

jobdrop-2.0.0/jobdrop/__init__.py

@@ -0,0 +1,256 @@
+from __future__ import annotations
+
+# Apply compiled defaults BEFORE any scraper module runs its module-level
+# os.environ reads. User-set env vars are preserved (setdefault semantics).
+from jobdrop import _defaults  # noqa: F401
+
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from typing import Tuple
+
+import pandas as pd
+
+from jobdrop.adzuna import Adzuna
+from jobdrop.bayt import BaytScraper
+from jobdrop.bdjobs import BDJobs
+from jobdrop.clearancejobs import ClearanceJobs
+from jobdrop.collabwork import CollabWork
+from jobdrop.findwork import Findwork
+from jobdrop.glassdoor import Glassdoor
+from jobdrop.google import Google
+from jobdrop.greenhouse import Greenhouse
+from jobdrop.hiring_cafe import HiringCafe
+from jobdrop.indeed import Indeed
+from jobdrop.insightglobal import InsightGlobal
+from jobdrop.jooble import Jooble
+from jobdrop.kforce import Kforce
+from jobdrop.linkedin import LinkedIn
+from jobdrop.naukri import Naukri
+from jobdrop.the_muse import TheMuse
+from jobdrop.usajobs import USAJobs
+from jobdrop.wellfound import Wellfound
+from jobdrop.model import JobType, Location, JobResponse, Country
+from jobdrop.model import SalarySource, ScraperInput, Site
+from jobdrop.util import (
+    set_logger_level,
+    extract_salary,
+    create_logger,
+    get_enum_from_value,
+    map_str_to_site,
+    convert_to_annual,
+    desired_order,
+)
+from jobdrop.ziprecruiter import ZipRecruiter
+
+
+# Update the SCRAPER_MAPPING dictionary in the scrape_jobs function
+
+def scrape_jobs(
+    site_name: str | list[str] | Site | list[Site] | None = None,
+    search_term: str | None = None,
+    google_search_term: str | None = None,
+    location: str | None = None,
+    distance: int | None = 50,
+    is_remote: bool = False,
+    job_type: str | None = None,
+    easy_apply: bool | None = None,
+    results_wanted: int = 15,
+    country_indeed: str = "usa",
+    proxies: list[str] | str | None = None,
+    ca_cert: str | None = None,
+    description_format: str = "markdown",
+    linkedin_fetch_description: bool | None = False,
+    linkedin_company_ids: list[int] | None = None,
+    offset: int | None = 0,
+    hours_old: int = None,
+    enforce_annual_salary: bool = False,
+    verbose: int = 0,
+    user_agent: str = None,
+    **kwargs,
+) -> pd.DataFrame:
+    """
+    Scrapes job data from job boards concurrently
+    :return: Pandas DataFrame containing job data
+    """
+    SCRAPER_MAPPING = {
+        Site.LINKEDIN: LinkedIn,
+        Site.INDEED: Indeed,
+        Site.ZIP_RECRUITER: ZipRecruiter,
+        Site.GLASSDOOR: Glassdoor,
+        Site.GOOGLE: Google,
+        Site.BAYT: BaytScraper,
+        Site.NAUKRI: Naukri,
+        Site.BDJOBS: BDJobs,
+        # API-based sources added in the kbwhodat fork
+        Site.USAJOBS: USAJobs,
+        Site.ADZUNA: Adzuna,
+        Site.JOOBLE: Jooble,
+        Site.FINDWORK: Findwork,
+        Site.THE_MUSE: TheMuse,
+        Site.INSIGHT_GLOBAL: InsightGlobal,
+        Site.CLEARANCE_JOBS: ClearanceJobs,
+        Site.KFORCE: Kforce,
+        Site.GREENHOUSE: Greenhouse,
+        Site.COLLAB_WORK: CollabWork,
+        Site.WELLFOUND: Wellfound,
+        Site.HIRING_CAFE: HiringCafe,
+    }
+    set_logger_level(verbose)
+    job_type = get_enum_from_value(job_type) if job_type else None
+
+    def get_site_type():
+        site_types = list(Site)
+        if isinstance(site_name, str):
+            site_types = [map_str_to_site(site_name)]
+        elif isinstance(site_name, Site):
+            site_types = [site_name]
+        elif isinstance(site_name, list):
+            site_types = [
+                map_str_to_site(site) if isinstance(site, str) else site
+                for site in site_name
+            ]
+        return site_types
+
+    country_enum = Country.from_string(country_indeed)
+
+    scraper_input = ScraperInput(
+        site_type=get_site_type(),
+        country=country_enum,
+        search_term=search_term,
+        google_search_term=google_search_term,
+        location=location,
+        distance=distance,
+        is_remote=is_remote,
+        job_type=job_type,
+        easy_apply=easy_apply,
+        description_format=description_format,
+        linkedin_fetch_description=linkedin_fetch_description,
+        results_wanted=results_wanted,
+        linkedin_company_ids=linkedin_company_ids,
+        offset=offset,
+        hours_old=hours_old,
+    )
+
+    def scrape_site(site: Site) -> Tuple[str, JobResponse]:
+        scraper_class = SCRAPER_MAPPING[site]
+        scraper = scraper_class(proxies=proxies, ca_cert=ca_cert, user_agent=user_agent)
+        scraped_data: JobResponse = scraper.scrape(scraper_input)
+        cap_name = site.value.capitalize()
+        site_name = "ZipRecruiter" if cap_name == "Zip_recruiter" else cap_name
+        site_name = "LinkedIn" if cap_name == "Linkedin" else cap_name
+        create_logger(site_name).info(f"finished scraping")
+        return site.value, scraped_data
+
+    site_to_jobs_dict = {}
+
+    def worker(site):
+        site_val, scraped_info = scrape_site(site)
+        return site_val, scraped_info
+
+    with ThreadPoolExecutor() as executor:
+        future_to_site = {
+            executor.submit(worker, site): site for site in scraper_input.site_type
+        }
+
+        for future in as_completed(future_to_site):
+            site_value, scraped_data = future.result()
+            site_to_jobs_dict[site_value] = scraped_data
+
+    jobs_dfs: list[pd.DataFrame] = []
+
+    for site, job_response in site_to_jobs_dict.items():
+        for job in job_response.jobs:
+            job_data = job.dict()
+            job_url = job_data["job_url"]
+            job_data["site"] = site
+            job_data["company"] = job_data["company_name"]
+            job_data["job_type"] = (
+                ", ".join(job_type.value[0] for job_type in job_data["job_type"])
+                if job_data["job_type"]
+                else None
+            )
+            job_data["emails"] = (
+                ", ".join(job_data["emails"]) if job_data["emails"] else None
+            )
+            if job_data["location"]:
+                job_data["location"] = Location(
+                    **job_data["location"]
+                ).display_location()
+
+            # Handle compensation
+            compensation_obj = job_data.get("compensation")
+            if compensation_obj and isinstance(compensation_obj, dict):
+                job_data["interval"] = (
+                    compensation_obj.get("interval").value
+                    if compensation_obj.get("interval")
+                    else None
+                )
+                job_data["min_amount"] = compensation_obj.get("min_amount")
+                job_data["max_amount"] = compensation_obj.get("max_amount")
+                job_data["currency"] = compensation_obj.get("currency", "USD")
+                job_data["salary_source"] = SalarySource.DIRECT_DATA.value
+                if enforce_annual_salary and (
+                    job_data["interval"]
+                    and job_data["interval"] != "yearly"
+                    and job_data["min_amount"]
+                    and job_data["max_amount"]
+                ):
+                    convert_to_annual(job_data)
+            else:
+                if country_enum == Country.USA:
+                    (
+                        job_data["interval"],
+                        job_data["min_amount"],
+                        job_data["max_amount"],
+                        job_data["currency"],
+                    ) = extract_salary(
+                        job_data["description"],
+                        enforce_annual_salary=enforce_annual_salary,
+                    )
+                    job_data["salary_source"] = SalarySource.DESCRIPTION.value
+
+            job_data["salary_source"] = (
+                job_data["salary_source"]
+                if "min_amount" in job_data and job_data["min_amount"]
+                else None
+            )
+
+            #naukri-specific fields
+            job_data["skills"] = (
+                ", ".join(job_data["skills"]) if job_data["skills"] else None
+            )
+            job_data["experience_range"] = job_data.get("experience_range")
+            job_data["company_rating"] = job_data.get("company_rating")
+            job_data["company_reviews_count"] = job_data.get("company_reviews_count")
+            job_data["vacancy_count"] = job_data.get("vacancy_count")
+            job_data["work_from_home_type"] = job_data.get("work_from_home_type")
+
+            job_df = pd.DataFrame([job_data])
+            jobs_dfs.append(job_df)
+
+    if jobs_dfs:
+        # Step 1: Filter out all-NA columns from each DataFrame before concatenation
+        filtered_dfs = [df.dropna(axis=1, how="all") for df in jobs_dfs]
+
+        # Step 2: Concatenate the filtered DataFrames
+        jobs_df = pd.concat(filtered_dfs, ignore_index=True)
+
+        # Step 3: Ensure all desired columns are present, adding missing ones as empty
+        for column in desired_order:
+            if column not in jobs_df.columns:
+                jobs_df[column] = None  # Add missing columns as empty
+
+        # Reorder the DataFrame according to the desired order
+        jobs_df = jobs_df[desired_order]
+
+        # Step 4: Sort the DataFrame as required
+        return jobs_df.sort_values(
+            by=["site", "date_posted"], ascending=[True, False]
+        ).reset_index(drop=True)
+    else:
+        return pd.DataFrame()
+
+
+# Add BDJobs to __all__
+__all__ = [
+    "BDJobs",
+]