eksi-scraper 0.1.0__tar.gz

eksi_scraper-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+__pycache__/
+*.py[cod]
+*.log
+*.csv
+*.json
+venv/
+.venv/
+dist/
+*.egg-info/
+

eksi_scraper-0.1.0/CLAUDE.md

@@ -0,0 +1,38 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Setup
+
+```bash
+pip3 install -r requirements.txt
+```
+
+## Running the scraper
+
+```bash
+# Run from the src/ directory
+cd src
+
+# Scrape specific threads
+python3 main.py -t murat-kurum--2582131 ekrem-imamoglu--2577439 -o json
+
+# Scrape from a file (one thread slug per line)
+python3 main.py -f threads.txt -o csv
+```
+
+Thread slugs are the URL path segment after `eksisozluk.com/` and before any `?` (e.g., `murat-kurum--2582131`).
+
+Output files are written to the working directory where the script is run. Logs go to `eksisozluk_scraper.log`.
+
+## Architecture
+
+Three modules in `src/`:
+
+- **`main.py`** — Entry point. Parses CLI args, creates one async task per thread, runs them concurrently via `asyncio.gather`. Uses `curl_cffi` with Chrome browser impersonation (`impersonate="chrome124"`) to bypass bot detection.
+
+- **`eksisozluk_scraper.py`** — `EksiSozlukScraper` class. First fetches the thread's first page to read `data-pagecount` from the `.pager` div, then concurrently scrapes all pages using an `asyncio.Semaphore` (default: 15 concurrent requests). BeautifulSoup parsing is offloaded to a `ThreadPoolExecutor` to avoid blocking the event loop. Page requests retry with exponential backoff (up to 8 tries / 300s) via the `backoff` library.
+
+- **`data_writer.py`** — `DataWriter` static class. Writes scraped entries (Content, Author, Date Created, Last Changed) to CSV or JSON using `aiofiles` for async I/O.
+
+Each scraped entry dict has keys: `Content`, `Author`, `Date Created`, `Last Changed`.

eksi_scraper-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Ibrahim Berkay Ceylan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

eksi_scraper-0.1.0/PKG-INFO

@@ -0,0 +1,59 @@
+Metadata-Version: 2.4
+Name: eksi-scraper
+Version: 0.1.0
+Summary: asynchronously scrapes eksisozluk threads and exports to csv or json
+Project-URL: Homepage, https://github.com/iberkayC/eksi-scraper
+Author-email: Ibrahim Berkay Ceylan <ceylaniberkay@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: backoff>=2.2
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: curl-cffi>=0.7
+Requires-Dist: lxml>=5.1
+Description-Content-Type: text/markdown
+
+# eksi-scraper
+
+asynchronously scrapes eksisozluk threads in python, and puts entries in a csv or json file, named after the thread. intended for educational purposes only.
+
+## installation
+
+```
+uv pip install eksi-scraper
+```
+or with pip:
+```
+pip install eksi-scraper
+```
+
+## usage
+
+```
+eksi-scraper -t [thread1] [thread2] ... -f [inputFile.txt] -o (csv or json)
+```
+you can pass full URLs or just the slug (the part of the url after '/' and before '?'). for example:
+
+```
+eksi-scraper -t https://eksisozluk.com/murat-kurum--2582131 https://eksisozluk.com/ekrem-imamoglu--2577439 -o json
+```
+or using slugs:
+```
+eksi-scraper -t murat-kurum--2582131 ekrem-imamoglu--2577439 -o json
+```
+or from a file:
+```
+eksi-scraper -f threads.txt -o csv
+```
+
+where in threads.txt, threads are listed as URLs or slugs, one per line:
+
+```
+https://eksisozluk.com/murat-kurum--2582131
+ekrem-imamoglu--2577439
+...
+```
+
+## contact
+
+reach out to me at ceylaniberkay@gmail.com

eksi_scraper-0.1.0/README.md

@@ -0,0 +1,44 @@
+# eksi-scraper
+
+asynchronously scrapes eksisozluk threads in python, and puts entries in a csv or json file, named after the thread. intended for educational purposes only.
+
+## installation
+
+```
+uv pip install eksi-scraper
+```
+or with pip:
+```
+pip install eksi-scraper
+```
+
+## usage
+
+```
+eksi-scraper -t [thread1] [thread2] ... -f [inputFile.txt] -o (csv or json)
+```
+you can pass full URLs or just the slug (the part of the url after '/' and before '?'). for example:
+
+```
+eksi-scraper -t https://eksisozluk.com/murat-kurum--2582131 https://eksisozluk.com/ekrem-imamoglu--2577439 -o json
+```
+or using slugs:
+```
+eksi-scraper -t murat-kurum--2582131 ekrem-imamoglu--2577439 -o json
+```
+or from a file:
+```
+eksi-scraper -f threads.txt -o csv
+```
+
+where in threads.txt, threads are listed as URLs or slugs, one per line:
+
+```
+https://eksisozluk.com/murat-kurum--2582131
+ekrem-imamoglu--2577439
+...
+```
+
+## contact
+
+reach out to me at ceylaniberkay@gmail.com

eksi_scraper-0.1.0/eksisozluk_scraper/__main__.py

@@ -0,0 +1,3 @@
+from eksisozluk_scraper.cli import cli
+
+cli()

eksi_scraper-0.1.0/eksisozluk_scraper/cli.py

@@ -0,0 +1,123 @@
+"""Scrape threads from eksisozluk
+
+Asynchronously scrapes threads from eksisozluk, 
+taking threads as command line arguments and 
+writes them to csv files. Some variables are too low 
+for any real scraping, but it's good for educational
+purposes.
+"""
+from typing import List, Literal
+import argparse
+import asyncio
+import sys
+import logging
+from urllib.parse import urlparse
+from curl_cffi import requests
+from .scraper import EksiSozlukScraper
+from .data_writer import DataWriter
+
+BASE_URL = 'https://www.eksisozluk.com/'
+
+
+def extract_slug(value: str) -> str:
+    """Return the thread slug from either a full URL or a bare slug.
+
+    Examples:
+        'https://eksisozluk.com/murat-kurum--2582131?p=2' -> 'murat-kurum--2582131'
+        'murat-kurum--2582131' -> 'murat-kurum--2582131'
+    """
+    parsed = urlparse(value)
+    if parsed.scheme in ('http', 'https'):
+        return parsed.path.strip('/')
+    return value
+
+logging.basicConfig(filename='eksisozluk_scraper.log', level=logging.INFO,
+                    format='%(asctime)s - %(message)s')
+
+async def process_thread(scraper: EksiSozlukScraper,
+                         session: requests.AsyncSession,
+                         thread: str,
+                         output_format: Literal['csv', 'json']) -> None:
+    """
+    Process a thread, scrape it and write it to a file.
+
+    Args:
+        scraper (EksiSozlukScraper): scraper object
+        session (requests.AsyncSession): session to make requests
+        thread (str): thread to scrape
+        output_format (csv or json): output format 
+    """
+    try:
+        logging.info('Started scraping thread %s', thread)
+        scraped_data = await scraper.scrape_thread(session, thread)
+
+        if scraped_data:
+            filename = f"{thread}.{output_format}"
+            DataWriter.write_data(filename, scraped_data, output_format)
+            logging.info(f"Successfully scraped and saved thread {thread} to {filename}")
+        else:
+            logging.warning(f"No data scraped for thread: {thread}")
+
+    except Exception as e:
+        logging.error(f"Unexpected error in process_thread: {e}")
+
+
+async def main(threads: List[str], output_format: Literal['csv', 'json'] = 'csv') -> None:
+    """
+    Main function to scrape threads from eksisozluk
+
+    Args:
+        threads (list): list of threads to scrape, part of the url after the /, before possibly ?.
+    """
+    scraper = EksiSozlukScraper(BASE_URL)
+    header = {
+        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like\
+            Gecko) Chrome/58.0.3029.110 Safari/537.3',
+        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
+    }
+
+    async with requests.AsyncSession(headers=header, impersonate="chrome124") as session:
+        tasks = [process_thread(scraper, session, thread, output_format) for thread in threads]
+        await asyncio.gather(*tasks)
+
+
+def cli():
+    parser = argparse.ArgumentParser(
+        description='Scrape threads from eksisozluk.com')
+    parser.add_argument('-t', '--threads',
+                        metavar='thread',
+                        required=False,
+                        type=str,
+                        nargs='+',
+                        help='Threads to scrape. Accepts full URLs or slugs (part of the URL after /).')
+    parser.add_argument('-f', '--file',
+                        metavar='file',
+                        required=False,
+                        type=str,
+                        help='File to read threads from, one thread per line.')
+    parser.add_argument('-o', '--output',
+                        choices=['csv', 'json'],
+                        default='csv',
+                        help='Output format (csv or json). Default is csv.')
+    args = parser.parse_args()
+
+    thread_list = [extract_slug(t) for t in args.threads] if args.threads else []
+
+    if args.file:
+        try:
+            with open(args.file, 'r', encoding='utf-8') as file:
+                thread_list.extend([extract_slug(line.strip()) for line in file.readlines()])
+        except IOError as e:
+            logging.error(f"Error reading file {args.file}: {e}")
+            sys.exit(1)
+
+    if thread_list:
+        asyncio.run(main(thread_list, args.output))
+    else:
+        logging.error('No threads provided. Exiting.')
+        parser.print_help()
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    cli()

eksi_scraper-0.1.0/eksisozluk_scraper/data_writer.py

@@ -0,0 +1,55 @@
+"""
+Module to write data to files in either CSV or JSON format.
+
+Raises:
+    ValueError: if an unsupported format is specified.
+"""
+import csv
+import json
+from typing import List, Dict, Any, Literal
+
+
+class DataWriter:
+    """
+    Class to write data to files in either CSV or JSON format.
+
+    Raises:
+        ValueError: if an unsupported format is specified.
+    """
+    @staticmethod
+    def write_data(filename: str,
+                   data: List[Dict[str, Any]],
+                   filetype: Literal['csv', 'json']) -> None:
+        """
+        Write data to a file in either CSV or JSON format.
+
+        Args:
+            filename (str): the name of the file to write.
+            data (List[Dict[str, Any]]): the data to write.
+            filetype (Literal['csv', 'json']): the output format ('csv' or 'json').
+
+        Raises:
+            ValueError: if an unsupported format is specified.
+        """
+        if filetype == 'csv':
+            DataWriter._write_csv(filename, data)
+        elif filetype == 'json':
+            DataWriter._write_json(filename, data)
+        else:
+            raise ValueError(f"Unsupported format: {filetype}")
+
+    @staticmethod
+    def _write_csv(filename: str, data: List[Dict[str, Any]]) -> None:
+        with open(filename, 'w', encoding='utf-8', newline='') as f:
+            if not data:
+                return
+            fieldnames = data[0].keys()
+            writer = csv.DictWriter(f, fieldnames=fieldnames)
+            writer.writeheader()
+            for row in data:
+                writer.writerow(row)
+
+    @staticmethod
+    def _write_json(filename: str, data: List[Dict[str, Any]]) -> None:
+        with open(filename, 'w', encoding='utf-8') as f:
+            json.dump(data, f, ensure_ascii=False, indent=2)

eksi_scraper-0.1.0/eksisozluk_scraper/scraper.py

@@ -0,0 +1,139 @@
+"""
+Contains the EksiSozlukScraper class. Scrapes threads from eksisozluk.com
+"""
+from typing import List, Dict, Any
+import logging
+import asyncio
+
+from bs4 import BeautifulSoup
+from curl_cffi import requests
+import backoff
+
+class EksiSozlukScraper:
+    """
+    Scraper class for EksiSozluk. Handles the scraping logic of threads.
+    """
+
+    def __init__(self, base_url: str):
+        """
+        Initializes the scraper with the base URL.
+
+        Args:
+            base_url (str): The base URL of EksiSozluk.
+        """
+        self.base_url = base_url
+
+    async def find_number_of_pages(self,
+                                   session: requests.AsyncSession,
+                                   url: str) -> int:
+        """Finds the number of pages in a thread
+
+        Args:
+            session (requests.AsyncSession): session to make requests
+            url (str): url of the thread
+
+        Returns:
+            int: number of pages in the thread.
+        """
+        try:
+            response = await session.get(url)
+            if response.status_code != 200:
+                logging.error("Failed to fetch %s (status %s)",
+                              url, response.status_code)
+                return 1
+            text = response.text
+            soup = BeautifulSoup(text, 'lxml')
+            pager_div = soup.find('div', class_='pager')
+            if pager_div and 'data-pagecount' in pager_div.attrs:
+                return int(pager_div['data-pagecount'])
+            return 1
+        except Exception as e:
+            logging.error(f"Unexpected error in find_number_of_pages: {e}")
+            return 1
+
+
+    def _parse_entry(self, entry: BeautifulSoup) -> Dict[str, Any]:
+        """
+        Parses an entry and returns a dictionary with the content, 
+        author, date created and last changed.
+
+        Args:
+            entry (BeautifulSoup): an entry in the thread
+        """
+        content_div = entry.find(class_='content')
+        # Replace shortened link text with the full URL from href
+        for a in content_div.find_all('a', href=True):
+            if a['href'].startswith('http'):
+                a.string = a['href']
+        content = content_div.get_text(separator=' ').strip()
+        author = entry.find(class_='entry-author').text.strip()
+        entry_date_text = entry.find(class_='entry-date').text.strip()
+
+        if '~' in entry_date_text:
+            date_created, last_changed = [
+                part.strip() for part in entry_date_text.split('~')]
+        else:
+            date_created = entry_date_text
+            last_changed = 'null'
+
+        return {
+            'Content': content,
+            'Author': author,
+            'Date Created': date_created,
+            'Last Changed': last_changed
+        }
+
+
+    @backoff.on_exception(backoff.expo,
+                          requests.RequestsError,
+                          max_tries=8,
+                          max_time=300)
+    async def scrape_page(self,
+                          session: requests.AsyncSession,
+                          url: str,
+                          semaphore: asyncio.Semaphore) -> List[Dict[str, Any]]:
+        """
+        Scrapes a page and appends the data to scraped_data
+
+        Args:
+            session (requests.AsyncSession): session to make requests
+            url (str): url of the page to scrape
+            semaphore (asyncio.Semaphore): semaphore to limit the number of concurrent requests
+            scraped_data (list): list to append the scraped data
+        """
+        async with semaphore:
+            try:
+                response = await session.get(url)
+                if response.status_code != 200:
+                    logging.error("Failed to fetch %s (status %s)",
+                                  url, response.status_code)
+                    return []
+                text = response.text
+                soup = BeautifulSoup(text, 'lxml')
+                entries = soup.find_all(id='entry-item')
+                return [self._parse_entry(entry) for entry in entries]
+            except Exception as e:
+                logging.error(f"Unexpected error in scrape_page {url}: {e}")
+                return []
+
+    async def scrape_thread(self,
+                            session: requests.AsyncSession,
+                            thread: str,
+                            max_concurrent_requests: int = 15):
+        """
+        Scrapes a thread and writes the data to a csv file
+
+        Args:
+            session (requests.AsyncSession): session to make requests
+            thread (str): thread to scrape, the part of the url after the /, before, if exists, ?.
+            max_concurrent_requests (int, optional): max # of concurrent requests. Defaults to 15.
+        """
+        semaphore = asyncio.Semaphore(max_concurrent_requests)
+
+        thread_url = self.base_url + thread
+        number_of_pages = await self.find_number_of_pages(session, thread_url)
+        tasks = [self.scrape_page(session, thread_url + '?p=' + str(page), semaphore)
+                 for page in range(1, int(number_of_pages) + 1)]
+        results = await asyncio.gather(*tasks)
+
+        return [entry for page in results for entry in page]

eksi_scraper-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "eksi-scraper"
+version = "0.1.0"
+description = "asynchronously scrapes eksisozluk threads and exports to csv or json"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [
+    { name = "Ibrahim Berkay Ceylan", email = "ceylaniberkay@gmail.com" }
+]
+dependencies = [
+    "backoff>=2.2",
+    "beautifulsoup4>=4.12",
+    "lxml>=5.1",
+    "curl_cffi>=0.7",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["eksisozluk_scraper"]
+
+[project.urls]
+Homepage = "https://github.com/iberkayC/eksi-scraper"
+
+[project.scripts]
+eksi-scraper = "eksisozluk_scraper.cli:cli"