flatscraper 0.1.0__tar.gz

flatscraper-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lukas
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

flatscraper-0.1.0/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.4
+Name: flatscraper
+Version: 0.1.0
+Summary: Multi-platform flat search automation
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: playwright>=1.40.0
+Requires-Dist: groq>=0.4.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: questionary>=2.0.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Dynamic: license-file
+
+# FlatScraper
+
+**Find WG rooms and apartments in hours, not weeks.** FlatScraper automates your search on WG-Gesucht: it discovers new listings, generates personalized messages with AI, and sends contact requests—so you can land multiple viewings while others are still refreshing the page.
+
+---
+
+## Why FlatScraper?
+
+| The old way | With FlatScraper |
+|-------------|------------------|
+| Manually refresh WG-Gesucht | Automatically scans for new listings |
+| Copy-paste generic messages | AI writes personalized Anschreiben for each ad |
+| Hope your message stands out | Tailored tone: WG-friendly or landlord-professional |
+| One viewing per week if you're lucky | **Multiple viewings in hours** |
+
+---
+
+## Demo
+
+### Setup wizard (one-time, ~2 minutes)
+
+![Setup demo](public/setup-demo.gif)
+
+### FlatScraper in action
+
+![FlatScraper demo](public/flatscraper-demo.gif)
+
+---
+
+## Features
+
+- **Zero cost** – Runs entirely on [Groq's free tier](https://console.groq.com). No paid APIs.
+- **Switch models** – Choose from Groq models (compound, llama, etc.) in setup or `.env`.
+- **Personalized messages** – Your profile (age, job, personality, hobbies) shapes every Anschreiben.
+- **Smart extraction** – Strips LLM meta-commentary; only the message is sent.
+- **Rate limit handling** – Automatic retries with user feedback when Groq throttles.
+- **Headless by default** – Runs in the background; use `--visible` for debugging.
+- **Dry run** – `--no-send` to preview messages without sending.
+
+---
+
+## Quick start
+
+### 1. Install
+
+```powershell
+# With uv (recommended)
+uv sync
+
+# Or with pip
+pip install flatscraper
+# or from source: pip install -e .
+```
+
+### 2. Install Playwright browser
+
+```powershell
+playwright install chromium
+```
+
+### 3. Run setup
+
+```powershell
+uv run flatscraper setup
+# or: uv run setup
+```
+
+The wizard guides you through:
+
+1. WG-Gesucht credentials  
+2. [Groq API key](https://console.groq.com) (free)  
+3. Google Drive link for your documents  
+4. Your profile (age, city, job, personality, hobbies)  
+5. Model selection (Groq free-tier models)  
+6. Search URLs from WG-Gesucht  
+
+### 4. Run
+
+```powershell
+# Test run (no messages sent)
+uv run flatscraper --no-send
+
+# Live run
+uv run flatscraper
+```
+
+---
+
+## CLI reference
+
+| Command | Description |
+|---------|-------------|
+| `flatscraper` | Run once: find listings, generate messages, send |
+| `flatscraper --no-send` | Dry run: generate messages only, don't send |
+| `flatscraper --visible` | Show browser window (default: headless) |
+| `flatscraper --debug` | Include all listings (ignore age filter) |
+| `flatscraper --schedule` | Run repeatedly on an interval |
+| `flatscraper setup` | Run the setup wizard |
+
+---
+
+## Configuration
+
+### Environment (`.env`)
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `FLATSCRAPER_EMAIL` | Yes | WG-Gesucht login email |
+| `FLATSCRAPER_PASSWORD` | Yes | WG-Gesucht password |
+| `GROQ_API_KEY` | Yes | [Groq API key](https://console.groq.com) (free tier) |
+| `GOOGLE_DRIVE_LINK` | Yes | Google Drive folder with your documents |
+| `GROQ_MODEL` | No | Model (default: `llama-3.1-8b-instant`) |
+| `RUN_INTERVAL_MINUTES` | No | Schedule interval (default: `30`) |
+| `AUTO_RUN_ENABLED` | No | Enable schedule (default: `false`) |
+
+Copy `.env.example` to `.env` and fill in your values. **Never commit `.env` or `user_profile.json`**—they contain personal data.
+
+### Profile (`user_profile.json`)
+
+Created by the setup wizard. Contains your persona (for personalized messages) and search URLs. Edit manually or run `flatscraper setup` again.
+
+---
+
+## Groq models (free tier)
+
+You can switch models in setup or via `GROQ_MODEL` in `.env`:
+
+| Model | Notes |
+|-------|-------|
+| `groq/compound` | High throughput, good for many messages |
+| `groq/compound-mini` | Fast, good for high frequency |
+| `llama-3.3-70b-versatile` | Best text quality |
+| `meta-llama/llama-4-scout-17b-16e-instruct` | Good balance |
+| `llama-3.1-8b-instant` | Default, fast |
+
+---
+
+## Tests
+
+```powershell
+uv sync --extra dev
+uv run pytest tests/ -v
+```
+
+---
+
+## Project structure
+
+```
+flatscraper/
+├── run.py             # Main entry point
+├── config.py          # Settings + user profile
+├── groq_client.py     # LLM client (Groq)
+├── models.py          # Pydantic models
+├── setup_wizard.py    # Interactive setup
+├── .env.example       # Env template
+└── platforms/
+    └── wggesucht/     # WG-Gesucht implementation
+```
+
+---
+
+## Supported platforms
+
+| Platform   | Status |
+|------------|--------|
+| WG-Gesucht | ✅ Ready |
+
+---
+
+## License
+
+MIT

flatscraper-0.1.0/README.md

@@ -0,0 +1,173 @@
+# FlatScraper
+
+**Find WG rooms and apartments in hours, not weeks.** FlatScraper automates your search on WG-Gesucht: it discovers new listings, generates personalized messages with AI, and sends contact requests—so you can land multiple viewings while others are still refreshing the page.
+
+---
+
+## Why FlatScraper?
+
+| The old way | With FlatScraper |
+|-------------|------------------|
+| Manually refresh WG-Gesucht | Automatically scans for new listings |
+| Copy-paste generic messages | AI writes personalized Anschreiben for each ad |
+| Hope your message stands out | Tailored tone: WG-friendly or landlord-professional |
+| One viewing per week if you're lucky | **Multiple viewings in hours** |
+
+---
+
+## Demo
+
+### Setup wizard (one-time, ~2 minutes)
+
+![Setup demo](public/setup-demo.gif)
+
+### FlatScraper in action
+
+![FlatScraper demo](public/flatscraper-demo.gif)
+
+---
+
+## Features
+
+- **Zero cost** – Runs entirely on [Groq's free tier](https://console.groq.com). No paid APIs.
+- **Switch models** – Choose from Groq models (compound, llama, etc.) in setup or `.env`.
+- **Personalized messages** – Your profile (age, job, personality, hobbies) shapes every Anschreiben.
+- **Smart extraction** – Strips LLM meta-commentary; only the message is sent.
+- **Rate limit handling** – Automatic retries with user feedback when Groq throttles.
+- **Headless by default** – Runs in the background; use `--visible` for debugging.
+- **Dry run** – `--no-send` to preview messages without sending.
+
+---
+
+## Quick start
+
+### 1. Install
+
+```powershell
+# With uv (recommended)
+uv sync
+
+# Or with pip
+pip install flatscraper
+# or from source: pip install -e .
+```
+
+### 2. Install Playwright browser
+
+```powershell
+playwright install chromium
+```
+
+### 3. Run setup
+
+```powershell
+uv run flatscraper setup
+# or: uv run setup
+```
+
+The wizard guides you through:
+
+1. WG-Gesucht credentials  
+2. [Groq API key](https://console.groq.com) (free)  
+3. Google Drive link for your documents  
+4. Your profile (age, city, job, personality, hobbies)  
+5. Model selection (Groq free-tier models)  
+6. Search URLs from WG-Gesucht  
+
+### 4. Run
+
+```powershell
+# Test run (no messages sent)
+uv run flatscraper --no-send
+
+# Live run
+uv run flatscraper
+```
+
+---
+
+## CLI reference
+
+| Command | Description |
+|---------|-------------|
+| `flatscraper` | Run once: find listings, generate messages, send |
+| `flatscraper --no-send` | Dry run: generate messages only, don't send |
+| `flatscraper --visible` | Show browser window (default: headless) |
+| `flatscraper --debug` | Include all listings (ignore age filter) |
+| `flatscraper --schedule` | Run repeatedly on an interval |
+| `flatscraper setup` | Run the setup wizard |
+
+---
+
+## Configuration
+
+### Environment (`.env`)
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `FLATSCRAPER_EMAIL` | Yes | WG-Gesucht login email |
+| `FLATSCRAPER_PASSWORD` | Yes | WG-Gesucht password |
+| `GROQ_API_KEY` | Yes | [Groq API key](https://console.groq.com) (free tier) |
+| `GOOGLE_DRIVE_LINK` | Yes | Google Drive folder with your documents |
+| `GROQ_MODEL` | No | Model (default: `llama-3.1-8b-instant`) |
+| `RUN_INTERVAL_MINUTES` | No | Schedule interval (default: `30`) |
+| `AUTO_RUN_ENABLED` | No | Enable schedule (default: `false`) |
+
+Copy `.env.example` to `.env` and fill in your values. **Never commit `.env` or `user_profile.json`**—they contain personal data.
+
+### Profile (`user_profile.json`)
+
+Created by the setup wizard. Contains your persona (for personalized messages) and search URLs. Edit manually or run `flatscraper setup` again.
+
+---
+
+## Groq models (free tier)
+
+You can switch models in setup or via `GROQ_MODEL` in `.env`:
+
+| Model | Notes |
+|-------|-------|
+| `groq/compound` | High throughput, good for many messages |
+| `groq/compound-mini` | Fast, good for high frequency |
+| `llama-3.3-70b-versatile` | Best text quality |
+| `meta-llama/llama-4-scout-17b-16e-instruct` | Good balance |
+| `llama-3.1-8b-instant` | Default, fast |
+
+---
+
+## Tests
+
+```powershell
+uv sync --extra dev
+uv run pytest tests/ -v
+```
+
+---
+
+## Project structure
+
+```
+flatscraper/
+├── run.py             # Main entry point
+├── config.py          # Settings + user profile
+├── groq_client.py     # LLM client (Groq)
+├── models.py          # Pydantic models
+├── setup_wizard.py    # Interactive setup
+├── .env.example       # Env template
+└── platforms/
+    └── wggesucht/     # WG-Gesucht implementation
+```
+
+---
+
+## Supported platforms
+
+| Platform   | Status |
+|------------|--------|
+| WG-Gesucht | ✅ Ready |
+
+---
+
+## License
+
+MIT

flatscraper-0.1.0/config.py

@@ -0,0 +1,165 @@
+"""Shared configuration for FlatScraper. Type-safe loading from env and user_profile.json."""
+
+import json
+from pathlib import Path
+
+from pydantic import Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+from models import UserProfile
+
+PROJECT_ROOT = Path(__file__).parent
+PROFILE_PATH = PROJECT_ROOT / "user_profile.json"
+
+# Default persona block when no profile exists (placeholder – run "flatscraper setup" to configure)
+_DEFAULT_PERSONA_BLOCK = """DEINE PERSONA:
+- Alter: 28 Jahre
+- Herkunft: Berlin (zieht in Zielstadt)
+- Beruf: Software Engineer
+- Einzugstermin: Flexibel, gerne ab nächstem Monat
+- Persönlichkeit: Ruhig, ordentlich, gesellig
+- Hobbys: Lesen, Sport, Kochen
+- Dokumente: Alle Unterlagen im Google Drive Link (bitte in .env konfigurieren)"""
+
+
+class AppSettings(BaseSettings):
+    """Environment-based settings (from .env or os.environ)."""
+
+    model_config = SettingsConfigDict(
+        env_file=str(PROJECT_ROOT / ".env"),
+        env_file_encoding="utf-8",
+        extra="ignore",
+        populate_by_name=True,
+    )
+
+    email: str = Field(default="", validation_alias="FLATSCRAPER_EMAIL")
+    password: str = Field(default="", validation_alias="FLATSCRAPER_PASSWORD")
+    groq_api_key: str = Field(default="", validation_alias="GROQ_API_KEY")
+    groq_model: str = Field(default="llama-3.1-8b-instant", validation_alias="GROQ_MODEL")
+    google_drive_link: str = Field(default="", validation_alias="GOOGLE_DRIVE_LINK")
+    run_interval_minutes: int = Field(default=30, validation_alias="RUN_INTERVAL_MINUTES")
+    auto_run_enabled: bool = Field(
+        default=False,
+        validation_alias="AUTO_RUN_ENABLED",
+    )
+
+
+_user_profile: UserProfile | None = None
+
+
+def _load_user_profile() -> UserProfile | None:
+    global _user_profile
+    if _user_profile is not None:
+        return _user_profile
+    if PROFILE_PATH.exists():
+        try:
+            data = json.loads(PROFILE_PATH.read_text(encoding="utf-8"))
+            _user_profile = UserProfile.model_validate(data)
+            return _user_profile
+        except Exception:
+            pass
+    return None
+
+
+# --- Public API ---
+
+def get_settings() -> AppSettings:
+    return AppSettings()
+
+
+def get_search_urls() -> list[str]:
+    profile = _load_user_profile()
+    if profile and profile.search_urls:
+        return profile.search_urls
+    return []
+
+
+def get_persona_name() -> str:
+    profile = _load_user_profile()
+    if profile and profile.persona_name:
+        return profile.persona_name
+    return "Nutzer"
+
+
+def get_persona_block() -> str:
+    profile = _load_user_profile()
+    if profile and profile.persona_block:
+        return profile.persona_block
+    return _DEFAULT_PERSONA_BLOCK
+
+
+# Backward-compatible module-level exports (loaded at import)
+_settings_instance = AppSettings()
+
+EMAIL = _settings_instance.email
+PASSWORD = _settings_instance.password
+GROQ_API_KEY = _settings_instance.groq_api_key
+GROQ_MODEL = _settings_instance.groq_model
+GOOGLE_DRIVE_LINK = _settings_instance.google_drive_link
+RUN_INTERVAL_MINUTES = _settings_instance.run_interval_minutes
+AUTO_RUN_ENABLED = _settings_instance.auto_run_enabled
+
+# Parse AUTO_RUN_ENABLED - pydantic-settings should handle "true"/"false" for bool
+# Let me check - by default BaseSettings will coerce "false" to False. Good.
+
+# LLM prompts - built from profile
+_persona_block = get_persona_block()
+_persona_name = get_persona_name()
+
+LLM_SYSTEM_PROMPT = f"""Du bist ein charmanter, professioneller Assistent, der dabei hilft, ein WG-Zimmer ODER eine Wohnung zu finden. Deine Aufgabe ist es, basierend auf einer Wohnungsanzeige ein kurzes, sympathisches und persönliches Anschreiben auf Deutsch zu verfassen. Das Anschreiben passt sich dem Anzeigentyp an (WG-Zimmer vs. Wohnung).
+
+{_persona_block}"""
+
+# ---------------------------------------------------------------------------
+# AD-TYPE INSTRUCTIONS
+# ---------------------------------------------------------------------------
+
+LLM_AD_TYPE_INSTRUCTIONS_WG = """\
+ANZEIGENTYP: WG-Zimmer
+- Ton: locker, herzlich, "passen wir zusammen?"-Gefühl.
+- Fokus: gemeinsames WG-Leben, Interessen der Mitbewohner, Atmosphäre.
+- Anrede: "Hallo [Name]," oder "Hi [Name]," – nie formell."""
+
+LLM_AD_TYPE_INSTRUCTIONS_WOHNUNG = """\
+ANZEIGENTYP: Wohnung (privat oder professionell)
+- Ton: freundlich-professionell, seriös, verbindlich.
+- Fokus: stabiles Einkommen, Zuverlässigkeit, Einzugstermin, vollständige Unterlagen.
+- Anrede: "Hallo [Name]," oder "Sehr geehrte/r [Name]," – je nach Tonalität der Anzeige."""
+
+# ---------------------------------------------------------------------------
+# MESSAGE PROMPT
+# ---------------------------------------------------------------------------
+
+LLM_MESSAGE_PROMPT_TEMPLATE = """\
+ANZEIGENDATEN (Typ: {ad_type_label}):
+
+Titel:        {title}
+Adresse:      {address}
+Anbieter:     {publisher_name}
+Beschreibung:
+\"\"\"
+{description}
+\"\"\"
+
+ANZEIGENTYP-ANWEISUNGEN:
+{ad_type_instructions}
+
+ANREDE-REGELN (Priorität 1 → 3):
+1. Name unter "Anbieter" vorhanden → nutze ihn: "Hallo Marco," / "Hi Lisa,"
+2. Namen im Beschreibungstext ("Wir sind Jonas und Lisa") → nutze den ersten Namen.
+3. Kein Name bekannt → WG: "Hallo liebe WG," | Wohnung: "Hallo,"
+
+AUFGABE:
+Schreibe das Anschreiben für {persona_name} nach dieser Struktur:
+
+1. Persönliche Anrede (siehe Anrede-Regeln – niemals generisch wenn ein Name bekannt ist).
+2. Kurzer Icebreaker: ein konkretes Detail aus der Anzeige aufgreifen.
+3. Kurzvorstellung: Beruf, Herkunft, Einzugstermin.
+4. Call to Action: Verweis auf den Google Drive Link ({google_drive_link}) für alle Unterlagen + Freude auf Besichtigung.
+
+STIL: persönlich, lebendig, kurz – kein Werbejargon, kein Fülltext.
+LÄNGE: unter 150 Wörter.
+FORMAT: Fließtext, kein Markdown, keine Signatur, kein Betreff.
+
+WICHTIG: Antworte AUSSCHLIESSLICH mit dem fertigen Nachrichtentext. Keine Erklärungen, keine Gedanken, keine Meta-Kommentare (z.B. "Kurzfassung meiner Überlegungen"), keine Anleitung – nur die Nachricht selbst.\
+"""