flru-parser 0.3.0__py3-none-any.whl

flru_parser-0.3.0.dist-info/METADATA

@@ -0,0 +1,421 @@
+Metadata-Version: 2.4
+Name: flru-parser
+Version: 0.3.0
+Summary: Simple, typed and resilient async/sync parser client for public FL.ru pages
+Author: Andrey
+License-Expression: MIT
+Project-URL: Documentation, https://github.com/N3ffus/flru-parser#readme
+Project-URL: Changelog, https://github.com/N3ffus/flru-parser/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/N3ffus/flru-parser/issues
+Project-URL: Repository, https://github.com/N3ffus/flru-parser
+Keywords: fl.ru,freelance,parser,scraper,asyncio,httpx
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: beautifulsoup4<5,>=4.12
+Requires-Dist: httpx[http2]<1,>=0.28
+Requires-Dist: lxml<7,>=5
+Requires-Dist: pydantic<3,>=2.10
+Provides-Extra: postgres
+Requires-Dist: asyncpg<1,>=0.30; extra == "postgres"
+Provides-Extra: redis
+Requires-Dist: redis<8,>=5.2; extra == "redis"
+Provides-Extra: observability
+Requires-Dist: opentelemetry-api<2,>=1.29; extra == "observability"
+Requires-Dist: opentelemetry-sdk<2,>=1.29; extra == "observability"
+Requires-Dist: prometheus-client<1,>=0.21; extra == "observability"
+Dynamic: license-file
+
+# flru-parser
+
+[![PyPI](https://img.shields.io/pypi/v/flru-parser.svg)](https://pypi.org/project/flru-parser/)
+[![Python](https://img.shields.io/pypi/pyversions/flru-parser.svg)](https://pypi.org/project/flru-parser/)
+[![License](https://img.shields.io/pypi/l/flru-parser.svg)](LICENSE)
+[![Typing](https://img.shields.io/badge/typing-PEP%20561-blue)](src/flru/py.typed)
+[![Coverage](https://img.shields.io/badge/branch%20coverage-85.76%25-brightgreen)](#quality)
+[![Tests](https://img.shields.io/badge/tests-47%20passed-brightgreen)](#quality)
+[![CI](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml/badge.svg)](https://github.com/N3ffus/flru-parser/actions/workflows/ci.yml)
+
+A typed, resilient, read-only Python client for public pages on [FL.ru](https://www.fl.ru/).
+The default API is intentionally small; production controls remain available when needed.
+
+> **Unofficial project.** This package is not affiliated with or endorsed by FL.ru. FL.ru does not expose a stable public API for all supported data. HTML selectors may require maintenance after a site redesign. The parser detects likely selector drift instead of silently returning an empty catalog.
+
+[Русская документация](docs/ru/README.md) · [Simple API](docs/SIMPLE_API.md) · [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md)
+
+## Features
+
+- Simple async API: `projects()`, `project()`, `user()`, `freelancers()`, `new_projects()`.
+- Matching synchronous API in `flru.sync`.
+- Projects, details, categories, freelancers, profiles, reviews and portfolios.
+- Typed Pydantic models with PEP 561 support.
+- Retries, jitter, `Retry-After`, shared HTTP 429 cooldown and retry budgets.
+- RPS and concurrency limits, endpoint/proxy circuit breakers and proxy health tracking.
+- HTTP, HTTPS and SOCKS proxy pools with credential redaction.
+- Safe redirects, allowed-host enforcement, `robots.txt` support and block-page detection.
+- Incremental crawling with Memory, SQLite, PostgreSQL or Redis state.
+- Parse diagnostics, confidence, field provenance and page fingerprints.
+- Prometheus, OpenTelemetry and structured event integrations.
+- CI, branch-coverage gate, dependency audit, live canary and PyPI Trusted Publishing.
+
+## Installation
+
+```bash
+uv add flru-parser
+```
+
+Optional integrations:
+
+```bash
+uv add "flru-parser[postgres]"
+uv add "flru-parser[redis]"
+uv add "flru-parser[observability]"
+```
+
+Supported Python versions: **3.11–3.13**.
+
+## Five-minute API
+
+### Async
+
+```python
+import asyncio
+
+from flru import Client
+
+
+async def main() -> None:
+    async with Client() as fl:
+        projects = await fl.projects(pages=5)
+
+        for project in projects:
+            print(project.title, project.budget_min, project.currency, project.url)
+
+
+asyncio.run(main())
+```
+
+### Sync
+
+```python
+from flru.sync import Client
+
+with Client() as fl:
+    projects = fl.projects(pages=5)
+
+for project in projects:
+    print(project.title, project.budget_min, project.url)
+```
+
+### One-shot helpers
+
+```python
+import asyncio
+
+from flru import fetch_project, fetch_projects
+
+projects = asyncio.run(fetch_projects(pages=3, query="FastAPI"))
+project = asyncio.run(fetch_project(projects[0].id))
+```
+
+For synchronous scripts:
+
+```python
+from flru.sync import project, projects
+
+items = projects(pages=3, query="FastAPI")
+detail = project(items[0].id)
+```
+
+## Common operations
+
+### Search and filter projects
+
+No separate filter object is needed for the common path:
+
+```python
+projects = await fl.projects(
+    pages=10,
+    query="Python FastAPI",
+    category="programmirovanie/python",
+    min_budget=30_000,
+    max_budget=200_000,
+    types="order",            # order, vacancy, contest
+    with_budget=True,
+    concurrency=3,
+)
+```
+
+`types` accepts one value or a sequence. English and Russian aliases are supported:
+
+```python
+projects = await fl.projects(types=["заказ", "вакансия"])
+```
+
+### Parse the complete catalog
+
+```python
+projects = await fl.projects(pages="all")
+```
+
+For a large catalog, stream results instead of retaining everything in memory:
+
+```python
+async for project in fl.stream_projects(pages="all", concurrency=3):
+    print(project.id, project.title)
+```
+
+### Load full project cards
+
+```python
+details = await fl.projects(pages=3, details=True)
+
+for project in details:
+    print(project.full_description, project.attachments)
+```
+
+Or retrieve one project by ID or URL:
+
+```python
+project = await fl.project(5500001)
+project = await fl.project("https://www.fl.ru/projects/5500001/example.html")
+```
+
+### Profiles and freelancers
+
+```python
+profile = await fl.user("username")
+full_profile = await fl.user("username", full=True, pages=3)
+freelancers = await fl.freelancers(pages=5, category="programmirovanie")
+```
+
+`full=True` concurrently loads projects, reviews and portfolio sections.
+
+### Incremental crawling
+
+The simplest durable mode uses a local SQLite file automatically:
+
+```python
+new_or_changed = await fl.new_projects(
+    "flru-state.db",
+    pages=30,
+    stop_after_known=20,
+)
+```
+
+The state tracks content hashes, `first_seen_at`, `last_seen_at` and crawl checkpoints. Repeated runs stop after enough already-known records.
+
+Advanced stores remain available:
+
+```python
+from flru import PostgresStateStore
+
+state = PostgresStateStore("postgresql://user:pass@localhost/app")
+projects = await fl.new_projects(state, pages="all")
+await state.close()
+```
+
+### Proxies, cookies and request limits
+
+Common production options are flat constructor arguments:
+
+```python
+async with Client(
+    concurrency=3,
+    rps=0.8,
+    retries=6,
+    timeout=45,
+    proxy=[
+        "http://user:password@proxy-1.example:8080",
+        "socks5://user:password@proxy-2.example:1080",
+    ],
+    cookies="cookies.txt",
+) as fl:
+    projects = await fl.projects(pages=10)
+```
+
+`cookies` can be a mapping or a Netscape-format browser export path.
+Proxy credentials are redacted from metrics and events.
+
+### Models
+
+Models are regular Pydantic models:
+
+```python
+project = projects[0]
+
+print(project.budget_min)
+print(project.budget_max)
+print(project.currency)
+print(project.customer_username)
+print(project.to_dict())
+print(project.model_dump_json(indent=2))
+```
+
+## Advanced API
+
+The simple `Client` subclasses the full `FLClient`, so low-level methods remain directly available:
+
+```python
+from flru import FLClient, ProjectFilters
+
+async with FLClient() as client:
+    page = await client.get_projects_page(page=1)
+    batch = await client.get_projects_batch_result(range(1, 11), concurrency=4)
+    categories = await client.get_categories()
+```
+
+Use `FLClient` and the immutable configuration dataclasses when every transport setting must be controlled:
+
+```python
+from flru import (
+    CircuitBreakerConfig,
+    ClientConfig,
+    FLClient,
+    ProxyConfig,
+    RateLimitConfig,
+    RetryConfig,
+)
+
+config = ClientConfig(
+    retry=RetryConfig(
+        max_attempts=6,
+        base_delay=1,
+        max_delay=45,
+        total_timeout=120,
+        max_total_delay=60,
+    ),
+    rate_limit=RateLimitConfig(
+        requests_per_second=0.8,
+        max_concurrency=4,
+        min_interval=0.4,
+    ),
+    circuit_breaker=CircuitBreakerConfig(
+        failure_threshold=8,
+        recovery_timeout=90,
+        scope="endpoint_proxy",
+    ),
+    proxies=ProxyConfig(
+        urls=("http://user:password@proxy.example:8080",),
+        direct_fallback=True,
+    ),
+)
+
+client = FLClient(config)
+```
+
+See [docs/SIMPLE_API.md](docs/SIMPLE_API.md) for the complete high-level API and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for internals.
+
+## Reliability behavior
+
+- Empty output is not automatically treated as the end of the catalog.
+- Candidate links with no parsed cards raise `SelectorDriftError`.
+- Unclassified empty pages raise `EmptyPageError`.
+- A recognized end page contains the `catalog_end` diagnostic warning.
+- HTTP 429 can pause all concurrent workers through a shared cooldown.
+- Circuit breakers are scoped by endpoint and proxy by default.
+- CAPTCHA and block pages are detected, not bypassed.
+- Requests are restricted to allowed FL.ru hosts and safe redirects.
+
+## Observability
+
+```python
+metrics = await fl.stats()
+print(metrics.requests_total, metrics.retries_total, metrics.endpoints)
+```
+
+Structured logging:
+
+```python
+from flru import Client, StructuredLogHandler
+
+fl = Client(event_handler=StructuredLogHandler())
+```
+
+Optional adapters:
+
+```python
+from flru.integrations.opentelemetry import OpenTelemetryEventHandler
+from flru.integrations.prometheus import PrometheusEventHandler
+```
+
+Install `flru-parser[observability]` before using these adapters.
+
+## Exceptions
+
+| Exception | Meaning |
+|---|---|
+| `BlockedError` | CAPTCHA, anti-bot or access-block page detected |
+| `AuthenticationRequired` | Authenticated cookies are required |
+| `RateLimitedError` | HTTP 429 after retry policy exhaustion |
+| `CircuitOpenError` | A scoped circuit breaker is open |
+| `SecurityError` | Host, scheme or redirect violates policy |
+| `RobotsDeniedError` | `robots.txt` denies access |
+| `SelectorDriftError` | Candidate records exist but current selectors parsed none |
+| `EmptyPageError` | Empty page cannot be classified as catalog end |
+| `ParseError` | HTML cannot be converted into the requested model |
+
+## Quality
+
+Current verified result for **0.3.0**:
+
+```text
+47 passed, 1 optional Hypothesis suite skipped locally
+Branch coverage: 85.76%
+Coverage gate: 85%
+```
+
+Run the complete local checks:
+
+```bash
+uv sync --group dev
+uv run ruff check .
+uv run ruff format --check .
+uv run mypy src/flru
+uv run pytest --cov=flru --cov-branch --cov-report=term-missing --cov-report=html
+uv build
+uvx twine check dist/*
+```
+
+The repository also contains:
+
+- Python 3.11–3.13 CI matrix;
+- scheduled dependency vulnerability audit;
+- read-only live selector canary;
+- pre-commit configuration;
+- Trusted Publishing workflows for TestPyPI and PyPI.
+
+## Publishing
+
+The release workflow uses PyPI Trusted Publishing and checks that the Git tag matches both package version declarations.
+
+```bash
+uv run python scripts/configure_project.py YOUR_GITHUB_USERNAME
+git tag -a v0.3.0 -m "flru-parser 0.3.0"
+git push origin main v0.3.0
+```
+
+See [docs/PUBLISHING.md](docs/PUBLISHING.md) for the one-time PyPI and GitHub setup.
+
+## Responsible use
+
+- Respect FL.ru terms, `robots.txt`, applicable law and personal-data requirements.
+- Prefer incremental crawling and reasonable request rates.
+- Do not use proxies or this package to bypass access controls or CAPTCHA.
+- Do not automate account actions through undocumented endpoints.
+- Public indexability does not automatically grant unlimited collection or redistribution rights.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

flru_parser-0.3.0.dist-info/RECORD

@@ -0,0 +1,33 @@
+flru/__init__.py,sha256=lFrVs-PRWp67zd5EOLaJN2yxFfl-2g8m-lkg3gmhQ9I,2704
+flru/batch.py,sha256=kYDgVJJf-1gy64T5PxPqQYIB0q2fJDz8UbJ3RsQomjk,625
+flru/canary.py,sha256=ENeuztMB4G6P355nBUoLyUAy3o84lwyzD1nfj64AmZM,914
+flru/client.py,sha256=fE_xCjB61nT7sXd9uY4MKKGipP6xi5_CkpaoeQ0Vb_4,30376
+flru/config.py,sha256=-xLydcgEY-z7lMCZTvRcsW8kKaFBIFifUzgPyuf6ZE0,4226
+flru/cookies.py,sha256=pov7XZMZyEJGmlMiRCfB7GyAoDiEGxS2tf7LAeEJBKo,500
+flru/easy.py,sha256=6vKsYCVlPOH8Y_NJcn9joyTJPnHLj5k_qmQWsxppPeA,16990
+flru/exceptions.py,sha256=aWzFv4lHIZXMoAH_jl7Vt1Y-KpEGh6QPZyQVTMz57Og,1598
+flru/filters.py,sha256=07Zm8XCQP31B3IRl3Wf8L4aEGc8IQ4WYQVvQfpxp_8Q,1251
+flru/models.py,sha256=eEO9KXEhSdpdVirf28Pe1N9ZucZR-hnlGujp-T-DFyg,8474
+flru/observability.py,sha256=NyWZuJPEs8gor9Uln2JjxHG77QPqWhi3RSxwJn3mxkM,3960
+flru/proxy.py,sha256=lklKtZu_GO35VPOz-ocPWgrRkSw39MzoXaEn2R7z9M4,2878
+flru/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+flru/resilience.py,sha256=a5WH6ljiFzjKAME5j4PNIf5pJ_oWRl3JahIpPWuatTk,5387
+flru/robots.py,sha256=CTQPe4BCm-SxUHZt73fcqWTmmwNulBj2ox3ccIpyO08,1973
+flru/security.py,sha256=WFBF98hnlefSwlLnLuhXSSEEw5vlJlMw9MNi40BUHRM,1455
+flru/state.py,sha256=8T337jecGKScKv_IU4Sf1S4-0gQgsLlroa1hNUFBotA,12383
+flru/sync.py,sha256=u3rJt2cbnnEWtdfHMm0J2Kef1cmU8JgcsexDme7XOJE,6710
+flru/transport.py,sha256=SBtN2ZP1NwXgdItOgiBbrEbN8CwBArlJVzrzuyyaB9k,14025
+flru/integrations/__init__.py,sha256=cXG129B1iZSQ3O39LJART1PFK_3p6abMMrxoQEJeCmY,43
+flru/integrations/opentelemetry.py,sha256=0-WGJPFnuON5Te21ZFNa1TIVdHu974MgK2KsFp6aKHE,990
+flru/integrations/prometheus.py,sha256=JYRoM2NFYcDWQzGQ0RHqLdhGpEE7pBGe8pn1EoFBnkQ,1198
+flru/parsers/__init__.py,sha256=lEFj6Be5gHHvaCtWQFH0BiRoVzyGnVoOQczCNTwfvwQ,365
+flru/parsers/common.py,sha256=_Uef1_yPorBSr-9NXoGaIYmaIYnsopoKz-N1ovkF1NQ,9270
+flru/parsers/freelancers.py,sha256=Ukq2UwZGQiknuKNGEexULTROi4Hi-agvDHAatYvwIbQ,4936
+flru/parsers/projects.py,sha256=PwmkU2RHF1Ow28Xfo4tTjkKOFb-_RWNOPPGdLGCA9CI,13414
+flru/parsers/users.py,sha256=HRBftGXW3Ju1TKZZJtQJ833_eRkg0Jnu0pMcGN-8JXE,7380
+flru_parser-0.3.0.dist-info/licenses/LICENSE,sha256=1zNNV91mmCQYfzAOrjvPh0c18NR6BImrUBS18w3yvFc,1063
+flru_parser-0.3.0.dist-info/METADATA,sha256=3m4FQqjRQ9dKHn7HWAXN6Uic5jHyzjICdXnkLQ_jZa8,12430
+flru_parser-0.3.0.dist-info/WHEEL,sha256=K260EYznzXsJYBQGqmI8VTxEdiZYNvDZwW9cBh9-_MA,91
+flru_parser-0.3.0.dist-info/entry_points.txt,sha256=pA1nzHVJnt6IjNusyCW8paiu4oUbRfTwzp669owuEAY,49
+flru_parser-0.3.0.dist-info/top_level.txt,sha256=1oNJlhBZ1TyECwrj5InG0Kfi1P_KeEwJO2ASSBD-9uw,5
+flru_parser-0.3.0.dist-info/RECORD,,

flru_parser-0.3.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (83.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

flru_parser-0.3.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+flru-canary = flru.canary:main

flru_parser-0.3.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Andrey
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

flru_parser-0.3.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+flru