search-parser 0.5.0__tar.gz → 0.5.2__tar.gz

{search_parser-0.5.0 → search_parser-0.5.2}/CHANGELOG.md

@@ -9,6 +9,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.5.2] - 2026-04-26
+
+### Added
+
+- **`GoogleFinanceParser`** — parses Google Finance HTML pages by extracting the
+  embedded `AF_initDataCallback` data blocks (no network calls required).
+  Supports stocks, ETFs, indices, crypto, and FX pairs.  Returns a
+  `SearchResults` object with dedicated fields:
+  - `stock_quote` — price, change, change %, previous close, currency, timezone,
+    after-hours data
+  - `company_info` — description, CEO, employees, market cap, P/E ratio,
+    52-week range, sector, headquarters
+  - `stock_chart` — time-series of date / price / volume points
+  - `financial_news` — news items with title, URL, source, and Unix timestamp
+  - `financial_statements` — income statement, balance sheet, and cash flow
+    data (stocks only; uses recursive positional-array detection)
+
+- **`GoogleFinanceScraper`** — standalone class that fetches live data from
+  Google Finance's internal `batchexecute` RPC endpoint using only the Python
+  standard library (`urllib`).  No API key required.  Returns typed dataclasses
+  (`GoogleFinanceData`, `FinancialQuote`, `CompanyInfo`, `ChartData`,
+  `ChartPoint`, `NewsItem`, `FinancialStatement`).
+
+- **New `result_type` literals** — `stock_quote`, `company_info`, `stock_chart`,
+  `financials`, `financial_news` added to `SearchResult.result_type`.
+
+- **New `SearchResults` fields** — `stock_quote`, `company_info`, `stock_chart`,
+  `financial_statements`, `financial_news` (all default to `None` / `[]` so
+  existing code is unaffected).
+
+- **Auto-detection** — `SearchEngineDetector` now recognises Google Finance
+  pages (canonical URL or `AF_dataServiceRequests` script marker) before
+  falling through to the generic Google check.
+
+### Usage
+
+```python
+# Parse a saved Google Finance HTML page
+from search_parser import GoogleFinanceParser
+
+parser = GoogleFinanceParser()
+results = parser.parse(html)
+
+print(results.query)                          # "GOOGL:NASDAQ"
+print(results.stock_quote.metadata["price"]) # 339.32
+print(results.company_info.metadata["ceo"])  # "Sundar Pichai"
+print(results.financial_news[0].title)
+
+# Fetch live data (no API key needed)
+from search_parser import GoogleFinanceScraper
+
+scraper = GoogleFinanceScraper()
+data = scraper.scrape("GOOGL:NASDAQ")   # stocks
+data = scraper.scrape("BTC-USD")        # crypto
+data = scraper.scrape("EUR-USD")        # FX
+
+print(data.quote.price)
+print(data.company.sector)
+print(data.chart.points[0].date)
+```
+
+---
+
+## [0.5.1] - 2026-04-08
+
+### Fixed
+
+- **JSON output: non-ASCII characters no longer escaped** — `JSONFormatter`, `SearchResults.to_json()`, and the `"dict"` output path in `SearchParser.parse()` all previously relied on Pydantic's `model_dump_json()` / `serde_json`, which escapes every non-ASCII character as `\uXXXX`. They now use `model_dump(mode="json")` + `json.dumps(..., ensure_ascii=False)`, so Korean, Japanese, Arabic, and all other non-Latin scripts are emitted as literal UTF-8 characters.
+
+---
+
 ## [0.5.0] - 2026-04-08
 
 ### Added

{search_parser-0.5.0 → search_parser-0.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: search-parser
-Version: 0.5.0
+Version: 0.5.2
 Summary: Parse Google, Bing, and DuckDuckGo HTML search results into JSON, Markdown, or dict — with auto-detection
 Project-URL: Homepage, https://github.com/getlinksc/search-parser
 Project-URL: Documentation, https://getlinksc.github.io/search-parser
@@ -24,7 +24,7 @@ Requires-Dist: beautifulsoup4>=4.12.0
 Requires-Dist: lxml>=5.0.0
 Requires-Dist: markdownify>=0.11.0
 Requires-Dist: pydantic>=2.0.0
-Requires-Dist: pytest<10.0.0,>=9.0.2
+Requires-Dist: pytest>=7.4.0
 Provides-Extra: cli
 Requires-Dist: click>=8.1.0; extra == 'cli'
 Requires-Dist: rich>=13.0.0; extra == 'cli'
@@ -123,6 +123,7 @@ pip install search-parser
 | Jobs | `jobs` | ✓ | — | — |
 | Discussions and forums | `discussions` | ✓ | — | — |
 | Shopping ads | `shopping_ads` | ✓ | — | — |
+| News tab articles | `news` | ✓ | — | — |
 
 ---
 
@@ -177,6 +178,11 @@ for disc in data["discussions"]:
 for ad in data["shopping_ads"]:
     print(ad["title"], ad["metadata"]["price"], ad["metadata"]["merchant"])
 
+# News tab articles (Google News tab, tbm=nws) — metadata["source"] and metadata["published_time"]
+for article in data["news"]:
+    print(article["title"], article["url"])
+    print(article["metadata"]["source"], article["metadata"]["published_time"])
+
 # Metadata
 print(data["search_engine"])        # "google"
 print(data["query"])                # "python web scraping"
@@ -215,6 +221,10 @@ for post in results.people_saying:
 for ad in results.shopping_ads:
     print(ad.title, ad.metadata["price"], ad.metadata["merchant"])
 
+for article in results.news:
+    print(article.title, article.url)
+    print(article.metadata["source"], article.metadata["published_time"])
+
 # Convert to JSON or Markdown directly on the model
 json_str = results.to_json()
 json_str = results.to_json(indent=4)  # custom indent
@@ -301,6 +311,7 @@ md_str = results.to_markdown()
       }
     }
   ],
+  "news": [],
   "detection_confidence": 0.95,
   "parsed_at": "2026-02-21T00:00:00Z",
   "metadata": {}
@@ -361,6 +372,16 @@ I work for a mid-sized company as a Procurement Manager...
 **Price:** $51.19
 **Merchant:** Contacts Direct
 **URL:** http://www.google.com/aclk?sa=L&ai=...
+
+## News Results
+
+### 1. Python web scraping tutorial released
+
+**Source:** Real Python · 2 days ago
+
+Learn how to scrape websites with Python using BeautifulSoup and Requests...
+
+**URL:** https://realpython.com/python-web-scraping/
 ```
 
 ---

{search_parser-0.5.0 → search_parser-0.5.2}/README.md

@@ -80,6 +80,7 @@ pip install search-parser
 | Jobs | `jobs` | ✓ | — | — |
 | Discussions and forums | `discussions` | ✓ | — | — |
 | Shopping ads | `shopping_ads` | ✓ | — | — |
+| News tab articles | `news` | ✓ | — | — |
 
 ---
 
@@ -134,6 +135,11 @@ for disc in data["discussions"]:
 for ad in data["shopping_ads"]:
     print(ad["title"], ad["metadata"]["price"], ad["metadata"]["merchant"])
 
+# News tab articles (Google News tab, tbm=nws) — metadata["source"] and metadata["published_time"]
+for article in data["news"]:
+    print(article["title"], article["url"])
+    print(article["metadata"]["source"], article["metadata"]["published_time"])
+
 # Metadata
 print(data["search_engine"])        # "google"
 print(data["query"])                # "python web scraping"
@@ -172,6 +178,10 @@ for post in results.people_saying:
 for ad in results.shopping_ads:
     print(ad.title, ad.metadata["price"], ad.metadata["merchant"])
 
+for article in results.news:
+    print(article.title, article.url)
+    print(article.metadata["source"], article.metadata["published_time"])
+
 # Convert to JSON or Markdown directly on the model
 json_str = results.to_json()
 json_str = results.to_json(indent=4)  # custom indent
@@ -258,6 +268,7 @@ md_str = results.to_markdown()
       }
     }
   ],
+  "news": [],
   "detection_confidence": 0.95,
   "parsed_at": "2026-02-21T00:00:00Z",
   "metadata": {}
@@ -318,6 +329,16 @@ I work for a mid-sized company as a Procurement Manager...
 **Price:** $51.19
 **Merchant:** Contacts Direct
 **URL:** http://www.google.com/aclk?sa=L&ai=...
+
+## News Results
+
+### 1. Python web scraping tutorial released
+
+**Source:** Real Python · 2 days ago
+
+Learn how to scrape websites with Python using BeautifulSoup and Requests...
+
+**URL:** https://realpython.com/python-web-scraping/
 ```
 
 ---

{search_parser-0.5.0 → search_parser-0.5.2}/docs/examples/advanced_usage.md

@@ -106,6 +106,12 @@ for ad in results.shopping_ads:
     print(ad.title)
     print(f"  {ad.metadata['price']} from {ad.metadata['merchant']}")
     print(f"  {ad.url}")
+
+# News tab articles — metadata["source"] (publisher) and metadata["published_time"]
+for article in results.news:
+    print(article.title)
+    print(f"  {article.metadata['source']} · {article.metadata['published_time']}")
+    print(f"  {article.url}")
 ```
 
 ## Using to_json() and to_markdown()

{search_parser-0.5.0 → search_parser-0.5.2}/docs/examples/basic_usage.md

@@ -102,6 +102,13 @@ for ad in data["shopping_ads"]:
     print(ad["title"])
     print("  Price:", ad["metadata"]["price"])
     print("  Merchant:", ad["metadata"]["merchant"])
+
+# News tab articles (Google only — tbm=nws pages)
+for article in data["news"]:
+    print(article["title"])
+    print("  URL:", article["url"])
+    print("  Source:", article["metadata"]["source"])
+    print("  Published:", article["metadata"]["published_time"])
 ```
 
 ## Specifying the Engine

{search_parser-0.5.0 → search_parser-0.5.2}/docs/getting_started.md

@@ -105,6 +105,14 @@ for ad in data["shopping_ads"]:
     print("  Price:", ad["metadata"]["price"])
     print("  Merchant:", ad["metadata"]["merchant"])
 
+# News tab articles (list) — Google only (tbm=nws pages)
+# Each entry has metadata["source"] (publisher) and metadata["published_time"]
+for article in data["news"]:
+    print("News:", article["title"])
+    print("  URL:", article["url"])
+    print("  Source:", article["metadata"]["source"])
+    print("  Published:", article["metadata"]["published_time"])
+
 # Metadata
 print(data["query"])           # "python web scraping"
 print(data["total_results"])   # 26200000 or None
@@ -145,6 +153,10 @@ for disc in results.discussions:
 for ad in results.shopping_ads:
     print(ad.title, ad.metadata["price"], ad.metadata["merchant"])
 
+for article in results.news:
+    print(article.title, article.url)
+    print(article.metadata["source"], article.metadata["published_time"])
+
 # Serialize without going through SearchParser
 json_str = results.to_json()  # JSON string, indent=2 by default
 json_str = results.to_json(indent=4)  # custom indent

{search_parser-0.5.0 → search_parser-0.5.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "search-parser"
-version = "0.5.0"
+version = "0.5.2"
 description = "Parse Google, Bing, and DuckDuckGo HTML search results into JSON, Markdown, or dict — with auto-detection"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -29,7 +29,7 @@ dependencies = [
     "lxml>=5.0.0",
     "markdownify>=0.11.0",
     "pydantic>=2.0.0",
-    "pytest (>=9.0.2,<10.0.0)",
+    "pytest (>=7.4.0)",
 ]
 
 [project.optional-dependencies]

search_parser-0.5.2/src/search_parser/__init__.py

@@ -0,0 +1,32 @@
+"""Search Engine Parser - Parse search engine HTML results into structured data."""
+
+from search_parser.__version__ import __version__
+from search_parser.core.models import SearchResult, SearchResults
+from search_parser.core.parser import SearchParser
+from search_parser.parsers.google_finance import GoogleFinanceParser
+from search_parser.scrapers.google_finance import (
+    ChartData,
+    ChartPoint,
+    CompanyInfo,
+    FinancialQuote,
+    FinancialStatement,
+    GoogleFinanceData,
+    GoogleFinanceScraper,
+    NewsItem,
+)
+
+__all__ = [
+    "__version__",
+    "SearchParser",
+    "SearchResult",
+    "SearchResults",
+    "GoogleFinanceParser",
+    "GoogleFinanceScraper",
+    "GoogleFinanceData",
+    "FinancialQuote",
+    "CompanyInfo",
+    "ChartData",
+    "ChartPoint",
+    "NewsItem",
+    "FinancialStatement",
+]

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/__version__.py

@@ -1,3 +1,3 @@
 """Version information."""
 
-__version__ = "0.5.0"
+__version__ = "0.5.3"

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/core/detector.py

@@ -44,6 +44,7 @@ class SearchEngineDetector:
         candidates: list[DetectionResult] = []
 
         for check in [
+            self._check_google_finance,
             self._check_meta_tags,
             self._check_dom_structure,
             self._check_url_patterns,
@@ -60,6 +61,16 @@ class SearchEngineDetector:
             return None
         return best
 
+    def _check_google_finance(self, soup: BeautifulSoup) -> DetectionResult | None:
+        """Detect Google Finance pages before the generic Google check."""
+        canonical = soup.find("link", attrs={"rel": "canonical"})
+        if isinstance(canonical, Tag) and "google.com/finance" in str(canonical.get("href", "")):
+            return DetectionResult(engine="google_finance", confidence=0.97)
+        for script in soup.find_all("script"):
+            if script.string and "AF_dataServiceRequests" in script.string:
+                return DetectionResult(engine="google_finance", confidence=0.85)
+        return None
+
     def _check_meta_tags(self, soup: BeautifulSoup) -> DetectionResult | None:
         """Check HTML meta tags for search engine identification."""
         # Google: <meta property="og:site_name" content="Google">

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/core/models.py

@@ -30,6 +30,12 @@ class SearchResult(BaseModel):
         "job",
         "discussion",
         "shopping_ad",
+        "stock_quote",
+        "company_info",
+        "stock_chart",
+        "financials",
+        "financial_news",
+        "local_business",
     ] = "organic"
     metadata: dict[str, object] = Field(default_factory=dict)
 
@@ -60,6 +66,14 @@ class SearchResults(BaseModel):
     discussions: list[SearchResult] = Field(default_factory=list)
     shopping_ads: list[SearchResult] = Field(default_factory=list)
     news: list[SearchResult] = Field(default_factory=list)
+    local_businesses: list[SearchResult] = Field(default_factory=list)
+
+    # Google Finance dedicated fields
+    stock_quote: Optional[SearchResult] = None  # noqa: UP045
+    company_info: Optional[SearchResult] = None  # noqa: UP045
+    stock_chart: Optional[SearchResult] = None  # noqa: UP045
+    financial_statements: Optional[SearchResult] = None  # noqa: UP045
+    financial_news: list[SearchResult] = Field(default_factory=list)
 
     detection_confidence: float = Field(ge=0.0, le=1.0)
     parsed_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
@@ -74,7 +88,9 @@ class SearchResults(BaseModel):
         Returns:
             JSON string representation of all fields.
         """
-        return self.model_dump_json(indent=indent)
+        import json  # noqa: PLC0415
+
+        return json.dumps(self.model_dump(mode="json"), indent=indent, ensure_ascii=False)
 
     def to_markdown(self) -> str:
         """Render results as a Markdown string suitable for human or LLM consumption.

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/core/parser.py

@@ -118,7 +118,7 @@ class SearchParser:
         elif output_format == "markdown":
             return self._markdown_formatter.format(results)
         elif output_format == "dict":
-            result: dict[str, Any] = json.loads(results.model_dump_json())
+            result: dict[str, Any] = results.model_dump(mode="json")
             return result
         else:
             raise ValueError(f"Unknown output format: {output_format!r}")

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/formatters/json_formatter.py

@@ -2,6 +2,8 @@
 
 from __future__ import annotations
 
+import json
+
 from search_parser.core.models import SearchResults
 from search_parser.formatters.base import BaseFormatter
 
@@ -18,4 +20,4 @@ class JSONFormatter(BaseFormatter):
         Returns:
             JSON string representation.
         """
-        return results.model_dump_json(indent=2)
+        return json.dumps(results.model_dump(mode="json"), indent=2, ensure_ascii=False)

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/formatters/markdown_formatter.py

@@ -75,6 +75,12 @@ class MarkdownFormatter(BaseFormatter):
             for result in results.jobs:
                 lines.extend(self._format_job(result))
 
+        if results.local_businesses:
+            lines.append("## Local Businesses")
+            lines.append("")
+            for result in results.local_businesses:
+                lines.extend(self._format_local_business(result))
+
         if results.discussions:
             lines.append("## Discussions and Forums")
             lines.append("")
@@ -169,6 +175,28 @@ class MarkdownFormatter(BaseFormatter):
         lines.append("")
         return lines
 
+    def _format_local_business(self, result: SearchResult) -> list[str]:
+        """Format a local business pack result."""
+        lines: list[str] = []
+        sponsored = " (Sponsored)" if result.metadata.get("sponsored") else ""
+        lines.append(f"### {result.title}{sponsored}")
+        lines.append("")
+        if result.metadata.get("rating"):
+            rating_str = f"**Rating:** {result.metadata['rating']}"
+            if result.metadata.get("reviews"):
+                rating_str += f" ({result.metadata['reviews']} reviews)"
+            lines.append(rating_str)
+        if result.metadata.get("category"):
+            lines.append(f"**Category:** {result.metadata['category']}")
+        if result.metadata.get("location"):
+            lines.append(f"**Location:** {result.metadata['location']}")
+        if result.metadata.get("hours"):
+            lines.append(f"**Hours:** {result.metadata['hours']}")
+        if result.metadata.get("phone"):
+            lines.append(f"**Phone:** {result.metadata['phone']}")
+        lines.append("")
+        return lines
+
     def _format_result(self, result: SearchResult) -> list[str]:
         """Format a generic result."""
         lines: list[str] = []

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/parsers/__init__.py

@@ -4,12 +4,14 @@ from search_parser.parsers.base import BaseParser
 from search_parser.parsers.bing import BingParser
 from search_parser.parsers.duckduckgo import DuckDuckGoParser
 from search_parser.parsers.google import GoogleParser
+from search_parser.parsers.google_finance import GoogleFinanceParser
 
 # Registry of all available parsers
 PARSER_REGISTRY: dict[str, type[BaseParser]] = {
     "google": GoogleParser,
     "bing": BingParser,
     "duckduckgo": DuckDuckGoParser,
+    "google_finance": GoogleFinanceParser,
 }
 
 __all__ = [
@@ -17,5 +19,6 @@ __all__ = [
     "BingParser",
     "DuckDuckGoParser",
     "GoogleParser",
+    "GoogleFinanceParser",
     "PARSER_REGISTRY",
 ]

{search_parser-0.5.0 → search_parser-0.5.2}/src/search_parser/parsers/google.py

@@ -76,6 +76,7 @@ class GoogleParser(BaseParser):
             discussions=self._extract_discussions(soup),
             shopping_ads=self._extract_shopping_ads(soup),
             news=self._extract_news_results(soup),
+            local_businesses=self._extract_local_businesses(soup),
             detection_confidence=confidence,
         )
 
@@ -688,6 +689,96 @@ class GoogleParser(BaseParser):
             metadata=metadata,
         )
 
+    def _extract_local_businesses(self, soup: BeautifulSoup) -> list[SearchResult]:
+        """Extract local business pack results (div.cXedhc cards)."""
+        results: list[SearchResult] = []
+        for item in soup.find_all("div", class_="cXedhc"):
+            if not isinstance(item, Tag):
+                continue
+            result = self._parse_local_business(item)
+            if result:
+                results.append(result)
+        return results
+
+    def _parse_local_business(self, item: Tag) -> SearchResult | None:
+        """Parse a single local business pack card (div.cXedhc)."""
+        name_el = item.find("span", class_="OSrXXb")
+        if not isinstance(name_el, Tag):
+            return None
+        name = clean_text(name_el.get_text())
+        if not name:
+            return None
+
+        rating_el = item.find("span", class_="yi40Hd")
+        rating = clean_text(rating_el.get_text()) if isinstance(rating_el, Tag) else None
+
+        reviews_el = item.find("span", class_="RDApEe")
+        reviews_raw = clean_text(reviews_el.get_text()) if isinstance(reviews_el, Tag) else None
+        reviews = re.sub(r"[()]", "", reviews_raw).strip() if reviews_raw else None
+
+        sponsored_el = item.find("span", class_="gghBu")
+        sponsored = isinstance(sponsored_el, Tag) and "sponsored" in sponsored_el.get_text().lower()
+
+        # Un-classed leaf divs carry category, location, hours, and phone
+        leaf_divs = [
+            d
+            for d in item.find_all("div")
+            if isinstance(d, Tag) and not d.get("class") and not d.find("div") and d.get_text(strip=True)
+        ]
+
+        category: str | None = None
+        location: str | None = None
+        hours: str | None = None
+        phone: str | None = None
+
+        for div in leaf_divs:
+            text = div.get_text(separator="|", strip=True)
+            # "4.9|(813)|· Personal injury attorney" — category follows "· "
+            cat_match = re.search(r"·\s+(.+)$", text.replace("|", " "))
+            if cat_match and not category:
+                candidate = cat_match.group(1).strip()
+                if not re.search(r"\d+\s*(hours?|year|min)", candidate, re.I):
+                    category = candidate
+                    continue
+            # "15+ years in business · Chantilly, VA" — split on " · "
+            if "years in business" in text or re.search(r"\d+\+?\s+year", text):
+                parts = text.replace("|", " ").split(" · ", 1)
+                if len(parts) == 2:
+                    location = parts[1].strip()
+                continue
+            # "Open 24 hours · (703) 952-3191"
+            if re.search(r"open|closed|\d{1,2}:\d{2}", text, re.I):
+                parts = text.replace("|", " ").split("· ", 1)
+                hours = parts[0].strip()
+                if len(parts) == 2:
+                    phone = parts[1].strip()
+                continue
+
+        metadata: dict[str, object] = {}
+        if rating:
+            metadata["rating"] = rating
+        if reviews:
+            metadata["reviews"] = reviews
+        if category:
+            metadata["category"] = category
+        if location:
+            metadata["location"] = location
+        if hours:
+            metadata["hours"] = hours
+        if phone:
+            metadata["phone"] = phone
+        if sponsored:
+            metadata["sponsored"] = True
+
+        return SearchResult(
+            title=name,
+            url="",
+            description=None,
+            position=0,
+            result_type="local_business",
+            metadata=metadata,
+        )
+
     def _extract_featured_snippet(self, soup: BeautifulSoup) -> SearchResult | None:
         """Extract featured snippet if present."""
         snippet_container = soup.find("div", class_="xpdopen")

{search_parser-0.5.0 → search_parser-0.5.2}/tests/conftest.py

@@ -82,3 +82,23 @@ def google_seven_cloak_html() -> str:
 @pytest.fixture()
 def google_news_tab_html() -> str:
     return (FIXTURES_DIR / "google" / "news_tab_search.html").read_text()
+
+
+@pytest.fixture()
+def google_personal_injury_lawyer_html() -> str:
+    return (FIXTURES_DIR / "google" / "personal_injury_lawyer_20260528_203801.html").read_text()
+
+
+@pytest.fixture()
+def google_iphone_15_review_html() -> str:
+    return (FIXTURES_DIR / "google" / "iphone_15_review_20260528_204042.html").read_text()
+
+
+@pytest.fixture()
+def google_finance_quote_html() -> str:
+    return (FIXTURES_DIR / "google_finance" / "quote.html").read_text()
+
+
+@pytest.fixture()
+def google_finance_crypto_html() -> str:
+    return (FIXTURES_DIR / "google_finance" / "crypto.html").read_text()