cloro-haystack 0.2.0__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,7 +1,7 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: cloro-haystack
3
- Version: 0.2.0
4
- Summary: Haystack components for cloro: Google search results with the AI Overview, and ChatGPT, Google AI Mode, Gemini and Bing AI Mode answers with sources.
3
+ Version: 0.3.0
4
+ Summary: Haystack components for cloro: Google and Bing search results with the AI Overview, and ChatGPT, Google AI Mode, Gemini and Bing AI Mode answers with sources.
5
5
  Project-URL: Homepage, https://cloro.dev/?utm_source=cloro-haystack&utm_medium=byok
6
6
  Project-URL: Documentation, https://github.com/cloro-dev/cloro-haystack#readme
7
7
  Project-URL: Issues, https://github.com/cloro-dev/cloro-haystack/issues
@@ -28,7 +28,7 @@ Description-Content-Type: text/markdown
28
28
 
29
29
  | Component | cloro endpoint | Output | Credits per run |
30
30
  |---|---|---|--:|
31
- | `CloroWebSearch` | `POST /v1/monitor/google` | `documents` and `links` from Google organic results. With `include_ai_overview=True`, the Google AI Overview is the first Document. | 5, or 7 with the AI Overview |
31
+ | `CloroWebSearch` | `POST /v1/monitor/google` or `/bing` | `documents` and `links` from Google or Bing organic results. With `include_ai_overview=True`, the AI Overview is the first Document. | 5, or 7 with the AI Overview |
32
32
  | `CloroAIAnswer` | `POST /v1/monitor/chatgpt`, `/aimode`, `/gemini` or `/bing/aimode` | `replies` (the answer as Markdown), `documents` and `links` (the cited sources) | ChatGPT 7, Google AI Mode 6, Gemini 6, Bing AI Mode 6 |
33
33
 
34
34
  ## Installation
@@ -55,9 +55,10 @@ for doc in result["documents"]:
55
55
  print(doc.meta.get("title"), doc.meta.get("link"))
56
56
  ```
57
57
 
58
- Each organic result is a `Document`. Its `content` is the Google snippet and its `meta` has `title`,
59
- `link` and `position`. The AI Overview Document has `meta["source"] == "google_ai_overview"` and the
60
- URLs it cites in `meta["links"]`. Results with an empty link or a Google redirect link are skipped.
58
+ Each organic result is a `Document`. Its `content` is the result snippet and its `meta` has `title`,
59
+ `link` and `position`. The AI Overview Document has `meta["source"] == "google_ai_overview"` (or
60
+ `"bing_ai_overview"`) and the URLs it cites in `meta["links"]`. Results with an empty link or a Google
61
+ redirect link are skipped.
61
62
 
62
63
  Parameters:
63
64
 
@@ -66,9 +67,19 @@ Parameters:
66
67
  results (about 10). Set `search_params={"pages": 2}` for more (2 credits for each extra page).
67
68
  - `country`: ISO 3166-1 alpha-2 country code. Defaults to `"US"`.
68
69
  - `allowed_domains`: domains to limit the search to, sent as `site:` filters.
69
- - `include_ai_overview`: request the Google AI Overview. Defaults to `False`.
70
- - `search_params`: other request fields, for example `device`, `location`, `hl` or `pages`.
70
+ - `include_ai_overview`: request the AI Overview. Defaults to `False`.
71
+ - `search_params`: other request fields, for example `device`, `location`, `hl` or `pages` for Google.
71
72
  - `timeout`: request timeout in seconds. Defaults to 60.
73
+ - `engine`: `"google"` (Google Search) or `"bing"` (Bing Search). Defaults to `"google"`.
74
+
75
+ With `engine="bing"`, the component searches Bing. The country sets the Bing market. Bing Search accepts
76
+ all countries except CN, KR and RU. It has no `device`, `location`, `hl` or `pages` fields and returns
77
+ one page of results. The query can have a maximum of 3,500 characters after URL encoding, and fewer for
78
+ non-Latin scripts.
79
+
80
+ ```python
81
+ websearch = CloroWebSearch(engine="bing", country="GB", include_ai_overview=True)
82
+ ```
72
83
 
73
84
  ### CloroAIAnswer
74
85
 
@@ -5,7 +5,7 @@
5
5
 
6
6
  | Component | cloro endpoint | Output | Credits per run |
7
7
  |---|---|---|--:|
8
- | `CloroWebSearch` | `POST /v1/monitor/google` | `documents` and `links` from Google organic results. With `include_ai_overview=True`, the Google AI Overview is the first Document. | 5, or 7 with the AI Overview |
8
+ | `CloroWebSearch` | `POST /v1/monitor/google` or `/bing` | `documents` and `links` from Google or Bing organic results. With `include_ai_overview=True`, the AI Overview is the first Document. | 5, or 7 with the AI Overview |
9
9
  | `CloroAIAnswer` | `POST /v1/monitor/chatgpt`, `/aimode`, `/gemini` or `/bing/aimode` | `replies` (the answer as Markdown), `documents` and `links` (the cited sources) | ChatGPT 7, Google AI Mode 6, Gemini 6, Bing AI Mode 6 |
10
10
 
11
11
  ## Installation
@@ -32,9 +32,10 @@ for doc in result["documents"]:
32
32
  print(doc.meta.get("title"), doc.meta.get("link"))
33
33
  ```
34
34
 
35
- Each organic result is a `Document`. Its `content` is the Google snippet and its `meta` has `title`,
36
- `link` and `position`. The AI Overview Document has `meta["source"] == "google_ai_overview"` and the
37
- URLs it cites in `meta["links"]`. Results with an empty link or a Google redirect link are skipped.
35
+ Each organic result is a `Document`. Its `content` is the result snippet and its `meta` has `title`,
36
+ `link` and `position`. The AI Overview Document has `meta["source"] == "google_ai_overview"` (or
37
+ `"bing_ai_overview"`) and the URLs it cites in `meta["links"]`. Results with an empty link or a Google
38
+ redirect link are skipped.
38
39
 
39
40
  Parameters:
40
41
 
@@ -43,9 +44,19 @@ Parameters:
43
44
  results (about 10). Set `search_params={"pages": 2}` for more (2 credits for each extra page).
44
45
  - `country`: ISO 3166-1 alpha-2 country code. Defaults to `"US"`.
45
46
  - `allowed_domains`: domains to limit the search to, sent as `site:` filters.
46
- - `include_ai_overview`: request the Google AI Overview. Defaults to `False`.
47
- - `search_params`: other request fields, for example `device`, `location`, `hl` or `pages`.
47
+ - `include_ai_overview`: request the AI Overview. Defaults to `False`.
48
+ - `search_params`: other request fields, for example `device`, `location`, `hl` or `pages` for Google.
48
49
  - `timeout`: request timeout in seconds. Defaults to 60.
50
+ - `engine`: `"google"` (Google Search) or `"bing"` (Bing Search). Defaults to `"google"`.
51
+
52
+ With `engine="bing"`, the component searches Bing. The country sets the Bing market. Bing Search accepts
53
+ all countries except CN, KR and RU. It has no `device`, `location`, `hl` or `pages` fields and returns
54
+ one page of results. The query can have a maximum of 3,500 characters after URL encoding, and fewer for
55
+ non-Latin scripts.
56
+
57
+ ```python
58
+ websearch = CloroWebSearch(engine="bing", country="GB", include_ai_overview=True)
59
+ ```
49
60
 
50
61
  ### CloroAIAnswer
51
62
 
@@ -4,8 +4,8 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "cloro-haystack"
7
- version = "0.2.0"
8
- description = "Haystack components for cloro: Google search results with the AI Overview, and ChatGPT, Google AI Mode, Gemini and Bing AI Mode answers with sources."
7
+ version = "0.3.0"
8
+ description = "Haystack components for cloro: Google and Bing search results with the AI Overview, and ChatGPT, Google AI Mode, Gemini and Bing AI Mode answers with sources."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.11"
11
11
  license = "MIT"
@@ -2,7 +2,7 @@
2
2
  #
3
3
  # SPDX-License-Identifier: MIT
4
4
 
5
- from typing import Any
5
+ from typing import Any, Literal
6
6
 
7
7
  from haystack import Document, component, default_from_dict, default_to_dict, logging
8
8
  from haystack.utils import Secret, deserialize_secrets_inplace
@@ -11,19 +11,23 @@ from haystack_integrations.components.websearch.cloro._api import is_direct_link
11
11
 
12
12
  logger = logging.getLogger(__name__)
13
13
 
14
- GOOGLE_PATH = "/v1/monitor/google"
14
+ Engine = Literal["google", "bing"]
15
+
16
+ # The API path and the display name for each engine.
17
+ _PATHS: dict[str, str] = {"google": "/v1/monitor/google", "bing": "/v1/monitor/bing"}
18
+ _NAMES: dict[str, str] = {"google": "Google", "bing": "Bing"}
15
19
 
16
20
 
17
21
  @component
18
22
  class CloroWebSearch:
19
23
  """
20
- Searches Google with the cloro API and returns the results as Documents.
24
+ Searches Google or Bing with the cloro API and returns the results as Documents.
21
25
 
22
26
  cloro: https://cloro.dev/?utm_source=cloro-haystack&utm_medium=byok
23
27
 
24
- Each organic result becomes a Document. Its content is the Google snippet. Its meta has
25
- `title`, `link` and `position`. With `include_ai_overview=True`, the Google AI Overview
26
- (when Google shows one) is the first Document.
28
+ Each organic result becomes a Document. Its content is the result snippet. Its meta has
29
+ `title`, `link` and `position`. With `include_ai_overview=True`, the AI Overview of the
30
+ engine (when the engine shows one) is the first Document.
27
31
 
28
32
  ### Usage example
29
33
 
@@ -46,6 +50,7 @@ class CloroWebSearch:
46
50
  include_ai_overview: bool = False,
47
51
  search_params: dict[str, Any] | None = None,
48
52
  timeout: float = 60.0,
53
+ engine: Engine = "google",
49
54
  ) -> None:
50
55
  """
51
56
  Initialize the CloroWebSearch component.
@@ -53,22 +58,28 @@ class CloroWebSearch:
53
58
  :param api_key:
54
59
  cloro API key. Defaults to the `CLORO_API_KEY` environment variable.
55
60
  :param top_k:
56
- Maximum number of organic results to return. One request returns one page of Google
57
- results (about 10). Set `search_params={"pages": 2}` or more to get more results.
61
+ Maximum number of organic results to return. One request returns one page of results
62
+ (about 10). For Google, set `search_params={"pages": 2}` or more to get more results.
58
63
  :param country:
59
- ISO 3166-1 alpha-2 country code for the Google location.
64
+ ISO 3166-1 alpha-2 country code for the engine location.
60
65
  :param allowed_domains:
61
66
  Domains to limit the search to. They are added to the query as `site:` filters.
62
67
  :param include_ai_overview:
63
- If True, request the Google AI Overview and return it as the first Document.
68
+ If True, request the AI Overview and return it as the first Document.
64
69
  :param search_params:
65
- Other fields for the `POST /v1/monitor/google` request body, for example `device`,
66
- `location`, `hl` or `pages`. See the cloro API reference.
70
+ Other fields for the engine request body, for example `device`, `location`, `hl` or
71
+ `pages` for Google. See the cloro API reference.
67
72
  :param timeout:
68
73
  Request timeout in seconds.
74
+ :param engine:
75
+ The search engine: `"google"` (Google Search) or `"bing"` (Bing Search).
69
76
  """
77
+ if engine not in _PATHS:
78
+ msg = f"engine must be 'google' or 'bing', not {engine!r}"
79
+ raise ValueError(msg)
70
80
  self.api_key = api_key
71
81
  self.top_k = top_k
82
+ self.engine = engine
72
83
  self.country = country
73
84
  self.allowed_domains = allowed_domains
74
85
  self.include_ai_overview = include_ai_overview
@@ -90,6 +101,7 @@ class CloroWebSearch:
90
101
  include_ai_overview=self.include_ai_overview,
91
102
  search_params=self.search_params,
92
103
  timeout=self.timeout,
104
+ engine=self.engine,
93
105
  )
94
106
 
95
107
  @classmethod
@@ -106,7 +118,7 @@ class CloroWebSearch:
106
118
  @component.output_types(documents=list[Document], links=list[str])
107
119
  def run(self, query: str) -> dict[str, Any]:
108
120
  """
109
- Search Google with cloro.
121
+ Search Google or Bing with cloro.
110
122
 
111
123
  :param query: Search query.
112
124
  :returns: A dictionary with:
@@ -115,13 +127,13 @@ class CloroWebSearch:
115
127
  :raises TimeoutError: If the request times out.
116
128
  :raises CloroError: If the cloro API returns an error.
117
129
  """
118
- result = post(GOOGLE_PATH, self._payload(query), self.api_key, self.timeout)
130
+ result = post(_PATHS[self.engine], self._payload(query), self.api_key, self.timeout)
119
131
  return self._parse(result)
120
132
 
121
133
  @component.output_types(documents=list[Document], links=list[str])
122
134
  async def run_async(self, query: str) -> dict[str, Any]:
123
135
  """
124
- Search Google with cloro without blocking.
136
+ Search Google or Bing with cloro without blocking.
125
137
 
126
138
  This is the asynchronous version of `run`, with the same parameters and return values.
127
139
 
@@ -132,7 +144,7 @@ class CloroWebSearch:
132
144
  :raises TimeoutError: If the request times out.
133
145
  :raises CloroError: If the cloro API returns an error.
134
146
  """
135
- result = await post_async(GOOGLE_PATH, self._payload(query), self.api_key, self.timeout)
147
+ result = await post_async(_PATHS[self.engine], self._payload(query), self.api_key, self.timeout)
136
148
  return self._parse(result)
137
149
 
138
150
  def _payload(self, query: str) -> dict[str, Any]:
@@ -167,7 +179,11 @@ class CloroWebSearch:
167
179
  0,
168
180
  Document(
169
181
  content=overview["text"],
170
- meta={"title": "Google AI Overview", "source": "google_ai_overview", "links": sources},
182
+ meta={
183
+ "title": f"{_NAMES[self.engine]} AI Overview",
184
+ "source": f"{self.engine}_ai_overview",
185
+ "links": sources,
186
+ },
171
187
  ),
172
188
  )
173
189
 
@@ -0,0 +1,59 @@
1
+ {
2
+ "success": true,
3
+ "result": {
4
+ "organicResults": [
5
+ {
6
+ "position": 1,
7
+ "title": "How to Season a Wok",
8
+ "link": "https://www.wokandskillet.com/how-to-season-a-wok/",
9
+ "displayedLink": "https://www.wokandskillet.com › how-to-season-a-wok",
10
+ "snippet": "Wash the wok, heat it until the water evaporates, then coat it with a thin layer of oil.",
11
+ "page": 1
12
+ },
13
+ {
14
+ "position": 2,
15
+ "title": "How to Season a Wok - The Woks of Life",
16
+ "link": "https://thewoksoflife.com/how-to-season-a-wok/",
17
+ "displayedLink": "https://thewoksoflife.com › how-to-season-a-wok",
18
+ "snippet": "Seasoning a carbon steel wok gives it a natural nonstick surface.",
19
+ "page": 1
20
+ },
21
+ {
22
+ "position": 3,
23
+ "title": "3 Ways to Season a Wok - wikiHow",
24
+ "link": "https://www.wikihow.com/Season-a-Wok",
25
+ "displayedLink": "https://www.wikihow.com › Season-a-Wok",
26
+ "snippet": "Heat the wok over high heat, add oil and swirl it to coat the inside evenly.",
27
+ "page": 1,
28
+ "date": "Apr 5, 2024"
29
+ }
30
+ ],
31
+ "aioverview": {
32
+ "text": "Seasoning a carbon steel or cast iron wok means heating a thin layer of oil until it polymerizes.",
33
+ "sources": [
34
+ {
35
+ "position": 1,
36
+ "label": "Wok & Skillet",
37
+ "url": "https://www.wokandskillet.com/how-to-season-a-wok/",
38
+ "description": null
39
+ },
40
+ {
41
+ "position": 2,
42
+ "label": "Umamicart",
43
+ "url": "https://www.umamicart.com/how-to-season-a-wok/",
44
+ "description": null
45
+ }
46
+ ],
47
+ "citationPills": [
48
+ {
49
+ "position": 1,
50
+ "label": "Wok & Skillet",
51
+ "url": "https://www.wokandskillet.com/how-to-season-a-wok/",
52
+ "domain": "wokandskillet.com",
53
+ "citationPillId": 1
54
+ }
55
+ ],
56
+ "markdown": "Seasoning a carbon steel or cast iron wok means **heating a thin layer of oil until it polymerizes**.[Wok & Skillet](https://www.wokandskillet.com/how-to-season-a-wok/)"
57
+ }
58
+ }
59
+ }
@@ -124,6 +124,27 @@ def test_websearch_top_k_domains_and_params(api):
124
124
  assert len(result["links"]) == 2
125
125
 
126
126
 
127
+ def test_websearch_bing_request_and_ai_overview(api):
128
+ payload = _load("bing.json")
129
+ api.respond(httpx.Response(200, json=payload))
130
+ result = CloroWebSearch(engine="bing", country="GB", include_ai_overview=True).run(query="season a wok")
131
+ request = api.calls[0]
132
+ assert str(request.url) == "https://api.cloro.dev/v1/monitor/bing"
133
+ assert _body(request) == {"query": "season a wok", "country": "GB", "include": {"aioverview": True}}
134
+ overview = result["documents"][0]
135
+ assert overview.content == payload["result"]["aioverview"]["text"]
136
+ assert overview.meta["title"] == "Bing AI Overview"
137
+ assert overview.meta["source"] == "bing_ai_overview"
138
+ assert overview.meta["links"] == [s["url"] for s in payload["result"]["aioverview"]["sources"]]
139
+ assert result["links"] == [r["link"] for r in payload["result"]["organicResults"]]
140
+ assert len(result["documents"]) == 4
141
+
142
+
143
+ def test_websearch_rejects_other_engines():
144
+ with pytest.raises(ValueError, match="engine"):
145
+ CloroWebSearch(engine="other") # type: ignore[arg-type]
146
+
147
+
127
148
  def test_websearch_run_async(api):
128
149
  api.respond(httpx.Response(200, json=_load("google_aioverview.json")))
129
150
  result = asyncio.run(CloroWebSearch().run_async(query="q"))
@@ -241,6 +262,7 @@ def test_error_message_has_no_key(api):
241
262
  "component",
242
263
  [
243
264
  CloroWebSearch(top_k=3, country="GB", include_ai_overview=True, search_params={"device": "mobile"}),
265
+ CloroWebSearch(engine="bing", country="DE"),
244
266
  CloroAIAnswer(engine="gemini", country="FR"),
245
267
  ],
246
268
  )
File without changes