pytrends-next 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pytrends_next-0.1.0/LICENSE +21 -0
- pytrends_next-0.1.0/MANIFEST.in +1 -0
- pytrends_next-0.1.0/PKG-INFO +137 -0
- pytrends_next-0.1.0/README.md +113 -0
- pytrends_next-0.1.0/pyproject.toml +32 -0
- pytrends_next-0.1.0/setup.cfg +4 -0
- pytrends_next-0.1.0/src/pytrends_next/__init__.py +6 -0
- pytrends_next-0.1.0/src/pytrends_next/exceptions.py +17 -0
- pytrends_next-0.1.0/src/pytrends_next/hosted.py +48 -0
- pytrends_next-0.1.0/src/pytrends_next/request.py +526 -0
- pytrends_next-0.1.0/src/pytrends_next.egg-info/PKG-INFO +137 -0
- pytrends_next-0.1.0/src/pytrends_next.egg-info/SOURCES.txt +13 -0
- pytrends_next-0.1.0/src/pytrends_next.egg-info/dependency_links.txt +1 -0
- pytrends_next-0.1.0/src/pytrends_next.egg-info/requires.txt +3 -0
- pytrends_next-0.1.0/src/pytrends_next.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Ctrio Lab
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
prune tests
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pytrends-next
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Maintained drop-in replacement for pytrends (archived): Google Trends with the TrendReq API, standard library only, optional hosted fallback on HTTP 429.
|
|
5
|
+
Author-email: Ctrio Lab <myhouse9600@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/ctriolab/google-trends-python
|
|
8
|
+
Project-URL: Issues, https://github.com/ctriolab/google-trends-python/issues
|
|
9
|
+
Project-URL: Hosted fallback, https://apify.com/ctriolab/google-trends-scraper
|
|
10
|
+
Keywords: google trends,pytrends,trends,seo,keyword research,429
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Topic :: Internet
|
|
17
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
18
|
+
Requires-Python: >=3.9
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Provides-Extra: pandas
|
|
22
|
+
Requires-Dist: pandas>=1.3; extra == "pandas"
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
# pytrends-next
|
|
26
|
+
|
|
27
|
+
A maintained drop-in replacement for [pytrends](https://github.com/GeneralMills/pytrends), which was archived
|
|
28
|
+
(read-only) in April 2025. Same `TrendReq` API, same DataFrames, talks to Google Trends directly.
|
|
29
|
+
|
|
30
|
+
- Standard library only. pandas is optional: with pandas you get the same DataFrames as pytrends,
|
|
31
|
+
without it you get lists of dicts.
|
|
32
|
+
- Python 3.9+.
|
|
33
|
+
- Built-in handling of HTTP 429: new cookie session and exponential backoff with jitter.
|
|
34
|
+
- Optional, paid, off by default: route a call through a hosted scraper when Google keeps answering 429.
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
pip install "pytrends-next[pandas]" # or: pip install pytrends-next (no pandas)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
```python
|
|
41
|
+
from pytrends_next.request import TrendReq # was: from pytrends.request import TrendReq
|
|
42
|
+
|
|
43
|
+
pytrends = TrendReq(hl="en-US", tz=360)
|
|
44
|
+
pytrends.build_payload(["coffee", "tea"], timeframe="today 3-m", geo="US")
|
|
45
|
+
print(pytrends.interest_over_time().head(3))
|
|
46
|
+
# coffee tea isPartial
|
|
47
|
+
# date
|
|
48
|
+
# 2026-07-23 75 35 False
|
|
49
|
+
# 2026-07-24 81 38 False
|
|
50
|
+
# 2026-07-25 94 40 False
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
For most code the migration is one search and replace: `pytrends.` to `pytrends_next.` in imports.
|
|
54
|
+
`pytrends_next.exceptions` has the same `ResponseError` and `TooManyRequestsError`.
|
|
55
|
+
|
|
56
|
+
## Moving from pytrends
|
|
57
|
+
|
|
58
|
+
| pytrends 4.9.2 | pytrends-next | Notes |
|
|
59
|
+
|---|---|---|
|
|
60
|
+
| `TrendReq(hl, tz, geo, timeout, proxies, retries, backoff_factor, requests_args)` | same | `timeout` tuple: the larger value is used (urllib has one timeout). `requests_args`: `headers`, `proxies` and `verify` are used, other keys are ignored with a warning. `retries` defaults to 3 (pytrends: 0). |
|
|
61
|
+
| `build_payload(kw_list, cat, timeframe, geo, gprop)` | same | Lists of geos and timeframes work as in pytrends. |
|
|
62
|
+
| `interest_over_time()` | same | Index `date`, one int column per keyword, `isPartial`. MultiIndex columns for several geos. |
|
|
63
|
+
| `interest_by_region(resolution, inc_low_vol, inc_geo_code)` | same | pytrends' check meant to apply `resolution` worldwide and for sub-levels never matched; here it does. |
|
|
64
|
+
| `related_queries()` | same | `{kw: {"top": DataFrame[query, value] or None, "rising": ...}}` |
|
|
65
|
+
| `related_topics()` | same | Google now returns topics only for single-term requests, so each term is asked alone. Often empty, see Limits. |
|
|
66
|
+
| `trending_searches(pn="united_states")` | same | The old endpoint is gone. Now served from Google's "Trending now" list. `pn` takes the pytrends country name or an ISO code (`"US"`). |
|
|
67
|
+
| `today_searches(pn="US")` | same | Series of titles, past 24 hours. |
|
|
68
|
+
| `realtime_trending_searches(pn, cat, count)` | same | Trends started in the past 4 hours, columns `entityNames`, `title`. `cat` is ignored. |
|
|
69
|
+
| `suggestions(keyword)` | same | |
|
|
70
|
+
| `categories()` | same | |
|
|
71
|
+
| `top_charts()` | raises `NotImplementedError` | Google removed the endpoint. |
|
|
72
|
+
| `multirange_interest_over_time()` | raises `NotImplementedError` | Not ported yet. |
|
|
73
|
+
| (none) | `trending_now(geo="US", hours=24)` | New. Rows with `title`, `volume`, `increase_percentage`, `started`, `ended`, `active`, `categories`, `related_queries`. `hours`: 4, 24, 48 or 168. Falls back to the Trending RSS feed if the main endpoint changes. |
|
|
74
|
+
|
|
75
|
+
## HTTP 429 (TooManyRequestsError)
|
|
76
|
+
|
|
77
|
+
Google Trends has no public API. It rate-limits by IP and by session, and a laptop or a single
|
|
78
|
+
server often gets HTTP 429 after a few dozen requests. pytrends raised `TooManyRequestsError` at once.
|
|
79
|
+
|
|
80
|
+
pytrends-next does what our hosted scraper does on every request:
|
|
81
|
+
|
|
82
|
+
1. Visits trends.google.com first to get the `NID` cookie a browser visitor has (cookieless clients are limited sooner).
|
|
83
|
+
2. On 429, 403 or 5xx: drops the session, takes the next proxy from `proxies=[...]` if you gave any,
|
|
84
|
+
waits about 0.75 s, 1.5 s, 3 s ... (capped at 30 s, with jitter) and tries again, `retries` times.
|
|
85
|
+
3. 400 and 404 are not retried (bad geo, category or timeframe).
|
|
86
|
+
|
|
87
|
+
If you still get `TooManyRequestsError`: send fewer requests (batch up to 5 terms per `build_payload`),
|
|
88
|
+
wait a few minutes, use your own proxies, or use the hosted fallback below.
|
|
89
|
+
|
|
90
|
+
## Hosted fallback (optional, paid)
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
pytrends = TrendReq(fallback="apify", apify_token="apify_api_...") # or set APIFY_TOKEN
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
When Google still answers 429 after the local retries, the same call is sent to the
|
|
97
|
+
[Google Trends Scraper Actor on Apify](https://apify.com/ctriolab/google-trends-scraper?utm_source=github&utm_medium=readme&utm_campaign=pytrends-next),
|
|
98
|
+
which rotates IPs on Apify's proxy network. The result comes back in the same shape.
|
|
99
|
+
|
|
100
|
+
- **It costs money.** The Actor is billed per result to your own Apify account (about $1.50 per 1,000
|
|
101
|
+
results; one result is one term's timeline, one term's region list, one term's related list or one trending topic).
|
|
102
|
+
Apify's free plan includes a small monthly credit.
|
|
103
|
+
- It is never on by default. Setting `APIFY_TOKEN` alone does nothing; you must pass `fallback="apify"`.
|
|
104
|
+
- A warning is logged (logger `pytrends_next`) every time a call goes to the Actor.
|
|
105
|
+
- If `build_payload` itself gets 429, every method on that payload goes to the Actor.
|
|
106
|
+
- Supported: `interest_over_time`, `interest_by_region`, `related_queries`, `related_topics`, `trending_now`
|
|
107
|
+
and the pytrends trending methods. Not supported: several geos or timeframes in one payload,
|
|
108
|
+
commas inside a keyword, `suggestions`, `categories`.
|
|
109
|
+
- Difference: in hosted mode each term's region map is scaled on its own (locally, the terms share one scale).
|
|
110
|
+
- Timeframes: the named ranges (`now 1-H` to `today 5-y`, `all`) and `YYYY-MM-DD YYYY-MM-DD`. Others are rejected before a run starts.
|
|
111
|
+
|
|
112
|
+
Disclosure: Ctrio Lab maintains this library and also builds and sells that Actor. The library is MIT
|
|
113
|
+
licensed and fully works without it.
|
|
114
|
+
|
|
115
|
+
## Limits
|
|
116
|
+
|
|
117
|
+
- Values are relative (0 to 100) within one payload, as on trends.google.com. Separate calls are not on the same scale.
|
|
118
|
+
- Related topics are usually empty: Google hides them from clients it classifies as automated. Empty lists come back as `None`, like pytrends.
|
|
119
|
+
- Very small search volumes come back as all zeros (Google's own behaviour).
|
|
120
|
+
- This is an unofficial client for undocumented endpoints. Google can change them at any time.
|
|
121
|
+
|
|
122
|
+
## Legacy single-file client
|
|
123
|
+
|
|
124
|
+
`gtrends.py` (the earlier version of this repo) is still here and unchanged. It calls the hosted Actor for
|
|
125
|
+
every request. New code should use `pytrends_next.request.TrendReq`: `gtrends.interest_over_time_df(terms, ...)`
|
|
126
|
+
becomes `build_payload(terms, ...)` plus `interest_over_time()`.
|
|
127
|
+
|
|
128
|
+
## Development
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
python -m unittest discover -s tests # no network, recorded responses in tests/fixtures
|
|
132
|
+
python tests/live_smoke.py # about 12 live requests to Google
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
## License
|
|
136
|
+
|
|
137
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# pytrends-next
|
|
2
|
+
|
|
3
|
+
A maintained drop-in replacement for [pytrends](https://github.com/GeneralMills/pytrends), which was archived
|
|
4
|
+
(read-only) in April 2025. Same `TrendReq` API, same DataFrames, talks to Google Trends directly.
|
|
5
|
+
|
|
6
|
+
- Standard library only. pandas is optional: with pandas you get the same DataFrames as pytrends,
|
|
7
|
+
without it you get lists of dicts.
|
|
8
|
+
- Python 3.9+.
|
|
9
|
+
- Built-in handling of HTTP 429: new cookie session and exponential backoff with jitter.
|
|
10
|
+
- Optional, paid, off by default: route a call through a hosted scraper when Google keeps answering 429.
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
pip install "pytrends-next[pandas]" # or: pip install pytrends-next (no pandas)
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
```python
|
|
17
|
+
from pytrends_next.request import TrendReq # was: from pytrends.request import TrendReq
|
|
18
|
+
|
|
19
|
+
pytrends = TrendReq(hl="en-US", tz=360)
|
|
20
|
+
pytrends.build_payload(["coffee", "tea"], timeframe="today 3-m", geo="US")
|
|
21
|
+
print(pytrends.interest_over_time().head(3))
|
|
22
|
+
# coffee tea isPartial
|
|
23
|
+
# date
|
|
24
|
+
# 2026-07-23 75 35 False
|
|
25
|
+
# 2026-07-24 81 38 False
|
|
26
|
+
# 2026-07-25 94 40 False
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
For most code the migration is one search and replace: `pytrends.` to `pytrends_next.` in imports.
|
|
30
|
+
`pytrends_next.exceptions` has the same `ResponseError` and `TooManyRequestsError`.
|
|
31
|
+
|
|
32
|
+
## Moving from pytrends
|
|
33
|
+
|
|
34
|
+
| pytrends 4.9.2 | pytrends-next | Notes |
|
|
35
|
+
|---|---|---|
|
|
36
|
+
| `TrendReq(hl, tz, geo, timeout, proxies, retries, backoff_factor, requests_args)` | same | `timeout` tuple: the larger value is used (urllib has one timeout). `requests_args`: `headers`, `proxies` and `verify` are used, other keys are ignored with a warning. `retries` defaults to 3 (pytrends: 0). |
|
|
37
|
+
| `build_payload(kw_list, cat, timeframe, geo, gprop)` | same | Lists of geos and timeframes work as in pytrends. |
|
|
38
|
+
| `interest_over_time()` | same | Index `date`, one int column per keyword, `isPartial`. MultiIndex columns for several geos. |
|
|
39
|
+
| `interest_by_region(resolution, inc_low_vol, inc_geo_code)` | same | pytrends' check meant to apply `resolution` worldwide and for sub-levels never matched; here it does. |
|
|
40
|
+
| `related_queries()` | same | `{kw: {"top": DataFrame[query, value] or None, "rising": ...}}` |
|
|
41
|
+
| `related_topics()` | same | Google now returns topics only for single-term requests, so each term is asked alone. Often empty, see Limits. |
|
|
42
|
+
| `trending_searches(pn="united_states")` | same | The old endpoint is gone. Now served from Google's "Trending now" list. `pn` takes the pytrends country name or an ISO code (`"US"`). |
|
|
43
|
+
| `today_searches(pn="US")` | same | Series of titles, past 24 hours. |
|
|
44
|
+
| `realtime_trending_searches(pn, cat, count)` | same | Trends started in the past 4 hours, columns `entityNames`, `title`. `cat` is ignored. |
|
|
45
|
+
| `suggestions(keyword)` | same | |
|
|
46
|
+
| `categories()` | same | |
|
|
47
|
+
| `top_charts()` | raises `NotImplementedError` | Google removed the endpoint. |
|
|
48
|
+
| `multirange_interest_over_time()` | raises `NotImplementedError` | Not ported yet. |
|
|
49
|
+
| (none) | `trending_now(geo="US", hours=24)` | New. Rows with `title`, `volume`, `increase_percentage`, `started`, `ended`, `active`, `categories`, `related_queries`. `hours`: 4, 24, 48 or 168. Falls back to the Trending RSS feed if the main endpoint changes. |
|
|
50
|
+
|
|
51
|
+
## HTTP 429 (TooManyRequestsError)
|
|
52
|
+
|
|
53
|
+
Google Trends has no public API. It rate-limits by IP and by session, and a laptop or a single
|
|
54
|
+
server often gets HTTP 429 after a few dozen requests. pytrends raised `TooManyRequestsError` at once.
|
|
55
|
+
|
|
56
|
+
pytrends-next does what our hosted scraper does on every request:
|
|
57
|
+
|
|
58
|
+
1. Visits trends.google.com first to get the `NID` cookie a browser visitor has (cookieless clients are limited sooner).
|
|
59
|
+
2. On 429, 403 or 5xx: drops the session, takes the next proxy from `proxies=[...]` if you gave any,
|
|
60
|
+
waits about 0.75 s, 1.5 s, 3 s ... (capped at 30 s, with jitter) and tries again, `retries` times.
|
|
61
|
+
3. 400 and 404 are not retried (bad geo, category or timeframe).
|
|
62
|
+
|
|
63
|
+
If you still get `TooManyRequestsError`: send fewer requests (batch up to 5 terms per `build_payload`),
|
|
64
|
+
wait a few minutes, use your own proxies, or use the hosted fallback below.
|
|
65
|
+
|
|
66
|
+
## Hosted fallback (optional, paid)
|
|
67
|
+
|
|
68
|
+
```python
|
|
69
|
+
pytrends = TrendReq(fallback="apify", apify_token="apify_api_...") # or set APIFY_TOKEN
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
When Google still answers 429 after the local retries, the same call is sent to the
|
|
73
|
+
[Google Trends Scraper Actor on Apify](https://apify.com/ctriolab/google-trends-scraper?utm_source=github&utm_medium=readme&utm_campaign=pytrends-next),
|
|
74
|
+
which rotates IPs on Apify's proxy network. The result comes back in the same shape.
|
|
75
|
+
|
|
76
|
+
- **It costs money.** The Actor is billed per result to your own Apify account (about $1.50 per 1,000
|
|
77
|
+
results; one result is one term's timeline, one term's region list, one term's related list or one trending topic).
|
|
78
|
+
Apify's free plan includes a small monthly credit.
|
|
79
|
+
- It is never on by default. Setting `APIFY_TOKEN` alone does nothing; you must pass `fallback="apify"`.
|
|
80
|
+
- A warning is logged (logger `pytrends_next`) every time a call goes to the Actor.
|
|
81
|
+
- If `build_payload` itself gets 429, every method on that payload goes to the Actor.
|
|
82
|
+
- Supported: `interest_over_time`, `interest_by_region`, `related_queries`, `related_topics`, `trending_now`
|
|
83
|
+
and the pytrends trending methods. Not supported: several geos or timeframes in one payload,
|
|
84
|
+
commas inside a keyword, `suggestions`, `categories`.
|
|
85
|
+
- Difference: in hosted mode each term's region map is scaled on its own (locally, the terms share one scale).
|
|
86
|
+
- Timeframes: the named ranges (`now 1-H` to `today 5-y`, `all`) and `YYYY-MM-DD YYYY-MM-DD`. Others are rejected before a run starts.
|
|
87
|
+
|
|
88
|
+
Disclosure: Ctrio Lab maintains this library and also builds and sells that Actor. The library is MIT
|
|
89
|
+
licensed and fully works without it.
|
|
90
|
+
|
|
91
|
+
## Limits
|
|
92
|
+
|
|
93
|
+
- Values are relative (0 to 100) within one payload, as on trends.google.com. Separate calls are not on the same scale.
|
|
94
|
+
- Related topics are usually empty: Google hides them from clients it classifies as automated. Empty lists come back as `None`, like pytrends.
|
|
95
|
+
- Very small search volumes come back as all zeros (Google's own behaviour).
|
|
96
|
+
- This is an unofficial client for undocumented endpoints. Google can change them at any time.
|
|
97
|
+
|
|
98
|
+
## Legacy single-file client
|
|
99
|
+
|
|
100
|
+
`gtrends.py` (the earlier version of this repo) is still here and unchanged. It calls the hosted Actor for
|
|
101
|
+
every request. New code should use `pytrends_next.request.TrendReq`: `gtrends.interest_over_time_df(terms, ...)`
|
|
102
|
+
becomes `build_payload(terms, ...)` plus `interest_over_time()`.
|
|
103
|
+
|
|
104
|
+
## Development
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
python -m unittest discover -s tests # no network, recorded responses in tests/fixtures
|
|
108
|
+
python tests/live_smoke.py # about 12 live requests to Google
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## License
|
|
112
|
+
|
|
113
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=77"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "pytrends-next"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Maintained drop-in replacement for pytrends (archived): Google Trends with the TrendReq API, standard library only, optional hosted fallback on HTTP 429."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.9"
|
|
11
|
+
license = "MIT"
|
|
12
|
+
license-files = ["LICENSE"]
|
|
13
|
+
authors = [{ name = "Ctrio Lab", email = "myhouse9600@gmail.com" }]
|
|
14
|
+
keywords = ["google trends", "pytrends", "trends", "seo", "keyword research", "429"]
|
|
15
|
+
classifiers = [
|
|
16
|
+
"Development Status :: 4 - Beta",
|
|
17
|
+
"Intended Audience :: Developers",
|
|
18
|
+
"Intended Audience :: Science/Research",
|
|
19
|
+
"Operating System :: OS Independent",
|
|
20
|
+
"Programming Language :: Python :: 3",
|
|
21
|
+
"Topic :: Internet",
|
|
22
|
+
"Topic :: Scientific/Engineering :: Information Analysis",
|
|
23
|
+
]
|
|
24
|
+
dependencies = []
|
|
25
|
+
|
|
26
|
+
[project.optional-dependencies]
|
|
27
|
+
pandas = ["pandas>=1.3"]
|
|
28
|
+
|
|
29
|
+
[project.urls]
|
|
30
|
+
Homepage = "https://github.com/ctriolab/google-trends-python"
|
|
31
|
+
Issues = "https://github.com/ctriolab/google-trends-python/issues"
|
|
32
|
+
"Hosted fallback" = "https://apify.com/ctriolab/google-trends-scraper"
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
"""Maintained drop-in replacement for pytrends: `from pytrends_next.request import TrendReq`."""
|
|
2
|
+
from .exceptions import ResponseError, TooManyRequestsError
|
|
3
|
+
from .request import TrendReq
|
|
4
|
+
|
|
5
|
+
__version__ = "0.1.0"
|
|
6
|
+
__all__ = ["TrendReq", "ResponseError", "TooManyRequestsError"]
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
"""Same exception names as pytrends.exceptions, so existing `except` blocks keep working."""
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
class ResponseError(Exception):
|
|
5
|
+
"""Google (or the hosted fallback) answered with an error. `response` has status_code, text and url."""
|
|
6
|
+
|
|
7
|
+
def __init__(self, message, response=None):
|
|
8
|
+
super().__init__(message)
|
|
9
|
+
self.response = response
|
|
10
|
+
|
|
11
|
+
@classmethod
|
|
12
|
+
def from_response(cls, response):
|
|
13
|
+
return cls(f"The request failed: Google returned a response with code {response.status_code}", response)
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
class TooManyRequestsError(ResponseError):
|
|
17
|
+
"""HTTP 429: Google rate-limited this IP or session."""
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
"""Opt-in hosted fallback: the Ctrio Lab "Google Trends Scraper" Actor on Apify.
|
|
2
|
+
|
|
3
|
+
This is a PAID service billed per result to the caller's own Apify account (APIFY_TOKEN).
|
|
4
|
+
It is only used when TrendReq(fallback="apify") is set and Google answered HTTP 429.
|
|
5
|
+
Disclosure: Ctrio Lab builds both this library and the Actor.
|
|
6
|
+
"""
|
|
7
|
+
from __future__ import annotations
|
|
8
|
+
|
|
9
|
+
import json
|
|
10
|
+
import re
|
|
11
|
+
import urllib.error
|
|
12
|
+
import urllib.request
|
|
13
|
+
from types import SimpleNamespace
|
|
14
|
+
|
|
15
|
+
from .exceptions import ResponseError
|
|
16
|
+
|
|
17
|
+
ACTOR = "ctriolab~google-trends-scraper"
|
|
18
|
+
API = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items"
|
|
19
|
+
ACTOR_URL = "https://apify.com/ctriolab/google-trends-scraper"
|
|
20
|
+
TIMEFRAMES = {"now 1-H", "now 4-H", "now 1-d", "now 7-d", "today 1-m", "today 3-m", "today 12-m", "today 5-y", "all"}
|
|
21
|
+
_CUSTOM = re.compile(r"^\d{4}-\d{2}-\d{2}(T\d{2})? \d{4}-\d{2}-\d{2}(T\d{2})?$")
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
def timeframe(tf: str) -> dict:
|
|
25
|
+
"""pytrends timeframe -> Actor input fields. Rejects what the Actor would reject, before a paid run."""
|
|
26
|
+
if tf in TIMEFRAMES:
|
|
27
|
+
return {"timeframe": tf}
|
|
28
|
+
if isinstance(tf, str) and _CUSTOM.match(tf.strip()):
|
|
29
|
+
return {"timeframe": "custom", "customTimeRange": tf.strip()}
|
|
30
|
+
raise ValueError(f'The hosted fallback does not support timeframe "{tf}". '
|
|
31
|
+
f'Use one of {sorted(TIMEFRAMES)} or "YYYY-MM-DD YYYY-MM-DD".')
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def run(actor_input: dict, token: str, timeout: int = 300) -> list:
|
|
35
|
+
"""Run the Actor synchronously and return its dataset items."""
|
|
36
|
+
req = urllib.request.Request(
|
|
37
|
+
f"{API}?timeout={timeout}",
|
|
38
|
+
data=json.dumps(actor_input).encode("utf-8"),
|
|
39
|
+
headers={"Content-Type": "application/json", "Authorization": f"Bearer {token}"},
|
|
40
|
+
method="POST",
|
|
41
|
+
)
|
|
42
|
+
try:
|
|
43
|
+
with urllib.request.urlopen(req, timeout=timeout + 30) as res:
|
|
44
|
+
return json.loads(res.read().decode("utf-8"))
|
|
45
|
+
except urllib.error.HTTPError as e:
|
|
46
|
+
body = e.read().decode("utf-8", "replace")[:500]
|
|
47
|
+
raise ResponseError(f"Apify API returned HTTP {e.code}: {body}",
|
|
48
|
+
SimpleNamespace(status_code=e.code, text=body, url=API)) from None
|
|
@@ -0,0 +1,526 @@
|
|
|
1
|
+
"""TrendReq: the pytrends API, talking to Google Trends directly with the standard library.
|
|
2
|
+
|
|
3
|
+
Differences from pytrends 4.9.2 are listed in README.md ("Moving from pytrends").
|
|
4
|
+
"""
|
|
5
|
+
from __future__ import annotations
|
|
6
|
+
|
|
7
|
+
import http.cookiejar
|
|
8
|
+
import json
|
|
9
|
+
import logging
|
|
10
|
+
import os
|
|
11
|
+
import random
|
|
12
|
+
import ssl
|
|
13
|
+
import time
|
|
14
|
+
import urllib.error
|
|
15
|
+
import urllib.parse
|
|
16
|
+
import urllib.request
|
|
17
|
+
import warnings
|
|
18
|
+
import xml.etree.ElementTree as ET
|
|
19
|
+
from datetime import datetime, timezone
|
|
20
|
+
from email.utils import parsedate_to_datetime
|
|
21
|
+
from itertools import product
|
|
22
|
+
from types import SimpleNamespace
|
|
23
|
+
|
|
24
|
+
from . import hosted
|
|
25
|
+
from .exceptions import ResponseError, TooManyRequestsError
|
|
26
|
+
|
|
27
|
+
try:
|
|
28
|
+
import pandas as pd
|
|
29
|
+
except ImportError: # pandas is optional: without it, methods return lists of dicts
|
|
30
|
+
pd = None
|
|
31
|
+
|
|
32
|
+
log = logging.getLogger("pytrends_next")
|
|
33
|
+
|
|
34
|
+
BASE = "https://trends.google.com"
|
|
35
|
+
BASE_TRENDS_URL = f"{BASE}/trends"
|
|
36
|
+
USER_AGENT = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
|
37
|
+
"(KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36")
|
|
38
|
+
|
|
39
|
+
TRENDING_CATEGORIES = {
|
|
40
|
+
1: "Autos and Vehicles", 2: "Beauty and Fashion", 3: "Business and Finance", 4: "Entertainment",
|
|
41
|
+
5: "Food and Drink", 6: "Games", 7: "Health", 8: "Hobbies and Leisure", 9: "Jobs and Education",
|
|
42
|
+
10: "Law and Government", 11: "Other", 13: "Pets and Animals", 14: "Politics", 15: "Science",
|
|
43
|
+
16: "Shopping", 17: "Sports", 18: "Technology", 19: "Travel and Transportation", 20: "Climate",
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
# pytrends trending_searches(pn=...) used country names; Trending now uses ISO codes.
|
|
47
|
+
PN_GEO = {
|
|
48
|
+
"argentina": "AR", "australia": "AU", "austria": "AT", "belgium": "BE", "brazil": "BR", "canada": "CA",
|
|
49
|
+
"chile": "CL", "colombia": "CO", "czech_republic": "CZ", "denmark": "DK", "egypt": "EG", "finland": "FI",
|
|
50
|
+
"france": "FR", "germany": "DE", "greece": "GR", "hong_kong": "HK", "hungary": "HU", "india": "IN",
|
|
51
|
+
"indonesia": "ID", "ireland": "IE", "israel": "IL", "italy": "IT", "japan": "JP", "kenya": "KE",
|
|
52
|
+
"malaysia": "MY", "mexico": "MX", "netherlands": "NL", "new_zealand": "NZ", "nigeria": "NG", "norway": "NO",
|
|
53
|
+
"peru": "PE", "philippines": "PH", "poland": "PL", "portugal": "PT", "romania": "RO", "russia": "RU",
|
|
54
|
+
"saudi_arabia": "SA", "singapore": "SG", "south_africa": "ZA", "south_korea": "KR", "spain": "ES",
|
|
55
|
+
"sweden": "SE", "switzerland": "CH", "taiwan": "TW", "thailand": "TH", "turkey": "TR", "ukraine": "UA",
|
|
56
|
+
"united_kingdom": "GB", "united_states": "US", "vietnam": "VN",
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
def _json(text):
|
|
61
|
+
"""Google prefixes JSON with )]}' to stop JSON hijacking; drop everything before the first brace."""
|
|
62
|
+
i = text.find("{")
|
|
63
|
+
if i < 0:
|
|
64
|
+
raise ValueError("no JSON in response")
|
|
65
|
+
return json.loads(text[i:])
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
def _ts(seconds):
|
|
69
|
+
return datetime.fromtimestamp(int(seconds), timezone.utc).replace(tzinfo=None)
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
def _iso(s):
|
|
73
|
+
return datetime.fromisoformat(s.replace("Z", "+00:00")).replace(tzinfo=None) if s else None
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
def _frame(rows, index=None):
|
|
77
|
+
"""DataFrame like pytrends when pandas is installed, else the rows themselves."""
|
|
78
|
+
if pd is None:
|
|
79
|
+
return rows
|
|
80
|
+
df = pd.DataFrame(rows)
|
|
81
|
+
if index and not df.empty:
|
|
82
|
+
df = df.set_index(index).sort_index()
|
|
83
|
+
return df
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
def _flat(d, prefix=""):
|
|
87
|
+
"""pd.json_normalize(sep='_') for one dict, without pandas."""
|
|
88
|
+
out = {}
|
|
89
|
+
for k, v in d.items():
|
|
90
|
+
if isinstance(v, dict):
|
|
91
|
+
out.update(_flat(v, f"{prefix}{k}_"))
|
|
92
|
+
else:
|
|
93
|
+
out[f"{prefix}{k}"] = v
|
|
94
|
+
return out
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
def _ranked(data, i, flat):
|
|
98
|
+
"""rankedList[i] (0 = top, 1 = rising) as pytrends returns it: None when Google sent no list."""
|
|
99
|
+
try:
|
|
100
|
+
rows = data["rankedList"][i]["rankedKeyword"]
|
|
101
|
+
except (KeyError, IndexError, TypeError):
|
|
102
|
+
return None
|
|
103
|
+
if flat:
|
|
104
|
+
return _frame([_flat(r) for r in rows])
|
|
105
|
+
if not rows:
|
|
106
|
+
return None # pytrends: empty DataFrame[['query','value']] raised KeyError -> None
|
|
107
|
+
return _frame([{"query": r.get("query"), "value": r.get("value")} for r in rows])
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
class TrendReq:
|
|
111
|
+
"""Google Trends client with the pytrends TrendReq interface.
|
|
112
|
+
|
|
113
|
+
fallback="apify": when Google answers HTTP 429 after `retries`, run the same request through the
|
|
114
|
+
hosted Google Trends Scraper Actor on Apify. Paid per result on your own Apify account
|
|
115
|
+
(apify_token=... or the APIFY_TOKEN environment variable). Off by default.
|
|
116
|
+
"""
|
|
117
|
+
|
|
118
|
+
GENERAL_URL = f"{BASE_TRENDS_URL}/api/explore"
|
|
119
|
+
INTEREST_OVER_TIME_URL = f"{BASE_TRENDS_URL}/api/widgetdata/multiline"
|
|
120
|
+
INTEREST_BY_REGION_URL = f"{BASE_TRENDS_URL}/api/widgetdata/comparedgeo"
|
|
121
|
+
RELATED_QUERIES_URL = f"{BASE_TRENDS_URL}/api/widgetdata/relatedsearches"
|
|
122
|
+
SUGGESTIONS_URL = f"{BASE_TRENDS_URL}/api/autocomplete/"
|
|
123
|
+
CATEGORIES_URL = f"{BASE_TRENDS_URL}/api/explore/pickers/category"
|
|
124
|
+
TRENDING_NOW_URL = f"{BASE}/_/TrendsUi/data/batchexecute"
|
|
125
|
+
TRENDING_RSS_URL = f"{BASE}/trending/rss"
|
|
126
|
+
|
|
127
|
+
def __init__(self, hl="en-US", tz=360, geo="", timeout=(2, 5), proxies="", retries=3, backoff_factor=0,
|
|
128
|
+
requests_args=None, fallback=None, apify_token=None):
|
|
129
|
+
if fallback not in (None, "apify"):
|
|
130
|
+
raise ValueError('fallback must be None or "apify"')
|
|
131
|
+
self.hl = hl
|
|
132
|
+
self.tz = tz
|
|
133
|
+
self.geo = geo
|
|
134
|
+
self.kw_list = []
|
|
135
|
+
# urllib has one timeout; pytrends passed (connect, read) to requests.
|
|
136
|
+
self.timeout = max(timeout) if isinstance(timeout, (tuple, list)) else timeout
|
|
137
|
+
self.proxies = list(proxies) if proxies else []
|
|
138
|
+
self.proxy_index = 0
|
|
139
|
+
self.retries = retries
|
|
140
|
+
self.backoff_factor = backoff_factor
|
|
141
|
+
args = dict(requests_args or {})
|
|
142
|
+
self.headers = {"accept-language": hl, "user-agent": USER_AGENT}
|
|
143
|
+
self.headers.update(args.pop("headers", {}))
|
|
144
|
+
self._proxy_map = args.pop("proxies", None)
|
|
145
|
+
self._ssl = None if args.pop("verify", True) else ssl._create_unverified_context()
|
|
146
|
+
if args:
|
|
147
|
+
warnings.warn(f"pytrends_next ignores these requests_args: {sorted(args)}")
|
|
148
|
+
self.fallback = fallback
|
|
149
|
+
self.apify_token = apify_token or os.environ.get("APIFY_TOKEN")
|
|
150
|
+
if fallback and not self.apify_token:
|
|
151
|
+
raise ValueError('fallback="apify" needs apify_token=... or the APIFY_TOKEN environment variable')
|
|
152
|
+
self.cookies = {}
|
|
153
|
+
self._opener = None
|
|
154
|
+
self._hosted = False
|
|
155
|
+
self._query = {}
|
|
156
|
+
self.token_payload = {}
|
|
157
|
+
self.interest_over_time_widget = {}
|
|
158
|
+
self.interest_by_region_widget = {}
|
|
159
|
+
self.related_topics_widget_list = []
|
|
160
|
+
self.related_queries_widget_list = []
|
|
161
|
+
|
|
162
|
+
# HTTP -------------------------------------------------------------------------------------------
|
|
163
|
+
|
|
164
|
+
def _fetch(self, url, data=None, headers=None):
|
|
165
|
+
"""One HTTP request, returns (status, text). Unit tests replace this method."""
|
|
166
|
+
req = urllib.request.Request(url, data=data, headers={**self.headers, **(headers or {})})
|
|
167
|
+
try:
|
|
168
|
+
with self._opener.open(req, timeout=self.timeout) as res:
|
|
169
|
+
return res.status, res.read().decode("utf-8", "replace")
|
|
170
|
+
except urllib.error.HTTPError as e:
|
|
171
|
+
return e.code, e.read().decode("utf-8", "replace")
|
|
172
|
+
|
|
173
|
+
def _new_session(self):
|
|
174
|
+
"""Fresh cookie jar (and the next proxy), then fetch the NID cookie a browser visitor would have.
|
|
175
|
+
Google rate-limits cookieless clients much sooner."""
|
|
176
|
+
jar = http.cookiejar.CookieJar()
|
|
177
|
+
handlers = [urllib.request.HTTPCookieProcessor(jar), urllib.request.HTTPSHandler(context=self._ssl)]
|
|
178
|
+
proxy = self._proxy_map or ({"https": self.proxies[self.proxy_index]} if self.proxies else None)
|
|
179
|
+
if proxy:
|
|
180
|
+
handlers.append(urllib.request.ProxyHandler(proxy))
|
|
181
|
+
if self.proxies:
|
|
182
|
+
self.proxy_index = (self.proxy_index + 1) % len(self.proxies)
|
|
183
|
+
self._opener = urllib.request.build_opener(*handlers)
|
|
184
|
+
for url in (f"{BASE_TRENDS_URL}/explore?geo={self.hl[-2:]}&hl={self.hl}", "https://www.google.com/?hl=en"):
|
|
185
|
+
try:
|
|
186
|
+
self._fetch(url)
|
|
187
|
+
except OSError:
|
|
188
|
+
pass # best effort
|
|
189
|
+
if any(c.name == "NID" for c in jar):
|
|
190
|
+
break
|
|
191
|
+
self.cookies = {c.name: c.value for c in jar if c.name == "NID"}
|
|
192
|
+
|
|
193
|
+
def _backoff(self, attempt):
|
|
194
|
+
base = self.backoff_factor * 2 ** attempt if self.backoff_factor else min(30.0, 1.5 * 2 ** attempt)
|
|
195
|
+
return base / 2 + random.random() * base / 2
|
|
196
|
+
|
|
197
|
+
def _get_data(self, url, params=None, parse=_json, data=None, headers=None):
|
|
198
|
+
"""Request with retries. 429, 403 and 5xx: new session, exponential backoff with jitter.
|
|
199
|
+
400 and 404 mean bad input and are raised at once."""
|
|
200
|
+
if params:
|
|
201
|
+
url = f"{url}?{urllib.parse.urlencode(params)}"
|
|
202
|
+
err, parse_fails = None, 0
|
|
203
|
+
for attempt in range(self.retries + 1):
|
|
204
|
+
if self._opener is None:
|
|
205
|
+
self._new_session()
|
|
206
|
+
try:
|
|
207
|
+
status, text = self._fetch(url, data=data, headers=headers)
|
|
208
|
+
except OSError as e:
|
|
209
|
+
err = ResponseError(f"Network error: {e}")
|
|
210
|
+
else:
|
|
211
|
+
resp = SimpleNamespace(status_code=status, text=text[:2000], url=url)
|
|
212
|
+
if status == 200:
|
|
213
|
+
try:
|
|
214
|
+
return parse(text)
|
|
215
|
+
except (ValueError, KeyError, IndexError, TypeError) as e:
|
|
216
|
+
# A 200 without data is usually deterministic (bad geo, unsupported country).
|
|
217
|
+
err = ResponseError(f"Unexpected response from Google: {e}", resp)
|
|
218
|
+
parse_fails += 1
|
|
219
|
+
if parse_fails >= 3:
|
|
220
|
+
raise err from None
|
|
221
|
+
elif status in (400, 404):
|
|
222
|
+
raise ResponseError.from_response(resp)
|
|
223
|
+
elif status == 429:
|
|
224
|
+
err = TooManyRequestsError.from_response(resp)
|
|
225
|
+
else:
|
|
226
|
+
err = ResponseError.from_response(resp)
|
|
227
|
+
if attempt < self.retries:
|
|
228
|
+
wait = self._backoff(attempt)
|
|
229
|
+
log.debug("%s; new session, retry in %.1fs", err, wait)
|
|
230
|
+
self._opener = None
|
|
231
|
+
time.sleep(wait)
|
|
232
|
+
raise err
|
|
233
|
+
|
|
234
|
+
def _widget(self, url, widget):
|
|
235
|
+
return self._get_data(url, {"hl": self.hl, "tz": self.tz, "req": json.dumps(widget["request"]),
|
|
236
|
+
"token": widget["token"]})["default"]
|
|
237
|
+
|
|
238
|
+
def _either(self, local, remote):
|
|
239
|
+
"""Run locally; on HTTP 429 use the hosted Actor if fallback="apify"."""
|
|
240
|
+
if not self._hosted:
|
|
241
|
+
try:
|
|
242
|
+
return local()
|
|
243
|
+
except TooManyRequestsError:
|
|
244
|
+
if not self.fallback:
|
|
245
|
+
raise
|
|
246
|
+
log.warning("Google rate-limited this client (HTTP 429). Using the hosted Apify Actor %s "
|
|
247
|
+
"(billed per result to your Apify account).", hosted.ACTOR_URL)
|
|
248
|
+
return remote()
|
|
249
|
+
|
|
250
|
+
# Explore ----------------------------------------------------------------------------------------
|
|
251
|
+
|
|
252
|
+
def build_payload(self, kw_list, cat=0, timeframe="today 5-y", geo="", gprop=""):
|
|
253
|
+
"""Create the payload for interest over time, by region and related queries/topics."""
|
|
254
|
+
if gprop not in ("", "images", "news", "youtube", "froogle"):
|
|
255
|
+
raise ValueError("gprop must be empty (to indicate web), images, news, youtube, or froogle")
|
|
256
|
+
self.kw_list = [kw_list] if isinstance(kw_list, str) else list(kw_list)
|
|
257
|
+
self.geo = geo or self.geo
|
|
258
|
+
if not isinstance(self.geo, list):
|
|
259
|
+
self.geo = [self.geo]
|
|
260
|
+
self._query = {"cat": cat, "timeframe": timeframe, "gprop": gprop}
|
|
261
|
+
self.token_payload = self._payload(self.kw_list)
|
|
262
|
+
self._hosted = False
|
|
263
|
+
try:
|
|
264
|
+
self._tokens(self._get_data(self.GENERAL_URL, self.token_payload)["widgets"])
|
|
265
|
+
except TooManyRequestsError:
|
|
266
|
+
if not self.fallback:
|
|
267
|
+
raise
|
|
268
|
+
self._hosted = True # every call on this payload goes to the hosted Actor
|
|
269
|
+
|
|
270
|
+
def _payload(self, kws):
|
|
271
|
+
q = self._query
|
|
272
|
+
times = q["timeframe"] if isinstance(q["timeframe"], list) else None
|
|
273
|
+
items = [{"keyword": kw, "time": times[i] if times else q["timeframe"], "geo": g}
|
|
274
|
+
for i, (kw, g) in enumerate(product(kws, self.geo))]
|
|
275
|
+
return {"hl": self.hl, "tz": self.tz,
|
|
276
|
+
"req": json.dumps({"comparisonItem": items, "category": q["cat"], "property": q["gprop"]})}
|
|
277
|
+
|
|
278
|
+
def _tokens(self, widgets):
|
|
279
|
+
self.interest_over_time_widget = {}
|
|
280
|
+
self.interest_by_region_widget = {}
|
|
281
|
+
self.related_queries_widget_list = []
|
|
282
|
+
self.related_topics_widget_list = []
|
|
283
|
+
for w in widgets:
|
|
284
|
+
if w["id"] == "TIMESERIES":
|
|
285
|
+
self.interest_over_time_widget = w
|
|
286
|
+
if w["id"] == "GEO_MAP" and not self.interest_by_region_widget:
|
|
287
|
+
self.interest_by_region_widget = w
|
|
288
|
+
if "RELATED_TOPICS" in w["id"]:
|
|
289
|
+
self.related_topics_widget_list.append(w)
|
|
290
|
+
if "RELATED_QUERIES" in w["id"]:
|
|
291
|
+
self.related_queries_widget_list.append(w)
|
|
292
|
+
|
|
293
|
+
def _columns(self):
|
|
294
|
+
return [kw if len(self.geo) == 1 else (kw, g) for kw, g in product(self.kw_list, self.geo)]
|
|
295
|
+
|
|
296
|
+
def interest_over_time(self):
|
|
297
|
+
"""DataFrame indexed by date: one int column per keyword (0-100) and isPartial."""
|
|
298
|
+
def local():
|
|
299
|
+
data = self._widget(self.INTEREST_OVER_TIME_URL, self.interest_over_time_widget)
|
|
300
|
+
cols = self._columns()
|
|
301
|
+
rows = [{"date": _ts(p["time"]), **{c: int(v) for c, v in zip(cols, p["value"])},
|
|
302
|
+
"isPartial": bool(p.get("isPartial", False))} for p in data["timelineData"]]
|
|
303
|
+
return self._over_time_frame(rows)
|
|
304
|
+
|
|
305
|
+
def remote():
|
|
306
|
+
points = {}
|
|
307
|
+
for item in self._hosted_items("interest_over_time", includeInterestOverTime=True):
|
|
308
|
+
for p in item.get("timeline", []):
|
|
309
|
+
row = points.setdefault(p["timestamp"], {"date": _ts(p["timestamp"])})
|
|
310
|
+
row[item["term"]] = int(p["value"])
|
|
311
|
+
row["isPartial"] = bool(p.get("isPartial", False))
|
|
312
|
+
rows = [{"date": r["date"], **{k: r.get(k, 0) for k in self.kw_list}, "isPartial": r["isPartial"]}
|
|
313
|
+
for _, r in sorted(points.items())]
|
|
314
|
+
return self._over_time_frame(rows)
|
|
315
|
+
|
|
316
|
+
return self._either(local, remote)
|
|
317
|
+
|
|
318
|
+
def _over_time_frame(self, rows):
|
|
319
|
+
df = _frame(rows, "date")
|
|
320
|
+
if pd is not None and len(self.geo) > 1 and not df.empty:
|
|
321
|
+
df.columns = pd.MultiIndex.from_tuples([c if isinstance(c, tuple) else (c,) for c in df],
|
|
322
|
+
names=["keyword", "region"])
|
|
323
|
+
return df
|
|
324
|
+
|
|
325
|
+
def _region_resolution(self, resolution):
|
|
326
|
+
# pytrends meant to apply `resolution` only worldwide or for US sub-levels (its check never matched).
|
|
327
|
+
geo = self.geo[0] if self.geo else ""
|
|
328
|
+
return resolution if (not geo or resolution != "COUNTRY") else None
|
|
329
|
+
|
|
330
|
+
def interest_by_region(self, resolution="COUNTRY", inc_low_vol=False, inc_geo_code=False):
|
|
331
|
+
"""DataFrame indexed by geoName: one int column per keyword (plus geoCode if inc_geo_code)."""
|
|
332
|
+
res = self._region_resolution(resolution)
|
|
333
|
+
|
|
334
|
+
def local():
|
|
335
|
+
w = self.interest_by_region_widget
|
|
336
|
+
req = {**w["request"], "includeLowSearchVolumeGeos": inc_low_vol, **({"resolution": res} if res else {})}
|
|
337
|
+
data = self._widget(self.INTEREST_BY_REGION_URL, {"request": req, "token": w["token"]})
|
|
338
|
+
rows = []
|
|
339
|
+
for g in data["geoMapData"]:
|
|
340
|
+
row = {"geoName": g["geoName"]}
|
|
341
|
+
if inc_geo_code:
|
|
342
|
+
row["geoCode"] = g.get("geoCode", g.get("coordinates"))
|
|
343
|
+
row.update({kw: int(v) for kw, v in zip(self.kw_list, g["value"])})
|
|
344
|
+
rows.append(row)
|
|
345
|
+
return _frame(rows, "geoName")
|
|
346
|
+
|
|
347
|
+
def remote():
|
|
348
|
+
# ponytail: the Actor scales each term on its own map; locally pytrends' shared scale is used.
|
|
349
|
+
merged = {}
|
|
350
|
+
items = self._hosted_items("interest_by_region", includeInterestByRegion=True,
|
|
351
|
+
regionResolution=res or "auto", includeLowVolumeRegions=inc_low_vol)
|
|
352
|
+
for item in items:
|
|
353
|
+
for g in item.get("regions", []):
|
|
354
|
+
row = merged.setdefault(g["geoName"], {"geoName": g["geoName"],
|
|
355
|
+
**({"geoCode": g.get("geoCode")} if inc_geo_code else {}),
|
|
356
|
+
**{kw: 0 for kw in self.kw_list}})
|
|
357
|
+
row[item["term"]] = int(g.get("value", 0))
|
|
358
|
+
return _frame(list(merged.values()), "geoName")
|
|
359
|
+
|
|
360
|
+
return self._either(local, remote)
|
|
361
|
+
|
|
362
|
+
def _related(self, widgets, flat):
|
|
363
|
+
by_kw = {}
|
|
364
|
+
for w in widgets:
|
|
365
|
+
try:
|
|
366
|
+
by_kw[w["request"]["restriction"]["complexKeywordsRestriction"]["keyword"][0]["value"]] = w
|
|
367
|
+
except (KeyError, IndexError):
|
|
368
|
+
by_kw[""] = w
|
|
369
|
+
prefix = "RELATED_TOPICS" if flat else "RELATED_QUERIES"
|
|
370
|
+
for kw in self.kw_list:
|
|
371
|
+
if kw not in by_kw and len(self.geo) == 1:
|
|
372
|
+
# Google now sends related topics only for single-term explores: ask for this term alone.
|
|
373
|
+
widgets = self._get_data(self.GENERAL_URL, self._payload([kw]))["widgets"]
|
|
374
|
+
w = next((x for x in widgets if x["id"] == prefix), None)
|
|
375
|
+
if w:
|
|
376
|
+
by_kw[kw] = w
|
|
377
|
+
result = {}
|
|
378
|
+
for kw, w in by_kw.items():
|
|
379
|
+
data = self._widget(self.RELATED_QUERIES_URL, w)
|
|
380
|
+
result[kw] = {"top": _ranked(data, 0, flat), "rising": _ranked(data, 1, flat)}
|
|
381
|
+
return result
|
|
382
|
+
|
|
383
|
+
def related_queries(self):
|
|
384
|
+
"""{keyword: {'top': DataFrame[query, value] or None, 'rising': ...}}"""
|
|
385
|
+
def remote():
|
|
386
|
+
out = {kw: {"top": None, "rising": None} for kw in self.kw_list}
|
|
387
|
+
for item in self._hosted_items("related_queries", includeRelatedQueries=True):
|
|
388
|
+
out[item["term"]] = {k: _frame([{"query": r["query"], "value": r["value"]} for r in item.get(k, [])])
|
|
389
|
+
if item.get(k) else None for k in ("top", "rising")}
|
|
390
|
+
return out
|
|
391
|
+
|
|
392
|
+
return self._either(lambda: self._related(self.related_queries_widget_list, False), remote)
|
|
393
|
+
|
|
394
|
+
def related_topics(self):
|
|
395
|
+
"""{keyword: {'top': DataFrame or None, 'rising': ...}}. Google often hides topics from scripts."""
|
|
396
|
+
def remote():
|
|
397
|
+
out = {kw: {"top": None, "rising": None} for kw in self.kw_list}
|
|
398
|
+
for item in self._hosted_items("related_topics", includeRelatedTopics=True):
|
|
399
|
+
out[item["term"]] = {k: _frame([{"value": r.get("value"), "formattedValue": r.get("formattedValue"),
|
|
400
|
+
"link": r.get("link"), "topic_mid": r.get("topicId"),
|
|
401
|
+
"topic_title": r.get("topic"), "topic_type": r.get("topicType")}
|
|
402
|
+
for r in item.get(k, [])]) for k in ("top", "rising")}
|
|
403
|
+
return out
|
|
404
|
+
|
|
405
|
+
return self._either(lambda: self._related(self.related_topics_widget_list, True), remote)
|
|
406
|
+
|
|
407
|
+
def _hosted_items(self, kind, **flags):
|
|
408
|
+
if len(self.geo) != 1 or isinstance(self._query.get("timeframe"), list):
|
|
409
|
+
raise ValueError("The hosted fallback supports one geo and one timeframe per build_payload")
|
|
410
|
+
if any("," in kw for kw in self.kw_list):
|
|
411
|
+
raise ValueError("The hosted fallback does not support commas inside a keyword")
|
|
412
|
+
actor_input = {"searchTerms": [", ".join(self.kw_list)], "geo": self.geo[0],
|
|
413
|
+
**hosted.timeframe(self._query["timeframe"]), "category": self._query["cat"],
|
|
414
|
+
"gprop": self._query["gprop"] or "web", "language": self.hl, "timezoneOffset": self.tz,
|
|
415
|
+
"includeInterestOverTime": False, **flags}
|
|
416
|
+
return [i for i in hosted.run(actor_input, self.apify_token) if i.get("type") == kind]
|
|
417
|
+
|
|
418
|
+
# Trending now -----------------------------------------------------------------------------------
|
|
419
|
+
|
|
420
|
+
def trending_now(self, geo="US", hours=24):
|
|
421
|
+
"""Google Trends 'Trending now' for one country (hours: 4, 24, 48 or 168).
|
|
422
|
+
Rows: title, volume, increase_percentage, started, ended, active, categories, related_queries.
|
|
423
|
+
Falls back to the RSS feed (24 hours, with news) when the main endpoint changes shape."""
|
|
424
|
+
def local():
|
|
425
|
+
try:
|
|
426
|
+
return self._trending_batch(geo, hours)
|
|
427
|
+
except TooManyRequestsError:
|
|
428
|
+
raise
|
|
429
|
+
except ResponseError as e:
|
|
430
|
+
log.info("Trending now endpoint failed (%s); using the RSS feed", e)
|
|
431
|
+
return self._trending_rss(geo)
|
|
432
|
+
|
|
433
|
+
def remote():
|
|
434
|
+
items = hosted.run({"searchTerms": [], "trendingNowCountries": [geo], "trendingNowHours": str(hours),
|
|
435
|
+
"language": self.hl}, self.apify_token)
|
|
436
|
+
return [{"title": t["title"], "volume": t.get("searchVolume"),
|
|
437
|
+
"increase_percentage": t.get("increasePercentage"), "started": _iso(t.get("startedAt")),
|
|
438
|
+
"ended": _iso(t.get("endedAt")), "active": t.get("active"), "categories": t.get("categories", []),
|
|
439
|
+
"related_queries": t.get("relatedQueries", [])} for t in items if t.get("type") == "trending_now"]
|
|
440
|
+
|
|
441
|
+
return self._either(local, remote)
|
|
442
|
+
|
|
443
|
+
def _trending_batch(self, geo, hours):
|
|
444
|
+
f_req = json.dumps([[["i0OFE", json.dumps([None, None, geo, 0, self.hl, hours, 1]), None, "generic"]]])
|
|
445
|
+
|
|
446
|
+
def parse(text):
|
|
447
|
+
line = next(ln for ln in text.split("\n") if ln.startswith('[["wrb.fr"'))
|
|
448
|
+
trends = json.loads(json.loads(line)[0][2])[1]
|
|
449
|
+
if not isinstance(trends, list):
|
|
450
|
+
raise ValueError("no trends list")
|
|
451
|
+
return trends
|
|
452
|
+
|
|
453
|
+
url = f"{self.TRENDING_NOW_URL}?rpcids=i0OFE&source-path=%2Ftrending&hl={self.hl}&rt=c"
|
|
454
|
+
rows = self._get_data(url, parse=lambda t: _guard(parse, t),
|
|
455
|
+
data=urllib.parse.urlencode({"f.req": f_req}).encode(),
|
|
456
|
+
headers={"content-type": "application/x-www-form-urlencoded;charset=UTF-8"})
|
|
457
|
+
return [{"title": t[0], "volume": t[6], "increase_percentage": t[8],
|
|
458
|
+
"started": _ts(t[3][0]) if t[3] else None, "ended": _ts(t[4][0]) if t[4] else None,
|
|
459
|
+
"active": not t[4], "categories": [TRENDING_CATEGORIES.get(c, str(c)) for c in (t[10] or [])],
|
|
460
|
+
"related_queries": t[9] if isinstance(t[9], list) else []} for t in rows]
|
|
461
|
+
|
|
462
|
+
def _trending_rss(self, geo):
|
|
463
|
+
def parse(text):
|
|
464
|
+
rows = []
|
|
465
|
+
for it in ET.fromstring(text).iter("item"):
|
|
466
|
+
traffic = it.findtext("{*}approx_traffic") or ""
|
|
467
|
+
pub = it.findtext("pubDate")
|
|
468
|
+
rows.append({"title": it.findtext("title"), "volume": int("".join(filter(str.isdigit, traffic)) or 0) or None,
|
|
469
|
+
"increase_percentage": None,
|
|
470
|
+
"started": parsedate_to_datetime(pub).astimezone(timezone.utc).replace(tzinfo=None) if pub else None,
|
|
471
|
+
"ended": None, "active": None, "categories": [], "related_queries": [],
|
|
472
|
+
"news": [{"title": n.findtext("{*}news_item_title"), "url": n.findtext("{*}news_item_url"),
|
|
473
|
+
"source": n.findtext("{*}news_item_source")} for n in it.iter("{*}news_item")]})
|
|
474
|
+
return rows
|
|
475
|
+
|
|
476
|
+
return self._get_data(self.TRENDING_RSS_URL, {"geo": geo}, parse=lambda t: _guard(parse, t))
|
|
477
|
+
|
|
478
|
+
def _trending(self, pn, hours=24):
|
|
479
|
+
geo = PN_GEO.get(pn, pn).upper()
|
|
480
|
+
return [t["title"] for t in self.trending_now(geo, hours)]
|
|
481
|
+
|
|
482
|
+
def trending_searches(self, pn="united_states"):
|
|
483
|
+
"""DataFrame with one column (0) of trending titles. pn: pytrends country name or ISO code."""
|
|
484
|
+
titles = self._trending(pn)
|
|
485
|
+
return pd.DataFrame(titles) if pd is not None else titles
|
|
486
|
+
|
|
487
|
+
def today_searches(self, pn="US"):
|
|
488
|
+
"""Series of today's trending titles (past 24 hours)."""
|
|
489
|
+
titles = self._trending(pn)
|
|
490
|
+
return pd.Series(titles) if pd is not None else titles
|
|
491
|
+
|
|
492
|
+
def realtime_trending_searches(self, pn="US", cat="all", count=300):
|
|
493
|
+
"""DataFrame[entityNames, title]: trends started in the past 4 hours. `cat` is not supported."""
|
|
494
|
+
if cat != "all":
|
|
495
|
+
warnings.warn("realtime_trending_searches(cat=...) is ignored: Trending now has no category filter")
|
|
496
|
+
rows = self.trending_now(PN_GEO.get(pn, pn).upper(), 4)[:count]
|
|
497
|
+
return _frame([{"entityNames": t["related_queries"], "title": t["title"]} for t in rows])
|
|
498
|
+
|
|
499
|
+
# Other ------------------------------------------------------------------------------------------
|
|
500
|
+
|
|
501
|
+
def suggestions(self, keyword):
|
|
502
|
+
"""Google's keyword suggestions: list of {'mid', 'title', 'type'}."""
|
|
503
|
+
return self._get_data(self.SUGGESTIONS_URL + urllib.parse.quote(keyword), {"hl": self.hl})["default"]["topics"]
|
|
504
|
+
|
|
505
|
+
def categories(self):
|
|
506
|
+
"""Category tree: {'name', 'id', 'children': [...]}"""
|
|
507
|
+
return self._get_data(self.CATEGORIES_URL, {"hl": self.hl})
|
|
508
|
+
|
|
509
|
+
def multirange_interest_over_time(self):
|
|
510
|
+
raise NotImplementedError("multirange_interest_over_time is not ported yet. Call build_payload once per range.")
|
|
511
|
+
|
|
512
|
+
def top_charts(self, *args, **kwargs):
|
|
513
|
+
raise NotImplementedError("Google removed the Top Charts endpoint. Use trending_now() for current trends.")
|
|
514
|
+
|
|
515
|
+
def get_historical_interest(self, *args, **kwargs):
|
|
516
|
+
raise NotImplementedError("Removed in pytrends for incorrectness (pytrends PR #542).")
|
|
517
|
+
|
|
518
|
+
|
|
519
|
+
def _guard(parse, text):
|
|
520
|
+
"""Turn StopIteration from a missing line into the ValueError _get_data retries on."""
|
|
521
|
+
try:
|
|
522
|
+
return parse(text)
|
|
523
|
+
except StopIteration:
|
|
524
|
+
raise ValueError("payload not found") from None
|
|
525
|
+
except ET.ParseError as e:
|
|
526
|
+
raise ValueError(f"not RSS: {e}") from None
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pytrends-next
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Maintained drop-in replacement for pytrends (archived): Google Trends with the TrendReq API, standard library only, optional hosted fallback on HTTP 429.
|
|
5
|
+
Author-email: Ctrio Lab <myhouse9600@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/ctriolab/google-trends-python
|
|
8
|
+
Project-URL: Issues, https://github.com/ctriolab/google-trends-python/issues
|
|
9
|
+
Project-URL: Hosted fallback, https://apify.com/ctriolab/google-trends-scraper
|
|
10
|
+
Keywords: google trends,pytrends,trends,seo,keyword research,429
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Topic :: Internet
|
|
17
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
18
|
+
Requires-Python: >=3.9
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Provides-Extra: pandas
|
|
22
|
+
Requires-Dist: pandas>=1.3; extra == "pandas"
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
# pytrends-next
|
|
26
|
+
|
|
27
|
+
A maintained drop-in replacement for [pytrends](https://github.com/GeneralMills/pytrends), which was archived
|
|
28
|
+
(read-only) in April 2025. Same `TrendReq` API, same DataFrames, talks to Google Trends directly.
|
|
29
|
+
|
|
30
|
+
- Standard library only. pandas is optional: with pandas you get the same DataFrames as pytrends,
|
|
31
|
+
without it you get lists of dicts.
|
|
32
|
+
- Python 3.9+.
|
|
33
|
+
- Built-in handling of HTTP 429: new cookie session and exponential backoff with jitter.
|
|
34
|
+
- Optional, paid, off by default: route a call through a hosted scraper when Google keeps answering 429.
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
pip install "pytrends-next[pandas]" # or: pip install pytrends-next (no pandas)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
```python
|
|
41
|
+
from pytrends_next.request import TrendReq # was: from pytrends.request import TrendReq
|
|
42
|
+
|
|
43
|
+
pytrends = TrendReq(hl="en-US", tz=360)
|
|
44
|
+
pytrends.build_payload(["coffee", "tea"], timeframe="today 3-m", geo="US")
|
|
45
|
+
print(pytrends.interest_over_time().head(3))
|
|
46
|
+
# coffee tea isPartial
|
|
47
|
+
# date
|
|
48
|
+
# 2026-07-23 75 35 False
|
|
49
|
+
# 2026-07-24 81 38 False
|
|
50
|
+
# 2026-07-25 94 40 False
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
For most code the migration is one search and replace: `pytrends.` to `pytrends_next.` in imports.
|
|
54
|
+
`pytrends_next.exceptions` has the same `ResponseError` and `TooManyRequestsError`.
|
|
55
|
+
|
|
56
|
+
## Moving from pytrends
|
|
57
|
+
|
|
58
|
+
| pytrends 4.9.2 | pytrends-next | Notes |
|
|
59
|
+
|---|---|---|
|
|
60
|
+
| `TrendReq(hl, tz, geo, timeout, proxies, retries, backoff_factor, requests_args)` | same | `timeout` tuple: the larger value is used (urllib has one timeout). `requests_args`: `headers`, `proxies` and `verify` are used, other keys are ignored with a warning. `retries` defaults to 3 (pytrends: 0). |
|
|
61
|
+
| `build_payload(kw_list, cat, timeframe, geo, gprop)` | same | Lists of geos and timeframes work as in pytrends. |
|
|
62
|
+
| `interest_over_time()` | same | Index `date`, one int column per keyword, `isPartial`. MultiIndex columns for several geos. |
|
|
63
|
+
| `interest_by_region(resolution, inc_low_vol, inc_geo_code)` | same | pytrends' check meant to apply `resolution` worldwide and for sub-levels never matched; here it does. |
|
|
64
|
+
| `related_queries()` | same | `{kw: {"top": DataFrame[query, value] or None, "rising": ...}}` |
|
|
65
|
+
| `related_topics()` | same | Google now returns topics only for single-term requests, so each term is asked alone. Often empty, see Limits. |
|
|
66
|
+
| `trending_searches(pn="united_states")` | same | The old endpoint is gone. Now served from Google's "Trending now" list. `pn` takes the pytrends country name or an ISO code (`"US"`). |
|
|
67
|
+
| `today_searches(pn="US")` | same | Series of titles, past 24 hours. |
|
|
68
|
+
| `realtime_trending_searches(pn, cat, count)` | same | Trends started in the past 4 hours, columns `entityNames`, `title`. `cat` is ignored. |
|
|
69
|
+
| `suggestions(keyword)` | same | |
|
|
70
|
+
| `categories()` | same | |
|
|
71
|
+
| `top_charts()` | raises `NotImplementedError` | Google removed the endpoint. |
|
|
72
|
+
| `multirange_interest_over_time()` | raises `NotImplementedError` | Not ported yet. |
|
|
73
|
+
| (none) | `trending_now(geo="US", hours=24)` | New. Rows with `title`, `volume`, `increase_percentage`, `started`, `ended`, `active`, `categories`, `related_queries`. `hours`: 4, 24, 48 or 168. Falls back to the Trending RSS feed if the main endpoint changes. |
|
|
74
|
+
|
|
75
|
+
## HTTP 429 (TooManyRequestsError)
|
|
76
|
+
|
|
77
|
+
Google Trends has no public API. It rate-limits by IP and by session, and a laptop or a single
|
|
78
|
+
server often gets HTTP 429 after a few dozen requests. pytrends raised `TooManyRequestsError` at once.
|
|
79
|
+
|
|
80
|
+
pytrends-next does what our hosted scraper does on every request:
|
|
81
|
+
|
|
82
|
+
1. Visits trends.google.com first to get the `NID` cookie a browser visitor has (cookieless clients are limited sooner).
|
|
83
|
+
2. On 429, 403 or 5xx: drops the session, takes the next proxy from `proxies=[...]` if you gave any,
|
|
84
|
+
waits about 0.75 s, 1.5 s, 3 s ... (capped at 30 s, with jitter) and tries again, `retries` times.
|
|
85
|
+
3. 400 and 404 are not retried (bad geo, category or timeframe).
|
|
86
|
+
|
|
87
|
+
If you still get `TooManyRequestsError`: send fewer requests (batch up to 5 terms per `build_payload`),
|
|
88
|
+
wait a few minutes, use your own proxies, or use the hosted fallback below.
|
|
89
|
+
|
|
90
|
+
## Hosted fallback (optional, paid)
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
pytrends = TrendReq(fallback="apify", apify_token="apify_api_...") # or set APIFY_TOKEN
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
When Google still answers 429 after the local retries, the same call is sent to the
|
|
97
|
+
[Google Trends Scraper Actor on Apify](https://apify.com/ctriolab/google-trends-scraper?utm_source=github&utm_medium=readme&utm_campaign=pytrends-next),
|
|
98
|
+
which rotates IPs on Apify's proxy network. The result comes back in the same shape.
|
|
99
|
+
|
|
100
|
+
- **It costs money.** The Actor is billed per result to your own Apify account (about $1.50 per 1,000
|
|
101
|
+
results; one result is one term's timeline, one term's region list, one term's related list or one trending topic).
|
|
102
|
+
Apify's free plan includes a small monthly credit.
|
|
103
|
+
- It is never on by default. Setting `APIFY_TOKEN` alone does nothing; you must pass `fallback="apify"`.
|
|
104
|
+
- A warning is logged (logger `pytrends_next`) every time a call goes to the Actor.
|
|
105
|
+
- If `build_payload` itself gets 429, every method on that payload goes to the Actor.
|
|
106
|
+
- Supported: `interest_over_time`, `interest_by_region`, `related_queries`, `related_topics`, `trending_now`
|
|
107
|
+
and the pytrends trending methods. Not supported: several geos or timeframes in one payload,
|
|
108
|
+
commas inside a keyword, `suggestions`, `categories`.
|
|
109
|
+
- Difference: in hosted mode each term's region map is scaled on its own (locally, the terms share one scale).
|
|
110
|
+
- Timeframes: the named ranges (`now 1-H` to `today 5-y`, `all`) and `YYYY-MM-DD YYYY-MM-DD`. Others are rejected before a run starts.
|
|
111
|
+
|
|
112
|
+
Disclosure: Ctrio Lab maintains this library and also builds and sells that Actor. The library is MIT
|
|
113
|
+
licensed and fully works without it.
|
|
114
|
+
|
|
115
|
+
## Limits
|
|
116
|
+
|
|
117
|
+
- Values are relative (0 to 100) within one payload, as on trends.google.com. Separate calls are not on the same scale.
|
|
118
|
+
- Related topics are usually empty: Google hides them from clients it classifies as automated. Empty lists come back as `None`, like pytrends.
|
|
119
|
+
- Very small search volumes come back as all zeros (Google's own behaviour).
|
|
120
|
+
- This is an unofficial client for undocumented endpoints. Google can change them at any time.
|
|
121
|
+
|
|
122
|
+
## Legacy single-file client
|
|
123
|
+
|
|
124
|
+
`gtrends.py` (the earlier version of this repo) is still here and unchanged. It calls the hosted Actor for
|
|
125
|
+
every request. New code should use `pytrends_next.request.TrendReq`: `gtrends.interest_over_time_df(terms, ...)`
|
|
126
|
+
becomes `build_payload(terms, ...)` plus `interest_over_time()`.
|
|
127
|
+
|
|
128
|
+
## Development
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
python -m unittest discover -s tests # no network, recorded responses in tests/fixtures
|
|
132
|
+
python tests/live_smoke.py # about 12 live requests to Google
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
## License
|
|
136
|
+
|
|
137
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
MANIFEST.in
|
|
3
|
+
README.md
|
|
4
|
+
pyproject.toml
|
|
5
|
+
src/pytrends_next/__init__.py
|
|
6
|
+
src/pytrends_next/exceptions.py
|
|
7
|
+
src/pytrends_next/hosted.py
|
|
8
|
+
src/pytrends_next/request.py
|
|
9
|
+
src/pytrends_next.egg-info/PKG-INFO
|
|
10
|
+
src/pytrends_next.egg-info/SOURCES.txt
|
|
11
|
+
src/pytrends_next.egg-info/dependency_links.txt
|
|
12
|
+
src/pytrends_next.egg-info/requires.txt
|
|
13
|
+
src/pytrends_next.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
pytrends_next
|