@mradex77/google-play-scraper 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,13 +1,55 @@
1
- # @mradex77/google-play-scraper
1
+ # google-play-scraper
2
2
 
3
3
  [![npm version](https://img.shields.io/npm/v/@mradex77/google-play-scraper.svg)](https://www.npmjs.com/package/@mradex77/google-play-scraper)
4
+ [![npm downloads](https://img.shields.io/npm/dm/@mradex77/google-play-scraper.svg)](https://www.npmjs.com/package/@mradex77/google-play-scraper)
4
5
  [![CI](https://github.com/MrAdex77/google-play-scraper/actions/workflows/ci.yml/badge.svg)](https://github.com/MrAdex77/google-play-scraper/actions/workflows/ci.yml)
5
6
  [![Live contract tests](https://github.com/MrAdex77/google-play-scraper/actions/workflows/e2e.yml/badge.svg)](https://github.com/MrAdex77/google-play-scraper/actions/workflows/e2e.yml)
7
+ [![types](https://img.shields.io/npm/types/@mradex77/google-play-scraper.svg)](https://www.npmjs.com/package/@mradex77/google-play-scraper)
8
+ [![node](https://img.shields.io/node/v/@mradex77/google-play-scraper.svg)](https://nodejs.org)
6
9
  [![license](https://img.shields.io/npm/l/@mradex77/google-play-scraper.svg)](LICENSE)
7
10
 
8
- A modern TypeScript rewrite of the unmaintained [`google-play-scraper`](https://github.com/facundoolano/google-play-scraper). It scrapes public app data from Google Play — app details, search, suggestions, lists, developer pages, similar apps, reviews, permissions, and data safety. Ships as both ESM and CommonJS, returns fully typed results validated with [zod](https://zod.dev), runs on native `fetch` with no runtime HTTP dependency, and isolates every fragile Google Play array path behind a resilient spec layer that is exercised by daily live contract tests.
9
-
10
- The public method names, options, and constants match the original library, so migrating is mostly a matter of swapping the import.
11
+ **Scrape Google Play app data in Node.js with a fully typed TypeScript API.** Fetch app details, search results, top charts, developer pages, similar apps, user reviews, permissions and data safety information directly from the Play Store. Built for ASO research, app market analysis, competitor tracking, review monitoring and data pipelines.
12
+
13
+ This is a modern TypeScript rewrite of the popular but unmaintained [`google-play-scraper`](https://github.com/facundoolano/google-play-scraper) package. The public method names, options and constants match the original, so migrating is usually just a matter of swapping the import.
14
+
15
+ ## Why this library
16
+
17
+ - **Fully typed results.** Every method returns a precise TypeScript type derived from a [zod](https://zod.dev) schema that validates the scraped data at runtime.
18
+ - **No HTTP dependency.** Runs on the native `fetch` of Node.js 22 and newer.
19
+ - **Resilient by design.** Every fragile Google Play array path lives behind a spec layer with ordered fallback paths, so a single moved index does not take a whole call down.
20
+ - **Verified against live Google Play daily.** Contract tests run against play.google.com on a daily schedule in CI and open a labeled issue the moment Google changes its layout.
21
+ - **Typed errors.** Branch on `NotFoundError`, `RateLimitError` or `SpecError` instead of parsing message strings.
22
+ - **Throttling, retries and caching included.** Rate limiting, exponential backoff with `Retry-After` support and an optional memoized client come standard.
23
+ - **ESM and CommonJS.** Ships both module formats plus type declarations from a single package.
24
+
25
+ ## Comparison with the original google-play-scraper
26
+
27
+ | Capability | @mradex77/google-play-scraper | facundoolano/google-play-scraper |
28
+ | ------------------ | -------------------------------------- | -------------------------------- |
29
+ | Language | TypeScript in strict mode | JavaScript |
30
+ | Type definitions | Generated from zod schemas | Community typings |
31
+ | Runtime validation | zod on every input and output boundary | None |
32
+ | Error handling | Typed error classes | Plain `Error` |
33
+ | Module formats | ESM and CommonJS with `.d.ts` | ESM only |
34
+ | Breakage detection | Daily live contract tests in CI | None |
35
+ | Maintenance | Actively maintained | Unmaintained |
36
+
37
+ ## Table of contents
38
+
39
+ - [Installation](#installation)
40
+ - [Quick start](#quick-start)
41
+ - [Common options](#common-options)
42
+ - [Methods](#methods)
43
+ - [Constants](#constants)
44
+ - [Error handling](#error-handling)
45
+ - [Throttling and requestOptions](#throttling-and-requestoptions)
46
+ - [Resilience](#resilience)
47
+ - [Migrating from google-play-scraper](#migrating-from-google-play-scraper)
48
+ - [FAQ](#faq)
49
+ - [Related projects](#related-projects)
50
+ - [Contributing](#contributing)
51
+ - [Disclaimer](#disclaimer)
52
+ - [License](#license)
11
53
 
12
54
  ## Installation
13
55
 
@@ -45,6 +87,8 @@ gplay.search({ term: 'panda' }).then((results) => {
45
87
  });
46
88
  ```
47
89
 
90
+ More runnable examples live in [examples/](examples/).
91
+
48
92
  ## Common options
49
93
 
50
94
  Every method accepts a single options object. These options are available on all of them:
@@ -58,17 +102,17 @@ Every method accepts a single options object. These options are available on all
58
102
 
59
103
  ## Methods
60
104
 
61
- - [app](#app) full detail of a single application
62
- - [search](#search) apps matching a search term
63
- - [suggest](#suggest) search term autocompletions
64
- - [list](#list) a ranked collection of apps
65
- - [developer](#developer) other apps by the same developer
66
- - [similar](#similar) apps related to a given app
67
- - [reviews](#reviews) user reviews for an app
68
- - [permissions](#permissions) permissions an app requests
69
- - [datasafety](#datasafety) the data safety section of an app
70
- - [categories](#categories) the Google Play category taxonomy
71
- - [memoized](#memoized) a client that caches identical calls
105
+ - [app](#app): full detail of a single application
106
+ - [search](#search): apps matching a search term
107
+ - [suggest](#suggest): search term autocompletions
108
+ - [list](#list): a ranked collection of apps
109
+ - [developer](#developer): other apps by the same developer
110
+ - [similar](#similar): apps related to a given app
111
+ - [reviews](#reviews): user reviews for an app
112
+ - [permissions](#permissions): permissions an app requests
113
+ - [datasafety](#datasafety): the data safety section of an app
114
+ - [categories](#categories): the Google Play category taxonomy
115
+ - [memoized](#memoized): a client that caches identical calls
72
116
 
73
117
  ### app
74
118
 
@@ -380,7 +424,7 @@ Returns `string[]`:
380
424
 
381
425
  ### memoized
382
426
 
383
- Returns a client whose methods share an in-memory LRU cache, so identical calls made within the TTL resolve from cache instead of hitting Google Play again.
427
+ Returns a client whose methods share an LRU cache held in memory, so identical calls made within the TTL resolve from cache instead of hitting Google Play again.
384
428
 
385
429
  | Option | Type | Default | Description |
386
430
  | ---------- | -------- | -------- | ---------------------------------------------- |
@@ -400,7 +444,7 @@ The returned client exposes every method above plus the exported constants.
400
444
 
401
445
  ## Constants
402
446
 
403
- The library re-exports the same constant sets as the original, frozen and typed.
447
+ The library exports the same constant sets as the original, frozen and typed.
404
448
 
405
449
  | Constant | Values |
406
450
  | ------------ | ----------------------------------------------------------------------------------------------------------------- |
@@ -418,16 +462,16 @@ import { category, collection, sort, age, permission } from '@mradex77/google-pl
418
462
 
419
463
  Every failure surfaces as a typed subclass of `GooglePlayError`, so you can branch on the exact cause instead of parsing message strings.
420
464
 
421
- | Error | Extends | Thrown when |
422
- | ----------------- | ----------------- | ----------------------------------------------------------------------------------------- |
423
- | `GooglePlayError` | `Error` | Base class for every error the library throws. |
424
- | `ValidationError` | `GooglePlayError` | The options you passed fail their zod schema. |
425
- | `HttpError` | `GooglePlayError` | A request fails with a non-success status or a network error. Carries `status` and `url`. |
426
- | `NotFoundError` | `HttpError` | Google Play responds `404`, e.g. an unknown `appId`. |
427
- | `RateLimitError` | `HttpError` | Google Play responds `429` after retries are exhausted. |
428
- | `BlockedError` | `GooglePlayError` | A consent wall or captcha interstitial is detected. |
429
- | `ParseError` | `GooglePlayError` | A batchexecute response cannot be parsed. |
430
- | `SpecError` | `ParseError` | Extraction fails; lists every failing field and the paths that were tried. |
465
+ | Error | Extends | Thrown when |
466
+ | ----------------- | ----------------- | ------------------------------------------------------------------------------------------- |
467
+ | `GooglePlayError` | `Error` | Base class for every error the library throws. |
468
+ | `ValidationError` | `GooglePlayError` | The options you passed fail their zod schema. |
469
+ | `HttpError` | `GooglePlayError` | A request fails with an unsuccessful status or a network error. Carries `status` and `url`. |
470
+ | `NotFoundError` | `HttpError` | Google Play responds `404`, e.g. an unknown `appId`. |
471
+ | `RateLimitError` | `HttpError` | Google Play responds `429` after retries are exhausted. |
472
+ | `BlockedError` | `GooglePlayError` | A consent wall or captcha interstitial is detected. |
473
+ | `ParseError` | `GooglePlayError` | A batchexecute response cannot be parsed. |
474
+ | `SpecError` | `ParseError` | Extraction fails; lists every failing field and the paths that were tried. |
431
475
 
432
476
  ```typescript
433
477
  import { app, NotFoundError, SpecError } from '@mradex77/google-play-scraper';
@@ -453,7 +497,7 @@ Pass `throttle` to cap requests per second, and `requestOptions` to override the
453
497
  | -------------------- | ------------------------ | -------------------------------------------------------------- |
454
498
  | `headers` | `Record<string, string>` | Extra headers merged into every request. |
455
499
  | `fetchImpl` | `typeof fetch` | A custom `fetch` implementation, useful for proxies and tests. |
456
- | `timeoutMs` | `number` | Per-request timeout, up to `120000`. Default `30000`. |
500
+ | `timeoutMs` | `number` | Timeout per request, up to `120000`. Default `30000`. |
457
501
  | `retries` | `number` | Retry count for `429` and `5xx`, `0` to `5`. Default `2`. |
458
502
 
459
503
  ```typescript
@@ -475,9 +519,9 @@ Retries use exponential backoff and honor a `Retry-After` header when present.
475
519
 
476
520
  ## Resilience
477
521
 
478
- Google Play serves its data as deeply nested, unlabeled arrays whose positions shift a few times a year. That is what breaks scrapers. Every positional path in this library lives as a typed constant in a per-feature `specs.ts` file, never inline in logic, and each field is resolved through an ordered list of candidate paths so a single moved index does not take the whole call down. Extraction collects all field failures in one pass and throws a single `SpecError` naming every broken field and the paths that were tried, which is exactly the input the maintenance runbook needs. Unknown data enters as `unknown` and only leaves through a zod schema, so a layout change fails loudly at the boundary rather than three layers up.
522
+ Google Play serves its data as deeply nested, unlabeled arrays whose positions shift a few times a year. That is what breaks scrapers. Every positional path in this library lives as a typed constant in a `specs.ts` file per feature, never inline in logic, and each field is resolved through an ordered list of candidate paths so a single moved index does not take the whole call down. Extraction collects all field failures in one pass and throws a single `SpecError` naming every broken field and the paths that were tried, which is exactly the input the maintenance runbook needs. Unknown data enters as `unknown` and only leaves through a zod schema, so a layout change fails loudly at the boundary rather than three layers up.
479
523
 
480
- To catch breakage before users do, the `e2e/` suite runs against live Google Play on a daily GitHub Actions schedule and opens a labeled issue on failure. Repairing a break is a one-file spec diff, walked through step by step in [docs/RUNBOOK.md](docs/RUNBOOK.md).
524
+ To catch breakage before users do, the `e2e/` suite runs against live Google Play on a daily GitHub Actions schedule and opens a labeled issue on failure. Repairing a break is a spec diff confined to one file, walked through step by step in [docs/RUNBOOK.md](docs/RUNBOOK.md).
481
525
 
482
526
  ## Migrating from google-play-scraper
483
527
 
@@ -486,11 +530,39 @@ The method names, options, and constants are the same, so most code keeps workin
486
530
  - `reviews` always returns the `{ data, nextPaginationToken }` envelope, never a bare array.
487
531
  - Dates are ISO 8601 strings (review `date`, `replyDate`), and `updated` is a millisecond timestamp.
488
532
  - Errors are the typed classes above instead of plain `Error`.
489
- - The package is ESM-first with a CommonJS build; the default export is the aggregate client and named exports are also available.
533
+ - The package is ESM first with a CommonJS build; the default export is the aggregate client and named exports are also available.
534
+
535
+ ## FAQ
536
+
537
+ ### Is there an official Google Play API for app data?
538
+
539
+ No. Google does not offer a public API for store listings, search results or reviews. Libraries like this one fill that gap by scraping the public web pages and internal endpoints that power play.google.com. This is the standard approach for ASO tools, market research and academic studies.
540
+
541
+ ### How do I scrape Google Play reviews for an app?
542
+
543
+ Call [reviews](#reviews) with the app id. Use `paginate: true` and the returned `nextPaginationToken` to walk through all pages. Combine it with `throttle` to stay under the rate limits.
544
+
545
+ ### How do I avoid getting rate limited or blocked?
546
+
547
+ Set the `throttle` option to cap requests per second, keep the default retry behavior, and reuse results through the [memoized](#memoized) client. If you run large jobs, spread them out over time. A `RateLimitError` or `BlockedError` tells you exactly when Google started pushing back.
548
+
549
+ ### Does this library work in the browser?
550
+
551
+ No. It targets Node.js 22.12 and newer. Browsers cannot scrape Google Play anyway because of CORS restrictions.
552
+
553
+ ### Is this library affiliated with Google?
554
+
555
+ No. It is an independent open source project. All app data belongs to its respective owners, and you are responsible for using it in compliance with the Google Play Terms of Service and applicable law.
556
+
557
+ ## Related projects
558
+
559
+ - [facundoolano/google-play-scraper](https://github.com/facundoolano/google-play-scraper): the original Node.js library this project is a rewrite of.
560
+ - [facundoolano/app-store-scraper](https://github.com/facundoolano/app-store-scraper): the same idea for the Apple App Store.
561
+ - [JoMingyu/google-play-scraper](https://github.com/JoMingyu/google-play-scraper): a Python implementation.
490
562
 
491
563
  ## Contributing
492
564
 
493
- This project uses [pnpm](https://pnpm.io), [Conventional Commits](https://www.conventionalcommits.org), and the conventional branch spec (`feature/`, `bugfix/`, `chore/`, ...). Versioning, changelog, and npm publishing are automated with Release Please.
565
+ Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, commit conventions and test workflow. Found a field that stopped resolving? Open an issue with the `SpecError` output, or follow [docs/RUNBOOK.md](docs/RUNBOOK.md) to fix the spec yourself.
494
566
 
495
567
  ```
496
568
  pnpm install
@@ -503,6 +575,10 @@ pnpm build emit dist/ with esm, cjs, and d.ts
503
575
  pnpm check:package build then verify the published package
504
576
  ```
505
577
 
578
+ ## Disclaimer
579
+
580
+ This project is not affiliated with, endorsed by or sponsored by Google. It accesses publicly available data only. Use it responsibly, respect the Google Play Terms of Service, and throttle your requests.
581
+
506
582
  ## License
507
583
 
508
- MIT
584
+ [MIT](LICENSE)
package/dist/index.cjs CHANGED
@@ -522,6 +522,36 @@ const appSchema = zod.z.object({
522
522
  url: zod.z.string()
523
523
  });
524
524
  //#endregion
525
+ //#region src/core/text.ts
526
+ const TAB = 9;
527
+ const LINE_FEED = 10;
528
+ const CARRIAGE_RETURN = 13;
529
+ const UNIT_SEPARATOR = 31;
530
+ const DELETE = 127;
531
+ const C1_END = 159;
532
+ const SURROGATE_START = 55296;
533
+ const SURROGATE_END = 57343;
534
+ function isPreservedWhitespace(code) {
535
+ return code === TAB || code === LINE_FEED || code === CARRIAGE_RETURN;
536
+ }
537
+ function isControlCharacter(code) {
538
+ if (isPreservedWhitespace(code)) return false;
539
+ return code <= UNIT_SEPARATOR || code >= DELETE && code <= C1_END;
540
+ }
541
+ function isLoneSurrogate(code) {
542
+ return code >= SURROGATE_START && code <= SURROGATE_END;
543
+ }
544
+ function sanitizeText(value) {
545
+ if (typeof value !== "string") return;
546
+ let result = "";
547
+ for (const character of value) {
548
+ const code = character.codePointAt(0) ?? 0;
549
+ if (isControlCharacter(code) || isLoneSurrogate(code)) continue;
550
+ result += character;
551
+ }
552
+ return result;
553
+ }
554
+ //#endregion
525
555
  //#region src/features/app/transforms.ts
526
556
  const COMMENT_ROOTS = ["ds:8", "ds:9"];
527
557
  const MAX_COMMENTS = 5;
@@ -538,12 +568,11 @@ function descriptionHtmlLocalized(value) {
538
568
  0,
539
569
  1
540
570
  ]);
541
- const resolved = typeof translated === "string" && translated.length > 0 ? translated : original;
542
- return typeof resolved === "string" ? resolved : void 0;
571
+ return sanitizeText(typeof translated === "string" && translated.length > 0 ? translated : original);
543
572
  }
544
573
  function descriptionText(html) {
545
574
  if (typeof html !== "string") return;
546
- return cheerio.load(`<div>${html.replace(/<br>/g, "\r\n")}</div>`)("div").text();
575
+ return sanitizeText(cheerio.load(`<div>${html.replace(/<br>/g, "\r\n")}</div>`)("div").text());
547
576
  }
548
577
  function priceText(value) {
549
578
  return typeof value === "string" && value.length > 0 ? value : "Free";
@@ -1322,7 +1351,8 @@ const appSpecs = {
1322
1351
  1,
1323
1352
  1
1324
1353
  ]],
1325
- schema: shape$6.recentChanges
1354
+ schema: shape$6.recentChanges,
1355
+ transform: sanitizeText
1326
1356
  },
1327
1357
  comments: {
1328
1358
  paths: [[]],
@@ -1880,6 +1910,17 @@ function priceGoogleValue(value) {
1880
1910
  default: return 0;
1881
1911
  }
1882
1912
  }
1913
+ function matchesPriceFilter(free, filter) {
1914
+ switch (filter) {
1915
+ case "free": return free;
1916
+ case "paid": return !free;
1917
+ default: return true;
1918
+ }
1919
+ }
1920
+ function filterByPrice(items, filter) {
1921
+ if (filter === "all") return [...items];
1922
+ return items.filter((item) => matchesPriceFilter(item.free, filter));
1923
+ }
1883
1924
  //#endregion
1884
1925
  //#region src/features/search/search.ts
1885
1926
  const searchOptionsSchema = baseOptionsSchema.extend({
@@ -1940,7 +1981,7 @@ function createSearch(getApp) {
1940
1981
  });
1941
1982
  const client = clientFromOptions(parsed);
1942
1983
  const page = firstPage(parseScriptData(await client.request({ url: `${SEARCH_URL}?${params.toString()}` })));
1943
- const sliced = (await fetchClusterApps({
1984
+ const sliced = filterByPrice(await fetchClusterApps({
1944
1985
  client,
1945
1986
  lang: parsed.lang,
1946
1987
  country: parsed.country,
@@ -1951,7 +1992,7 @@ function createSearch(getApp) {
1951
1992
  appsPath: CLUSTER_MAPPINGS$1.apps,
1952
1993
  tokenPath: CLUSTER_MAPPINGS$1.token,
1953
1994
  context: SEARCH_CONTEXT
1954
- })).slice(0, parsed.num);
1995
+ }), parsed.price).slice(0, parsed.num);
1955
1996
  if (parsed.fullDetail) return resolveFullDetail(sliced, parsed, getApp);
1956
1997
  return zod.z.array(searchResultSchema).parse(sliced);
1957
1998
  };
@@ -2574,17 +2615,35 @@ const developerOptionsSchema = baseOptionsSchema.extend({
2574
2615
  fullDetail: zod.z.boolean().default(false)
2575
2616
  });
2576
2617
  const DEVELOPER_CONTEXT = "developer";
2577
- function extractInitial(blocks, numeric) {
2578
- const mappings = numeric ? NUMERIC_INITIAL_MAPPINGS : NAME_INITIAL_MAPPINGS;
2579
- const itemSpecs = numeric ? numericItemSpecs : nameItemSpecs;
2580
- const appsData = getPath(blocks, mappings.apps);
2581
- const apps = Array.isArray(appsData) ? appsData.map((item) => extract(item, itemSpecs, DEVELOPER_CONTEXT)) : [];
2582
- const token = getPath(blocks, mappings.token);
2618
+ const NUMERIC_LAYOUT = {
2619
+ mappings: NUMERIC_INITIAL_MAPPINGS,
2620
+ itemSpecs: numericItemSpecs
2621
+ };
2622
+ const NAME_LAYOUT = {
2623
+ mappings: NAME_INITIAL_MAPPINGS,
2624
+ itemSpecs: nameItemSpecs
2625
+ };
2626
+ function extractLayout(blocks, layout) {
2627
+ const appsData = getPath(blocks, layout.mappings.apps);
2628
+ if (!Array.isArray(appsData) || appsData.length === 0) return;
2629
+ const apps = appsData.map((item) => extract(item, layout.itemSpecs, DEVELOPER_CONTEXT));
2630
+ const token = getPath(blocks, layout.mappings.token);
2583
2631
  return {
2584
2632
  apps,
2585
2633
  token: typeof token === "string" ? token : void 0
2586
2634
  };
2587
2635
  }
2636
+ function extractInitial(blocks, numeric) {
2637
+ const ordered = numeric ? [NUMERIC_LAYOUT, NAME_LAYOUT] : [NAME_LAYOUT, NUMERIC_LAYOUT];
2638
+ for (const layout of ordered) {
2639
+ const extracted = extractLayout(blocks, layout);
2640
+ if (extracted !== void 0) return extracted;
2641
+ }
2642
+ return {
2643
+ apps: [],
2644
+ token: void 0
2645
+ };
2646
+ }
2588
2647
  function createDeveloper(getApp) {
2589
2648
  return async function developer(options) {
2590
2649
  const parsed = parseOptions(developerOptionsSchema, options, DEVELOPER_CONTEXT);
@@ -2868,6 +2927,9 @@ function alwaysNull() {
2868
2927
  function emptyToUndefined(value) {
2869
2928
  return typeof value === "string" && value.length > 0 ? value : void 0;
2870
2929
  }
2930
+ function cleanReplyText(value) {
2931
+ return emptyToUndefined(sanitizeText(value));
2932
+ }
2871
2933
  function buildCriteria(entry) {
2872
2934
  if (!Array.isArray(entry)) return {
2873
2935
  criteria: void 0,
@@ -2919,7 +2981,8 @@ const reviewItemSpecs = {
2919
2981
  },
2920
2982
  text: {
2921
2983
  paths: [[4]],
2922
- schema: shape.text
2984
+ schema: shape.text,
2985
+ transform: sanitizeText
2923
2986
  },
2924
2987
  replyDate: {
2925
2988
  paths: [[7, 2]],
@@ -2929,7 +2992,7 @@ const reviewItemSpecs = {
2929
2992
  replyText: {
2930
2993
  paths: [[7, 1]],
2931
2994
  schema: shape.replyText,
2932
- transform: emptyToUndefined
2995
+ transform: cleanReplyText
2933
2996
  },
2934
2997
  version: {
2935
2998
  paths: [[10]],