@crawlee/core 3.3.4-beta.16 → 3.3.4-beta.17

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +108 -27
  2. package/package.json +19 -12
package/README.md CHANGED
@@ -1,46 +1,127 @@
1
- # `@crawlee/core`
1
+ <h1 align="center">
2
+ <a href="https://crawlee.dev">
3
+ <picture>
4
+ <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/apify/crawlee/master/website/static/img/crawlee-dark.svg?sanitize=true">
5
+ <img alt="Crawlee" src="https://raw.githubusercontent.com/apify/crawlee/master/website/static/img/crawlee-light.svg?sanitize=true" width="500">
6
+ </picture>
7
+ </a>
8
+ <br>
9
+ <small>A web scraping and browser automation library</small>
10
+ </h1>
2
11
 
3
- Core set of classes required for Crawlee.
12
+ <p align=center>
13
+ <a href="https://www.npmjs.com/package/@crawlee/core" rel="nofollow"><img src="https://img.shields.io/npm/v/@crawlee/core.svg" alt="NPM latest version" data-canonical-src="https://img.shields.io/npm/v/@crawlee/core/next.svg" style="max-width: 100%;"></a>
14
+ <a href="https://www.npmjs.com/package/@crawlee/core" rel="nofollow"><img src="https://img.shields.io/npm/dm/@crawlee/core.svg" alt="Downloads" data-canonical-src="https://img.shields.io/npm/dm/@crawlee/core.svg" style="max-width: 100%;"></a>
15
+ <a href="https://discord.gg/jyEM2PRvMU" rel="nofollow"><img src="https://img.shields.io/discord/801163717915574323?label=discord" alt="Chat on discord" data-canonical-src="https://img.shields.io/discord/801163717915574323?label=discord" style="max-width: 100%;"></a>
16
+ <a href="https://github.com/apify/crawlee/actions/workflows/test-and-release.yml"><img src="https://github.com/apify/crawlee/actions/workflows/test-and-release.yml/badge.svg?branch=master" alt="Build Status" style="max-width: 100%;"></a>
17
+ </p>
4
18
 
5
- The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:
19
+ > ℹ️ Crawlee is the successor to [Apify SDK](https://sdk.apify.com). 🎉 Fully rewritten in **TypeScript** for a better developer experience, and with even more powerful anti-blocking features. The interface is almost the same as Apify SDK so upgrading is a breeze. Read [the upgrading guide](https://crawlee.dev/docs/upgrading/upgrading-to-v3) to learn about the changes. ℹ️
6
20
 
7
- - [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
8
- - [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`
9
- - [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`
10
- - [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`
11
- - [`@crawlee/linkedom`](https://crawlee.dev/api/jsdom-crawler): exports `LinkeDOMCrawler`
12
- - [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`
13
- - [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`
14
- - [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))
15
- - [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))
16
- - [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
17
- - [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
18
- - [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods
19
- - [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)
21
+ Crawlee covers your crawling and scraping end-to-end and **helps you build reliable scrapers. Fast.**
20
22
 
21
- ## Installing Crawlee
23
+ Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.
22
24
 
23
- Most of the Crawlee packages are extending and reexporting each other, so it's enough to install just the one you plan on using, e.g. `@crawlee/playwright` if you plan on using `playwright` - it already contains everything from the `@crawlee/browser` package, which includes everything from `@crawlee/basic`, which includes everything from `@crawlee/core`.
25
+ Crawlee is available as the [`crawlee`](https://www.npmjs.com/package/crawlee) NPM package.
24
26
 
25
- If we don't care much about additional code being pulled in, we can just use the `crawlee` meta-package, which contains (re-exports) most of the `@crawlee/*` packages, and therefore contains all the crawler classes.
27
+ > 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈
28
+
29
+ ## Installation
30
+
31
+ We recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.
32
+
33
+ > Crawlee requires **Node.js 16 or higher**.
34
+
35
+ ### With Crawlee CLI
36
+
37
+ The fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.
26
38
 
27
39
  ```bash
28
- npm install crawlee
40
+ npx crawlee create my-crawler
29
41
  ```
30
42
 
31
- Or if all we need is cheerio support, we can install only `@crawlee/cheerio`.
32
-
33
43
  ```bash
34
- npm install @crawlee/cheerio
44
+ cd my-crawler
45
+ npm start
35
46
  ```
36
47
 
37
- When using `playwright` or `puppeteer`, we still need to install those dependencies explicitly - this allows the users to be in control of which version will be used.
48
+ ### Manual installation
49
+ If you prefer adding Crawlee **into your own project**, try the example below. Because it uses `PlaywrightCrawler` we also need to install [Playwright](https://playwright.dev). It's not bundled with Crawlee to reduce install size.
38
50
 
39
51
  ```bash
40
52
  npm install crawlee playwright
41
- # or npm install @crawlee/playwright playwright
42
53
  ```
43
54
 
44
- Alternatively we can also use the `crawlee` meta-package which contains (re-exports) most of the `@crawlee/*` packages, and therefore contains all the crawler classes.
55
+ ```js
56
+ import { PlaywrightCrawler, Dataset } from 'crawlee';
57
+
58
+ // PlaywrightCrawler crawls the web using a headless
59
+ // browser controlled by the Playwright library.
60
+ const crawler = new PlaywrightCrawler({
61
+ // Use the requestHandler to process each of the crawled pages.
62
+ async requestHandler({ request, page, enqueueLinks, log }) {
63
+ const title = await page.title();
64
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
65
+
66
+ // Save results as JSON to ./storage/datasets/default
67
+ await Dataset.pushData({ title, url: request.loadedUrl });
68
+
69
+ // Extract links from the current page
70
+ // and add them to the crawling queue.
71
+ await enqueueLinks();
72
+ },
73
+ // Uncomment this option to see the browser window.
74
+ // headless: false,
75
+ });
76
+
77
+ // Add first URL to the queue and start the crawl.
78
+ await crawler.run(['https://crawlee.dev']);
79
+ ```
80
+
81
+ By default, Crawlee stores data to `./storage` in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).
82
+
83
+ ## 🛠 Features
84
+
85
+ - Single interface for **HTTP and headless browser** crawling
86
+ - Persistent **queue** for URLs to crawl (breadth & depth first)
87
+ - Pluggable **storage** of both tabular data and files
88
+ - Automatic **scaling** with available system resources
89
+ - Integrated **proxy rotation** and session management
90
+ - Lifecycles customizable with **hooks**
91
+ - **CLI** to bootstrap your projects
92
+ - Configurable **routing**, **error handling** and **retries**
93
+ - **Dockerfiles** ready to deploy
94
+ - Written in **TypeScript** with generics
95
+
96
+ ### 👾 HTTP crawling
97
+
98
+ - Zero config **HTTP2 support**, even for proxies
99
+ - Automatic generation of **browser-like headers**
100
+ - Replication of browser **TLS fingerprints**
101
+ - Integrated fast **HTML parsers**. Cheerio and JSDOM
102
+ - Yes, you can scrape **JSON APIs** as well
103
+
104
+ ### 💻 Real browser crawling
105
+
106
+ - JavaScript **rendering** and **screenshots**
107
+ - **Headless** and **headful** support
108
+ - Zero-config generation of **human-like fingerprints**
109
+ - Automatic **browser management**
110
+ - Use **Playwright** and **Puppeteer** with the same interface
111
+ - **Chrome**, **Firefox**, **Webkit** and many others
112
+
113
+ ## Usage on the Apify platform
114
+
115
+ Crawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.
116
+
117
+ ## Support
118
+
119
+ If you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).
120
+
121
+ ## Contributing
122
+
123
+ Your code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).
124
+
125
+ ## License
45
126
 
46
- > Sometimes you might want to use some utility methods from `@crawlee/utils`, so you might want to install that as well. This package contains some utilities that were previously available under `Apify.utils`. Browser related utilities can be also found in the crawler packages (e.g. `@crawlee/playwright`).
127
+ This project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details.
package/package.json CHANGED
@@ -1,18 +1,18 @@
1
1
  {
2
2
  "name": "@crawlee/core",
3
- "version": "3.3.4-beta.16",
3
+ "version": "3.3.4-beta.17",
4
4
  "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
5
5
  "engines": {
6
6
  "node": ">=16.0.0"
7
7
  },
8
- "main": "./dist/index.js",
9
- "module": "./dist/index.mjs",
10
- "types": "./dist/index.d.ts",
8
+ "main": "./index.js",
9
+ "module": "./index.mjs",
10
+ "types": "./index.d.ts",
11
11
  "exports": {
12
12
  ".": {
13
- "import": "./dist/index.mjs",
14
- "require": "./dist/index.js",
15
- "types": "./dist/index.d.ts"
13
+ "import": "./index.mjs",
14
+ "require": "./index.js",
15
+ "types": "./index.d.ts"
16
16
  },
17
17
  "./package.json": "./package.json"
18
18
  },
@@ -46,7 +46,7 @@
46
46
  "scripts": {
47
47
  "build": "yarn clean && yarn compile && yarn copy",
48
48
  "clean": "rimraf ./dist",
49
- "compile": "tsc -p tsconfig.build.json && gen-esm-wrapper ./dist/index.js ./dist/index.mjs",
49
+ "compile": "tsc -p tsconfig.build.json && gen-esm-wrapper ./index.js ./index.mjs",
50
50
  "copy": "ts-node -T ../../scripts/copy.ts"
51
51
  },
52
52
  "publishConfig": {
@@ -59,9 +59,9 @@
59
59
  "@apify/pseudo_url": "^2.0.14",
60
60
  "@apify/timeout": "^0.3.0",
61
61
  "@apify/utilities": "^2.3.3",
62
- "@crawlee/memory-storage": "^3.3.4-beta.16",
63
- "@crawlee/types": "^3.3.4-beta.16",
64
- "@crawlee/utils": "^3.3.4-beta.16",
62
+ "@crawlee/memory-storage": "^3.3.4-beta.17",
63
+ "@crawlee/types": "^3.3.4-beta.17",
64
+ "@crawlee/utils": "^3.3.4-beta.17",
65
65
  "@sapphire/async-queue": "^1.5.0",
66
66
  "@types/tough-cookie": "^4.0.2",
67
67
  "@vladfrangu/async_event_emitter": "^2.0.0",
@@ -77,5 +77,12 @@
77
77
  "tslib": "^2.4.0",
78
78
  "type-fest": "^3.0.0"
79
79
  },
80
- "gitHead": "67a826c104e2e29a65ff89ccabaecf8b078f2c98"
80
+ "lerna": {
81
+ "command": {
82
+ "publish": {
83
+ "assets": []
84
+ }
85
+ }
86
+ },
87
+ "gitHead": "8bbc596f4dba0aa755df204d836c97c05fc94fca"
81
88
  }