@the-convocation/twitter-scraper 0.19.0 → 0.19.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/dist/default/cjs/index.js +25 -7
- package/dist/default/cjs/index.js.map +1 -1
- package/dist/default/esm/index.mjs +25 -7
- package/dist/default/esm/index.mjs.map +1 -1
- package/dist/node/cjs/index.cjs +25 -7
- package/dist/node/cjs/index.cjs.map +1 -1
- package/dist/node/esm/index.mjs +25 -7
- package/dist/node/esm/index.mjs.map +1 -1
- package/examples/cycletls/README.md +4 -10
- package/examples/cycletls/package.json +1 -0
- package/package.json +1 -1
|
@@ -4,11 +4,11 @@ This example demonstrates how to use the `@the-convocation/twitter-scraper/cycle
|
|
|
4
4
|
|
|
5
5
|
## Problem
|
|
6
6
|
|
|
7
|
-
Twitter's authentication endpoints may be protected by Cloudflare's
|
|
7
|
+
Twitter's authentication endpoints may be protected by Cloudflare's bot detection, which analyzes TLS fingerprints to detect non-browser clients. Standard Node.js TLS handshakes can trigger `403 Forbidden` errors during login.
|
|
8
8
|
|
|
9
9
|
## Solution
|
|
10
10
|
|
|
11
|
-
This example uses [CycleTLS](https://github.com/Danny-Dasilva/CycleTLS)
|
|
11
|
+
This example uses [CycleTLS](https://github.com/Danny-Dasilva/CycleTLS) to mimic Chrome browser TLS fingerprints, allowing requests to pass through Cloudflare's protection.
|
|
12
12
|
|
|
13
13
|
## Installation
|
|
14
14
|
|
|
@@ -18,7 +18,7 @@ yarn install
|
|
|
18
18
|
|
|
19
19
|
## Configuration
|
|
20
20
|
|
|
21
|
-
Create a `.env` file in
|
|
21
|
+
Create a `.env` file in this directory with your Twitter credentials:
|
|
22
22
|
|
|
23
23
|
```
|
|
24
24
|
TWITTER_USERNAME=your_username
|
|
@@ -41,14 +41,8 @@ import { Scraper } from '@the-convocation/twitter-scraper';
|
|
|
41
41
|
import { cycleTLSFetch, cycleTLSExit } from '@the-convocation/twitter-scraper/cycletls';
|
|
42
42
|
|
|
43
43
|
const scraper = new Scraper({
|
|
44
|
-
fetch: cycleTLSFetch
|
|
44
|
+
fetch: cycleTLSFetch,
|
|
45
45
|
});
|
|
46
46
|
```
|
|
47
47
|
|
|
48
48
|
This replaces the default fetch implementation with one that uses Chrome-like TLS fingerprints, bypassing Cloudflare's detection.
|
|
49
|
-
|
|
50
|
-
## Important Notes
|
|
51
|
-
|
|
52
|
-
- **Node.js only**: The `/cycletls` entrypoint requires Node.js and will not work in browsers
|
|
53
|
-
- **Cleanup required**: Always call `cycleTLSExit()` when done to cleanup golang resources
|
|
54
|
-
- **Optional dependency**: `cycletls` must be explicitly installed alongside the main package
|