fb-reels-scraper 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- fb_reels_scraper-1.0.0/LICENSE +21 -0
- fb_reels_scraper-1.0.0/MANIFEST.in +5 -0
- fb_reels_scraper-1.0.0/PKG-INFO +219 -0
- fb_reels_scraper-1.0.0/README.md +194 -0
- fb_reels_scraper-1.0.0/extension/content.js +185 -0
- fb_reels_scraper-1.0.0/extension/popup.html +51 -0
- fb_reels_scraper-1.0.0/extension/popup.js +36 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper/__init__.py +9 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper/__main__.py +4 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper/cli.py +124 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper/extension/content.js +185 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper/extension/popup.html +51 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper/extension/popup.js +36 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper/scraper.py +200 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper.egg-info/PKG-INFO +219 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper.egg-info/SOURCES.txt +21 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper.egg-info/dependency_links.txt +1 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper.egg-info/entry_points.txt +2 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper.egg-info/requires.txt +1 -0
- fb_reels_scraper-1.0.0/fb_reels_scraper.egg-info/top_level.txt +1 -0
- fb_reels_scraper-1.0.0/pyproject.toml +45 -0
- fb_reels_scraper-1.0.0/requirements.txt +1 -0
- fb_reels_scraper-1.0.0/setup.cfg +4 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Ishan Dutta
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,219 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: fb-reels-scraper
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Scrape Facebook Reels descriptions using your existing browser session via a WebSocket bridge.
|
|
5
|
+
Author-email: Ishan <ishan@example.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/ishan-kh/fb-reels-scraper
|
|
8
|
+
Project-URL: Repository, https://github.com/ishan-kh/fb-reels-scraper
|
|
9
|
+
Project-URL: Issues, https://github.com/ishan-kh/fb-reels-scraper/issues
|
|
10
|
+
Keywords: facebook,reels,scraper,crawler,browser-extension,websocket
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Operating System :: OS Independent
|
|
18
|
+
Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
|
|
19
|
+
Classifier: Topic :: Utilities
|
|
20
|
+
Requires-Python: >=3.8
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
Requires-Dist: websockets>=12.0
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
# FB Reels Scraper Bridge 🎬
|
|
27
|
+
|
|
28
|
+
[](https://pypi.org/project/fb-reels-scraper/)
|
|
29
|
+
[](https://pypi.org/project/fb-reels-scraper/)
|
|
30
|
+
[](https://opensource.org/licenses/MIT)
|
|
31
|
+
|
|
32
|
+
A robust, two-stage Facebook Reels scraper that operates through your **existing browser session** via a lightweight Chrome/Edge extension and a local Python WebSocket bridge.
|
|
33
|
+
|
|
34
|
+
By utilizing your real browser session, you completely bypass login hurdles, 2FA prompts, fingerprinting, and bot-detection challenges without needing credentials or headless automation drivers.
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## ✨ Features
|
|
39
|
+
|
|
40
|
+
- **No Bot Detection / Captchas**: Works directly inside your logged-in browser tab.
|
|
41
|
+
- **Two-Phase Architecture**:
|
|
42
|
+
1. **Phase 1 (Discovery & Harvest)**: Scrolls target profile or page reels feeds (`/reels/`) to collect reel permalinks.
|
|
43
|
+
2. **Phase 2 (Deep Extraction)**: Navigates to individual reel view pages, automatically clicks **"See more"** / **"...more"** to un-truncate descriptions, and extracts the full caption text.
|
|
44
|
+
- **Local Smart Caching (`reels_cache.json`)**: Automatically saves extracted reels on the fly. Re-running the script skips previously scraped URLs to save time and bandwidth.
|
|
45
|
+
- **Bundled Chrome Extension**: Extension files are packaged directly inside the package and can be exported with a single CLI command (`fb-reels-scraper --install-extension`).
|
|
46
|
+
- **Resilient WebSocket Bridge**: Gracefully handles page transitions, DOM reloads, and tab navigation events without throwing disconnect exceptions.
|
|
47
|
+
|
|
48
|
+
---
|
|
49
|
+
|
|
50
|
+
## 📦 Installation
|
|
51
|
+
|
|
52
|
+
### From PyPI
|
|
53
|
+
```bash
|
|
54
|
+
pip install fb-reels-scraper
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### Or from source (Editable mode)
|
|
58
|
+
```bash
|
|
59
|
+
git clone https://github.com/ishan-kh/fb-reels-scraper.git
|
|
60
|
+
cd fb-reels-scraper
|
|
61
|
+
pip install -e .
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## 🚀 Quick Setup & Usage
|
|
67
|
+
|
|
68
|
+
### Step 1: Export & Load the Browser Extension
|
|
69
|
+
|
|
70
|
+
If installed via `pip`, you can export the bundled extension directory into your current working folder:
|
|
71
|
+
```bash
|
|
72
|
+
fb-reels-scraper --install-extension
|
|
73
|
+
```
|
|
74
|
+
*(Or check the internal extension path via `fb-reels-scraper --extension-path`)*
|
|
75
|
+
|
|
76
|
+
1. Open your Chromium browser (Chrome, Brave, Edge, etc.) and go to `chrome://extensions/` (or `edge://extensions/`).
|
|
77
|
+
2. Toggle on **Developer mode** in the top-right corner.
|
|
78
|
+
3. Click **Load unpacked** and select the exported `./fb_reels_extension` folder (or `./extension` if running from source).
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
### Step 2: Open Facebook & Run the CLI
|
|
83
|
+
|
|
84
|
+
1. Ensure you have an active Facebook tab open in your browser where you are logged in.
|
|
85
|
+
2. In your terminal, run the CLI tool:
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
# Run with default fallback URL and scroll count
|
|
89
|
+
fb-reels-scraper
|
|
90
|
+
|
|
91
|
+
# Scrape a specific profile or page
|
|
92
|
+
fb-reels-scraper --url "https://www.facebook.com/<target_user_or_page>/reels/"
|
|
93
|
+
|
|
94
|
+
# Specify scroll count and custom cache file (short flags)
|
|
95
|
+
fb-reels-scraper -u "https://www.facebook.com/<target_user_or_page>/reels/" -s 25 -c my_reels.json
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## 🛠️ CLI Options Reference
|
|
101
|
+
|
|
102
|
+
| Flag | Short | Default | Description |
|
|
103
|
+
|---|---|---|---|
|
|
104
|
+
| `--url` | `-u` | `https://www.facebook.com/nadeemkhancyber/reels/` | Facebook reels page URL to scrape |
|
|
105
|
+
| `--scrolls` | `-s` | `20` | Number of scroll cycles on the grid to harvest links |
|
|
106
|
+
| `--cache-file` | `-c` | `reels_cache.json` | Path to JSON file where results are cached |
|
|
107
|
+
| `--host` | | `127.0.0.1` | WebSocket server bind host |
|
|
108
|
+
| `--port` | `-p` | `8765` | WebSocket server bind port |
|
|
109
|
+
| `--install-extension` | | `./fb_reels_extension` | Export the bundled extension to a local directory for browser loading |
|
|
110
|
+
| `--extension-path` | | — | Print absolute path to the bundled extension directory |
|
|
111
|
+
| `--help` | `-h` | — | Display help text and usage instructions |
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## 🐍 Python Library Usage
|
|
116
|
+
|
|
117
|
+
You can also import and use `fb-reels-scraper` programmatically in your own Python scripts:
|
|
118
|
+
|
|
119
|
+
```python
|
|
120
|
+
import asyncio
|
|
121
|
+
from fb_reels_scraper import scrape_fb_reels, start_server
|
|
122
|
+
|
|
123
|
+
async def main():
|
|
124
|
+
server = await start_server(host="127.0.0.1", port=8765)
|
|
125
|
+
try:
|
|
126
|
+
results = await scrape_fb_reels(
|
|
127
|
+
target_url="https://www.facebook.com/example/reels/",
|
|
128
|
+
max_scrolls=10,
|
|
129
|
+
cache_file="reels_cache.json"
|
|
130
|
+
)
|
|
131
|
+
print(f"Scraped {len(results)} reels!")
|
|
132
|
+
finally:
|
|
133
|
+
server.close()
|
|
134
|
+
await server.wait_closed()
|
|
135
|
+
|
|
136
|
+
asyncio.run(main())
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
---
|
|
140
|
+
|
|
141
|
+
## 📁 Repository Structure
|
|
142
|
+
|
|
143
|
+
```text
|
|
144
|
+
fb-reels-scraper/
|
|
145
|
+
├── pyproject.toml # Modern PEP 517/621 packaging metadata
|
|
146
|
+
├── MANIFEST.in # Manifest including non-Python extension assets
|
|
147
|
+
├── fb_reels_scraper/ # Core Python package
|
|
148
|
+
│ ├── __init__.py
|
|
149
|
+
│ ├── __main__.py # python -m fb_reels_scraper support
|
|
150
|
+
│ ├── cli.py # CLI entrypoint for fb-reels-scraper command
|
|
151
|
+
│ ├── scraper.py # WebSocket bridge & scraping logic
|
|
152
|
+
│ └── extension/ # Bundled Chrome Extension (Manifest V3)
|
|
153
|
+
│ ├── manifest.json
|
|
154
|
+
│ ├── content.js
|
|
155
|
+
│ ├── popup.html
|
|
156
|
+
│ └── popup.js
|
|
157
|
+
├── reels_cache.json # Cached results
|
|
158
|
+
└── README.md
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
## 💾 Cache Format
|
|
164
|
+
|
|
165
|
+
Descriptions are saved to `reels_cache.json` in UTF-8 format:
|
|
166
|
+
|
|
167
|
+
```json
|
|
168
|
+
{
|
|
169
|
+
"https://www.facebook.com/reel/123456789012345/": "Here is the full description and caption of the reel...",
|
|
170
|
+
"https://www.facebook.com/reel/987654321098765/": "[No description found]"
|
|
171
|
+
}
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## 🚀 Publishing to PyPI
|
|
177
|
+
|
|
178
|
+
This repository includes a GitHub Actions workflow ([.github/workflows/publish.yml](.github/workflows/publish.yml)) that automatically builds and publishes the package to PyPI whenever a new version tag is pushed.
|
|
179
|
+
|
|
180
|
+
### 1. Configure PyPI Authentication (One-time Setup)
|
|
181
|
+
You can authenticate via **Trusted Publishing (Recommended by PyPI)** or an **API Token**:
|
|
182
|
+
|
|
183
|
+
- **Option A: Trusted Publishing (OIDC - Recommended)**:
|
|
184
|
+
1. Go to your project on [PyPI](https://pypi.org/manage/project/fb-reels-scraper/settings/publishing/) (or add a pending publisher before your first release).
|
|
185
|
+
2. Add a new publisher with:
|
|
186
|
+
- Owner: your GitHub username or organization (e.g. `ishan-kh`)
|
|
187
|
+
- Repository: `fb-reels-scraper`
|
|
188
|
+
- Workflow name: `publish.yml`
|
|
189
|
+
- Environment name: `pypi`
|
|
190
|
+
|
|
191
|
+
- **Option B: PyPI API Token**:
|
|
192
|
+
1. Generate an API token on [PyPI Account Settings](https://pypi.org/manage/account/token/).
|
|
193
|
+
2. In your GitHub repository, navigate to **Settings** ➔ **Secrets and variables** ➔ **Actions**.
|
|
194
|
+
3. Create a secret named `PYPI_API_TOKEN` and paste your token value.
|
|
195
|
+
|
|
196
|
+
### 2. Release a New Version
|
|
197
|
+
When you are ready to publish an update:
|
|
198
|
+
|
|
199
|
+
1. Bump the version in `pyproject.toml` and `fb_reels_scraper/__init__.py`:
|
|
200
|
+
```python
|
|
201
|
+
# pyproject.toml
|
|
202
|
+
version = "1.0.1"
|
|
203
|
+
|
|
204
|
+
# fb_reels_scraper/__init__.py
|
|
205
|
+
__version__ = "1.0.1"
|
|
206
|
+
```
|
|
207
|
+
2. Commit the change, tag the release, and push to GitHub:
|
|
208
|
+
```bash
|
|
209
|
+
git commit -am "Bump version to 1.0.1"
|
|
210
|
+
git tag v1.0.1
|
|
211
|
+
git push origin main --tags
|
|
212
|
+
```
|
|
213
|
+
3. GitHub Actions will automatically trigger, build the wheel and source tarball, and publish them to PyPI.
|
|
214
|
+
|
|
215
|
+
---
|
|
216
|
+
|
|
217
|
+
## 🛡️ License
|
|
218
|
+
|
|
219
|
+
This project is licensed under the [MIT License](LICENSE).
|
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
# FB Reels Scraper Bridge 🎬
|
|
2
|
+
|
|
3
|
+
[](https://pypi.org/project/fb-reels-scraper/)
|
|
4
|
+
[](https://pypi.org/project/fb-reels-scraper/)
|
|
5
|
+
[](https://opensource.org/licenses/MIT)
|
|
6
|
+
|
|
7
|
+
A robust, two-stage Facebook Reels scraper that operates through your **existing browser session** via a lightweight Chrome/Edge extension and a local Python WebSocket bridge.
|
|
8
|
+
|
|
9
|
+
By utilizing your real browser session, you completely bypass login hurdles, 2FA prompts, fingerprinting, and bot-detection challenges without needing credentials or headless automation drivers.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## ✨ Features
|
|
14
|
+
|
|
15
|
+
- **No Bot Detection / Captchas**: Works directly inside your logged-in browser tab.
|
|
16
|
+
- **Two-Phase Architecture**:
|
|
17
|
+
1. **Phase 1 (Discovery & Harvest)**: Scrolls target profile or page reels feeds (`/reels/`) to collect reel permalinks.
|
|
18
|
+
2. **Phase 2 (Deep Extraction)**: Navigates to individual reel view pages, automatically clicks **"See more"** / **"...more"** to un-truncate descriptions, and extracts the full caption text.
|
|
19
|
+
- **Local Smart Caching (`reels_cache.json`)**: Automatically saves extracted reels on the fly. Re-running the script skips previously scraped URLs to save time and bandwidth.
|
|
20
|
+
- **Bundled Chrome Extension**: Extension files are packaged directly inside the package and can be exported with a single CLI command (`fb-reels-scraper --install-extension`).
|
|
21
|
+
- **Resilient WebSocket Bridge**: Gracefully handles page transitions, DOM reloads, and tab navigation events without throwing disconnect exceptions.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## 📦 Installation
|
|
26
|
+
|
|
27
|
+
### From PyPI
|
|
28
|
+
```bash
|
|
29
|
+
pip install fb-reels-scraper
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
### Or from source (Editable mode)
|
|
33
|
+
```bash
|
|
34
|
+
git clone https://github.com/ishan-kh/fb-reels-scraper.git
|
|
35
|
+
cd fb-reels-scraper
|
|
36
|
+
pip install -e .
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## 🚀 Quick Setup & Usage
|
|
42
|
+
|
|
43
|
+
### Step 1: Export & Load the Browser Extension
|
|
44
|
+
|
|
45
|
+
If installed via `pip`, you can export the bundled extension directory into your current working folder:
|
|
46
|
+
```bash
|
|
47
|
+
fb-reels-scraper --install-extension
|
|
48
|
+
```
|
|
49
|
+
*(Or check the internal extension path via `fb-reels-scraper --extension-path`)*
|
|
50
|
+
|
|
51
|
+
1. Open your Chromium browser (Chrome, Brave, Edge, etc.) and go to `chrome://extensions/` (or `edge://extensions/`).
|
|
52
|
+
2. Toggle on **Developer mode** in the top-right corner.
|
|
53
|
+
3. Click **Load unpacked** and select the exported `./fb_reels_extension` folder (or `./extension` if running from source).
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
### Step 2: Open Facebook & Run the CLI
|
|
58
|
+
|
|
59
|
+
1. Ensure you have an active Facebook tab open in your browser where you are logged in.
|
|
60
|
+
2. In your terminal, run the CLI tool:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
# Run with default fallback URL and scroll count
|
|
64
|
+
fb-reels-scraper
|
|
65
|
+
|
|
66
|
+
# Scrape a specific profile or page
|
|
67
|
+
fb-reels-scraper --url "https://www.facebook.com/<target_user_or_page>/reels/"
|
|
68
|
+
|
|
69
|
+
# Specify scroll count and custom cache file (short flags)
|
|
70
|
+
fb-reels-scraper -u "https://www.facebook.com/<target_user_or_page>/reels/" -s 25 -c my_reels.json
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## 🛠️ CLI Options Reference
|
|
76
|
+
|
|
77
|
+
| Flag | Short | Default | Description |
|
|
78
|
+
|---|---|---|---|
|
|
79
|
+
| `--url` | `-u` | `https://www.facebook.com/nadeemkhancyber/reels/` | Facebook reels page URL to scrape |
|
|
80
|
+
| `--scrolls` | `-s` | `20` | Number of scroll cycles on the grid to harvest links |
|
|
81
|
+
| `--cache-file` | `-c` | `reels_cache.json` | Path to JSON file where results are cached |
|
|
82
|
+
| `--host` | | `127.0.0.1` | WebSocket server bind host |
|
|
83
|
+
| `--port` | `-p` | `8765` | WebSocket server bind port |
|
|
84
|
+
| `--install-extension` | | `./fb_reels_extension` | Export the bundled extension to a local directory for browser loading |
|
|
85
|
+
| `--extension-path` | | — | Print absolute path to the bundled extension directory |
|
|
86
|
+
| `--help` | `-h` | — | Display help text and usage instructions |
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## 🐍 Python Library Usage
|
|
91
|
+
|
|
92
|
+
You can also import and use `fb-reels-scraper` programmatically in your own Python scripts:
|
|
93
|
+
|
|
94
|
+
```python
|
|
95
|
+
import asyncio
|
|
96
|
+
from fb_reels_scraper import scrape_fb_reels, start_server
|
|
97
|
+
|
|
98
|
+
async def main():
|
|
99
|
+
server = await start_server(host="127.0.0.1", port=8765)
|
|
100
|
+
try:
|
|
101
|
+
results = await scrape_fb_reels(
|
|
102
|
+
target_url="https://www.facebook.com/example/reels/",
|
|
103
|
+
max_scrolls=10,
|
|
104
|
+
cache_file="reels_cache.json"
|
|
105
|
+
)
|
|
106
|
+
print(f"Scraped {len(results)} reels!")
|
|
107
|
+
finally:
|
|
108
|
+
server.close()
|
|
109
|
+
await server.wait_closed()
|
|
110
|
+
|
|
111
|
+
asyncio.run(main())
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## 📁 Repository Structure
|
|
117
|
+
|
|
118
|
+
```text
|
|
119
|
+
fb-reels-scraper/
|
|
120
|
+
├── pyproject.toml # Modern PEP 517/621 packaging metadata
|
|
121
|
+
├── MANIFEST.in # Manifest including non-Python extension assets
|
|
122
|
+
├── fb_reels_scraper/ # Core Python package
|
|
123
|
+
│ ├── __init__.py
|
|
124
|
+
│ ├── __main__.py # python -m fb_reels_scraper support
|
|
125
|
+
│ ├── cli.py # CLI entrypoint for fb-reels-scraper command
|
|
126
|
+
│ ├── scraper.py # WebSocket bridge & scraping logic
|
|
127
|
+
│ └── extension/ # Bundled Chrome Extension (Manifest V3)
|
|
128
|
+
│ ├── manifest.json
|
|
129
|
+
│ ├── content.js
|
|
130
|
+
│ ├── popup.html
|
|
131
|
+
│ └── popup.js
|
|
132
|
+
├── reels_cache.json # Cached results
|
|
133
|
+
└── README.md
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## 💾 Cache Format
|
|
139
|
+
|
|
140
|
+
Descriptions are saved to `reels_cache.json` in UTF-8 format:
|
|
141
|
+
|
|
142
|
+
```json
|
|
143
|
+
{
|
|
144
|
+
"https://www.facebook.com/reel/123456789012345/": "Here is the full description and caption of the reel...",
|
|
145
|
+
"https://www.facebook.com/reel/987654321098765/": "[No description found]"
|
|
146
|
+
}
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
---
|
|
150
|
+
|
|
151
|
+
## 🚀 Publishing to PyPI
|
|
152
|
+
|
|
153
|
+
This repository includes a GitHub Actions workflow ([.github/workflows/publish.yml](.github/workflows/publish.yml)) that automatically builds and publishes the package to PyPI whenever a new version tag is pushed.
|
|
154
|
+
|
|
155
|
+
### 1. Configure PyPI Authentication (One-time Setup)
|
|
156
|
+
You can authenticate via **Trusted Publishing (Recommended by PyPI)** or an **API Token**:
|
|
157
|
+
|
|
158
|
+
- **Option A: Trusted Publishing (OIDC - Recommended)**:
|
|
159
|
+
1. Go to your project on [PyPI](https://pypi.org/manage/project/fb-reels-scraper/settings/publishing/) (or add a pending publisher before your first release).
|
|
160
|
+
2. Add a new publisher with:
|
|
161
|
+
- Owner: your GitHub username or organization (e.g. `ishan-kh`)
|
|
162
|
+
- Repository: `fb-reels-scraper`
|
|
163
|
+
- Workflow name: `publish.yml`
|
|
164
|
+
- Environment name: `pypi`
|
|
165
|
+
|
|
166
|
+
- **Option B: PyPI API Token**:
|
|
167
|
+
1. Generate an API token on [PyPI Account Settings](https://pypi.org/manage/account/token/).
|
|
168
|
+
2. In your GitHub repository, navigate to **Settings** ➔ **Secrets and variables** ➔ **Actions**.
|
|
169
|
+
3. Create a secret named `PYPI_API_TOKEN` and paste your token value.
|
|
170
|
+
|
|
171
|
+
### 2. Release a New Version
|
|
172
|
+
When you are ready to publish an update:
|
|
173
|
+
|
|
174
|
+
1. Bump the version in `pyproject.toml` and `fb_reels_scraper/__init__.py`:
|
|
175
|
+
```python
|
|
176
|
+
# pyproject.toml
|
|
177
|
+
version = "1.0.1"
|
|
178
|
+
|
|
179
|
+
# fb_reels_scraper/__init__.py
|
|
180
|
+
__version__ = "1.0.1"
|
|
181
|
+
```
|
|
182
|
+
2. Commit the change, tag the release, and push to GitHub:
|
|
183
|
+
```bash
|
|
184
|
+
git commit -am "Bump version to 1.0.1"
|
|
185
|
+
git tag v1.0.1
|
|
186
|
+
git push origin main --tags
|
|
187
|
+
```
|
|
188
|
+
3. GitHub Actions will automatically trigger, build the wheel and source tarball, and publish them to PyPI.
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## 🛡️ License
|
|
193
|
+
|
|
194
|
+
This project is licensed under the [MIT License](LICENSE).
|
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
(() => {
|
|
2
|
+
let socket = null;
|
|
3
|
+
let isConnected = false;
|
|
4
|
+
let reconnectTimer = null;
|
|
5
|
+
|
|
6
|
+
function updateStatus(status, connected = false) {
|
|
7
|
+
chrome.runtime.sendMessage({
|
|
8
|
+
type: "STATUS_UPDATE",
|
|
9
|
+
status: status,
|
|
10
|
+
connected: connected
|
|
11
|
+
}).catch(() => {});
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
// Collect reel links from the reels grid/listing page
|
|
15
|
+
function extractReelLinks() {
|
|
16
|
+
const urls = new Set();
|
|
17
|
+
const links = document.querySelectorAll("a[href*='/reel/']");
|
|
18
|
+
|
|
19
|
+
links.forEach(link => {
|
|
20
|
+
try {
|
|
21
|
+
const href = link.href;
|
|
22
|
+
if (!href) return;
|
|
23
|
+
const match = href.match(/https?:\/\/(?:www\.)?facebook\.com\/reel\/(\d+)/i);
|
|
24
|
+
if (match) {
|
|
25
|
+
urls.add(`https://www.facebook.com/reel/${match[1]}/`);
|
|
26
|
+
} else if (href.includes("/reel/")) {
|
|
27
|
+
// Fallback to cleaned URL without tracking params
|
|
28
|
+
const cleanUrl = href.split("?")[0].replace(/\/+$/, "") + "/";
|
|
29
|
+
urls.add(cleanUrl);
|
|
30
|
+
}
|
|
31
|
+
} catch (_) {}
|
|
32
|
+
});
|
|
33
|
+
|
|
34
|
+
return Array.from(urls);
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
// Extract full description from a single reel viewer page
|
|
38
|
+
async function extractSingleReelDescription() {
|
|
39
|
+
// 1. Expand "See more" or "more" buttons if present
|
|
40
|
+
const clickableButtons = Array.from(
|
|
41
|
+
document.querySelectorAll('div[role="button"], span[role="button"], a[role="button"]')
|
|
42
|
+
);
|
|
43
|
+
for (const btn of clickableButtons) {
|
|
44
|
+
const text = (btn.innerText || "").trim().toLowerCase();
|
|
45
|
+
if (text === "see more" || text === "… more" || text === "...more" || text === "more") {
|
|
46
|
+
try {
|
|
47
|
+
btn.click();
|
|
48
|
+
} catch (_) {}
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
// Wait a brief moment for dynamic text expansion
|
|
53
|
+
await new Promise(res => setTimeout(res, 500));
|
|
54
|
+
|
|
55
|
+
// 2. Select potential description containers
|
|
56
|
+
// Facebook reel captions are typically rendered in dir="auto" elements or within data-ad-preview="message"
|
|
57
|
+
const selectors = [
|
|
58
|
+
'div[data-ad-preview="message"]',
|
|
59
|
+
'div[dir="auto"]',
|
|
60
|
+
'span[dir="auto"]'
|
|
61
|
+
];
|
|
62
|
+
|
|
63
|
+
const blacklistedPrefixes = [
|
|
64
|
+
"like", "comment", "share", "follow", "remix", "audio", "reels",
|
|
65
|
+
"watch more reels", "send in whatsapp", "save video", "copy link",
|
|
66
|
+
"original audio", "suggested for you"
|
|
67
|
+
];
|
|
68
|
+
|
|
69
|
+
const candidates = [];
|
|
70
|
+
|
|
71
|
+
document.querySelectorAll(selectors.join(", ")).forEach(el => {
|
|
72
|
+
// Ignore if element is inside action buttons or navigation header
|
|
73
|
+
if (el.closest('header, nav, ul, [role="navigation"]')) return;
|
|
74
|
+
|
|
75
|
+
const txt = (el.innerText || "").trim();
|
|
76
|
+
if (!txt || txt.length < 5) return;
|
|
77
|
+
|
|
78
|
+
const lower = txt.toLowerCase();
|
|
79
|
+
// Skip common UI button strings and view counters
|
|
80
|
+
const isBlacklisted = blacklistedPrefixes.some(prefix => lower.startsWith(prefix) || lower === prefix);
|
|
81
|
+
if (isBlacklisted) return;
|
|
82
|
+
|
|
83
|
+
// Skip pure numeric / timestamp strings like "2d", "1.2K views"
|
|
84
|
+
if (/^(\d+(\.\d+)?[KM]?\s*(views|likes|shares|comments)?|\d+[smhdw])$/i.test(lower)) return;
|
|
85
|
+
|
|
86
|
+
candidates.push(txt);
|
|
87
|
+
});
|
|
88
|
+
|
|
89
|
+
// Remove duplicates while keeping order
|
|
90
|
+
const uniqueCandidates = Array.from(new Set(candidates));
|
|
91
|
+
|
|
92
|
+
// Longest element text block in the reel info column is usually the full description
|
|
93
|
+
uniqueCandidates.sort((a, b) => b.length - a.length);
|
|
94
|
+
|
|
95
|
+
return uniqueCandidates.length > 0 ? uniqueCandidates[0] : "";
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
function connect() {
|
|
99
|
+
if (socket && (socket.readyState === WebSocket.OPEN || socket.readyState === WebSocket.CONNECTING)) {
|
|
100
|
+
return;
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
try {
|
|
104
|
+
socket = new WebSocket("ws://127.0.0.1:8765");
|
|
105
|
+
|
|
106
|
+
socket.onopen = () => {
|
|
107
|
+
isConnected = true;
|
|
108
|
+
updateStatus("Connected to Python scraper", true);
|
|
109
|
+
socket.send(JSON.stringify({
|
|
110
|
+
action: "handshake",
|
|
111
|
+
url: window.location.href,
|
|
112
|
+
title: document.title
|
|
113
|
+
}));
|
|
114
|
+
};
|
|
115
|
+
|
|
116
|
+
socket.onmessage = async (event) => {
|
|
117
|
+
try {
|
|
118
|
+
const message = JSON.parse(event.data);
|
|
119
|
+
|
|
120
|
+
if (message.action === "scroll_and_collect_links") {
|
|
121
|
+
const reelUrls = extractReelLinks();
|
|
122
|
+
|
|
123
|
+
// Scroll down to trigger Facebook lazy-loading more reels
|
|
124
|
+
window.scrollTo(0, document.body.scrollHeight);
|
|
125
|
+
|
|
126
|
+
socket.send(JSON.stringify({
|
|
127
|
+
action: "collect_links_result",
|
|
128
|
+
urls: reelUrls,
|
|
129
|
+
currentScroll: message.scrollIndex,
|
|
130
|
+
maxScrolls: message.maxScrolls,
|
|
131
|
+
url: window.location.href
|
|
132
|
+
}));
|
|
133
|
+
} else if (message.action === "navigate") {
|
|
134
|
+
if (message.url && window.location.href !== message.url) {
|
|
135
|
+
window.location.href = message.url;
|
|
136
|
+
}
|
|
137
|
+
} else if (message.action === "extract_reel_description") {
|
|
138
|
+
const description = await extractSingleReelDescription();
|
|
139
|
+
socket.send(JSON.stringify({
|
|
140
|
+
action: "extract_reel_result",
|
|
141
|
+
url: window.location.href,
|
|
142
|
+
requestedUrl: message.url,
|
|
143
|
+
description: description
|
|
144
|
+
}));
|
|
145
|
+
}
|
|
146
|
+
} catch (err) {
|
|
147
|
+
console.error("Error processing websocket message:", err);
|
|
148
|
+
}
|
|
149
|
+
};
|
|
150
|
+
|
|
151
|
+
socket.onclose = () => {
|
|
152
|
+
isConnected = false;
|
|
153
|
+
updateStatus("Disconnected (reconnecting...)", false);
|
|
154
|
+
scheduleReconnect();
|
|
155
|
+
};
|
|
156
|
+
|
|
157
|
+
socket.onerror = (err) => {
|
|
158
|
+
isConnected = false;
|
|
159
|
+
updateStatus("Connection error (check if python script is running)", false);
|
|
160
|
+
try { socket.close(); } catch (_) {}
|
|
161
|
+
};
|
|
162
|
+
} catch (e) {
|
|
163
|
+
scheduleReconnect();
|
|
164
|
+
}
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
function scheduleReconnect() {
|
|
168
|
+
if (reconnectTimer) clearTimeout(reconnectTimer);
|
|
169
|
+
reconnectTimer = setTimeout(connect, 3000);
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
// Connect on Facebook pages
|
|
173
|
+
connect();
|
|
174
|
+
|
|
175
|
+
// Listen for queries from popup
|
|
176
|
+
chrome.runtime.onMessage.addListener((msg, sender, sendResponse) => {
|
|
177
|
+
if (msg.type === "GET_STATUS") {
|
|
178
|
+
sendResponse({
|
|
179
|
+
connected: isConnected,
|
|
180
|
+
url: window.location.href
|
|
181
|
+
});
|
|
182
|
+
}
|
|
183
|
+
return true;
|
|
184
|
+
});
|
|
185
|
+
})();
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
<!DOCTYPE html>
|
|
2
|
+
<html>
|
|
3
|
+
<head>
|
|
4
|
+
<meta charset="utf-8">
|
|
5
|
+
<style>
|
|
6
|
+
body {
|
|
7
|
+
width: 250px;
|
|
8
|
+
padding: 12px;
|
|
9
|
+
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
|
10
|
+
font-size: 13px;
|
|
11
|
+
margin: 0;
|
|
12
|
+
color: #1c1e21;
|
|
13
|
+
}
|
|
14
|
+
.header {
|
|
15
|
+
font-weight: 600;
|
|
16
|
+
font-size: 14px;
|
|
17
|
+
margin-bottom: 8px;
|
|
18
|
+
color: #1877f2;
|
|
19
|
+
}
|
|
20
|
+
.badge {
|
|
21
|
+
display: inline-block;
|
|
22
|
+
padding: 4px 8px;
|
|
23
|
+
border-radius: 12px;
|
|
24
|
+
font-size: 11px;
|
|
25
|
+
font-weight: 600;
|
|
26
|
+
margin-bottom: 8px;
|
|
27
|
+
}
|
|
28
|
+
.connected {
|
|
29
|
+
background-color: #e7f3ff;
|
|
30
|
+
color: #1877f2;
|
|
31
|
+
}
|
|
32
|
+
.disconnected {
|
|
33
|
+
background-color: #fce8e6;
|
|
34
|
+
color: #c5221f;
|
|
35
|
+
}
|
|
36
|
+
.status-text {
|
|
37
|
+
color: #65676b;
|
|
38
|
+
line-height: 1.4;
|
|
39
|
+
font-size: 12px;
|
|
40
|
+
}
|
|
41
|
+
</style>
|
|
42
|
+
</head>
|
|
43
|
+
<body>
|
|
44
|
+
<div class="header">FB Reels Scraper Bridge</div>
|
|
45
|
+
<div id="badge" class="badge disconnected">Checking status...</div>
|
|
46
|
+
<div id="status" class="status-text">
|
|
47
|
+
Make sure you have a Facebook tab open and run <code>python fb-reels-scraper.py</code>.
|
|
48
|
+
</div>
|
|
49
|
+
<script src="popup.js"></script>
|
|
50
|
+
</body>
|
|
51
|
+
</html>
|