bmad-plus 0.4.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/README.md +12 -56
- package/osint-agent-package/skills/bmad-osint-investigate/osint/SKILL.md +452 -452
- package/osint-agent-package/skills/bmad-osint-investigate/osint/assets/dossier-template.md +116 -116
- package/osint-agent-package/skills/bmad-osint-investigate/osint/references/content-extraction.md +100 -100
- package/osint-agent-package/skills/bmad-osint-investigate/osint/references/platforms.md +130 -130
- package/osint-agent-package/skills/bmad-osint-investigate/osint/references/psychoprofile.md +69 -69
- package/osint-agent-package/skills/bmad-osint-investigate/osint/references/tools.md +281 -281
- package/osint-agent-package/skills/bmad-osint-investigate/osint/scripts/mcp-client.py +136 -136
- package/package.json +1 -1
- package/readme-international/README.de.md +1 -1
- package/readme-international/README.es.md +1 -1
- package/readme-international/README.fr.md +1 -1
- package/tools/cli/commands/install.js +74 -46
- package/tools/cli/i18n.js +501 -0
- package/oveanet-pack/animated-website/DEPLOYMENT.md +0 -104
- package/oveanet-pack/animated-website/README.md +0 -63
- package/oveanet-pack/animated-website/agent/animated-website-agent.md +0 -325
- package/oveanet-pack/animated-website/agent.yaml +0 -63
- package/oveanet-pack/animated-website/templates/animated-website-workflow.md +0 -55
- package/oveanet-pack/seo-audit-360/DEPLOYMENT.md +0 -115
- package/oveanet-pack/seo-audit-360/README.md +0 -66
- package/oveanet-pack/seo-audit-360/SKILL.md +0 -171
- package/oveanet-pack/seo-audit-360/agent/seo-chief.md +0 -294
- package/oveanet-pack/seo-audit-360/agent/seo-judge.md +0 -241
- package/oveanet-pack/seo-audit-360/agent/seo-scout.md +0 -171
- package/oveanet-pack/seo-audit-360/agent.yaml +0 -70
- package/oveanet-pack/seo-audit-360/checklist.md +0 -140
- package/oveanet-pack/seo-audit-360/extensions/google-analytics/EXTENSION.md +0 -79
- package/oveanet-pack/seo-audit-360/extensions/google-analytics/ga4_client.py +0 -200
- package/oveanet-pack/seo-audit-360/extensions/google-analytics/requirements.txt +0 -4
- package/oveanet-pack/seo-audit-360/extensions/google-search-console/EXTENSION.md +0 -109
- package/oveanet-pack/seo-audit-360/extensions/google-search-console/gsc_client.py +0 -186
- package/oveanet-pack/seo-audit-360/extensions/google-search-console/requirements.txt +0 -4
- package/oveanet-pack/seo-audit-360/hooks/seo-check.sh +0 -95
- package/oveanet-pack/seo-audit-360/pagespeed-playbook.md +0 -320
- package/oveanet-pack/seo-audit-360/ref/audit-schema.json +0 -187
- package/oveanet-pack/seo-audit-360/ref/cwv-thresholds.md +0 -87
- package/oveanet-pack/seo-audit-360/ref/eeat-criteria.md +0 -123
- package/oveanet-pack/seo-audit-360/ref/geo-signals.md +0 -167
- package/oveanet-pack/seo-audit-360/ref/hreflang-rules.md +0 -153
- package/oveanet-pack/seo-audit-360/ref/quality-gates.md +0 -133
- package/oveanet-pack/seo-audit-360/ref/schema-catalog.md +0 -91
- package/oveanet-pack/seo-audit-360/ref/schema-templates.json +0 -356
- package/oveanet-pack/seo-audit-360/requirements.txt +0 -14
- package/oveanet-pack/seo-audit-360/scripts/__pycache__/seo_crawl.cpython-314.pyc +0 -0
- package/oveanet-pack/seo-audit-360/scripts/__pycache__/seo_parse.cpython-314.pyc +0 -0
- package/oveanet-pack/seo-audit-360/scripts/install.ps1 +0 -53
- package/oveanet-pack/seo-audit-360/scripts/install.sh +0 -48
- package/oveanet-pack/seo-audit-360/scripts/seo_apis.py +0 -464
- package/oveanet-pack/seo-audit-360/scripts/seo_crawl.py +0 -282
- package/oveanet-pack/seo-audit-360/scripts/seo_fetch.py +0 -231
- package/oveanet-pack/seo-audit-360/scripts/seo_parse.py +0 -255
- package/oveanet-pack/seo-audit-360/scripts/seo_report.py +0 -403
- package/oveanet-pack/seo-audit-360/scripts/seo_screenshot.py +0 -202
- package/oveanet-pack/seo-audit-360/templates/seo-audit-workflow.md +0 -241
- package/oveanet-pack/seo-audit-360/tests/__pycache__/test_crawl.cpython-314-pytest-9.0.2.pyc +0 -0
- package/oveanet-pack/seo-audit-360/tests/__pycache__/test_parse.cpython-314-pytest-9.0.2.pyc +0 -0
- package/oveanet-pack/seo-audit-360/tests/fixtures/sample_page.html +0 -62
- package/oveanet-pack/seo-audit-360/tests/test_apis.py +0 -75
- package/oveanet-pack/seo-audit-360/tests/test_crawl.py +0 -121
- package/oveanet-pack/seo-audit-360/tests/test_fetch.py +0 -70
- package/oveanet-pack/seo-audit-360/tests/test_parse.py +0 -184
- package/oveanet-pack/universal-backup/DEPLOYMENT.md +0 -80
- package/oveanet-pack/universal-backup/README.md +0 -58
- package/oveanet-pack/universal-backup/agent/backup-agent.md +0 -71
- package/oveanet-pack/universal-backup/agent.yaml +0 -45
- package/oveanet-pack/universal-backup/templates/backup-workflow.md +0 -51
|
@@ -1,281 +1,281 @@
|
|
|
1
|
-
# OSINT Toolkit Reference
|
|
2
|
-
|
|
3
|
-
## Table of Contents
|
|
4
|
-
1. [Search Tools](#search-tools)
|
|
5
|
-
2. [Scraping Tools](#scraping-tools)
|
|
6
|
-
3. [Apify Actor Runner](#apify-actor-runner-embedded)
|
|
7
|
-
4. [Apify Actor Catalog (55+)](#apify-actor-catalog)
|
|
8
|
-
5. [Actor Discovery](#actor-discovery)
|
|
9
|
-
6. [Shortcuts (apify.sh)](#shortcuts)
|
|
10
|
-
7. [Telegram](#telegram)
|
|
11
|
-
8. [Apify Troubleshooting](#apify-troubleshooting)
|
|
12
|
-
9. [Cost Reference](#cost-reference)
|
|
13
|
-
|
|
14
|
-
---
|
|
15
|
-
|
|
16
|
-
## Search Tools
|
|
17
|
-
|
|
18
|
-
### Brave Search (web_search built-in)
|
|
19
|
-
- No setup, FREE plan 2000 queries/month, rate limit 1/sec
|
|
20
|
-
- Best for: general web, articles, profile discovery
|
|
21
|
-
- Weakness: rate limited, no geo-targeting
|
|
22
|
-
|
|
23
|
-
### Jina s.reader (scripts/jina.sh search)
|
|
24
|
-
- Top 10 results as full markdown
|
|
25
|
-
- Best for: broad search with content extraction
|
|
26
|
-
- Weakness: can return wrong person on common names
|
|
27
|
-
|
|
28
|
-
### Parallel AI (scripts/parallel.sh search)
|
|
29
|
-
- AI-powered search with reasoning and citations
|
|
30
|
-
- Best for: social media discovery, cross-referencing
|
|
31
|
-
- Weakness: sometimes returns irrelevant AI-reasoned results
|
|
32
|
-
|
|
33
|
-
### Bright Data (scripts/brightdata.sh search)
|
|
34
|
-
- Google/Bing/Yandex with geo-targeting, bypasses CAPTCHA
|
|
35
|
-
- search-geo: `brightdata.sh search-geo ru "query"` (search as if from Russia)
|
|
36
|
-
- search-yandex: `brightdata.sh search-yandex "query"` (better for Russian targets)
|
|
37
|
-
- Batch: up to 10 queries
|
|
38
|
-
|
|
39
|
-
## Scraping Tools
|
|
40
|
-
|
|
41
|
-
### web_fetch (built-in)
|
|
42
|
-
- Basic URL → markdown. First attempt, fallback to better tools
|
|
43
|
-
- Fails on: LinkedIn, Facebook, Cloudflare
|
|
44
|
-
|
|
45
|
-
### Jina r.reader (scripts/jina.sh read)
|
|
46
|
-
- Clean markdown, faster than web_fetch
|
|
47
|
-
- Fails on: LinkedIn authwall, Facebook authwall, Instagram JS
|
|
48
|
-
|
|
49
|
-
### Bright Data (scripts/brightdata.sh scrape)
|
|
50
|
-
- **Bypasses EVERYTHING**: bot detection, CAPTCHA, authwall, Cloudflare
|
|
51
|
-
- The nuclear option. Use when others fail.
|
|
52
|
-
- Batch: up to 10 URLs via scrape-batch
|
|
53
|
-
|
|
54
|
-
---
|
|
55
|
-
|
|
56
|
-
## Apify Actor Runner (embedded)
|
|
57
|
-
|
|
58
|
-
`run_actor.js` — universal script that runs ANY Apify actor. Embedded from
|
|
59
|
-
[apify/agent-skills](https://github.com/apify/agent-skills) v1.3.0.
|
|
60
|
-
|
|
61
|
-
**Requirements:** Node.js 18+ (for global fetch), APIFY_API_TOKEN or APIFY_TOKEN env var.
|
|
62
|
-
|
|
63
|
-
### Via bash wrapper (recommended)
|
|
64
|
-
|
|
65
|
-
```bash
|
|
66
|
-
# Quick answer — top 5 results in chat
|
|
67
|
-
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT'
|
|
68
|
-
|
|
69
|
-
# Export to CSV
|
|
70
|
-
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.csv --format csv
|
|
71
|
-
|
|
72
|
-
# Export to JSON
|
|
73
|
-
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.json --format json
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
### Direct node call
|
|
77
|
-
|
|
78
|
-
```bash
|
|
79
|
-
APIFY_TOKEN=$APIFY_API_TOKEN node scripts/run_actor.js \
|
|
80
|
-
--actor "ACTOR_ID" --input 'JSON_INPUT'
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
---
|
|
84
|
-
|
|
85
|
-
## Apify Actor Catalog
|
|
86
|
-
|
|
87
|
-
### Instagram (12 actors)
|
|
88
|
-
|
|
89
|
-
| Actor ID | Best For | OSINT Use |
|
|
90
|
-
|----------|----------|-----------|
|
|
91
|
-
| `apify/instagram-profile-scraper` | Profile data, follower counts, bio | **Primary** — bio, location, links |
|
|
92
|
-
| `apify/instagram-post-scraper` | Individual post details, engagement | Content analysis |
|
|
93
|
-
| `apify/instagram-comment-scraper` | Comment extraction | Sentiment, social graph |
|
|
94
|
-
| `apify/instagram-hashtag-scraper` | Hashtag content, trending topics | Find target by hashtag |
|
|
95
|
-
| `apify/instagram-hashtag-stats` | Hashtag performance metrics | Audience analysis |
|
|
96
|
-
| `apify/instagram-reel-scraper` | Reels content and metrics | Video content style |
|
|
97
|
-
| `apify/instagram-search-scraper` | Search users, places, hashtags | Discovery |
|
|
98
|
-
| `apify/instagram-tagged-scraper` | Posts tagged with specific accounts | **Who tags them** = social graph |
|
|
99
|
-
| `apify/instagram-followers-count-scraper` | Follower count tracking | Influence assessment |
|
|
100
|
-
| `apify/instagram-scraper` | Comprehensive Instagram data | Full profile deep dive |
|
|
101
|
-
| `apify/instagram-api-scraper` | API-based Instagram access | Fallback scraper |
|
|
102
|
-
| `apify/export-instagram-comments-posts` | Bulk comment/post export | Large-scale extraction |
|
|
103
|
-
|
|
104
|
-
### Facebook (14 actors)
|
|
105
|
-
|
|
106
|
-
| Actor ID | Best For | OSINT Use |
|
|
107
|
-
|----------|----------|-----------|
|
|
108
|
-
| `apify/facebook-pages-scraper` | Page data, metrics, contact info | **Primary** for public pages |
|
|
109
|
-
| `apify/facebook-page-contact-information` | Emails, phones, addresses | Contact enrichment |
|
|
110
|
-
| `apify/facebook-posts-scraper` | Post content and engagement | Writing style analysis |
|
|
111
|
-
| `apify/facebook-comments-scraper` | Comment extraction | Sentiment analysis |
|
|
112
|
-
| `apify/facebook-likes-scraper` | Reaction analysis | Audience mapping |
|
|
113
|
-
| `apify/facebook-reviews-scraper` | Page reviews | Reputation analysis |
|
|
114
|
-
| `apify/facebook-groups-scraper` | Group content and members | Community membership |
|
|
115
|
-
| `apify/facebook-events-scraper` | Event data | Activity/interests |
|
|
116
|
-
| `apify/facebook-ads-scraper` | Ad creative and targeting | Competitor intel |
|
|
117
|
-
| `apify/facebook-search-scraper` | Search results | Discovery |
|
|
118
|
-
| `apify/facebook-reels-scraper` | Reels content | Video style |
|
|
119
|
-
| `apify/facebook-photos-scraper` | Photo extraction | Visual profile |
|
|
120
|
-
| `apify/facebook-marketplace-scraper` | Marketplace listings | Business activity |
|
|
121
|
-
| `apify/facebook-followers-following-scraper` | Follower/following lists | Social graph |
|
|
122
|
-
|
|
123
|
-
**Note:** Facebook personal profiles require Bright Data. These actors work with public Pages/Groups/Marketplace.
|
|
124
|
-
|
|
125
|
-
### TikTok (14 actors)
|
|
126
|
-
|
|
127
|
-
| Actor ID | Best For | OSINT Use |
|
|
128
|
-
|----------|----------|-----------|
|
|
129
|
-
| `clockworks/tiktok-scraper` | Comprehensive TikTok data | Full platform deep dive |
|
|
130
|
-
| `clockworks/free-tiktok-scraper` | Free TikTok extraction | **Budget fallback** |
|
|
131
|
-
| `clockworks/tiktok-profile-scraper` | Profile data, bio, stats | **Primary** — bio, follower data |
|
|
132
|
-
| `clockworks/tiktok-video-scraper` | Video details and metrics | Content analysis |
|
|
133
|
-
| `clockworks/tiktok-comments-scraper` | Comment extraction | Sentiment, audience |
|
|
134
|
-
| `clockworks/tiktok-followers-scraper` | Follower lists | Social graph |
|
|
135
|
-
| `clockworks/tiktok-user-search-scraper` | Find users by keywords | **Discovery** by name |
|
|
136
|
-
| `clockworks/tiktok-hashtag-scraper` | Hashtag content | Topic analysis |
|
|
137
|
-
| `clockworks/tiktok-sound-scraper` | Trending sounds | Cultural markers |
|
|
138
|
-
| `clockworks/tiktok-ads-scraper` | Ad content | Competitor ads |
|
|
139
|
-
| `clockworks/tiktok-discover-scraper` | Discover page content | Trending content |
|
|
140
|
-
| `clockworks/tiktok-explore-scraper` | Explore content | Algorithm analysis |
|
|
141
|
-
| `clockworks/tiktok-trends-scraper` | Trending content | Trend tracking |
|
|
142
|
-
| `clockworks/tiktok-live-scraper` | Live stream data | Real-time activity |
|
|
143
|
-
|
|
144
|
-
### YouTube (5 actors)
|
|
145
|
-
|
|
146
|
-
| Actor ID | Best For | OSINT Use |
|
|
147
|
-
|----------|----------|-----------|
|
|
148
|
-
| `streamers/youtube-scraper` | Video data and metrics | Content analysis |
|
|
149
|
-
| `streamers/youtube-channel-scraper` | Channel info, subscriber data | **Primary** — channel metadata |
|
|
150
|
-
| `streamers/youtube-comments-scraper` | Comment extraction | Audience sentiment |
|
|
151
|
-
| `streamers/youtube-shorts-scraper` | Shorts content | Short-form style |
|
|
152
|
-
| `streamers/youtube-video-scraper-by-hashtag` | Videos by hashtag | Topic discovery |
|
|
153
|
-
|
|
154
|
-
### Google Maps (4 actors)
|
|
155
|
-
|
|
156
|
-
| Actor ID | Best For | OSINT Use |
|
|
157
|
-
|----------|----------|-----------|
|
|
158
|
-
| `compass/crawler-google-places` | Business listings, ratings, contact | **Primary** — verify business owners |
|
|
159
|
-
| `compass/google-maps-extractor` | Detailed business data | Deep business intel |
|
|
160
|
-
| `compass/Google-Maps-Reviews-Scraper` | Review extraction | Writing style in responses |
|
|
161
|
-
| `poidata/google-maps-email-extractor` | Email discovery from listings | Contact enrichment |
|
|
162
|
-
|
|
163
|
-
### Other Platforms (6 actors)
|
|
164
|
-
|
|
165
|
-
| Actor ID | Best For | OSINT Use |
|
|
166
|
-
|----------|----------|-----------|
|
|
167
|
-
| `apify/google-search-scraper` | Google search results | Structured search |
|
|
168
|
-
| `apify/google-trends-scraper` | Google Trends data | Interest tracking |
|
|
169
|
-
| `voyager/booking-scraper` | Booking.com hotel data | Travel/business intel |
|
|
170
|
-
| `voyager/booking-reviews-scraper` | Booking.com reviews | Reputation |
|
|
171
|
-
| `maxcopell/tripadvisor-reviews` | TripAdvisor reviews | Hospitality intel |
|
|
172
|
-
| `vdrmota/contact-info-scraper` | Contact enrichment from URLs | **Emails/phones from any site** |
|
|
173
|
-
|
|
174
|
-
### LinkedIn
|
|
175
|
-
|
|
176
|
-
LinkedIn actors are volatile on Apify. Current primary:
|
|
177
|
-
- `supreme_coder~linkedin-profile-scraper` via `apify.sh linkedin` → $0.005/profile
|
|
178
|
-
- Fallback: `brightdata.sh scrape` (always works, higher cost)
|
|
179
|
-
- If primary fails: `bash scripts/apify.sh store-search "linkedin scraper"`
|
|
180
|
-
|
|
181
|
-
---
|
|
182
|
-
|
|
183
|
-
## Actor Selection by OSINT Task
|
|
184
|
-
|
|
185
|
-
| OSINT Phase | What You Need | Primary Actors |
|
|
186
|
-
|-------------|---------------|----------------|
|
|
187
|
-
| **Profile discovery** | Find accounts | `apify/instagram-search-scraper`, `clockworks/tiktok-user-search-scraper`, `apify/facebook-search-scraper` |
|
|
188
|
-
| **Profile deep dive** | Extract bio/stats | `apify/instagram-profile-scraper`, `clockworks/tiktok-profile-scraper`, `streamers/youtube-channel-scraper` |
|
|
189
|
-
| **Social graph** | Who they interact with | `apify/instagram-tagged-scraper`, `apify/instagram-comment-scraper`, `clockworks/tiktok-followers-scraper`, `apify/facebook-followers-following-scraper` |
|
|
190
|
-
| **Content analysis** | Posts, videos, style | `apify/instagram-post-scraper`, `clockworks/tiktok-video-scraper`, `streamers/youtube-scraper` |
|
|
191
|
-
| **Contact enrichment** | Emails, phones | `vdrmota/contact-info-scraper`, `apify/facebook-page-contact-information`, `poidata/google-maps-email-extractor` |
|
|
192
|
-
| **Business verification** | Company, location | `compass/crawler-google-places`, `compass/google-maps-extractor` |
|
|
193
|
-
| **Psychoprofile signals** | Sentiment, style | `apify/instagram-comment-scraper`, `clockworks/tiktok-comments-scraper`, `streamers/youtube-comments-scraper` |
|
|
194
|
-
|
|
195
|
-
### Multi-Actor Workflows (OSINT-specific)
|
|
196
|
-
|
|
197
|
-
| Workflow | Step 1 → | Step 2 |
|
|
198
|
-
|----------|----------|--------|
|
|
199
|
-
| **Full Instagram** | `apify/instagram-profile-scraper` → | `apify/instagram-tagged-scraper` + `apify/instagram-comment-scraper` |
|
|
200
|
-
| **Business owner** | `compass/crawler-google-places` → | `vdrmota/contact-info-scraper` on website |
|
|
201
|
-
| **Content creator** | `streamers/youtube-channel-scraper` → | `streamers/youtube-comments-scraper` |
|
|
202
|
-
| **TikTok target** | `clockworks/tiktok-user-search-scraper` → | `clockworks/tiktok-profile-scraper` + `clockworks/tiktok-comments-scraper` |
|
|
203
|
-
| **Facebook page** | `apify/facebook-pages-scraper` → | `apify/facebook-posts-scraper` + `apify/facebook-page-contact-information` |
|
|
204
|
-
|
|
205
|
-
---
|
|
206
|
-
|
|
207
|
-
## Actor Discovery
|
|
208
|
-
|
|
209
|
-
When none of the 55+ actors fit, search the Apify Store dynamically:
|
|
210
|
-
|
|
211
|
-
```bash
|
|
212
|
-
# Via mcpc CLI (if installed)
|
|
213
|
-
APIFY_TOKEN=$APIFY_API_TOKEN mcpc --json mcp.apify.com \
|
|
214
|
-
--header "Authorization: Bearer $APIFY_TOKEN" \
|
|
215
|
-
tools-call search-actors keywords:="SEARCH_KEYWORDS" limit:=10
|
|
216
|
-
|
|
217
|
-
# Via Apify API directly
|
|
218
|
-
bash scripts/apify.sh store-search "keyword1 keyword2"
|
|
219
|
-
```
|
|
220
|
-
|
|
221
|
-
Tips:
|
|
222
|
-
- Use 1-3 simple keywords (e.g., "LinkedIn profiles", "Twitter scraper")
|
|
223
|
-
- Sort by: PAY_PER_EVENT (works with small budgets), high rating, high run count
|
|
224
|
-
- Actors on Apify are volatile — always have a Bright Data fallback
|
|
225
|
-
|
|
226
|
-
---
|
|
227
|
-
|
|
228
|
-
## Shortcuts
|
|
229
|
-
|
|
230
|
-
`apify.sh` provides quick shortcuts for common operations:
|
|
231
|
-
|
|
232
|
-
```bash
|
|
233
|
-
bash scripts/apify.sh linkedin "https://linkedin.com/in/..." # $0.005
|
|
234
|
-
bash scripts/apify.sh instagram "handle" # free tier
|
|
235
|
-
bash scripts/apify.sh run <actor_id> '<json>' # any actor
|
|
236
|
-
bash scripts/apify.sh results <run_id> # get results
|
|
237
|
-
bash scripts/apify.sh run-status <run_id> # check status
|
|
238
|
-
bash scripts/apify.sh store-search "query" # find actors
|
|
239
|
-
```
|
|
240
|
-
|
|
241
|
-
For anything beyond shortcuts, use `run-actor.sh` — it handles polling,
|
|
242
|
-
timeout, and CSV/JSON export automatically.
|
|
243
|
-
|
|
244
|
-
---
|
|
245
|
-
|
|
246
|
-
## Telegram
|
|
247
|
-
|
|
248
|
-
- Public channel posts: `web_fetch https://t.me/s/{channel}` (free, last ~20 posts)
|
|
249
|
-
- Profile bio: `web_fetch https://t.me/{username}` (free, shows bio/title)
|
|
250
|
-
- Private messages: tg.py (Telegram skill) or mcporter telegram tools
|
|
251
|
-
|
|
252
|
-
---
|
|
253
|
-
|
|
254
|
-
## Apify Troubleshooting
|
|
255
|
-
|
|
256
|
-
When an Apify actor fails:
|
|
257
|
-
1. Check if it blocks API on free plan ("run through UI only" error)
|
|
258
|
-
2. Check input field names (profileUrls vs urls vs user_name — varies per actor)
|
|
259
|
-
3. Check run status: `bash scripts/apify.sh run-status <run_id>`
|
|
260
|
-
4. Search for alternatives: `bash scripts/apify.sh store-search "platform scraper"`
|
|
261
|
-
5. Try the free-tier variant (e.g., `clockworks/free-tiktok-scraper`)
|
|
262
|
-
6. Fall back to Bright Data: `bash scripts/brightdata.sh scrape "<url>"`
|
|
263
|
-
|
|
264
|
-
---
|
|
265
|
-
|
|
266
|
-
## Cost Reference
|
|
267
|
-
|
|
268
|
-
| Tool | Cost | Notes |
|
|
269
|
-
|------|------|-------|
|
|
270
|
-
| LinkedIn (Apify supreme_coder) | $0.005/profile | |
|
|
271
|
-
| Instagram (Apify) | free tier | Limited runs |
|
|
272
|
-
| TikTok (Apify clockworks) | $0.01-0.05/profile | free-tiktok-scraper = free |
|
|
273
|
-
| YouTube (Apify streamers) | $0.01/channel | |
|
|
274
|
-
| Google Maps (Apify compass) | $0.01/listing | |
|
|
275
|
-
| Contact enrichment | $0.01/URL | |
|
|
276
|
-
| Facebook pages (Apify) | $0.01/page | Personal profiles = Bright Data |
|
|
277
|
-
| Facebook personal (Bright Data) | per-request | Check account balance |
|
|
278
|
-
| Jina | free tier with key | |
|
|
279
|
-
| Parallel | free tier with key | |
|
|
280
|
-
| Brave | free (2000/month) | |
|
|
281
|
-
| **Budget rule** | **≤$0.50 without asking** | **>$0.50 ask user** |
|
|
1
|
+
# OSINT Toolkit Reference
|
|
2
|
+
|
|
3
|
+
## Table of Contents
|
|
4
|
+
1. [Search Tools](#search-tools)
|
|
5
|
+
2. [Scraping Tools](#scraping-tools)
|
|
6
|
+
3. [Apify Actor Runner](#apify-actor-runner-embedded)
|
|
7
|
+
4. [Apify Actor Catalog (55+)](#apify-actor-catalog)
|
|
8
|
+
5. [Actor Discovery](#actor-discovery)
|
|
9
|
+
6. [Shortcuts (apify.sh)](#shortcuts)
|
|
10
|
+
7. [Telegram](#telegram)
|
|
11
|
+
8. [Apify Troubleshooting](#apify-troubleshooting)
|
|
12
|
+
9. [Cost Reference](#cost-reference)
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## Search Tools
|
|
17
|
+
|
|
18
|
+
### Brave Search (web_search built-in)
|
|
19
|
+
- No setup, FREE plan 2000 queries/month, rate limit 1/sec
|
|
20
|
+
- Best for: general web, articles, profile discovery
|
|
21
|
+
- Weakness: rate limited, no geo-targeting
|
|
22
|
+
|
|
23
|
+
### Jina s.reader (scripts/jina.sh search)
|
|
24
|
+
- Top 10 results as full markdown
|
|
25
|
+
- Best for: broad search with content extraction
|
|
26
|
+
- Weakness: can return wrong person on common names
|
|
27
|
+
|
|
28
|
+
### Parallel AI (scripts/parallel.sh search)
|
|
29
|
+
- AI-powered search with reasoning and citations
|
|
30
|
+
- Best for: social media discovery, cross-referencing
|
|
31
|
+
- Weakness: sometimes returns irrelevant AI-reasoned results
|
|
32
|
+
|
|
33
|
+
### Bright Data (scripts/brightdata.sh search)
|
|
34
|
+
- Google/Bing/Yandex with geo-targeting, bypasses CAPTCHA
|
|
35
|
+
- search-geo: `brightdata.sh search-geo ru "query"` (search as if from Russia)
|
|
36
|
+
- search-yandex: `brightdata.sh search-yandex "query"` (better for Russian targets)
|
|
37
|
+
- Batch: up to 10 queries
|
|
38
|
+
|
|
39
|
+
## Scraping Tools
|
|
40
|
+
|
|
41
|
+
### web_fetch (built-in)
|
|
42
|
+
- Basic URL → markdown. First attempt, fallback to better tools
|
|
43
|
+
- Fails on: LinkedIn, Facebook, Cloudflare
|
|
44
|
+
|
|
45
|
+
### Jina r.reader (scripts/jina.sh read)
|
|
46
|
+
- Clean markdown, faster than web_fetch
|
|
47
|
+
- Fails on: LinkedIn authwall, Facebook authwall, Instagram JS
|
|
48
|
+
|
|
49
|
+
### Bright Data (scripts/brightdata.sh scrape)
|
|
50
|
+
- **Bypasses EVERYTHING**: bot detection, CAPTCHA, authwall, Cloudflare
|
|
51
|
+
- The nuclear option. Use when others fail.
|
|
52
|
+
- Batch: up to 10 URLs via scrape-batch
|
|
53
|
+
|
|
54
|
+
---
|
|
55
|
+
|
|
56
|
+
## Apify Actor Runner (embedded)
|
|
57
|
+
|
|
58
|
+
`run_actor.js` — universal script that runs ANY Apify actor. Embedded from
|
|
59
|
+
[apify/agent-skills](https://github.com/apify/agent-skills) v1.3.0.
|
|
60
|
+
|
|
61
|
+
**Requirements:** Node.js 18+ (for global fetch), APIFY_API_TOKEN or APIFY_TOKEN env var.
|
|
62
|
+
|
|
63
|
+
### Via bash wrapper (recommended)
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
# Quick answer — top 5 results in chat
|
|
67
|
+
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT'
|
|
68
|
+
|
|
69
|
+
# Export to CSV
|
|
70
|
+
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.csv --format csv
|
|
71
|
+
|
|
72
|
+
# Export to JSON
|
|
73
|
+
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.json --format json
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
### Direct node call
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
APIFY_TOKEN=$APIFY_API_TOKEN node scripts/run_actor.js \
|
|
80
|
+
--actor "ACTOR_ID" --input 'JSON_INPUT'
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Apify Actor Catalog
|
|
86
|
+
|
|
87
|
+
### Instagram (12 actors)
|
|
88
|
+
|
|
89
|
+
| Actor ID | Best For | OSINT Use |
|
|
90
|
+
|----------|----------|-----------|
|
|
91
|
+
| `apify/instagram-profile-scraper` | Profile data, follower counts, bio | **Primary** — bio, location, links |
|
|
92
|
+
| `apify/instagram-post-scraper` | Individual post details, engagement | Content analysis |
|
|
93
|
+
| `apify/instagram-comment-scraper` | Comment extraction | Sentiment, social graph |
|
|
94
|
+
| `apify/instagram-hashtag-scraper` | Hashtag content, trending topics | Find target by hashtag |
|
|
95
|
+
| `apify/instagram-hashtag-stats` | Hashtag performance metrics | Audience analysis |
|
|
96
|
+
| `apify/instagram-reel-scraper` | Reels content and metrics | Video content style |
|
|
97
|
+
| `apify/instagram-search-scraper` | Search users, places, hashtags | Discovery |
|
|
98
|
+
| `apify/instagram-tagged-scraper` | Posts tagged with specific accounts | **Who tags them** = social graph |
|
|
99
|
+
| `apify/instagram-followers-count-scraper` | Follower count tracking | Influence assessment |
|
|
100
|
+
| `apify/instagram-scraper` | Comprehensive Instagram data | Full profile deep dive |
|
|
101
|
+
| `apify/instagram-api-scraper` | API-based Instagram access | Fallback scraper |
|
|
102
|
+
| `apify/export-instagram-comments-posts` | Bulk comment/post export | Large-scale extraction |
|
|
103
|
+
|
|
104
|
+
### Facebook (14 actors)
|
|
105
|
+
|
|
106
|
+
| Actor ID | Best For | OSINT Use |
|
|
107
|
+
|----------|----------|-----------|
|
|
108
|
+
| `apify/facebook-pages-scraper` | Page data, metrics, contact info | **Primary** for public pages |
|
|
109
|
+
| `apify/facebook-page-contact-information` | Emails, phones, addresses | Contact enrichment |
|
|
110
|
+
| `apify/facebook-posts-scraper` | Post content and engagement | Writing style analysis |
|
|
111
|
+
| `apify/facebook-comments-scraper` | Comment extraction | Sentiment analysis |
|
|
112
|
+
| `apify/facebook-likes-scraper` | Reaction analysis | Audience mapping |
|
|
113
|
+
| `apify/facebook-reviews-scraper` | Page reviews | Reputation analysis |
|
|
114
|
+
| `apify/facebook-groups-scraper` | Group content and members | Community membership |
|
|
115
|
+
| `apify/facebook-events-scraper` | Event data | Activity/interests |
|
|
116
|
+
| `apify/facebook-ads-scraper` | Ad creative and targeting | Competitor intel |
|
|
117
|
+
| `apify/facebook-search-scraper` | Search results | Discovery |
|
|
118
|
+
| `apify/facebook-reels-scraper` | Reels content | Video style |
|
|
119
|
+
| `apify/facebook-photos-scraper` | Photo extraction | Visual profile |
|
|
120
|
+
| `apify/facebook-marketplace-scraper` | Marketplace listings | Business activity |
|
|
121
|
+
| `apify/facebook-followers-following-scraper` | Follower/following lists | Social graph |
|
|
122
|
+
|
|
123
|
+
**Note:** Facebook personal profiles require Bright Data. These actors work with public Pages/Groups/Marketplace.
|
|
124
|
+
|
|
125
|
+
### TikTok (14 actors)
|
|
126
|
+
|
|
127
|
+
| Actor ID | Best For | OSINT Use |
|
|
128
|
+
|----------|----------|-----------|
|
|
129
|
+
| `clockworks/tiktok-scraper` | Comprehensive TikTok data | Full platform deep dive |
|
|
130
|
+
| `clockworks/free-tiktok-scraper` | Free TikTok extraction | **Budget fallback** |
|
|
131
|
+
| `clockworks/tiktok-profile-scraper` | Profile data, bio, stats | **Primary** — bio, follower data |
|
|
132
|
+
| `clockworks/tiktok-video-scraper` | Video details and metrics | Content analysis |
|
|
133
|
+
| `clockworks/tiktok-comments-scraper` | Comment extraction | Sentiment, audience |
|
|
134
|
+
| `clockworks/tiktok-followers-scraper` | Follower lists | Social graph |
|
|
135
|
+
| `clockworks/tiktok-user-search-scraper` | Find users by keywords | **Discovery** by name |
|
|
136
|
+
| `clockworks/tiktok-hashtag-scraper` | Hashtag content | Topic analysis |
|
|
137
|
+
| `clockworks/tiktok-sound-scraper` | Trending sounds | Cultural markers |
|
|
138
|
+
| `clockworks/tiktok-ads-scraper` | Ad content | Competitor ads |
|
|
139
|
+
| `clockworks/tiktok-discover-scraper` | Discover page content | Trending content |
|
|
140
|
+
| `clockworks/tiktok-explore-scraper` | Explore content | Algorithm analysis |
|
|
141
|
+
| `clockworks/tiktok-trends-scraper` | Trending content | Trend tracking |
|
|
142
|
+
| `clockworks/tiktok-live-scraper` | Live stream data | Real-time activity |
|
|
143
|
+
|
|
144
|
+
### YouTube (5 actors)
|
|
145
|
+
|
|
146
|
+
| Actor ID | Best For | OSINT Use |
|
|
147
|
+
|----------|----------|-----------|
|
|
148
|
+
| `streamers/youtube-scraper` | Video data and metrics | Content analysis |
|
|
149
|
+
| `streamers/youtube-channel-scraper` | Channel info, subscriber data | **Primary** — channel metadata |
|
|
150
|
+
| `streamers/youtube-comments-scraper` | Comment extraction | Audience sentiment |
|
|
151
|
+
| `streamers/youtube-shorts-scraper` | Shorts content | Short-form style |
|
|
152
|
+
| `streamers/youtube-video-scraper-by-hashtag` | Videos by hashtag | Topic discovery |
|
|
153
|
+
|
|
154
|
+
### Google Maps (4 actors)
|
|
155
|
+
|
|
156
|
+
| Actor ID | Best For | OSINT Use |
|
|
157
|
+
|----------|----------|-----------|
|
|
158
|
+
| `compass/crawler-google-places` | Business listings, ratings, contact | **Primary** — verify business owners |
|
|
159
|
+
| `compass/google-maps-extractor` | Detailed business data | Deep business intel |
|
|
160
|
+
| `compass/Google-Maps-Reviews-Scraper` | Review extraction | Writing style in responses |
|
|
161
|
+
| `poidata/google-maps-email-extractor` | Email discovery from listings | Contact enrichment |
|
|
162
|
+
|
|
163
|
+
### Other Platforms (6 actors)
|
|
164
|
+
|
|
165
|
+
| Actor ID | Best For | OSINT Use |
|
|
166
|
+
|----------|----------|-----------|
|
|
167
|
+
| `apify/google-search-scraper` | Google search results | Structured search |
|
|
168
|
+
| `apify/google-trends-scraper` | Google Trends data | Interest tracking |
|
|
169
|
+
| `voyager/booking-scraper` | Booking.com hotel data | Travel/business intel |
|
|
170
|
+
| `voyager/booking-reviews-scraper` | Booking.com reviews | Reputation |
|
|
171
|
+
| `maxcopell/tripadvisor-reviews` | TripAdvisor reviews | Hospitality intel |
|
|
172
|
+
| `vdrmota/contact-info-scraper` | Contact enrichment from URLs | **Emails/phones from any site** |
|
|
173
|
+
|
|
174
|
+
### LinkedIn
|
|
175
|
+
|
|
176
|
+
LinkedIn actors are volatile on Apify. Current primary:
|
|
177
|
+
- `supreme_coder~linkedin-profile-scraper` via `apify.sh linkedin` → $0.005/profile
|
|
178
|
+
- Fallback: `brightdata.sh scrape` (always works, higher cost)
|
|
179
|
+
- If primary fails: `bash scripts/apify.sh store-search "linkedin scraper"`
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
## Actor Selection by OSINT Task
|
|
184
|
+
|
|
185
|
+
| OSINT Phase | What You Need | Primary Actors |
|
|
186
|
+
|-------------|---------------|----------------|
|
|
187
|
+
| **Profile discovery** | Find accounts | `apify/instagram-search-scraper`, `clockworks/tiktok-user-search-scraper`, `apify/facebook-search-scraper` |
|
|
188
|
+
| **Profile deep dive** | Extract bio/stats | `apify/instagram-profile-scraper`, `clockworks/tiktok-profile-scraper`, `streamers/youtube-channel-scraper` |
|
|
189
|
+
| **Social graph** | Who they interact with | `apify/instagram-tagged-scraper`, `apify/instagram-comment-scraper`, `clockworks/tiktok-followers-scraper`, `apify/facebook-followers-following-scraper` |
|
|
190
|
+
| **Content analysis** | Posts, videos, style | `apify/instagram-post-scraper`, `clockworks/tiktok-video-scraper`, `streamers/youtube-scraper` |
|
|
191
|
+
| **Contact enrichment** | Emails, phones | `vdrmota/contact-info-scraper`, `apify/facebook-page-contact-information`, `poidata/google-maps-email-extractor` |
|
|
192
|
+
| **Business verification** | Company, location | `compass/crawler-google-places`, `compass/google-maps-extractor` |
|
|
193
|
+
| **Psychoprofile signals** | Sentiment, style | `apify/instagram-comment-scraper`, `clockworks/tiktok-comments-scraper`, `streamers/youtube-comments-scraper` |
|
|
194
|
+
|
|
195
|
+
### Multi-Actor Workflows (OSINT-specific)
|
|
196
|
+
|
|
197
|
+
| Workflow | Step 1 → | Step 2 |
|
|
198
|
+
|----------|----------|--------|
|
|
199
|
+
| **Full Instagram** | `apify/instagram-profile-scraper` → | `apify/instagram-tagged-scraper` + `apify/instagram-comment-scraper` |
|
|
200
|
+
| **Business owner** | `compass/crawler-google-places` → | `vdrmota/contact-info-scraper` on website |
|
|
201
|
+
| **Content creator** | `streamers/youtube-channel-scraper` → | `streamers/youtube-comments-scraper` |
|
|
202
|
+
| **TikTok target** | `clockworks/tiktok-user-search-scraper` → | `clockworks/tiktok-profile-scraper` + `clockworks/tiktok-comments-scraper` |
|
|
203
|
+
| **Facebook page** | `apify/facebook-pages-scraper` → | `apify/facebook-posts-scraper` + `apify/facebook-page-contact-information` |
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## Actor Discovery
|
|
208
|
+
|
|
209
|
+
When none of the 55+ actors fit, search the Apify Store dynamically:
|
|
210
|
+
|
|
211
|
+
```bash
|
|
212
|
+
# Via mcpc CLI (if installed)
|
|
213
|
+
APIFY_TOKEN=$APIFY_API_TOKEN mcpc --json mcp.apify.com \
|
|
214
|
+
--header "Authorization: Bearer $APIFY_TOKEN" \
|
|
215
|
+
tools-call search-actors keywords:="SEARCH_KEYWORDS" limit:=10
|
|
216
|
+
|
|
217
|
+
# Via Apify API directly
|
|
218
|
+
bash scripts/apify.sh store-search "keyword1 keyword2"
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
Tips:
|
|
222
|
+
- Use 1-3 simple keywords (e.g., "LinkedIn profiles", "Twitter scraper")
|
|
223
|
+
- Sort by: PAY_PER_EVENT (works with small budgets), high rating, high run count
|
|
224
|
+
- Actors on Apify are volatile — always have a Bright Data fallback
|
|
225
|
+
|
|
226
|
+
---
|
|
227
|
+
|
|
228
|
+
## Shortcuts
|
|
229
|
+
|
|
230
|
+
`apify.sh` provides quick shortcuts for common operations:
|
|
231
|
+
|
|
232
|
+
```bash
|
|
233
|
+
bash scripts/apify.sh linkedin "https://linkedin.com/in/..." # $0.005
|
|
234
|
+
bash scripts/apify.sh instagram "handle" # free tier
|
|
235
|
+
bash scripts/apify.sh run <actor_id> '<json>' # any actor
|
|
236
|
+
bash scripts/apify.sh results <run_id> # get results
|
|
237
|
+
bash scripts/apify.sh run-status <run_id> # check status
|
|
238
|
+
bash scripts/apify.sh store-search "query" # find actors
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
For anything beyond shortcuts, use `run-actor.sh` — it handles polling,
|
|
242
|
+
timeout, and CSV/JSON export automatically.
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
|
|
246
|
+
## Telegram
|
|
247
|
+
|
|
248
|
+
- Public channel posts: `web_fetch https://t.me/s/{channel}` (free, last ~20 posts)
|
|
249
|
+
- Profile bio: `web_fetch https://t.me/{username}` (free, shows bio/title)
|
|
250
|
+
- Private messages: tg.py (Telegram skill) or mcporter telegram tools
|
|
251
|
+
|
|
252
|
+
---
|
|
253
|
+
|
|
254
|
+
## Apify Troubleshooting
|
|
255
|
+
|
|
256
|
+
When an Apify actor fails:
|
|
257
|
+
1. Check if it blocks API on free plan ("run through UI only" error)
|
|
258
|
+
2. Check input field names (profileUrls vs urls vs user_name — varies per actor)
|
|
259
|
+
3. Check run status: `bash scripts/apify.sh run-status <run_id>`
|
|
260
|
+
4. Search for alternatives: `bash scripts/apify.sh store-search "platform scraper"`
|
|
261
|
+
5. Try the free-tier variant (e.g., `clockworks/free-tiktok-scraper`)
|
|
262
|
+
6. Fall back to Bright Data: `bash scripts/brightdata.sh scrape "<url>"`
|
|
263
|
+
|
|
264
|
+
---
|
|
265
|
+
|
|
266
|
+
## Cost Reference
|
|
267
|
+
|
|
268
|
+
| Tool | Cost | Notes |
|
|
269
|
+
|------|------|-------|
|
|
270
|
+
| LinkedIn (Apify supreme_coder) | $0.005/profile | |
|
|
271
|
+
| Instagram (Apify) | free tier | Limited runs |
|
|
272
|
+
| TikTok (Apify clockworks) | $0.01-0.05/profile | free-tiktok-scraper = free |
|
|
273
|
+
| YouTube (Apify streamers) | $0.01/channel | |
|
|
274
|
+
| Google Maps (Apify compass) | $0.01/listing | |
|
|
275
|
+
| Contact enrichment | $0.01/URL | |
|
|
276
|
+
| Facebook pages (Apify) | $0.01/page | Personal profiles = Bright Data |
|
|
277
|
+
| Facebook personal (Bright Data) | per-request | Check account balance |
|
|
278
|
+
| Jina | free tier with key | |
|
|
279
|
+
| Parallel | free tier with key | |
|
|
280
|
+
| Brave | free (2000/month) | |
|
|
281
|
+
| **Budget rule** | **≤$0.50 without asking** | **>$0.50 ask user** |
|