conectese 0.1.14

package/_conectese/core/prompts/sherlock-twitter.md

@@ -0,0 +1,78 @@
+# Sherlock — Twitter/X Extractor
+
+Load `sherlock-shared.md` before using this extractor.
+
+This file contains the Twitter/X-specific extraction process. The Architect loads this file (alongside `sherlock-shared.md`) when the reference URL is a Twitter/X profile or post.
+
+---
+
+## URL Patterns
+
+- Profile: `x.com/{username}`, `twitter.com/{username}`
+- Post: `x.com/{username}/status/{id}`
+
+---
+
+## Timeline Extraction
+
+1. Navigate to the user's profile using browser automation with the saved Twitter session:
+   ```bash
+   npx playwright open --load-storage=_conectese/_browser_profile/twitter.json https://x.com/{username}
+   ```
+
+2. Take a snapshot to read the timeline.
+
+3. From the snapshot, identify individual tweets and threads. Thread indicators include "Show this thread" links or reply chains from the same author.
+
+---
+
+## Tweet Extraction (per tweet)
+
+1. From the timeline snapshot, record for each tweet:
+   - Full tweet text
+   - Post date and time
+   - Engagement metrics: likes, retweets, replies, bookmarks, views (if visible)
+   - Whether it is a standalone tweet or part of a thread
+
+2. If the tweet text is truncated in the timeline view, click the tweet to open it.
+
+3. Take a snapshot of the full tweet.
+
+4. Record the complete text and all metrics.
+
+5. Navigate back to the timeline using browser back navigation.
+
+---
+
+## Thread Extraction
+
+1. Identify threads in the timeline (tweets with "Show this thread" or self-reply chains).
+
+2. Click the first tweet in the thread to open it.
+
+3. Take a snapshot to read the full thread.
+
+4. Scroll down within the thread to capture all tweets in the chain (e.g., `window.scrollBy(0, 800)` or equivalent scroll action).
+
+5. Take additional snapshots as needed to capture the complete thread.
+
+6. Record all tweets in the thread as a single content unit, preserving their order.
+
+7. Navigate back to the timeline using browser back navigation.
+
+---
+
+## Scrolling for More Tweets
+
+To load older tweets beyond the initial timeline view, scroll down the page using browser automation (e.g., `window.scrollBy(0, 1000)` via evaluate, or equivalent scroll action).
+
+After scrolling, take a new snapshot and continue extraction.
+
+---
+
+## Configuration Defaults
+
+- Default tweet count: up to 3 most recent tweets (excluding replies to others)
+- Content types: standalone tweets and threads (skip replies to other users unless they are high-engagement)
+- Priority: threads first (higher content density), then standalone tweets by engagement
+- Stop condition: target count reached OR 3 consecutive scrolls with no new tweets

package/_conectese/core/prompts/sherlock-youtube.md

@@ -0,0 +1,85 @@
+# Sherlock — YouTube Extractor
+
+Load `sherlock-shared.md` before using this extractor.
+
+This file contains the YouTube-specific extraction process. The Architect loads this file (alongside `sherlock-shared.md`) when the reference URL is a YouTube channel or video.
+
+---
+
+## URL Patterns
+
+- Channel: `youtube.com/@{handle}`, `youtube.com/channel/{id}`, `youtube.com/@{handle}/videos`
+- Video: `youtube.com/watch?v={id}`
+- Short URL: `youtu.be/{id}`
+
+---
+
+## Channel Video List Extraction
+
+1. Navigate to the channel's videos page using browser automation with the saved YouTube session:
+   ```bash
+   npx playwright open --load-storage=_conectese/_browser_profile/youtube.json https://www.youtube.com/@{handle}/videos
+   ```
+
+2. Take a snapshot to read the video list.
+
+3. From the snapshot, identify video titles, view counts, and upload dates.
+
+4. Select videos to extract based on configuration (most recent, most viewed, or a mix).
+
+---
+
+## Individual Video Extraction
+
+1. Click the video title to open it.
+
+2. Take a snapshot to read the title, view count, like count, and upload date.
+
+3. Expand the video description by clicking "...more" or the description area.
+
+4. Take another snapshot to capture the full description text.
+
+5. Record: title, full description, view count, like count, comment count, upload date.
+
+---
+
+## Video Transcription — Primary Method (Subtitles)
+
+Use yt-dlp to download auto-generated or manual subtitles:
+
+```bash
+yt-dlp --write-auto-sub --sub-lang en --skip-download -o "squads/{squad-name}/_investigations/{channel}/%(id)s" "https://www.youtube.com/watch?v={video-id}"
+```
+
+If subtitles are available, read the `.vtt` or `.srt` file and extract the text content (strip timestamps).
+
+---
+
+## Video Transcription — Fallback Method (Audio + Whisper)
+
+If no subtitles are available, download the audio and transcribe:
+
+```bash
+yt-dlp -x --audio-format wav -o "squads/{squad-name}/_investigations/{channel}/%(id)s.%(ext)s" "https://www.youtube.com/watch?v={video-id}"
+```
+
+```bash
+whisper "squads/{squad-name}/_investigations/{channel}/{video-id}.wav" --model small --output_dir "squads/{squad-name}/_investigations/{channel}/" --output_format txt
+```
+
+Read the transcription file and include it in the raw content output.
+
+---
+
+## Navigating Back to Video List
+
+After extracting a video, navigate back to the channel videos page using browser automation (navigate back in history or navigate directly to the channel URL).
+
+---
+
+## Configuration Defaults
+
+- Default video count: up to 3 most recent videos
+- Transcription method: subtitles first, audio+whisper fallback
+- Priority: most recent uploads first, then highest viewed
+- Stop condition: target count reached OR end of uploads page