openartemis 0.2.0__tar.gz

openartemis-0.2.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: openartemis
+Version: 0.2.0
+Summary: CLI to harvest transcripts from YouTube, TikTok, Instagram, and X — chat with Artemis, an AI that researches for you
+License: MIT
+Keywords: transcripts,youtube,tiktok,whisper,cli,research
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Multimedia :: Video
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: google-api-python-client>=2.100.0
+Requires-Dist: youtube-transcript-api>=0.6.0
+Requires-Dist: yt-dlp>=2024.1.0
+Requires-Dist: ddgs>=9.0.0
+Requires-Dist: openai-whisper>=20231117
+Requires-Dist: requests>=2.28.0
+Requires-Dist: typer[all]>=0.9.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: questionary>=2.0.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: bcrypt>=4.0.0
+Requires-Dist: trafilatura>=2.0.0
+Requires-Dist: playwright>=1.40.0
+
+# OpenArtemis
+
+CLI tool to harvest transcripts from YouTube, TikTok, Instagram, and X (Twitter). User-friendly design inspired by Claude CLI — no coding required.
+
+**Login required.** First run creates an admin account. Other users register and wait for admin approval. Approval requests are emailed to the admin.
+
+**Chat mode** (default): Run `openartemis` for a persistent chat with **Artemis**, an AI that researches for you. Say "research X" or "find out about Y" and Artemis uses tools (transcripts, web search, webpage fetch). Use `/research <query>` for multi-agent deep research. Type `/` for mode menu, `/logout` or `/exit` to quit.
+
+**Harvest mode**: `openartemis harvest` for guided step-by-step transcript download.
+
+**Admin**: `openartemis admin` — manage users, approve requests, view usage. Admins have unlimited API access.
+
+## Requirements
+
+- **Python 3.10+**
+- **ffmpeg** (required for yt-dlp and Whisper; must be on PATH)
+- **YouTube Data API key** (for YouTube search; set `YOUTUBE_API_KEY` or use `--youtube-api-key`)
+- **ScrapingDog API key** (optional, for faster YouTube transcripts; set `SCRAPINGDOG_API_KEY`)
+- **OpenAI API key** (for Artemis chat; set `OPENAI_API_KEY` — never commit your key)
+- **Brave Search API key** (optional, for web search; set `BRAVE_SEARCH_API_KEY` — search.brave.com)
+
+## Installation
+
+**From PyPI** (run anywhere):
+
+```bash
+pip install openartemis
+```
+
+Then run:
+
+```bash
+openartemis
+```
+
+**From source** (development):
+
+```bash
+pip install -e .
+playwright install chromium   # For browse_webpage (JS-heavy sites)
+```
+
+## Usage
+
+### Interactive mode (no coding required)
+
+Run without arguments for chat (login required):
+
+```bash
+openartemis
+```
+
+### Command-line mode
+
+```bash
+openartemis harvest   # Guided harvest (login required)
+```
+
+### Arguments
+
+| Argument | Description | Default |
+|----------|-------------|---------|
+| `query` | Search query (omit for interactive mode) | - |
+| `--platforms`, `-p` | Comma-separated: youtube,tiktok,instagram,x | youtube |
+| `--max-results`, `-n` | Max results per platform | 10 |
+| `--out-dir`, `-o` | Output directory | ./transcripts |
+| `--youtube-api-key` | YouTube Data API key (or YOUTUBE_API_KEY env) | - |
+| `--scrapingdog-api-key` | ScrapingDog API key (or SCRAPINGDOG_API_KEY env) | - |
+| `--whisper-model`, `-m` | Whisper model: tiny,base,small,medium,large | base |
+| `--interactive`, `-i` | Run in interactive mode | - |
+
+### Artemis Detective (multi-agent research)
+
+Tell Artemis what to find — it uses a multi-agent pipeline: one orchestrator plans up to 5 tasks, five workers run in parallel (Brave search, transcript harvest, webpage fetch/browse), then results are synthesized into a report:
+
+```bash
+openartemis investigate "Find out everything about what happened to John Smith"
+# Or run without args for interactive prompt:
+openartemis investigate
+```
+
+Options: `--model gpt-4o-mini` (default, cheapest), `--out-dir ./detective_output`
+
+Tools: `harvest_transcripts`, `brave_search`, `fetch_webpage`, `browse_webpage`, `list_transcripts`, `read_transcript`
+
+### Admin panel
+
+```bash
+openartemis admin
+```
+
+Approve users, view usage, revoke access, create users directly.
+
+### Other commands
+
+```bash
+openartemis logout   # Log out
+openartemis config   # Show configuration help
+```
+
+Shows required env vars and where to get API keys.
+
+### Output
+
+Per-post files in `out_dir`:
+- `{platform}_{id}.json` - Metadata + segments + full text
+- `{platform}_{id}.txt` - Plain text transcript
+- `{platform}_{id}.srt` - SRT subtitles
+
+## Pipeline
+
+- **YouTube**: Data API search → video IDs → ScrapingDog API (if key set) or youtube-transcript-api for captions; if none, download audio via yt-dlp → Whisper
+- **TikTok/Instagram/X**: Web search (site: operator) → yt-dlp download + extract audio → Whisper
+- **Research**: Multi-agent (orchestrator + 5 workers) — Brave Search, trafilatura (fetch_webpage), Playwright (browse_webpage), transcript harvest
+
+## Auth
+
+- Session expiry: 7 days
+- Rate limit: 5 failed logins per 15 min per username
+- Password policy: minimum 8 characters
+
+## Publishing to PyPI
+
+To publish a new version:
+
+```bash
+pip install build twine
+python -m build
+twine upload dist/*
+```

openartemis-0.2.0/README.md

@@ -0,0 +1,130 @@
+# OpenArtemis
+
+CLI tool to harvest transcripts from YouTube, TikTok, Instagram, and X (Twitter). User-friendly design inspired by Claude CLI — no coding required.
+
+**Login required.** First run creates an admin account. Other users register and wait for admin approval. Approval requests are emailed to the admin.
+
+**Chat mode** (default): Run `openartemis` for a persistent chat with **Artemis**, an AI that researches for you. Say "research X" or "find out about Y" and Artemis uses tools (transcripts, web search, webpage fetch). Use `/research <query>` for multi-agent deep research. Type `/` for mode menu, `/logout` or `/exit` to quit.
+
+**Harvest mode**: `openartemis harvest` for guided step-by-step transcript download.
+
+**Admin**: `openartemis admin` — manage users, approve requests, view usage. Admins have unlimited API access.
+
+## Requirements
+
+- **Python 3.10+**
+- **ffmpeg** (required for yt-dlp and Whisper; must be on PATH)
+- **YouTube Data API key** (for YouTube search; set `YOUTUBE_API_KEY` or use `--youtube-api-key`)
+- **ScrapingDog API key** (optional, for faster YouTube transcripts; set `SCRAPINGDOG_API_KEY`)
+- **OpenAI API key** (for Artemis chat; set `OPENAI_API_KEY` — never commit your key)
+- **Brave Search API key** (optional, for web search; set `BRAVE_SEARCH_API_KEY` — search.brave.com)
+
+## Installation
+
+**From PyPI** (run anywhere):
+
+```bash
+pip install openartemis
+```
+
+Then run:
+
+```bash
+openartemis
+```
+
+**From source** (development):
+
+```bash
+pip install -e .
+playwright install chromium   # For browse_webpage (JS-heavy sites)
+```
+
+## Usage
+
+### Interactive mode (no coding required)
+
+Run without arguments for chat (login required):
+
+```bash
+openartemis
+```
+
+### Command-line mode
+
+```bash
+openartemis harvest   # Guided harvest (login required)
+```
+
+### Arguments
+
+| Argument | Description | Default |
+|----------|-------------|---------|
+| `query` | Search query (omit for interactive mode) | - |
+| `--platforms`, `-p` | Comma-separated: youtube,tiktok,instagram,x | youtube |
+| `--max-results`, `-n` | Max results per platform | 10 |
+| `--out-dir`, `-o` | Output directory | ./transcripts |
+| `--youtube-api-key` | YouTube Data API key (or YOUTUBE_API_KEY env) | - |
+| `--scrapingdog-api-key` | ScrapingDog API key (or SCRAPINGDOG_API_KEY env) | - |
+| `--whisper-model`, `-m` | Whisper model: tiny,base,small,medium,large | base |
+| `--interactive`, `-i` | Run in interactive mode | - |
+
+### Artemis Detective (multi-agent research)
+
+Tell Artemis what to find — it uses a multi-agent pipeline: one orchestrator plans up to 5 tasks, five workers run in parallel (Brave search, transcript harvest, webpage fetch/browse), then results are synthesized into a report:
+
+```bash
+openartemis investigate "Find out everything about what happened to John Smith"
+# Or run without args for interactive prompt:
+openartemis investigate
+```
+
+Options: `--model gpt-4o-mini` (default, cheapest), `--out-dir ./detective_output`
+
+Tools: `harvest_transcripts`, `brave_search`, `fetch_webpage`, `browse_webpage`, `list_transcripts`, `read_transcript`
+
+### Admin panel
+
+```bash
+openartemis admin
+```
+
+Approve users, view usage, revoke access, create users directly.
+
+### Other commands
+
+```bash
+openartemis logout   # Log out
+openartemis config   # Show configuration help
+```
+
+Shows required env vars and where to get API keys.
+
+### Output
+
+Per-post files in `out_dir`:
+- `{platform}_{id}.json` - Metadata + segments + full text
+- `{platform}_{id}.txt` - Plain text transcript
+- `{platform}_{id}.srt` - SRT subtitles
+
+## Pipeline
+
+- **YouTube**: Data API search → video IDs → ScrapingDog API (if key set) or youtube-transcript-api for captions; if none, download audio via yt-dlp → Whisper
+- **TikTok/Instagram/X**: Web search (site: operator) → yt-dlp download + extract audio → Whisper
+- **Research**: Multi-agent (orchestrator + 5 workers) — Brave Search, trafilatura (fetch_webpage), Playwright (browse_webpage), transcript harvest
+
+## Auth
+
+- Session expiry: 7 days
+- Rate limit: 5 failed logins per 15 min per username
+- Password policy: minimum 8 characters
+
+## Publishing to PyPI
+
+To publish a new version:
+
+```bash
+pip install build twine
+python -m build
+twine upload dist/*
+```

openartemis-0.2.0/openartemis/__init__.py

@@ -0,0 +1,3 @@
+"""OpenArtemis - Harvest transcripts from YouTube, TikTok, Instagram, and X."""
+
+__version__ = "0.2.0"

openartemis-0.2.0/openartemis/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point for python -m openartemis."""
+
+from openartemis.cli import main
+
+if __name__ == "__main__":
+    main()

openartemis-0.2.0/openartemis/agent/__init__.py

@@ -0,0 +1,8 @@
+"""Multi-agent research pipeline — orchestrator, workers, synthesis."""
+
+from openartemis.agent.detective import DetectiveAgent, TOOLS
+from openartemis.agent.orchestrator import create_plan
+from openartemis.agent.worker import run_worker
+from openartemis.agent.pipeline import ResearchPipeline
+
+__all__ = ["DetectiveAgent", "TOOLS", "create_plan", "run_worker", "ResearchPipeline"]

openartemis-0.2.0/openartemis/agent/detective.py

@@ -0,0 +1,344 @@
+"""AI detective agent — uses transcriptharvest as tools to research and find answers."""
+
+import json
+import os
+from pathlib import Path
+from typing import Any, Callable
+
+from dotenv import load_dotenv
+from openai import OpenAI
+
+load_dotenv(Path(__file__).resolve().parents[2] / ".env")
+
+from openartemis.harvesters.social import SocialHarvester
+from openartemis.harvesters.youtube import (
+    TranscriptAPIInsufficientCreditsError,
+    YouTubeHarvester,
+    extract_youtube_video_id,
+)
+from openartemis.tools.web import brave_search, fetch_webpage, browse_webpage
+
+# Tools the AI can call (harvest + web)
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "harvest_transcripts",
+            "description": "Search and harvest transcripts from video/social platforms. Use this to find content about a person, event, or topic. Returns the output directory path and count of harvested posts.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "query": {
+                        "type": "string",
+                        "description": "Search query (e.g. person name, event, topic)",
+                    },
+                    "platforms": {
+                        "type": "array",
+                        "items": {"type": "string", "enum": ["youtube", "tiktok", "instagram", "x"]},
+                        "description": "Platforms to search",
+                        "default": ["youtube"],
+                    },
+                    "max_results": {
+                        "type": "integer",
+                        "description": "Max results per platform",
+                        "default": 5,
+                    },
+                },
+                "required": ["query"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "list_transcripts",
+            "description": "List all harvested transcript files in the output directory. Returns file paths and titles.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "out_dir": {
+                        "type": "string",
+                        "description": "Path to the transcripts directory",
+                    },
+                },
+                "required": ["out_dir"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "read_transcript",
+            "description": "Read the full text of a harvested transcript (JSON or TXT file).",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "file_path": {
+                        "type": "string",
+                        "description": "Full path to the .json or .txt transcript file",
+                    },
+                },
+                "required": ["file_path"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "brave_search",
+            "description": "Search the web via Brave Search API. Returns results with title, url, description.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "q": {"type": "string", "description": "Search query"},
+                    "count": {"type": "integer", "description": "Max results (default 10)", "default": 10},
+                    "freshness": {"type": "string", "enum": ["pd", "pw", "pm", "py"], "description": "pd=day, pw=week, pm=month, py=year"},
+                },
+                "required": ["q"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "fetch_webpage",
+            "description": "Fetch URL and extract main text (trafilatura). Fast for static HTML.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "url": {"type": "string", "description": "URL to fetch"},
+                },
+                "required": ["url"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "browse_webpage",
+            "description": "Fetch URL with headless browser (Playwright). Use for JS-heavy sites.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "url": {"type": "string", "description": "URL to browse"},
+                    "selector": {"type": "string", "description": "Optional CSS selector for specific element"},
+                },
+                "required": ["url"],
+            },
+        },
+    },
+]
+
+SYSTEM_PROMPT = """You are Artemis, a research detective. Your job is to find out everything you can about what the user asks.
+
+You have access to tools that let you:
+1. **harvest_transcripts** — Search YouTube, TikTok, Instagram, X for videos and download their transcripts
+2. **list_transcripts** — See what transcripts have been harvested
+3. **read_transcript** — Read the full text of any transcript
+4. **brave_search** — Web search via Brave API
+5. **fetch_webpage** — Fast text extraction from static HTML (trafilatura)
+6. **browse_webpage** — Full render with headless browser for JS-heavy sites
+
+Work systematically:
+- Start by harvesting transcripts with relevant search queries (person names, events, dates, related terms)
+- Read the transcripts to extract facts, quotes, and leads
+- If you find new names, places, or angles, harvest more with follow-up searches
+- Keep going until you have a comprehensive picture or hit diminishing returns
+
+Be thorough but efficient. Prioritize high-value sources. Summarize findings clearly.
+When done, provide a complete report with sources (URLs from the transcripts)."""
+
+
+class DetectiveAgent:
+    """AI agent that uses transcriptharvest to research."""
+
+    def __init__(
+        self,
+        openai_api_key: str | None = None,
+        youtube_api_key: str | None = None,
+        scrapingdog_api_key: str | None = None,
+        transcriptapi_api_key: str | None = None,
+        model: str = "gpt-4o-mini",
+        out_dir: Path | None = None,
+    ):
+        self.client = OpenAI(api_key=openai_api_key or os.environ.get("OPENAI_API_KEY"))
+        self.youtube_api_key = youtube_api_key or os.environ.get("YOUTUBE_API_KEY")
+        self.scrapingdog_api_key = scrapingdog_api_key or os.environ.get("SCRAPINGDOG_API_KEY")
+        self.transcriptapi_api_key = transcriptapi_api_key or os.environ.get("TRANSCRIPTAPI_API_KEY")
+        self.model = model
+        self.out_dir = Path(out_dir or "./detective_output").resolve()
+        self.out_dir.mkdir(parents=True, exist_ok=True)
+
+    def _harvest_transcripts(
+        self,
+        query: str,
+        platforms: list[str] | None = None,
+        max_results: int = 5,
+    ) -> dict[str, Any]:
+        """Tool: harvest transcripts."""
+        platforms = platforms or ["youtube"]
+        total = 0
+
+        try:
+            for platform in platforms:
+                if platform == "youtube":
+                    if not self.youtube_api_key and not self.transcriptapi_api_key:
+                        return {"error": "YouTube requires YOUTUBE_API_KEY (Google) or TRANSCRIPTAPI_API_KEY. Skipping YouTube."}
+                    harvester = YouTubeHarvester(
+                        api_key=self.youtube_api_key,
+                        scrapingdog_api_key=self.scrapingdog_api_key,
+                        transcriptapi_api_key=self.transcriptapi_api_key,
+                    )
+                else:
+                    harvester = SocialHarvester(platform=platform)
+
+                for _ in harvester.harvest(
+                    query=query,
+                    max_results=max_results,
+                    out_dir=self.out_dir,
+                    whisper_model="base",
+                ):
+                    total += 1
+        except TranscriptAPIInsufficientCreditsError as e:
+            return {"error": str(e), "harvested_count": 0}
+
+        return {
+            "out_dir": str(self.out_dir),
+            "query": query,
+            "platforms": platforms,
+            "harvested_count": total,
+        }
+
+    def harvest_url(self, url: str) -> dict[str, Any]:
+        """Harvest transcript for a single YouTube URL. Returns result dict."""
+        if not extract_youtube_video_id(url):
+            return {"error": f"Invalid YouTube URL: {url}"}
+        if not self.youtube_api_key and not self.transcriptapi_api_key:
+            return {"error": "YouTube requires YOUTUBE_API_KEY or TRANSCRIPTAPI_API_KEY."}
+        harvester = YouTubeHarvester(
+            api_key=self.youtube_api_key,
+            scrapingdog_api_key=self.scrapingdog_api_key,
+            transcriptapi_api_key=self.transcriptapi_api_key,
+        )
+        try:
+            data = harvester.harvest_url(url, self.out_dir, whisper_model="base")
+        except TranscriptAPIInsufficientCreditsError as e:
+            return {"harvested_count": 0, "error": str(e)}
+        if data:
+            return {"harvested_count": 1, "out_dir": str(self.out_dir), "title": data.get("title", "")}
+        return {"harvested_count": 0, "error": "Failed to fetch transcript"}
+
+    def _list_transcripts(self, out_dir: str) -> dict[str, Any]:
+        """Tool: list transcript files."""
+        path = Path(out_dir)
+        if not path.exists():
+            return {"error": f"Directory not found: {out_dir}"}
+        files = []
+        for f in path.glob("*.json"):
+            try:
+                with open(f, encoding="utf-8") as fp:
+                    data = json.load(fp)
+                files.append({
+                    "path": str(f),
+                    "title": data.get("title", ""),
+                    "url": data.get("url", ""),
+                    "platform": data.get("platform", ""),
+                })
+            except Exception:
+                files.append({"path": str(f), "title": "(parse error)"})
+        return {"files": files, "count": len(files)}
+
+    def _read_transcript(self, file_path: str) -> dict[str, Any]:
+        """Tool: read a transcript file."""
+        path = Path(file_path)
+        if not path.exists():
+            return {"error": f"File not found: {file_path}"}
+        try:
+            if path.suffix == ".json":
+                with open(path, encoding="utf-8") as f:
+                    data = json.load(f)
+                return {
+                    "title": data.get("title", ""),
+                    "url": data.get("url", ""),
+                    "platform": data.get("platform", ""),
+                    "full_text": data.get("full_text", ""),
+                    "segments_count": len(data.get("segments", [])),
+                }
+            if path.suffix == ".txt":
+                with open(path, encoding="utf-8") as f:
+                    return {"full_text": f.read()}
+            return {"error": "Unsupported file type. Use .json or .txt"}
+        except Exception as e:
+            return {"error": str(e)}
+
+    def _execute_tool(self, name: str, args: dict[str, Any]) -> str:
+        """Execute a tool and return JSON string result."""
+        if name == "harvest_transcripts":
+            result = self._harvest_transcripts(
+                query=args["query"],
+                platforms=args.get("platforms", ["youtube"]),
+                max_results=args.get("max_results", 5),
+            )
+        elif name == "list_transcripts":
+            result = self._list_transcripts(args["out_dir"])
+        elif name == "read_transcript":
+            result = self._read_transcript(args["file_path"])
+        elif name == "brave_search":
+            result = brave_search(
+                q=args["q"],
+                count=args.get("count", 10),
+                freshness=args.get("freshness"),
+            )
+        elif name == "fetch_webpage":
+            result = fetch_webpage(url=args["url"])
+        elif name == "browse_webpage":
+            result = browse_webpage(url=args["url"], selector=args.get("selector"))
+        else:
+            result = {"error": f"Unknown tool: {name}"}
+        return json.dumps(result, indent=2)
+
+    def run(
+        self,
+        user_prompt: str,
+        max_turns: int = 20,
+        on_tool_call: Callable[[str, dict], None] | None = None,
+    ) -> str:
+        """Run the agent. Returns the final report."""
+        messages = [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": user_prompt},
+        ]
+
+        for turn in range(max_turns):
+            response = self.client.chat.completions.create(
+                model=self.model,
+                messages=messages,
+                tools=TOOLS,
+                tool_choice="auto",
+            )
+            choice = response.choices[0]
+            msg = choice.message
+
+            # Convert to dict for API (handles model objects)
+            msg_dict = (
+                msg.model_dump() if hasattr(msg, "model_dump") else
+                {"role": msg.role, "content": msg.content or "", "tool_calls": getattr(msg, "tool_calls", None)}
+            )
+            messages.append(msg_dict)
+
+            if not msg.tool_calls:
+                return (msg.content or "").strip()
+
+            for tc in msg.tool_calls:
+                name = tc.function.name
+                args = json.loads(tc.function.arguments)
+                if on_tool_call:
+                    on_tool_call(name, args)
+                result = self._execute_tool(name, args)
+                messages.append({
+                    "role": "tool",
+                    "tool_call_id": tc.id,
+                    "content": result,
+                })
+
+        return "Max turns reached. Review harvested transcripts manually."