playlist-admin 2.7.13 → 2.15.14

package/README.md

@@ -1,37 +1,84 @@
 # playlist-admin
 
-A Flutter desktop app for music library management and Spotube playlist automation.
+A Flutter desktop app for Spotify-native music library management: browse the
+Spotify home feed, search & stream, playlist sync, batch downloads, and stats.
 
 ## Features
 
-- **Spotube Automation** — Automatically download playlists via Spotube GUI
-- **Audio Conversion** — M4A → MP3 with batch processing
-- **Spotify Scraping** — Fetch playlists via Spotify Embed API
-- **Playlist Management** — Track completeness, prune missing files
-- **Statistics** — Library size, format distribution, coverage charts
+- **Spotify 原生整合** — 個人化主頁 / 搜尋串流 / 你的歌單（內部 GQL API，WebView 登入）
+- **串流播放** — Spotify 搜尋 → yt-dlp 解析 → ffmpeg VBR 0 轉碼 → 本地 HTTP 串流（可選快取 cache\stream\）
+- **原生下載** — yt-dlp 256k Opus → mp3 VBR 0（`preferredquality: 0`），批量一鍵補全
+- **播放清單同步** — Spotify embed API 抓取 → m3u8 + 自動下載缺漏曲目
+- **工作列媒體控制器** — SMTC 進度條 + 拖動 seek + 播放/暫停/上下首
+- **播放佇列** — 拖曳排序 / 移除 / 跳轉 / play next，重啟還原（cache\queue.json）
+- **播放統計** — 50%/4min scrobble 規則 → Top 曲目 / 聆聽分鐘 / 最近播放
+- **下載統計** — 每輪下載成功率紀錄（cache\downloads_log.json）
+- **Sleep timer / Crossfade / 全域熱鍵**（Space、←/→ seek、N/P 切歌）
+- **Podcast + RAG** — RSS 下載、Groq 轉錄、Ollama chroma 問答
+- **Discord RPC** — 現播歌曲 + 封面
+- **一起聽（Jam）** — 免費版 Spotify Jam：開房間（免註冊/免外部伺服器），
+  同一個 Wi-Fi 下大家加入，共享播放佇列、即時同步播放進度、投票跳歌、
+  聊天室。手機（Android/iOS）也能加入房主一起聽。
+
+## 資料夾結構（`Music\playlist-admin\`）
+
+```
+music\        ← 唯一音樂庫（mp3）
+playlists\    ← m3u8
+lyrics\       ← LRC
+podcasts\     ← 播客音檔
+cache\spotify ← Spotify 快取 / snapshot / session
+cache\podcast ← 播客處理快取 / chroma_db
+cache\lufs    ← LUFS 快取
+cache\stream  ← 串流快取
+cache\thumbnails
+exports\      ← USB 匯出
+logs\         ← 日誌
+tools\        ← bridge 腳本
+config.json   ← 設定
+```
 
 ## Build
 
 ```bash
 flutter pub get
-flutter create --platforms=windows .
-flutter build windows --release
+flutter build windows --release   # 桌面版
+flutter build apk --release       # Android（手機版）
+flutter build ios --release       # iOS（需 macOS + Xcode）
 ```
 
-## CLI
+## 一起聽（Jam）
 
-```bash
-dart run lib/cli_main.dart pipeline
-dart run lib/cli_main.dart spotube-download "Daily Mix 1"
-dart run lib/cli_main.dart spotube-download-all
-dart run lib/cli_main.dart spotube-move
-dart run lib/cli_main.dart status
-```
+在側邊欄選「一起聽」：
+
+1. **建立房間** — 輸入暱稱 →「建立新房間」。房主端會自動開啟
+   WebSocket + 音訊串流伺服器，並顯示房主位址（`IP:埠口`）與 6 位房間代碼。
+2. **加入房間** — 對方輸入房主位址 + 房間代碼 + 暱稱即可加入
+   （手機/桌機皆可；手機需與房主連同一個 Wi-Fi）。
+3. 任何人都能**搜尋 Spotify 加歌**（房主需先在本機「搜尋」頁登入 Spotify）、
+   **投票**（上下箭頭排序）、**投票跳歌**（達半數跳過）、**聊天**。
+   播放進度由房主掌控，成員每幾秒做 drift 校正，確保同步。
+
+技術重點：
 
-## Calibration
+- 房主是「伺服器」：`lib/services/jam_service.dart` 用 `dart:io` 自架
+  HttpServer + WebSocket，成員連 `ws://<ip>:<port>/jam/ws`。
+- 歌曲來源：本機音樂庫（`local`，`/jam/audio` 支援 Range 串流）或
+  Spotify 搜尋後由 yt-dlp 串流（`stream`，走既有的 `StreamServer`）。
+- 成員端透過 `PlayerController.playJamUrl()` 播放房主提供的 URL，
+  `jamFollowMode` 讓播放/暫停/上下首/seek 指令自動轉送給房主。
+- 手機相容：Windows 專用程式碼（win32 單一實例、SMTC、WebView 登入）
+  都已加平台守衛；Android/iOS 已加入 `flutter create` 平台資料夾與
+  網路權限（INTERNET / cleartext / 本地網路描述）。
+
+## CLI
 
 ```bash
-dart run lib/calibrate.dart
+playlist-admin pipeline [--step N]     # 全管線（抓取→下載→修剪→整理→LUFS）
+playlist-admin podcast                 # Podcast 管線
+playlist-admin status                  # 狀態
+playlist-admin favorite list|toggle    # 我的最愛
+playlist-admin rag build|query         # Podcast RAG
 ```
 
 ## Release
@@ -39,13 +86,15 @@ dart run lib/calibrate.dart
 Push a `v*` tag to trigger GitHub Actions:
 
 ```bash
-git tag v2.0.0
-git push origin v2.0.0
+git tag v2.9.0
+git push origin v2.9.0
 ```
 
 ## Dependencies
 
-- `win32` — Windows API (Spotube automation)
-- `http` + `html` — Spotify scraping
-- `fl_chart` — Charts
-- `ffi` — FFI support
+- `http` + `html` — Spotify embed API
+- `crypto` — TOTP（Spotify token）
+- `webview_windows`（vendor）— 登入 cookie 擷取
+- `audioplayers` — 播放
+- `fl_chart` — 圖表
+- `flutter_discord_rpc` — Discord presence

package/cli/index.js

@@ -16,10 +16,8 @@ Usage:
   playlist-admin pipeline [--step N]     Run full pipeline (or from step N)
   playlist-admin podcast                 Run podcast pipeline
   playlist-admin status                  Show status
-  playlist-admin spotube-download <name> Download one playlist
-  playlist-admin spotube-download-all    Download all playlists
-  playlist-admin spotube-move            Move M4A files
-  playlist-admin spotube-cleanup         Remove metadata-renamed mp3 duplicates
+  playlist-admin favorite list           List favorite songs
+  playlist-admin favorite toggle <song>  Toggle favorite (我的最愛)
   playlist-admin rag build [--reset]     Build podcast RAG vector DB
   playlist-admin rag query "問題" [--topk N] [--show 節目] [--json]
 `;

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "playlist-admin",
-  "version": "2.7.13",
-  "description": "playlist-admin CLI - thin launcher for the Flutter app engine (pipeline, podcast, spotube) + podcast RAG query",
+  "version": "2.15.14",
+  "description": "playlist-admin CLI - thin launcher for the Flutter app engine (pipeline, podcast) + podcast RAG query",
   "type": "module",
   "bin": {
     "playlist-admin": "./cli/index.js"
@@ -9,11 +9,7 @@
   "scripts": {
     "pipeline": "node cli/index.js pipeline",
     "podcast": "node cli/index.js podcast",
-    "status": "node cli/index.js status",
-    "spotube-download": "node cli/index.js spotube-download",
-    "spotube-download-all": "node cli/index.js spotube-download-all",
-    "spotube-move": "node cli/index.js spotube-move",
-    "spotube-cleanup": "node cli/index.js spotube-cleanup"
+    "status": "node cli/index.js status"
   },
   "files": [
     "cli/index.js",

package/rag/_resolve.py

@@ -1,13 +1,42 @@
 """Resolve podcast RAG paths from the app config (same as the CLI/GUI)."""
 import json
 import os
+import platform
 from pathlib import Path
 
 
+def _is_wsl() -> bool:
+    """偵測是否在 WSL 中執行。"""
+    if "microsoft" in platform.release().lower():
+        return True
+    if os.path.exists("/proc/sys/fs/binfmt_misc/WSLInterop"):
+        return True
+    return False
+
+
+def _win_to_wsl(path: Path) -> Path:
+    """將 Windows 路徑轉為 WSL /mnt/c/... 路徑。"""
+    s = str(path)
+    if len(s) >= 2 and s[1] == ":":
+        drive = s[0].lower()
+        rest = s[2:].replace("\\", "/")
+        return Path(f"/mnt/{drive}{rest}")
+    return path
+
+
 def find_base() -> Path:
     env_bp = os.environ.get("BASE_PATH")
     if env_bp:
-        return Path(env_bp)
+        bp = Path(env_bp)
+        if _is_wsl():
+            bp = _win_to_wsl(bp)
+        return bp
+
+    # WSL: 直接用 /mnt/c/ 路徑
+    if _is_wsl():
+        return Path("/mnt/c/Users/CPXru/Music/playlist-admin")
+
+    # Windows: 讀 config.json
     local = os.environ.get("LOCALAPPDATA")
     pointer = None
     if local:
@@ -22,15 +51,15 @@ def find_base() -> Path:
                 return Path(bp)
     except Exception:
         pass
-    return Path(r"C:\Users\CPXru\Music\Spotube")
+    return Path(r"C:\Users\CPXru\Music\playlist-admin")
 
 
 def podcast_downloads_dir() -> Path:
-    return find_base() / "podcast_downloads"
+    return find_base() / "podcasts"
 
 
 def rag_data_dir() -> Path:
-    return find_base() / "podcast_downloads" / "podcast_rag"
+    return find_base() / "cache" / "podcast"
 
 
 def chroma_db_dir() -> Path:

package/rag/build_db.py

@@ -114,13 +114,14 @@ def make_chunks(sentences: list[str], max_len: int = 600, overlap: int = 60) ->
 
 
 def detect_meta(path: Path) -> dict:
-    """從檔名/路徑猜節目與日期"""
+    """從檔名/路徑猜節目、日期與來源類型"""
     parts = path.parts
     show = parts[-2] if len(parts) >= 2 else "unknown"
     name = path.stem
     m = re.search(r"(20\d{2})[_\-_](\d{1,2})[_\-_](\d{1,2})", name)
     date = f"{m.group(1)}-{m.group(2).zfill(2)}-{m.group(3).zfill(2)}" if m else ""
-    return {"show": show, "file": name, "date": date}
+    source = "course" if re.search(r"夜工智|機械群|課程", str(path)) else "podcast"
+    return {"show": show, "file": name, "date": date, "source": source}
 
 
 def main() -> None:
@@ -131,6 +132,7 @@ def main() -> None:
     ap.add_argument("--reset", action="store_true", help="重建資料庫")
     ap.add_argument("--limit", type=int, default=0, help="只索引前 N 篇 (測試用)")
     ap.add_argument("--workers", type=int, default=6, help="併發 embedding 數")
+    ap.add_argument("--batch", type=int, default=64, help="每批 embedding 數量")
     args = ap.parse_args()
 
     client = chromadb.PersistentClient(path=args.db)
@@ -144,24 +146,99 @@ def main() -> None:
         metadata={"hnsw:space": "cosine"},
     )
 
-    # 已索引的檔名 (resume 用) - 分段讀取避免 SQL 變數超限
-    done_files = set()
+    # ── Dedup: remove duplicate sig entries, keep first only ──────
+    total = col.count()
+    if total > 0:
+        sig_to_ids: dict[str, list[str]] = {}
+        offset = 0
+        while offset < total:
+            batch = col.get(limit=5000, offset=offset, include=["metadatas"])
+            ids = batch["ids"]
+            metas = batch["metadatas"]
+            for rid, m in zip(ids, metas):
+                sig = (m or {}).get("sig", "")
+                if sig:
+                    sig_to_ids.setdefault(sig, []).append(rid)
+            offset += 5000
+        to_delete = [ids[1:] for ids in sig_to_ids.values() if len(ids) > 1]
+        flat_delete = [item for sublist in to_delete for item in sublist]
+        if flat_delete:
+            print(f"Dedup: removing {len(flat_delete)} duplicate records...")
+            for i in range(0, len(flat_delete), 4500):
+                col.delete(ids=flat_delete[i:i+4500])
+            print(f"Dedup done. New count: {col.count()}")
+
+    # 已索引內容 hash 去重 - 分段讀取避免 SQL 變數超限
+    done_sigs: set[str] = set()
     offset = 0
     while True:
         batch = col.get(limit=5000, offset=offset, include=["metadatas"])["metadatas"]
         if not batch:
             break
         for m in batch:
-            if m:
-                done_files.add(m.get("file", ""))
+            if m and m.get("sig"):
+                done_sigs.add(m["sig"])
+        offset += 5000
+        if offset > 1000000:
+            break
+
+    # 一次性遷移: 舊版 chunks 沒有 sig — 從 metadata 的 path 讀檔補上，
+    # 避免 1662 篇全部被當新的重嵌入。
+    def file_sig(path: Path) -> str:
+        raw = path.read_bytes()
+        # Strip BOM and normalize whitespace to avoid false re-embeds
+        if raw.startswith(b'\xef\xbb\xbf'):
+            raw = raw[3:]
+        return hashlib.md5(raw).hexdigest()
+
+    migrated = 0
+    offset = 0
+    while True:
+        batch = col.get(limit=5000, offset=offset,
+                        include=["metadatas"])["metadatas"]
+        if not batch:
+            break
+        for m in batch:
+            if not m or m.get("sig"):
+                continue
+            p = m.get("path")
+            if not p or not os.path.exists(p):
+                continue
+            try:
+                sig = file_sig(Path(p))
+                done_sigs.add(sig)
+                migrated += 1
+            except Exception:
+                pass
         offset += 5000
         if offset > 1000000:
             break
+    if migrated:
+        print(f"migrated {migrated} legacy chunks to content-sig dedup")
 
-    files = [f for f in sorted(Path(args.data).rglob("*.txt")) if f.stem not in done_files]
+    # 無內容/失敗檔案的跳過清單:0 chunks 的檔案不會寫進 DB,
+    # 若沒有持久狀態會每次 pipeline 都被當「待索引」重試。
+    skip_state_file = Path(args.db) / "_skipped_files.json"
+    skipped: set[str] = set()
+    if skip_state_file.exists():
+        try:
+            import json
+            skipped = set(json.loads(skip_state_file.read_text(encoding="utf-8")))
+        except Exception:
+            skipped = set()
+
+    # 檔案內容 hash — 內容更新(來源切換)時 sig 變 → 重嵌入
+    def has_content(path: Path) -> bool:
+        return bool(make_chunks(split_sentences(read_text_file(path))))
+
+    files = [
+        f for f in sorted(Path(args.data).rglob("*.txt"))
+        if file_sig(f) not in done_sigs
+        and not (f.stem in skipped and not has_content(f))
+    ]
     if args.limit:
         files = files[: args.limit]
-    print(f"待索引 {len(files)} 篇 (已跳過 {len(done_files)} 篇)")
+    print(f"待索引 {len(files)} 篇 (已嵌入 {len(done_sigs)} 篇, 無內容跳過 {len(skipped)} 篇)")
 
     total_chunks = 0
     t_start = time.time()
@@ -176,10 +253,19 @@ def main() -> None:
             )
         chunks = final_chunks
         if not chunks:
+            # 空/太短內容: 標記為已跳過, 避免每次重試
+            skipped.add(f.stem)
+            import json
+            skip_state_file.parent.mkdir(parents=True, exist_ok=True)
+            skip_state_file.write_text(
+                json.dumps(sorted(skipped), ensure_ascii=False, indent=1),
+                encoding="utf-8",
+            )
             continue
         meta = detect_meta(f)
+        sig = file_sig(f)
         try:
-            kept_chunks, vectors = embed_texts(args.model, chunks, workers=args.workers)
+            kept_chunks, vectors = embed_texts(args.model, chunks, batch=args.batch, workers=args.workers)
         except requests.exceptions.HTTPError as e:
             print(f"!! HTTP 錯誤 檔案={meta['file']} 前300字={chunks[0][:300] if chunks else ''}")
             raise
@@ -193,7 +279,7 @@ def main() -> None:
             for j in range(len(kept_chunks))
         ]
         metadatas = [
-            {**meta, "chunk": j, "chars": len(c), "path": str(f)}
+            {**meta, "sig": sig, "chunk": j, "chars": len(c), "path": str(f)}
             for j, c in enumerate(kept_chunks)
         ]
         col.add(ids=ids, documents=kept_chunks, metadatas=metadatas, embeddings=vectors)

package/rag/query.py

@@ -83,15 +83,29 @@ def generate_answer(question: str, hits: list[dict]) -> str:
                 json={
                     "model": model,
                     "messages": [{"role": "user", "content": prompt}],
-                    "stream": False,
+                    "stream": True,
                 },
                 timeout=600,
+                stream=True,
             )
             if resp.status_code in (400, 401, 404):
                 errors.append(f"{model}: HTTP {resp.status_code}")
                 continue
             resp.raise_for_status()
-            return resp.json()["message"]["content"].strip()
+            answer = []
+            for line in resp.iter_lines():
+                if not line:
+                    continue
+                try:
+                    chunk = json.loads(line)
+                    token = chunk.get("message", {}).get("content", "")
+                    if token:
+                        answer.append(token)
+                        print(token, end="", flush=True)
+                except json.JSONDecodeError:
+                    pass
+            print()  # 換行
+            return "".join(answer).strip()
         except requests.RequestException as e:
             errors.append(f"{model}: {e}")
     if errors: