thoughtleaders-cli 0.5.0__py3-none-any.whl

tl_cli/_plugin/skills/tl/references/elasticsearch-schema.md

@@ -0,0 +1,259 @@
+# ThoughtLeaders Elasticsearch Schema Reference
+
+## How to query
+
+All ES access goes through the `tl` CLI:
+
+```bash
+tl db es '{"size": 1, "query": {"match_all": {}}}' --json
+
+# Read body from stdin
+cat query.json | tl db es -
+```
+
+The index is **fixed server-side** (defaults to `tl-platform`). The client cannot select an index — there is no `--index` flag. To narrow a query to a smaller time window, scope it inside the body with a `publication_date` range filter rather than picking a different alias.
+
+Cost grows non-linearly with result size (raw db queries use the list curve at `mult=1.4`). Aggregation queries bill on `min(hits.total, 200)` instead of `len(hits)`. See `SKILL.md` for the curve formula and the row-count → credits table.
+
+Output flags: `--json`, `--csv`, `--md`, `--toon`. The CLI flattens hits into rows of `{_id, _score, ...source}`; aggregations come back in the response envelope and are rendered after the rows in TTY mode.
+
+## Accepted query bodies
+
+Read `SKILL.md` → "Raw query reference → `tl db es`" for the full list. Highlights:
+
+- **Top-level keys** accepted: `query`, `aggs`/`aggregations`, `sort`, `_source`, `size`, `from`, `track_total_hits`, `highlight`, `fields`, `min_score`, `search_after`, `timeout`, `collapse`, `post_filter`. Anything else (incl. `scroll`, `pit`, `runtime_mappings`, `knn`) is not accepted.
+- `size` ≤ 500. `from + size` ≤ 10,000. Use `search_after` to page deeper.
+- **Accepted query types** include `term`/`terms`/`match`/`bool`/`nested`/`range`/`exists`/`match_phrase`. `query_string`, `regexp`, `wildcard`, `fuzzy`, `more_like_this`, `has_child`, `has_parent`, `parent_id` are not accepted.
+- **No scripts** — any key whose name contains `script` is not accepted.
+- **At most one aggregation total** counted recursively (top-level + sub-agg = 2 = not accepted). Run multiple calls for multi-metric work.
+
+## Index Structure
+
+### `tl-platform-{year}-{quarter}` — Main Content Index
+
+The primary index. Contains videos AND channels as parent-child documents (`doc_type` join field).
+
+Sharded by quarter going back to 2015. **~15.6M docs in Q1 2026 alone.**
+
+Through `tl db es`, all queries hit a server-fixed alias (typically `tl-platform`, which fans out across every quarter). **Always add `publication_date` range filters** when narrowing to a time window — that's the only knob the client has, since the alias itself isn't selectable.
+
+The underlying physical layout (one index per quarter, e.g. `tl-platform-2026-q1`, with year and full-platform aliases on top) is for context only.
+
+Raw mappings (read-only links — out of band, not via `tl`):
+- [articles](https://github.com/ThoughtLeaders-io/elk-stack-resources/blob/main/elasticsearch/templates/_mappings_article.kibana)
+- [channels](https://github.com/ThoughtLeaders-io/elk-stack-resources/blob/main/elasticsearch/templates/_mappings_channel.kibana)
+- [shared configuration](https://github.com/ThoughtLeaders-io/elk-stack-resources/blob/main/elasticsearch/templates/_mappings_common.kibana)
+- [vector indexes](https://github.com/ThoughtLeaders-io/elk-stack-resources/blob/main/elasticsearch/templates/vectors.kibana)
+
+#### Video Fields (selected — 73 total)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | keyword | Video/article ID. Compound form `<channel_id>:<youtube_id>` (matches PG `adlink.article_id` and ES `_id`). |
+| `title` | text | Video title |
+| `description` | text | Video description |
+| `content` | text | Full content/transcript text |
+| `transcript` | text | Raw transcript |
+| `transcript_language` | keyword | Transcript language code |
+| `summary` | text | AI-generated summary |
+| `publication_date` | date | When video was published |
+| `discovery_time` | date | When TL discovered/indexed it |
+| `url` | object | Video URL |
+| `image_url` | object | Thumbnail URL |
+| `views` | long | View count |
+| `total_views` | long | Total views |
+| `projected_views` | long | Projected views |
+| `likes` | long | Like count |
+| `comments` | integer | Comment count |
+| `engagement` | long | Engagement metric |
+| `duration` | integer | Duration in seconds |
+| `duration_live` | integer | Live stream duration |
+| `duration_longform` | integer | Long-form duration |
+| `duration_shorts` | integer | Shorts duration |
+| `content_type` | keyword | longform / short / live |
+| `content_category` | keyword | Content category |
+| `content_aspects` | keyword | Content features/aspects |
+| `language` | keyword | Content language |
+| `country` | keyword | Creator country |
+| `format` | keyword | Platform format |
+| `hashtags` | keyword | Hashtags used |
+| `face_on_screen` | boolean | Whether creator shows face |
+
+#### Brand Mention Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `brand_mentions` | nested | Full brand mention objects |
+| `all_brand_mentions` | keyword | All brand IDs mentioned |
+| `sponsored_brand_mentions` | keyword | Sponsored brand IDs |
+| `organic_brand_mentions` | keyword | Organic brand IDs |
+| `banner_ads` | object | Banner ad data |
+| `not_sponsored_by` | object | Explicitly not sponsored by |
+
+#### Channel Fields (on video docs via `channel.id`, or on channel parent docs)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | text | Channel name |
+| `channel` | object | Channel metadata (nested on article docs) |
+| `reach` | long | Subscriber count |
+| `impression` | long | View count |
+| `impression_live` | long | Live view count |
+| `impression_shorts` | long | Shorts view count |
+| `is_tl_channel` | boolean | TPP partner channel |
+| `is_active` | boolean | Channel is active |
+| `media_selling_network_join_date` | date | MSN join date |
+| `has_outreach_email` | boolean | Has outreach email |
+| `outreach_email` | text | Contact email |
+| `social_links` | text | Social media links |
+| `male_share` | byte | Male audience % |
+| `usa_share` | byte | US audience % |
+| `sponsorship_price` | scaled_float | Sponsorship price |
+| `sponsorship_score` | scaled_float | Sponsorship quality score |
+| `evergreenness` | float | Evergreen score |
+| `evergreenness_live` | scaled_float | Live evergreen score |
+| `evergreenness_longform` | scaled_float | Longform evergreen score |
+| `evergreenness_shorts` | scaled_float | Shorts evergreen score |
+| `trend` | float | Growth trend |
+| `trend_live` | scaled_float | Live trend |
+| `trend_shorts` | scaled_float | Shorts trend |
+| `posts_per_90_days` | integer | Upload frequency |
+| `posts_per_90_days_live` | integer | Live frequency |
+| `posts_per_90_days_shorts` | integer | Shorts frequency |
+| `fulfillment_rate` | scaled_float | Fulfillment rate |
+| `renewal_rate` | scaled_float | Renewal rate |
+| `metrics_update_period` | byte | How often metrics update |
+| `offline_since` | date | When channel went offline |
+
+#### AI & Enrichment Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `ai` | object | AI-generated metadata |
+| `applied_enrichments` | keyword | Which enrichments have been applied |
+| `article_category` | object | Categorization data |
+
+#### System Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `@timestamp` | date | Index timestamp |
+| `doc_type` | join | Parent-child join (channel→video) |
+| `es_index_tag` | object | Index routing metadata |
+
+### Other indices
+
+- `tl-ingest` — ingestion queue. **Don't query.** Internal pipeline state.
+- `tl-feature-vectors-channel`, `tl-feature-vectors-channel-profile` — channel similarity vectors.
+- `tl-vectors-brand-company-descriptions-*` — brand similarity vectors.
+- `tl-vectors-channel-audience-*`, `tl-vectors-channel-topic-descriptions-*`, `tl-vectors-channel-features` — channel feature vectors.
+
+Note: `knn` queries against vector indices are **not currently accepted** as a top-level key. For "find similar" results, use `tl channels similar` / `tl brands similar` — they wrap the vector search server-side.
+
+## Common Query Patterns
+
+### Search videos by sponsored brand mention
+
+```bash
+tl db es '{
+  "size": 50,
+  "query": {"term": {"sponsored_brand_mentions": "5612"}},
+  "_source": ["title", "channel.id", "publication_date", "views"],
+  "sort": [{"publication_date": "desc"}]
+}'
+```
+
+### Search videos for a single channel
+
+```bash
+tl db es '{
+  "size": 100,
+  "query": {"term": {"channel.id": 12345}},
+  "sort": [{"publication_date": "desc"}]
+}'
+```
+
+### Count sponsored mentions for a brand (size:0 + track_total_hits)
+
+```bash
+tl db es '{
+  "size": 0,
+  "track_total_hits": true,
+  "query": {"term": {"sponsored_brand_mentions": "5612"}}
+}'
+```
+
+### Full-text search on title/description/summary/content
+
+```bash
+tl db es '{
+  "size": 20,
+  "query": {
+    "multi_match": {
+      "query": "ergonomic keyboard review",
+      "fields": ["title^3", "description", "summary", "content"]
+    }
+  },
+  "_source": ["title", "channel.id", "publication_date"]
+}'
+```
+
+### Filter by date range
+
+```bash
+tl db es '{
+  "size": 100,
+  "query": {
+    "bool": {
+      "filter": [
+        {"term": {"channel.id": 12345}},
+        {"range": {"publication_date": {"gte": "2026-01-01", "lte": "2026-03-31"}}}
+      ]
+    }
+  }
+}'
+```
+
+### Single top-level aggregation (only one aggregation per request is accepted)
+
+```bash
+tl db es '{
+  "size": 0,
+  "aggs": {
+    "by_channel": {
+      "terms": {"field": "channel.id", "size": 20}
+    }
+  },
+  "query": {"term": {"sponsored_brand_mentions": "5612"}}
+}'
+```
+
+For more dimensions, run multiple `tl db es` calls and join client-side.
+
+### Deep pagination via `search_after`
+
+```bash
+# First page — sort must include a tiebreaker on _id for stability
+tl db es '{
+  "size": 500,
+  "query": {"term": {"channel.id": 12345}},
+  "sort": [{"publication_date": "desc"}, {"_id": "asc"}]
+}'
+
+# Subsequent pages — pass the last hit's sort values as search_after
+tl db es '{
+  "size": 500,
+  "query": {"term": {"channel.id": 12345}},
+  "sort": [{"publication_date": "desc"}, {"_id": "asc"}],
+  "search_after": ["2025-09-14", "12345:abc123"]
+}'
+```
+
+## Notes & gotchas
+
+- **Composite IDs:** `tl-platform.id` and `_id` are `<channel_id>:<youtube_id>`. The `youtube_id` portion alone is what Firebolt's `article_metrics.id` stores.
+- **Add a `publication_date` range filter** whenever the question is time-bounded — the alias is fixed, so this is the only way to narrow the search.
+- `sponsored_brand_mentions` and `organic_brand_mentions` are keyword arrays — use `term` queries.
+- For brand mention details (position, snippet, detection_tool), the data is in the `brand_mentions` nested field.
+- **`publication_id` is deprecated** — don't use for joins.
+- No write access. The CLI only exposes `_search` against `tl-platform-*`.

tl_cli/_plugin/skills/tl/references/firebolt-schema.md

@@ -0,0 +1,208 @@
+# ThoughtLeaders Firebolt Schema Reference
+
+## How to query
+
+All Firebolt access goes through the `tl` CLI:
+
+```bash
+tl db fb "SELECT id, age, view_count FROM article_metrics
+          WHERE channel_id = 12345 AND id = 'dQw4w9WgXcQ'
+          ORDER BY age" --json
+
+# Read SQL from stdin
+cat curve.sql | tl db fb -
+```
+
+Cost grows non-linearly with result size (raw db queries use the list curve at `mult=1.4`). A tightly-scoped `WHERE channel_id = X AND id = Y` query rarely costs more than ~10 credits even with months of snapshots; a channel-wide `WHERE channel_id = X` over a busy channel can run into the hundreds. See `SKILL.md` for the curve formula and the row-count → credits table.
+
+Output flags: `--json`, `--csv`, `--md`, `--toon`.
+
+For **standard view-curve / channel-growth** questions, prefer the higher-level commands — they implement the project's interpolation/sparseness handling:
+
+```bash
+tl snapshots channel <channel_id> --json
+tl snapshots video <video_id> --channel <channel_id> --json   # --channel is mandatory
+```
+
+Drop to `tl db fb` only when you need a shape `tl snapshots` doesn't produce (custom aggregates, milestone-age slices, multi-channel growth comparisons, etc.).
+
+## Accepted queries
+
+(See `SKILL.md` → "Raw query reference → `tl db fb`" for full reasoning.)
+
+- **SELECT only.** No DDL/DML/transactions/SET/locks.
+- **Single table.** No JOIN, CTE (`WITH`), subquery, set operation, or `LATERAL`.
+- **Only known tables:** `article_metrics`, `channel_metrics`. Other names return `UNKNOWN_TABLE`.
+- **WHERE/HAVING may only reference indexed columns** (`channel_id`/`id` for `article_metrics`; `id` for `channel_metrics`). Filtering by `age`, `publication_date`, `view_count`, `duration`, `scrape_date`, etc. in WHERE returns `NON_INDEXED_FILTER:<col>`. Apply those constraints client-side after fetching.
+- **Leading index column must be equality-or-IN-filtered with literals** (`channel_id = 1` or `channel_id IN (1,2,3)`). Without it: `MISSING_INDEXED_FILTER`.
+- **Trivial-aggregation exception:** a SELECT whose projected expressions are all aggregates and which has no GROUP BY / HAVING may omit WHERE entirely. Use only for tiny sanity checks.
+- **No mandatory LIMIT/OFFSET** — but Firebolt will time out on bad plans, so keep the leading-index filter selective.
+
+## Tables
+
+### `article_metrics` — Video-Level Time-Series (PRIMARY TABLE)
+
+**7.4 billion rows** | 159 GiB compressed | Data from 2022-03-04 to present.
+
+Tracks YouTube video metrics over time. Each row = one scrape of one video on one date. Videos are scraped repeatedly, so a single video has many rows at different ages.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | TEXT | YouTube video ID (e.g., `'dQw4w9WgXcQ'`). **Bare YouTube ID** — NOT the compound `<channel_id>:<youtube_id>` form used in PG `adlink.article_id` and ES `_id`. |
+| `channel_id` | INT | TL channel ID (matches the channel ID returned by `tl channels list`) |
+| `channel_format` | INT | Platform format (4 = YouTube) |
+| `publication_date` | DATE | When the video was published |
+| `scrape_date` | DATE | When this data point was captured |
+| `age` | INT | Days since publication (`scrape_date - publication_date`) |
+| `view_count` | INT | Total view count at time of scrape |
+| `like_count` | INT | Total likes at time of scrape |
+| `comment_count` | INT | Total comments at time of scrape |
+| `duration` | INT | Video duration in seconds |
+
+**Primary Index: `(channel_id, id)`** — queries MUST filter by `channel_id` first.
+
+**Shorts vs Longform:** `duration < 61` = Short. `duration >= 61` = Longform. In code: `(duration or 100) < 61`. Filter client-side after fetching — `duration` isn't an indexed column so it can't go in WHERE.
+
+### `channel_metrics` — Channel-Level Time-Series
+
+**1.1 billion rows** | 6.9 GiB compressed.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | INT | TL channel ID |
+| `total_views` | INT | Channel total views at time of scrape |
+| `reach` | INT | Subscriber count at time of scrape |
+| `scrape_date` | DATE | When this data point was captured |
+
+**Primary Index: `(id)`**
+
+## When to use Firebolt (and when NOT to)
+
+Firebolt's **only advantage** is historical metric snapshots. For everything else, prefer ES (current state) or the structured `tl` commands (deal/pipeline data).
+
+| Need | Source |
+|------|--------|
+| Current view count on a video | **Elasticsearch** (`tl uploads show` or `tl db es`) |
+| Current subscriber count | **Elasticsearch** (`tl channels show`) |
+| Video metadata (title, tags, duration) | **Elasticsearch** |
+| Find channels by criteria | **`tl channels list`** (Postgres) |
+| Deal / pipeline / sponsorship data | **`tl sponsorships`** etc. (Postgres) |
+| View curve over time (age 7→30→90→180) | **Firebolt** ✅ |
+| Views at age 30 vs age 180 (evergreenness) | **Firebolt** ✅ |
+| Channel subscriber growth trend | **Firebolt** ✅ |
+| Detect view spikes / anomalies over time | **Firebolt** ✅ |
+
+**Rule: only query Firebolt when you need a value AT A POINT IN TIME that no longer exists in the current ES/PG snapshot.**
+
+## Index rules — verified timings
+
+`article_metrics` has 7.4B rows. The primary index is `(channel_id, id)`. Without `channel_id` in WHERE, the engine does a full table scan and times out — such queries are not accepted up front.
+
+| Query Pattern | Performance | Result |
+|--------------|-------------|--------|
+| `WHERE channel_id = X AND id = Y` | ~12s | ✅ Full index match |
+| `WHERE channel_id = X` | ~12s | ✅ Partial index, viable |
+| `WHERE channel_id IN (X, Y, Z)` | ~12–30s | ✅ Multiple lookups, viable |
+| `WHERE id = 'abc'` (no channel_id) | n/a | ❌ rejected (`MISSING_INDEXED_FILTER`) |
+| `WHERE publication_date > X` | n/a | ❌ rejected (`NON_INDEXED_FILTER:publication_date`) |
+| `WHERE age = 30 AND view_count > 50000` | n/a | ❌ rejected (multiple `NON_INDEXED_FILTER`) |
+
+For `channel_metrics`, primary index is `(id)`. Same rule: always filter by `id`.
+
+## Workflow: resolve IDs first, then query
+
+Every Firebolt workflow has two steps:
+
+**Step 1 — get `channel_id` and (optionally) video IDs from PG/ES.**
+
+```bash
+# Channels matching some criterion (PG side)
+tl channels list category:tech --json --limit 50 | jq '.results[].id'
+
+# Or videos for a specific brand's deals (Postgres side, via tl sponsorships)
+tl deals list brand:"Nike" --json --limit 500 \
+  | jq -r '.results[] | select(.article_id != null) | "\(.channel_id):\(.article_id)"'
+
+# Or videos via Elasticsearch content search
+tl db es '{
+  "size":100,
+  "query":{"term":{"sponsored_brand_mentions":"5612"}},
+  "_source":["channel.id","id"]
+}' --json | jq '.results[] | {channel_id: .channel.id, id: (.id | split(":")[1])}'
+```
+
+**Step 2 — query Firebolt with those IDs.**
+
+```bash
+# Best: full index
+tl db fb "SELECT id, age, view_count FROM article_metrics
+          WHERE channel_id IN (123, 456, 789)
+            AND id IN ('abc', 'def', 'ghi')
+          ORDER BY id, age"
+
+# Acceptable: channel_id only (scans all videos for that channel)
+tl db fb "SELECT id, age, view_count FROM article_metrics
+          WHERE channel_id = 12345
+          ORDER BY id, age"
+```
+
+For non-indexed filters (`age IN (30, 180)`, `duration > 60`), pull a slightly wider slice and filter in `jq`/Python.
+
+## Common Query Patterns
+
+### Get a video's full view curve
+
+```bash
+tl db fb "SELECT id, age, view_count, like_count, comment_count, duration
+          FROM article_metrics
+          WHERE channel_id = 12345 AND id = 'dQw4w9WgXcQ'
+          ORDER BY age"
+```
+
+### Get all videos for a channel, then filter client-side to milestone ages and longform
+
+```bash
+tl db fb "SELECT id, age, view_count, like_count, comment_count, duration, publication_date
+          FROM article_metrics
+          WHERE channel_id = 12345
+          ORDER BY id, age" --json \
+| jq '.results[] | select(.duration > 60 and (.age == 7 or .age == 30 or .age == 60 or .age == 90 or .age == 180 or .age == 365))'
+```
+
+### Channel subscriber/view growth over time
+
+```bash
+tl db fb "SELECT scrape_date, total_views, reach
+          FROM channel_metrics
+          WHERE id = 12345
+          ORDER BY scrape_date"
+```
+
+### Compare multiple channels' growth
+
+```bash
+tl db fb "SELECT id, scrape_date, total_views, reach
+          FROM channel_metrics
+          WHERE id IN (123, 456, 789)
+          ORDER BY id, scrape_date"
+```
+
+## Sparse data warning
+
+Snapshots are **sparse**, especially for older videos (the project's scrape cadence backs off as videos age). Channels are snapshotted daily. Do **not** assume two arbitrary dates have data points. For approximations between gaps, prefer `tl snapshots` (which implements project-internal interpolation logic) over hand-rolled raw queries.
+
+## How Firebolt powers other features
+
+- **Evergreenness:** `evergreenness = (views_at_180 - views_at_30) / views_at_30`. Falls back to `(views_at_90 × 1.265 - views_at_30) / views_at_30` if 180 isn't available. Threshold `views_180 ≥ views_30 × 2`. Min views: 5,000. Stored on the ES `evergreenness` field and on the channel record returned by `tl channels show` — read those first; only recompute from Firebolt for investigations.
+- **Growth/Trend stats** (initial → middle → current dynamics) are computed server-side.
+- **View-curve approximation:** linear (channel metrics, gap ≤ 10 days) or logarithmic (article metrics, `views = a · log(b · age)`). Server-side; not exposed to raw `tl db fb`.
+
+## Use cases (mostly underexplored)
+
+- Late viral detection: dormant videos that spike — what caused it?
+- Evergreen outliers: abnormally high/low evergreen scores — why?
+- Engagement shifts: like/comment ratios changing dramatically over time.
+- View projection: fit logarithmic curve, project future views.
+- Sponsorship timing: when in a video's lifecycle does a sponsorship deliver most views?
+- Anomaly alerts: detect unusual view patterns and notify.
+- Performance benchmarking: a video's curve vs its niche average.