thoughtleaders-cli 0.9.1__tar.gz → 0.9.2__tar.gz

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tl-cli",
-  "version": "0.9.1",
+  "version": "0.9.2",
   "description": "ThoughtLeaders CLI — query sponsorship deals, channels, brands, uploads, and intelligence from the terminal",
   "author": {
     "name": "ThoughtLeaders",

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/AGENTS.md

@@ -95,7 +95,7 @@ Warn the user if they are creating a release and the latest commit didn't bump t
 ## Coding
 
 * Do not reference internal architecture of the ThoughtLeaders app in comments or skills. Specifially: do not reference internal table names, field names, API endpoints, Python modules or functions (including the sanitizer).
-* Do not let server implementation details into skill files (anything under `skills/`). Skills describe *what the CLI does* from the user's seat — observable command surface, inputs, outputs, examples. Do not say "the server enforces X", "the API validates Y on its side", "the backend rejects Z" — those are mechanism notes that drift the moment the server changes. State the user-visible behaviour ("unknown keys come back as 400") without naming where it's enforced.
+* Do not let server implementation details or internal Django project terminology into skill files (anything under `skills/`). Skills describe *what the CLI does* from the user's seat — observable command surface, inputs, outputs, examples. Do not say "the server enforces X", "the API validates Y on its side", "the backend rejects Z" — those are mechanism notes that drift the moment the server changes. Do not name Django enums, model/class names, app labels, or settings keys (e.g. `ContentField`, `dashboard.enums.*`) — describe the concept in user-observable terms instead. State the user-visible behaviour ("unknown keys come back as 400") without naming where it's enforced.
 * **All `import` and `from X import Y` statements live at the top of the Python module file** — after the module docstring, before any code. No inline imports inside function bodies, no lazy imports for "speed" or "optional dependency" reasons. `from __future__ import …` goes at the very top (Python requires that). The only legitimate inline-import exception is **platform-conditional imports** that cannot succeed on the other platform (e.g. `import msvcrt` on Linux, `import termios`/`tty` on Windows) — those stay inside their `if sys.platform == …:` guard. If a circular-import problem makes a top-level import impossible, fix the circular dependency rather than working around it with an inline import.
 
 # Updating

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: thoughtleaders-cli
-Version: 0.9.1
+Version: 0.9.2
 Summary: ThoughtLeaders CLI — query sponsorship data, channels, brands, and intelligence
 Project-URL: Homepage, https://thoughtleaders.io
 Project-URL: Repository, https://github.com/ThoughtLeaders-io/thoughtleaders-cli

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "thoughtleaders-cli"
-version = "0.9.1"
+version = "0.9.2"
 description = "ThoughtLeaders CLI — query sponsorship data, channels, brands, and intelligence"
 readme = "README.md"
 license = "MIT"

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/skills/tl/SKILL.md

@@ -242,21 +242,19 @@ tl channels update 12345 '{"demographic_male_share": 62}'
 tl channels update 12345 '{"demographic_geo": {"US": 60, "UK": 12, "CA": 8}}'
 tl channels update 12345 '{"demographic_male_share": 55, "demographic_usa_share": 70}'
 tl channels update 12345 '{"outreach_email": "press@creator.com"}'
-tl channels update 12345 '{"all_emails": {"press@creator.com": {"timestamp": "2026-06-25T12:00:00+00:00", "source": "manual"}}}'
 tl channels update 12345 '{"media_selling_network_join_date": "2026-01-15"}'
 ```
 
-**Channel contact emails.** Besides demographics, `tl channels update` accepts two
-contact fields:
+**Channel contact emails.** Besides demographics, `tl channels update` accepts one
+contact field:
 
 - **`outreach_email`** — the channel's primary outreach address (a single email string, or `null`).
-- **`all_emails`** — the channel's full email archive. It is **not** a free-form blob: it's
-  a JSON object keyed by email address, each value an object `{"timestamp": <ISO-8601 datetime>, "source": <string>}`.
-  `timestamp` must parse as ISO-8601 (e.g. `"2026-06-25T12:00:00+00:00"`); `source` records
-  where the address came from. Known `source` values: `outreach_email` (the address that was
-  the primary outreach email, archived when the channel was blacklisted), `google_search`,
-  `scheduled_rescrape`. Use a descriptive token such as `manual` for hand-added entries.
-  An edit **replaces** the whole `all_emails` object, so include every entry you want to keep.
+
+The channel's full email archive (`all_emails` — a JSON object keyed by email address,
+each value recording when and where the address was found) is **not editable here**;
+sending it returns a 400. It is append-only: new addresses are added with the
+internal CLI's `tl-internal channels add-email` command (superuser-only), which never
+modifies or removes existing entries.
 
 **MSN membership.** `tl channels update` also accepts **`media_selling_network_join_date`** —
 the date the channel joined the Media Selling Network, as a `YYYY-MM-DD` string. This date is

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/skills/tl/references/business-glossary.md

@@ -97,7 +97,7 @@ Vocabulary that AMs use about channels, mapped to the actual DB encoding. Most o
 | **Brand on a deal** | `adlink → advertiser_profile_id → profile_brands.profile_id → profile_brands.brand_id → brand` | 3-table chain. There is NO direct `brand_id` on adlink. See [postgres-schema.md](postgres-schema.md). |
 | **Channel on a deal** | `adlink.ad_spot_id → adspot.channel_id → channel` | NO direct `channel_id` on adlink. |
 | **Brand-virgin / VPN-virgin (etc.)** | Channel has no `adlink` row joined to any of the target brand_ids | Used in candidate sourcing ("never sponsored by any VPN brand"). Caveat: only catches TL-brokered deals; channels that ran the brand directly (no TL involvement) appear "virgin" but aren't — cross-check ES `sponsored_brand_mentions` before final outreach. |
-| **Channel quality score** *(internal-only)* | `sponsorship_score` on the indexed channel doc + `thoughtleaders_channel.sponsorship_score` (PG) | TL-internal composite score combining engagement, fulfillment, and historical sponsorship performance. **Use it internally to rank/tiebreak candidates, but do NOT quote the raw decimal in AM-facing or external output** — the score isn't documented to AMs and the absolute value isn't meaningful without context. In AM-facing prose, translate to qualitative language: "top-quartile fit," "strongest quality score in the candidate set," "high sponsorship-quality signal." |
+| **Channel quality score** *(internal-only)* | `sponsorship_score` on the indexed channel doc + `thoughtleaders_channel.sponsorship_score` (PG) | TL-internal sponsorship track-record score, range 0–10: blends how many distinct brands sponsored the channel over the past 2 years (40%) with how many of them booked repeatedly (60%). Scores below 5 read as little/no track record (the low range is deliberately compressed). **Use it internally to rank/tiebreak candidates, but do NOT quote the raw decimal in AM-facing or external output** — the score isn't documented to AMs and the absolute value isn't meaningful without context. In AM-facing prose, translate to qualitative language: "top-quartile fit," "strongest quality score in the candidate set," "high sponsorship-quality signal." |
 
 ## Projected Views (PV) — three related but distinct fields
 
@@ -107,6 +107,7 @@ AMs use "PV" loosely. There are three different DB fields, each meaning somethin
 |---------|----------|---------------------|
 | **PV (channel baseline)** | `thoughtleaders_channel.projected_views` | Channel-level "typical views per video" used as CPM denominator. ⚠️ Coverage and freshness vary; cross-check Firebolt longform median for hero-tier deals. |
 | **PV (deal-specific)** | `thoughtleaders_adlink.projected_views_at_purchase_date` | Snapshot of projected views at the moment the deal was sold. Use this for historical CPM analysis. |
+| **PV (video, at upload)** — the web app's "video projected views" | `projected_views` on the ES **video** doc | The channel's format-specific PV **frozen when TL first indexed the video** (never updated afterwards). Compare to the channel's current PV to see growth/decline since the upload, or to the video's actual `views` to see over/under-performance vs expectation. See the ES schema reference for the calculation. |
 | **VG (View Guarantee)** | `thoughtleaders_adlink.views_guarantee` | The contractual minimum views the brand is guaranteed. 0/NULL = no guarantee. NOT the same as PV — VG is a contractual floor, PV is an estimate. |
 
 When an AM says "what's the PV on this channel?" — they almost always mean `channel.projected_views`. When they say "what was the PV on this deal?" — they mean `adlink.projected_views_at_purchase_date`. When they say "did we hit the VG?" — they mean `adlink.view_guarantee_hit_date IS NOT NULL`.
@@ -120,7 +121,7 @@ Two derived metrics on the indexed channel doc that AMs use to qualify a channel
 | **Fulfillment rate** | `fulfillment_rate` (channel doc, scaled_float) | The share of a channel's content that is sponsored — `sponsored / all` content over the measurement window, expressed as a fraction. Higher = the channel reliably delivers paid integrations. | Quality signal: a high fulfillment rate means past brands have actually run on this channel, not just been pitched. AMs use it to filter out "looks promising but never closes" channels. |
 | **Renewal rate** | `renewal_rate` (channel doc, scaled_float) | The rate at which a brand-channel sponsorship relationship repeats over time, computed from clusters of sponsorship deals between a single subject (channel or brand) and its linked entities, with date-distribution heuristics (default 365-day max interval). | Loyalty signal: a high renewal rate means brands keep coming back to this channel. AMs use it to identify "sticky" channels worth premium positioning, and to flag low-renewal channels as one-shots. |
 
-Both metrics live on the channel side of the indexed video docs (the `channel.*` nested object). Channel pages in TL's product surface these as quality scores; in AM-facing reports, you can quote them as percentages (`0.45 → "45% renewal rate"`).
+Both metrics live on the **channel docs** in the search index — not on video docs (the embedded `channel.*` object on video docs carries no metrics). Channel pages in TL's product surface these as quality scores; in AM-facing reports, you can quote them as percentages (`0.45 → "45% renewal rate"`).
 
 ## Industry Terms vs TL Vocabulary
 

thoughtleaders_cli-0.9.2/skills/tl/references/elasticsearch-schema.md

@@ -0,0 +1,286 @@
+# ThoughtLeaders Elasticsearch Schema Reference
+
+## How to query
+
+All ES access goes through the `tl` CLI:
+
+```bash
+tl db es '{"size": 1, "query": {"match_all": {}}}' --json
+
+# Read body from stdin
+cat query.json | tl db es -
+```
+
+The index is **fixed server-side**. The client cannot select an index — there is no `--index` flag.
+
+Cost grows non-linearly with result size (raw db queries use the list curve at `mult=1.4`). Aggregation queries bill on `min(hits.total, 200)` instead of `len(hits)`. See `SKILL.md` for the curve formula and the row-count → credits table.
+
+Output flags: `--json`, `--csv`, `--md`, `--toon`. The CLI flattens hits into rows of `{_id, _score, ...source}`; aggregations come back in the response envelope and are rendered after the rows in TTY mode.
+
+## Accepted query bodies
+
+See the output of `tl db es`" for the object schema. Highlights:
+
+- **Top-level keys** accepted: `query`, `aggs`/`aggregations`, `sort`, `_source`, `size`, `from`, `search_after`, `track_total_hits`, `highlight`, `fields`, `min_score`, `timeout`, `collapse`, `post_filter`. Anything else (incl. `scroll`, `pit`, `runtime_mappings`, `knn`) is not accepted.
+- `size` ≤ 10,000. `from + size` ≤ 10,000 — to page past 10,000 hits use `search_after` (see *Deep pagination* below), not `from`.
+- `search_after` must be a non-empty array of ≤ 10 scalar sort values, requires an explicit `sort`, and `from` must be 0 or omitted.
+- **Accepted query types** include `term`/`terms`/`match`/`bool`/`nested`/`range`/`exists`/`match_phrase`/`simple_query_string` (the sanctioned rich-Boolean text surface — `+` required, `|` OR, `-` NOT, `"phrase"`, `"phrase"~N` slop, trailing-`*` prefix, `~N` fuzzy, `(...)` grouping, per-field `^boosts`; always set `default_operator: "and"` and an explicit `fields` list or counts balloon). `query_string`, `regexp`, `wildcard`, `fuzzy`, `more_like_this`, `has_child`, `has_parent`, `parent_id` are not accepted.
+- **No scripts** — keys that start with `script` (e.g. `script_fields`, `script_score`, `scripted_metric`) or end with `_script` (e.g. `bucket_script`) are not accepted. A field whose name merely contains `script` as a substring (e.g. `transcript`, `description`) is fine.
+- **Aggregations are bounded, not forbidden**: up to ~50 agg nodes per body, bucket nesting ≤ 3 levels deep (single-bucket framing aggs like `filter`/`nested`/`missing` count toward depth), terms-like bucket `size` ≤ 10,000, and a worst-case total bucket count cap — so a `filter` agg wrapping a `cardinality` (the common counts-plus-recency shape) is fine in one call. Heavy/experimental agg types (e.g. `significant_terms`, script-based pipeline aggs) are not accepted.
+- **`highlight` is accepted in the body but highlight fragments are not returned** — result rows carry only `_id`, `_score`, and the `_source` fields. Request the `_source` fields you need for validation samples instead.
+
+### ElasticSearch document structure ("articles")
+
+The `doc_type` join field distinguishes video uploads ("articles") from channel data — channel docs are parents, article docs are their children. Filter with `{"term": {"doc_type": "article"}}` or `{"term": {"doc_type": "channel"}}`. ⚠️ Term-querying `doc_type.name` matches nothing — even though article docs' `_source` shows `doc_type` as an object with a `name` key, that's join-field syntax, not a queryable subfield.
+
+#### Upload/video Fields (selected — 73 total)
+
+Filter with `{"term": {"doc_type": "article"}}`. Coverage percentages are live `exists` counts (July 2026, ~676M video docs) — they drift slowly as the index grows.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | keyword | Video/article ID. Compound form `<channel_id>:<youtube_id>` (matches PG `adlink.article_id` and ES `_id`). |
+| `title` | text | Video title (~100%) |
+| `description` | text | ⚠️ **Does not exist on video docs** — `exists` matches 0 of ~676M (verified). The video's description text lives in `summary`; `description` is a channel-doc field (the channel's "About this channel" text). |
+| `content` | text | ⚠️ **Podcast episodes only** — the episode's show-notes/body text from the podcast feed (often HTML fragments). ~7% of docs overall; effectively absent on YouTube videos (~256k legacy docs holding flat transcript prose, and 0 YouTube docs since 2025). Never search it for YouTube content — use `summary` / `transcript`. |
+| `transcript` | text | Raw transcript — stored as YouTube timed-text **XML**, not plain text (see note below). ~57% of docs; present on both longform and shorts. |
+| `transcript_language` | keyword | Language code of the caption track the transcript came from (present when `transcript` is) |
+| `summary` | text | ⚠️ **Misleading name — this is the video's creator-written description** (the text under the video: promo links, hashtags, timestamps, subscribe blocks), NOT an AI summary. Verified by sampling old and recent docs. ~86% of docs. This is *the* field for searching video-description text. |
+| `evergreenness` | float | Per-video evergreen score: `(views at age 180d − views at age 30d) / views at age 30d`. ≥ 1 = evergreen (views at day 180 are at least double the day-30 views). Only computed for videos with ≥ 5,000 views published since 2022 (~16% of docs). |
+| `publication_date` | date | When the video was published (~100%) |
+| `discovery_time` | date | When TL first indexed the video. Only ~39% of docs — absent on older docs. |
+| `url` | keyword | Watch/episode URL. **Stored-only: retrievable in `_source` but not searchable** (`exists`/`term` match 0 docs). |
+| `image_url` | keyword | Thumbnail URL on podcast docs; absent on YouTube video docs. Stored-only, not searchable. |
+| `views` | long | View count at the last metrics update (~92%) |
+| `projected_views` | long | **The channel's projected views for this video's format, frozen at the time TL first indexed the video** (~45%) — the web app's "video projected views". TL's prediction of views at age = 30 days, computed **only from the channel's recent uploads of the same content type** (a short's value comes from the channel's shorts, a longform video's from its longform uploads — never mixed): ≥ 4 same-format videos' day-30 views, median-anchored with outliers trimmed. **Never updated after first index**, so it's a snapshot: compare it to the channel doc's current `impression*` to see whether the channel grew or declined since the upload, and to the video's `views` to see whether the video over- or under-performed expectations. Caveats: the format bucket at stamp time is duration-based (≤ 60s → shorts projection; anything longer — including live streams — gets the longform projection), and for channels added to TL after the fact "first indexed" is discovery time, not the publish date. |
+| `likes` | long | Like count (~87%) |
+| `comments` | integer | Comment count (~67%) |
+| `duration` | integer | Video duration in seconds (~100%) |
+| `content_type` | keyword | `longform` / `short` / `live` — the complete value set. ~71% of docs; older docs have none (missing ≠ longform). Podcast/RSS docs have no `content_type`. |
+| `content_aspects` | keyword | Flags: `podcast`, `paid_promotion`, `unlisted` — the complete value set. Only ~2.6% of docs carry any. |
+| `hashtags` | keyword | Hashtags from the video description, stored **without the leading `#`** and lowercase (e.g. `marchmadness`); non-Latin tags appear percent-encoded (`%D0%B0…`) (~32%) |
+| `channel` | object | Embedded channel subset: `channel.id`, `channel.content_category`, `channel.format`, `channel.publication_id`, `channel.country`, `channel.language` — no text fields, no metrics. This is where a video's language/country/format/category live (top-level `language`, `country`, `format`, `content_category` exist only on channel docs). |
+
+⚠️ **Channel-doc fields that look like video fields but match 0 video docs:** `total_views`, `engagement`, `duration_live`/`duration_longform`/`duration_shorts`, `language`, `country`, `format`, `content_category`, `face_on_screen`. They live on channel docs (see below); on video docs use the embedded `channel.*` subset where available.
+
+#### Brand Mention Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `brand_mentions` | nested | Detected brand-mention objects (`id`, `type` organic/sponsored, `field`, `snippet`, `start_ts`/`end_ts`, `position`, `probability`, `detection_tool`) (~12%). Being `nested`, it must be queried with a `nested` query — a plain `{"exists": {"field": "brand_mentions"}}` matches 0 docs. |
+| `all_brand_mentions` | keyword | Brand IDs with any mention — the union of sponsored + organic (~12%) |
+| `sponsored_brand_mentions` | keyword | Brand IDs with a sponsored mention |
+| `organic_brand_mentions` | keyword | Brand IDs with an organic mention |
+| `not_sponsored_by` | object | Brand IDs marked as explicitly *not* sponsoring this video. **Not searchable** (`exists` matches 0 docs). |
+
+#### Channel Fields
+
+Filter with `{"term": {"doc_type": "channel"}}`. ~45.5M channel docs (July 2026); coverage percentages below are live `exists` counts against that total.
+
+Contains a denormalized subset of the PostgreSQL channel data.
+
+⚠️ **Channel docs are duplicated** — one channel id can appear as several identical docs (a well-known channel showed 8+ copies). Doc counts ≠ channel counts; dedupe with `collapse` on `id` or aggregate with `cardinality`.
+
+⚠️ **Legacy field names** — PostgreSQL and Firebolt renamed these, Elasticsearch did not. Raw ES queries must use the old names; the new names match **0 docs** (verified live, they fail silently):
+
+| ES (use this) | PG `thoughtleaders_channel` | Firebolt `channel_metrics` |
+|---|---|---|
+| `reach` | `subscribers` | `subscribers` |
+| `impression` / `impression_live` / `impression_shorts` | `projected_views*` | `projected_views` |
+| `is_tl_channel` | `is_tpp` | — |
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | text | Channel display name (~100%) |
+| `description` | text | The channel's creator-written YouTube "About this channel" text (~97%): usually first-person, links, promo. Especially worth investigating if it contains creator contact information. |
+| `description.domains` | text | Same "About this channel" text, additionally indexed so **website domains are searchable as single terms** — `{"term": {"description.domains": "patreon.com"}}` matches channels whose About text *links to* patreon.com anywhere (including inside full URLs), while plain topic words match identically to `description` (verified: same counts). This is the field the platform's `channel_description` report filter actually searches. |
+| `reach` | long | Subscriber count (~98%). ⚠️ NOT ad-industry "reach" (unique audience exposed) — this is the channel's subscriber count. Same data as PG `thoughtleaders_channel.subscribers` / Firebolt `channel_metrics.subscribers`. |
+| `impression` | long | TL's projected views per longform video at age = 30 days — computed only from the channel's recent **longform** videos' day-30 views (≥ 4 required; median-anchored, outliers trimmed) (~27%). This is the channel's *current* projection; a video doc's `projected_views` is the same quantity frozen when that video was first indexed — diff them to see channel growth/decline since the upload. ⚠️ NOT actual views and NOT ad-industry "impressions"; for actual views see `total_views` / the video docs. |
+| `impression_live` | long | Projected views per live stream at age = 30 days, from the channel's live streams only (~6%) |
+| `impression_shorts` | long | Projected views per short at age = 30 days, from the channel's shorts only (~21%) |
+| `total_views` | long | Lifetime actual views across the channel (~68%) |
+| `engagement` | long | Views-per-comment ratio over the channel's last 30 days of uploads: `sum(views) / sum(comments)`, rounded (~25%). **Lower = more engaged audience** (fewer views per comment). Not an engagement count. |
+| `duration_longform` | integer | Average longform video duration in seconds, over the channel's uploads from the trailing 365 days (~46%) |
+| `duration_shorts` | integer | Average short duration, same window (~32%) |
+| `duration_live` | integer | Average live-stream duration, same window (~11%) |
+| `is_tl_channel` | boolean | TPP partner channel (100%) |
+| `is_active` | boolean | Channel is active (100%) |
+| `media_selling_network_join_date` | date | MSN join date; non-null = MSN member (~1%) |
+| `has_outreach_email` | boolean | Has contact email (100%) |
+| `outreach_email` | text | Contact email (~45%) |
+| `social_links` | text | Flat array of the channel's profile URLs (~47%), e.g. `["https://twitter.com/…", "https://instagram.com/…", "https://discord.gg/…"]`. Source is a per-platform map plus a catch-all `_other` map for unrecognized platforms; in ES all of it is flattened into this one URL array (the `_other` URLs are folded in, the platform names are dropped). Occasional stray entries (bare emails, nested arrays) exist. |
+| `male_share` | byte | Male audience % — only ~1.6% of channel docs have demographic data |
+| `usa_share` | byte | US audience % — same ~1.6% coverage |
+| `device` | object | Audience device demographics where known: `device.primary` (most common device) and `device.share` (per-device % map). Very sparse (~0.2%). |
+| `sponsorship_price` | scaled_float | Estimated price of a sponsored video on this channel, from **TL's sponsorship calculator** (~27%). Inputs: the channel's last-30-day views and comments, fulfillment rate, renewal rate, and longform evergreenness. Recomputed on the channel's regular metrics-update cycle, so it moves as the channel's data changes. |
+| `sponsorship_score` | scaled_float | TL-internal sponsorship track-record score, **range 0–10, higher = better** (~96%). Blends how many distinct brands sponsored the channel in the last 2 years (40%) with how many of them booked repeatedly (60%). The scale is deliberately skewed: raw low scores are compressed below 5, so **< 5 reads as weak/no track record and ≥ 5 as a real one**. Internal-only — don't quote the raw decimal externally (see business glossary). |
+| `evergreenness` | float | ⚠️ **Dead — 0 docs.** Use the per-format fields below. |
+| `evergreenness_longform` | scaled_float | Median per-video evergreenness of the channel's longform uploads from the trailing 365 days (~24%). Per-video evergreenness = `(views@180d − views@30d) / views@30d`; ≥ 1 = evergreen (day-180 views at least double day-30). Recomputed on the metrics-update cycle. |
+| `evergreenness_shorts` | scaled_float | Same, for shorts (~20%) |
+| `evergreenness_live` | scaled_float | Same, for live streams (~4%) |
+| `trend` | float | View-trend angle for longform uploads (~13%). Positive = growing views. |
+| `trend_shorts` | scaled_float | View-trend angle for shorts (~12%) |
+| `trend_live` | scaled_float | View-trend angle for live streams (~3%). There is **no `trend_longform`** — the longform trend is the bare `trend`. |
+| `posts_per_90_days` | integer | Longform uploads per 90 days, normalized from the trailing 365 days (~96%) |
+| `posts_per_90_days_shorts` | integer | Shorts per 90 days (~67%) |
+| `posts_per_90_days_live` | integer | Live streams per 90 days (~67%) |
+| `fulfillment_rate` | scaled_float | Share of the channel's longform uploads (trailing 365 days) that carry a sponsored mention (~40%) |
+| `renewal_rate` | scaled_float | Rate at which the channel's sponsoring brands come back (~96%) |
+| `metrics_update_period` | byte | ⚠️ Vestigial — populated on only ~2,300 docs (~0.005%). |
+| `offline_since` | date | ⚠️ **Dead — 0 docs.** Use `is_active`. |
+| `content_category` | integer | TL's own content-category code, 1–22 (~94%). **Not YouTube's categories** — see the category map in `postgres-schema.md`. |
+| `format` | integer | Platform format code (100%): 1 = Newsletter, 3 = Podcast, **4 = YouTube**, 5 = Blog, 7 = Twitch, 8 = TikTok, 9 = Instagram, 10 = LinkedIn. |
+| `face_on_screen` | boolean | ThoughtLeaders-sourced flag: whether the creator shows their face on screen when doing brand sponsorships (~1% of channel docs). |
+
+#### AI & Enrichment Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `ai` | object | **Channel docs only** (0 video docs). Holds exactly the three AI-generated fields below (~93% of channel docs). |
+| `ai.description` | text | AI-generated third-person channel profile, always in English regardless of the channel's language (~93%) |
+| `ai.topic_descriptions` | text | AI-generated prose paragraph describing the channel's content topics. A **single string, not an array**. Only ~37% of channel docs — absence means "not yet generated", not "off-topic". |
+| `ai.brand_safety` | keyword | Brand-safety letter grade `A`–`F` (~93%; A ≈ 91% of graded channels) |
+| `applied_enrichments` | keyword | Enrichment names applied to the video (e.g. `brand_extractor`) |
+| `article_category` | object | ⚠️ **Dead — 0 docs.** |
+
+#### System Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `@timestamp` | date | Index/update timestamp (~92% of video docs) |
+| `doc_type` | join | Parent-child join (channel→video) |
+| `es_index_tag` | keyword | Publication-period tag: quarterly from 2019 (`2025-q2`), yearly 2016–2018 (`2017`), `2015-and-before` for older. **Stored-only — not searchable** (`exists`/`term` match 0 docs). |
+
+## Common Query Patterns
+
+### Search videos by sponsored brand mention
+
+```bash
+tl db es '{
+  "size": 50,
+  "query": {"term": {"sponsored_brand_mentions": "5612"}},
+  "_source": ["title", "channel.id", "publication_date", "views"],
+  "sort": [{"publication_date": "desc"}]
+}'
+```
+
+### Search videos for a single channel
+
+```bash
+tl db es '{
+  "size": 100,
+  "query": {"term": {"channel.id": 12345}},
+  "sort": [{"publication_date": "desc"}]
+}'
+```
+
+### Count sponsored mentions for a brand (size:0 + track_total_hits)
+
+```bash
+tl db es '{
+  "size": 0,
+  "track_total_hits": true,
+  "query": {"term": {"sponsored_brand_mentions": "5612"}}
+}'
+```
+
+### Full-text search on title/summary/transcript
+
+(`summary` = the video's creator-written description. `description` field is not pupulated for articles, and `content` is podcast-only — see the field table.)
+
+```bash
+tl db es '{
+  "size": 20,
+  "query": {
+    "multi_match": {
+      "query": "ergonomic keyboard review",
+      "fields": ["title^3", "summary", "transcript"]
+    }
+  },
+  "_source": ["title", "channel.id", "publication_date"]
+}'
+```
+
+### Filter by date range
+
+```bash
+tl db es '{
+  "size": 100,
+  "query": {
+    "bool": {
+      "filter": [
+        {"term": {"channel.id": 12345}},
+        {"range": {"publication_date": {"gte": "2026-01-01", "lte": "2026-03-31"}}}
+      ]
+    }
+  }
+}'
+```
+
+### Aggregation example (aggregations are bounded — see *Accepted query bodies* above)
+
+```bash
+tl db es '{
+  "size": 0,
+  "aggs": {
+    "by_channel": {
+      "terms": {"field": "channel.id", "size": 20}
+    }
+  },
+  "query": {"term": {"sponsored_brand_mentions": "5612"}}
+}'
+```
+
+For more dimensions, run multiple `tl db es` calls and join client-side.
+
+### Deep pagination — `search_after`
+
+`from + size` is capped at 10,000, and the stateful cursors (`scroll`, `pit`) are not accepted. To page past 10,000 hits, use the stateless `search_after` cursor: sort deterministically with a unique tiebreaker (the `id` field — not `_id`), then pass each response's `next_search_after` envelope value back as `search_after` in the next request, keeping the same `query` and `sort`:
+
+```bash
+# First page
+tl db es '{
+  "size": 10000,
+  "query": {"term": {"channel.id": 12345}},
+  "sort": [{"publication_date": "asc"}, {"id": "asc"}]
+}'
+# → envelope includes "next_search_after": ["2025-09-14", "12345:abc123"]
+
+# Next page — identical query & sort, plus the cursor
+tl db es '{
+  "size": 10000,
+  "query": {"term": {"channel.id": 12345}},
+  "sort": [{"publication_date": "asc"}, {"id": "asc"}],
+  "search_after": ["2025-09-14", "12345:abc123"]
+}'
+```
+
+Repeat until a page comes back short (`next_search_after` is absent on an empty page). Pages are not a consistent snapshot — concurrent indexing can occasionally duplicate or skip a boundary row, which is fine for analytics sweeps. Date-range windowing (filtering by `publication_date` ranges) remains a good alternative when you want resumable, idempotent slices.
+
+## Text analyzer behavior
+
+`text` fields on article docs (`title`, `summary`, `transcript`) appear to use the `standard` analyzer (tokenize + lowercase, no stemmer, no English-possessive filter), so inflections, plurals, and possessives are each indexed as distinct terms. For example: `bitcoin` (4,466,300) vs `bitcoins` (489,262). For stemming-style recall, expand the query side with a `bool.should` over the variants.
+
+One consequence: URLs in article fields tokenize on punctuation (`substack.com` → `substack`, `com`), so you can't term-match a domain there. The exception is the channel-doc `description.domains` subfield, where whole domains are single searchable terms — use it to find channels by a linked domain (see the channel field table).
+
+## Transcript field format
+
+The `transcript` field's `_source` is **YouTube timed-text caption XML**, not plain prose. Each caption cue is a `<text start="…" dur="…">` element wrapped in `<transcript>`. The inner text is HTML-entity-encoded — on older docs **double-encoded** (an apostrophe is `&amp;#39;`, i.e. an escaped `&#39;`), on recent docs single-encoded (`&apos;`):
+
+```xml
+<?xml version="1.0" encoding="utf-8" ?>
+<transcript><text start="0.05" dur="4.33">I&amp;#39;m going to</text>...</transcript>
+```
+
+- **Searching is unaffected** — the field is analyzed as `text`, so `match` / `match_phrase` queries hit the words directly regardless of the markup. The XML only matters when you retrieve and read the raw `_source`.
+- **For plain prose**, strip the markup yourself, e.g. `jq -r '.results[0].transcript' | sed -E 's/<[^>]+>/ /g'`, then unescape entities (twice on older docs). Don't reach for the `content` field — it's podcast-only and absent on YouTube docs.
+
+## Notes & gotchas
+
+- **Composite IDs:** `tl-platform.id` and `_id` are `<channel_id>:<youtube_id>`. The `youtube_id` portion alone is what Firebolt's `article_metrics.id` stores.
+- **Add a `publication_date` range filter** whenever the question is time-bounded — the alias is fixed, so this is the only way to narrow the search.
+- `sponsored_brand_mentions` and `organic_brand_mentions` are keyword arrays — use `term` queries.
+- For brand mention details (position, snippet, detection_tool), the data is in the `brand_mentions` nested field.
+- **Stored-only fields** — retrievable in `_source` but invisible to `exists`/`term`/`match` (queries on them silently match 0 docs): `url`, `image_url`, `es_index_tag`, `not_sponsored_by`.
+- **`publication_id` is deprecated** — don't use for joins.
+- No write access. The CLI only exposes `_search` against `tl-platform-*`.

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/skills/tl/references/postgres-schema.md

@@ -126,11 +126,22 @@ A deal is **committed** when it is SOLD, or OPEN with `brand_approval_status` in
 
 ### `thoughtleaders_brand`
 
+> 🚨 **Columns that DO NOT exist on `thoughtleaders_brand` — common hallucinations:**
+> - ❌ `domain` / `url` — the website column is `website`.
+> - ❌ `brand_name` — it's plain `name` here (unlike channel, which prefixes its display fields).
+> - ❌ `is_active` — brands have no active/inactive flag; every row is live.
+> - ❌ `sponsored_topics` (or similar topic columns) — the closest fields are `keywords` and `ai_description`.
+> - ❌ `organization_id` — org lives on profile, not brand (see Key Relationships below).
+
 | Column | Type | Description |
 |--------|------|-------------|
 | `id` | int | Primary key |
-| `name` | varchar | Brand name |
+| `name` | varchar | Brand name. ⚠️ Plain `name`, not `brand_name`. |
 | `description` | text | Brand description |
+| `website` | varchar | Brand website URL. ⚠️ The column is `website`, NOT `domain`. |
+| `slug` | varchar | TL-platform slug |
+| `keywords` | varchar | Topic/detection keywords for the brand |
+| `ai_description` | JSONB | AI-generated descriptive info about the brand |
 | `creator_id` | int FK | User who created it |
 
 #### Junction Tables
@@ -169,14 +180,14 @@ A channel can have multiple adspots (different sellers: talent manager, direct,
 | `subscribers` | bigint | Subscriber count. |
 | `media_selling_network_join_date` | date/timestamptz | When the channel joined the MSN. **MSN membership = this column IS NOT NULL.** |
 | `is_tpp` | boolean | True = TPP channel — TL's closest-partner channels (~144 at 100k+ subscribers), a strict subset of MSN. Prefer when booking: fastest response, easiest to close. ⚠️ **This is not the MSN flag.** For MSN, use `media_selling_network_join_date IS NOT NULL`. |
-| `content_category` | int | Content category code (1–22), as assigned by YouTube. This assignment is too unreliable, do not use it for discovering channels. **For topic/category discovery, prefer `tl recommender top-channels "<tag>"` |
+| `content_category` | int | TL's own content-category code (1–22) — **not YouTube's categories**: 1 Backend Development, 2 Design, 3 Entrepreneurship, 4 Frontend Development, 5 Lifestyle, 6 Marketing, 7 Mobile Development, 8 Sales, 9 Travel, 10 Business, 11 Photography, 12 General Knowledge, 13 Personal Finance, 14 News & Politics, 15 Technology, 16 Gaming, 17 Food, 18 Sports, 19 How To & Crafts, 20 Entertainment, 21 Health & Fitness, 22 Music. The assignment is too unreliable for discovery — do not use it to find channels by topic. **For topic/category discovery, prefer `tl recommender top-channels "<tag>"` |
 | `is_active` | boolean | Whether the channel is active. ⚠️ **Always include `is_active = true` in channel queries** unless explicitly looking for archived rows. |
 | `country` | varchar | Channel's primary country (ISO 3166-1 alpha-2 code, e.g. `US`, `GB`, `BR`). This is often the cleanest answer to "geography" questions on sponsorships. May be NULL or blank. |
 | `language` | varchar | Primary content language. ⚠️ **Short ISO 639 codes — NOT BCP-47.** Mostly 2-letter ISO 639-1 (`en`, `pt`, `hi`) for major languages; occasionally 3-letter ISO 639-2/3 (`arc`, `arz`, `ase`, `ceb`) for languages without a 2-letter code. Filtering with `language = 'en-US'` returns zero rows. **Don't assume `LENGTH(language) = 2`** — that silently drops the 3-letter long-tail. May be NULL. |
 | `last_published` | date | Date of the channel's most recently seen video. Use for "is the channel still active?" filters — e.g. `last_published >= CURRENT_DATE - INTERVAL '120 days'`. |
 | `sponsorship_score` | double precision | TL-internal channel quality score (range 0-10, higher is better, if below 5, the channel is low quality). Useful as a tiebreaker when ranking candidate channels. |
 | `ai_description` | JSON | Descriptive information about a channel. Contains fields such as `description`, `audience`, `topic_descriptions`, and `brand_safety`. Useful as a regex-target for thematic filtering when the recommender results are too coarse (e.g. filtering "technology" down to actual tech reviewers via keywords like `tech|gadget|review|software`). |
-| `evergreenness` | float | Evergreen score |
+| `evergreenness` | float | Longform evergreen score: median per-video `(views@180d − views@30d) / views@30d` over the channel's trailing-year uploads; ≥ 1 = evergreen (day-180 views at least double day-30). In ES this lands on the channel doc as `evergreenness_longform`. |
 | `demographic_usa_share` | smallint (0–100) | Percentage of the channel's audience based in the US. Convenience for the common "is this a US-heavy channel?" filter — pre-computed from `demographic_geo['US']`. NULL when the channel has no demographic data. |
 | `demographic_male_share` | smallint (0–100) | Percentage of the channel's audience that's male. `female_share = 100 - demographic_male_share` (no separate column). NULL when the channel has no demographic data. |
 | `demographic_age_median_value` | varchar | The age-bucket label (e.g. `25-34`) corresponding to the median of `demographic_age`, pre-computed on save. Indexed; cheap to filter on. NULL when there's no age data. |
@@ -290,7 +301,42 @@ JOIN thoughtleaders_profile p ON p.user_id = adspot.publisher_id
 ```
 Joining `adspot.publisher_id → profile.id` directly mixes ID spaces and returns garbage.
 
-## `thoughtleaders_profile` persona constants
+## Key columns for the `thoughtleaders_profile` table
+
+The profile is the account record for a person/company on the platform. Every brand-side query hops through it (`adlink.advertiser_profile_id → profile → profile_brands → brand`), which makes it the most-joined table after the sponsorships themselves.
+
+> 🚨 **Columns that DO NOT exist on `thoughtleaders_profile` — common hallucinations:**
+> - ❌ `name` / `email` — the person's name and email live on `auth_user` (join via `user_id`: `first_name`, `last_name`, `email`). The company name is `organization_name`, denormalized onto the profile.
+> - ❌ `auth_user_id` — the FK column is `user_id`.
+> - ❌ `brand_id` — a profile can have several brands; join `thoughtleaders_profile_brands` on `profile_id`.
+> - ❌ `is_tpp` — that's a channel attribute; a profile's channels come via `thoughtleaders_profile_channels`.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | int | Primary key. ⚠️ NOT the same ID space as `auth_user.id` — never join a `*_id` pointing at users to `profile.id` directly. |
+| `user_id` | int FK | → `auth_user.id` (the person's name and email live there) |
+| `organization_id` | int FK | → `thoughtleaders_organization.id` |
+| `organization_name` | varchar | Denormalized company/organization display name |
+| `persona` | int | Account persona (see constants below) |
+| `is_advertiser` | boolean | Buyer (brand) side capability flag |
+| `is_publisher` | boolean | Seller (channel) side capability flag |
+| `owner_sales_id` | int FK | → `auth_user.id` — TL-side sales owner for this profile |
+| `owner_advertiser_id` | int FK | → `auth_user.id` — TL-side brand account owner |
+| `owner_publisher_id` | int FK | → `auth_user.id` — on advertiser profiles: the adops user in charge |
+| `media_buying_network_join_date` | date | MBN (brand-side buying network) join date. **MBN membership = this column IS NOT NULL.** ⚠️ Don't confuse with the channel's `media_selling_network_join_date` (MSN — the channel-side network). |
+| `superuser_notes` | text | Internal TL notes about the profile |
+| `buying_preference_*` | various | The brand's buying preferences (audience age/gender/geo shares, face-on-screen, custom instructions) |
+
+**Profile with the person's name and email:**
+```sql
+SELECT p.id, u.first_name, u.last_name, u.email, p.organization_name
+FROM thoughtleaders_profile p
+JOIN auth_user u ON u.id = p.user_id
+WHERE p.organization_id = 123
+LIMIT 50 OFFSET 0
+```
+
+### `thoughtleaders_profile` persona constants
 
 | Value | Label |
 |-------|-------|
@@ -298,7 +344,7 @@ Joining `adspot.publisher_id → profile.id` directly mixes ID spaces and return
 | 2 | Creator |
 | 3 | Talent Manager |
 | 4 | Media Agency |
-| 5 | Creator Service |
+| 5 | Other |
 
 ## `thoughtleaders_profile_channels` (Profile ↔ Channel M2M)
 

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/skills/tl-keyword-research/SKILL.md

@@ -563,8 +563,8 @@ raw `|`/`+`/`-` are literal text in a link; `*`/`~` are rejected, enumerate
 variants first), and emits:
 
 - `filter_set` — platform shape (`keywords`, `keyword_operator`,
-  `content_fields`, per-group field/exclude maps). Fields are **ContentField
-  enum names** (`title`, `summary`, `transcript`, `channel_description`,
+  `content_fields`, per-group field/exclude maps). Fields are **recognized
+  content field names** (`title`, `summary`, `transcript`, `channel_description`,
   `channel_topic_description`, …) — unknown names fail loudly.
 - `report_link` — paste-ready URL that opens the report with the filter applied
   (no saved record, no credits). **The default thing to hand the user.**

{thoughtleaders_cli-0.9.1 → thoughtleaders_cli-0.9.2}/skills/tl-keyword-research/references/elasticsearch-content-search.md

@@ -13,7 +13,7 @@ The content alias holds **both** article docs and channel docs, distinguished by
 
 | `doc_type` | ~count | what it is | descriptive text fields |
 |---|---|---|---|
-| `article` | ~670M | one video / upload | `title`, `summary`, `transcript`, `content` |
+| `article` | ~670M | one video / upload | `title`, `summary`, `transcript` (or in case of podcast records, `content`) |
 | `channel` | ~44M | one whole channel | `name`, `description`, `ai.description`, `ai.topic_descriptions` |
 
 **Always scope by `doc_type`** or your counts silently mix videos and channels:
@@ -49,9 +49,9 @@ level filters on `format` 4 alone.
 
 **Article docs** (`doc_type: article`):
 - `title` — video title. Strongest relevance signal (see weighting).
-- `summary` — AI-generated summary of the video.
+- `summary` — **the video's creator-written description** (the text under the video: promo links, hashtags, subscribe blocks). Misleading name — it is NOT an AI summary. This is the main field for searching video description text.
 - `transcript` — raw transcript. Stored as timed XML (`<text start="..." dur="...">`), so a transcript hit carries timestamps if you parse it.
-- `content` — full content/transcript text.
+- `content` — ⚠️ podcast episodes' only;  usually show notes text or rarely transcripts; effectively absent on YouTube docs. Don't search for YouTube content.
 - `content_type` — `longform` | `short` | `live`. The default probe scope is
   `longform` (best sponsorable-content signal).
 - `channel` — object with `id`, `content_category`, `country`, `language`, `format` (NO descriptive text — see channel docs for that). `format` 4 = YouTube.
@@ -60,8 +60,8 @@ level filters on `format` 4 alone.
 **Channel docs** (`doc_type: channel`):
 - `name` — channel name.
 - `description` — the channel's own (YouTube) description.
-- `ai.description` — AI-written channel description.
-- `ai.topic_descriptions` — AI-written description of the channel's topics. **This is the "topic description" field** — prefer it for channel-fit validation; it's a focused, on-topic summary.
+- `ai.description` — AI-written channel description: third-person, always English regardless of the channel's language (~93% of channel docs).
+- `ai.topic_descriptions` — AI-written description of the channel's topics, a **single prose string, not an array**. **This is the "topic description" field** — prefer it for channel-fit validation; it's a focused, on-topic summary. ⚠️ Only ~37% of channel docs have it — a no-match here doesn't prove the channel is off-topic.
 - `ai.brand_safety` — letter grade (A–F).
 - `content_category`, `country`, `language`, `format`, `reach` (subscribers), `total_views`, `sponsorship_price`, `sponsorship_score`, `outreach_email`, `social_links`, `is_tl_channel`.
 
@@ -70,35 +70,42 @@ level filters on `format` 4 alone.
 > `is_tl_channel`→`is_tpp`; the Elasticsearch index **keeps the old names**.
 > Every ES `_source` / `term` / `range` / `sort` on channel docs must use
 > `reach`, `impression`/`impression_live`/`impression_shorts`, `is_tl_channel`.
-> Do not "fix" a probe to the new names — it silently returns nulls. Skill
+> Do not "fix" a probe to the new names — it silently returns nulls (verified
+> live: `subscribers`, `projected_views`, `is_tpp` each match **0** docs). Skill
 > *output* uses the new vocabulary (`subscribers`, `is_tpp`); the translation
 > point is `search_channels.py`'s sponsorability block.
 
-### ContentField names for the deliverable vs ES field names for probes
+### Content field names for the deliverable vs ES field names for probes
 
-The platform FilterSet / report link uses **ContentField enum** names; raw
+The platform FilterSet / report link uses its own **content field** names; raw
 `tl db es` probes use the **actual ES field paths**. They differ for channel
 fields — map them:
 
-| ContentField (filter set / report link) | ES field path (probe query) | doc_type |
+| Content field (filter set / report link) | ES field path (probe query) | doc_type |
 |---|---|---|
 | `title` | `title` | article |
 | `summary` | `summary` | article |
 | `transcript` | `transcript` | article |
 | `content` | `content` | article |
 | `channel.channel_name` | `name` | channel |
-| `channel_description` | `description` | channel |
+| `channel_description` | `description.domains` | channel |
 | `channel_description_ai` | `ai.description` | channel |
 | `channel_topic_description` | `ai.topic_descriptions` | channel |
 | `hashtags` | `hashtags` | article |
 
-Use ES paths in `probe.py` / raw queries; use ContentField names in
+Use ES paths in `probe.py` / raw queries; use content field names in
 `build_report.py` / the report link. `build_report.py` rejects any `content_fields`
-value that isn't a ContentField enum name, so passing a raw ES path (e.g.
+value that isn't a recognized content field name, so passing a raw ES path (e.g.
 `ai.topic_descriptions`) fails loudly rather than silently mis-filtering.
-(One asymmetry to know: probe counts on the bare `description` field are
-approximate for the `channel_description` ContentField — the delivered filter
-does not match exactly the same text.)
+
+The `channel_description` content field targets `description.domains` — the
+"About this channel" text additionally indexed so **whole domains are single
+searchable terms** (verified live: `patreon.com` matches every channel whose
+About-this-channel text links to patreon.com, including inside full URLs,
+which bare `description` probes
+undercount since URLs there tokenize into word pieces). For domain-shaped
+keywords (e.g. `substack.com`), probe against `description.domains`, not bare
+`description`.
 
 ### The report link / FilterSet keyword grammar (NOT simple_query_string)
 
@@ -152,6 +159,10 @@ Verified live (topic level, all-time): `fable 5` = `fable-5` = **1,215** docs;
 `fable5` = **94**; `fable five` = **118**; `fablefive` = **1**. Four different
 document populations — collapsing them in your head silently drops coverage.
 
+(One exception to the punctuation-split rule: the channel-doc
+`description.domains` subfield keeps whole domains as single terms —
+`patreon.com` is one token there. See the content field mapping above.)
+
 **Rule:** for any candidate carrying a number, version, model name, or anything
 that could be written solid vs spaced vs hyphenated, **probe each spelling as its
 own candidate** (spaced, solid, hyphenated, spelled-out, and the handle/hashtag
@@ -420,7 +431,7 @@ id and read `ai.topic_descriptions` / `ai.description`:
 Or run the probe with `--level channel` to search channel docs directly.
 
 **Channel-field probes approximate the delivered filter, they don't equal it.**
-A delivered filter keyword targeting a channel ContentField
+A delivered filter keyword targeting a channel content field
 (`channel_topic_description`, `channel_description`, …) matches **articles**
 whose channel matches — its result counts are videos. Raw probes can't run
 that parent-child join (the join query types are not accepted), so you probe