keble-idea 0.3.0__tar.gz

keble_idea-0.3.0/.gitattributes

@@ -0,0 +1,2 @@
+# Auto detect text files and perform LF normalization
+* text=auto

keble_idea-0.3.0/.gitignore

@@ -0,0 +1,41 @@
+# Generated by Cargo
+# will have compiled files and executables
+debug/
+target/
+cert/
+
+# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
+# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
+Cargo.lock
+
+# These are backup files generated by rustfmt
+**/*.rs.bk
+
+# MSVC Windows builds of rustc generate these, which store debugging information
+*.pdb
+*__ignored__*
+
+*.env
+*.DS_Store
+**/.DS_Store
+
+.vscode
+.idea
+
+*.crt
+
+uploads/
+!strapi-v5/public/uploads/
+!strapi-v5/public/uploads/.gitkeep
+test/
+resources/
+Brewfile
+qdrant/
+*.pyc
+**.pyc.*
+*/traefik/certs
+*/traefik/certs/*
+gcp.json
+.history
+
+prod.env.*

keble_idea-0.3.0/AGENTS.md

@@ -0,0 +1,29 @@
+# Keble Idea Agent Rules
+
+- Read the umbrella `AGENTS.md`, `ENGINEERING_PRINCIPLES.md`, and
+  `TESTING_GUIDELINE.md` before changing this repo.
+- Keep `domain/` and `application/` independent from raw providers, FastAPI,
+  Celery, RabbitMQ, MongoDB, Redis, Qdrant, and Neo4j implementations.
+- Preserve one canonical schema/protocol/use case for each behavior. Remove
+  replaced contracts instead of retaining compatibility aliases.
+- Do not persist LOW candidate stories, concepts, evidence, scores, or tags.
+- Do not add idea/story vector collections or idea graph nodes.
+- Do not use `try/except`, `getattr`, `setattr`, in-function imports, pass-only
+  tests, test skips, broad scans, offset pagination, or total-count queries.
+- Every changed function, class, and schema needs a docstring with steps where
+  useful and a concise `Side effects if changes:` note for downstream contracts.
+- Tests use canonical layer folders and the umbrella marker vocabulary.
+- Update `README.md`, `CODE_GUIDELINES.md`, and `AiAgentChangesLogs.md` when a
+  completed milestone changes behavior or test proof.
+- Persist one `JobFailureSnapshot`; never restore loose `error_code`,
+  `retry_classification`, raw response bags, or provider exception types.
+- Recovery profiles are append-only. Every job stores the complete submitted
+  snapshot, automatic retry preserves it, and manual retry adopts the active
+  profile only through an explicit command flag.
+- Retry timestamps derive from immutable failure observation plus deterministic
+  envelope/attempt jitter. Re-evaluation must not move a schedule.
+
+Side effects if changes:
+
+- Platform queue transitions, Mongo indexes, provider gates, incidents,
+  financial attempts, generated APIs, and frontend labels consume these rules.

keble_idea-0.3.0/AiAgentChangesLogs.md

@@ -0,0 +1,296 @@
+# AI Agent Changes Logs
+
+## 2026-07-13 16:50 CST
+State: `refactor/financial-usage-receipts`; physical-attempt identity checkpoint,
+no version, tag, release, or main merge authorized.
+
+zma:
+- Execute the complete financial-accuracy and 100K-ingestion plan, preserving
+  provider retries while keeping broker replay idempotent.
+
+Ai:
+- Added `EnvelopeExecutionIdentity` and propagated the atomically leased
+  envelope key/attempt number through extraction and relationship protocols.
+- Updated canonical services and tests so platform adapters can create a new
+  spend identity for each physical retry without duplicating one lease attempt.
+
+## 2026-07-12 23:01 CST
+State: `feature/saas-admin-console`; owner accepted the integrated Data Platform
+review and authorized fast-forward integration to current remote main. No
+version, tag, or release is requested.
+
+zma: accepted for now; merge the verified work to main.
+
+Ai: refreshed origin and proved the lifecycle-contract branch is a strict
+fast-forward of `origin/main`; the standalone correction and foundation work are
+already ancestors. This records authorization before the non-force main update.
+
+## 2026-07-12 15:40 CST
+State: `feature/saas-admin-console`; canonical schema-version-2 operator
+lifecycle contracts land ahead of the platform SaaS admin console. No version
+bump, tag, release, or main merge.
+
+zma: Redesign the data platform + Keble Ideas admin frontend per
+`docs/plans/REWORK_DATA_PLATFORM_LAYOUT.md` — replace the monolithic developer
+console with a protected SaaS operator app (Truth-backed auth, typed lifecycle
+commands, controlled purge). This repo owns the canonical lifecycle schemas.
+
+Ai: Added schema-version-2 lifecycle contracts: `ContentLifecycle`
+(ACTIVE/ARCHIVED/PURGING) with `lifecycle` + `archived_at` on `ContentBody`,
+`IdeaLifecycle` (ACTIVE/RETIRED/PURGING) with `lifecycle` + `retired_at` on
+`IdeaBody`, both with timestamp-consistency validators and finite
+`*_LIFECYCLE_TRANSITIONS` maps (PURGING terminal). New content/idea records
+default `schema_version=2` (`CONTENT_SCHEMA_VERSION` / `IDEA_SCHEMA_VERSION`);
+stored v1 rows still validate via ACTIVE defaults until the platform backfill
+migration stamps them. Added `IdeaJobStatus.CANCELLED` +
+`IdeaJobType.PURGE` and canonical `TERMINAL_JOB_STATUSES` /
+`CANCELLABLE_JOB_STATUSES` sets. Added typed `LifecycleTransitionError`
+(`IdeaErrorCode.INVALID_LIFECYCLE_TRANSITION`) and the shared
+`assert_lifecycle_transition` guard in `domain/errors.py`. New
+`tests/unit/domain/test_lifecycle_contracts.py` (13 tests) plus `build_idea`
+helper; suite 40 passed, pyright clean.
+
+## 2026-07-12 10:10 CST
+State: `feature/idea-foundation`; the provider-neutral video payload now retains
+OCR-only and unknown-language observations without claiming they are English.
+
+zma:
+
+- Continue the complete Stand Out media integration with real OCR/ASR models,
+  preserve original content, and proceed until human quality inspection.
+
+Ai:
+
+- Added `VideoContentPayload.ocr_text_original` beside the existing normalized
+  English field. This prevents successful OCR from disappearing when ASR is
+  absent or reports a non-English language, while keeping `ocr_text_en`
+  semantically strict.
+- Added contract coverage for camel-case wire serialization and the rule that
+  original OCR does not implicitly populate English evidence.
+
+## 2026-07-11 15:51 CST
+State: `feature/idea-foundation` implements the provider-neutral foundation and
+portable vertical contracts from `docs/plans/COMPLETE_STAND_OUT_BY_CX.md`; no
+version bump, tag, or release label is authorized yet.
+
+zma:
+
+- Implement the complete Stand Out plan while avoiding concurrent work in
+  `keble-data-infra`, `keble-idea`, and `keble-data-platform` checkouts.
+- Keep idea provider-neutral, strongly typed, ACID-aware, concept-vector-only,
+  reversible, bounded, and executable by later workers without hidden context.
+
+Ai:
+
+- Added strict content, evidence, receipt, idea, eight-facet concept, scalar
+  posting, relationship, generation, queue ledger, and search contracts.
+- Added deterministic content/record identities, exact extraction batch
+  correction, composite confidence and LOW discard, idempotent publication,
+  reversible PURPOSE aliases, neighborhood hysteresis, invalidation planning,
+  and concept-first cursor search application services.
+- Added reusable portable in-memory adapters and 21 unit/contract tests covering
+  architecture boundaries, receipts, identities, duplicate delivery, aliases,
+  reruns, search sessions, and relationship retirement.
+- Verified portable pytest (21 passed), strict pyright (0 errors), and package
+  sdist/wheel build under Python 3.13.12. The prescribed
+  `keble-2.1.0-3.13` conda environment is not installed on this host, so proof
+  uses the repo-local `uv` Python 3.13 environment.
+
+## 2026-07-11 16:30 CST
+State: `feature/idea-foundation` generated-client contract correction before
+platform frontend composition; no version/tag/release change.
+
+zma: Implement the Stand Out plan through the typed platform and generated client.
+
+Ai: Kept `ideaKey` in the API search item for detail navigation while leaving
+display suppression to the frontend, and bound cursors to expiry plus the full
+session generation/profile identity hash so a valid cursor cannot cross a
+different frontier contract.
+
+## 2026-07-11 16:38 CST
+State: `feature/idea-foundation` extraction usage contract aligned with the
+platform PydanticAI runtime; no version/tag/release change.
+
+zma: Implement the approved Stand Out plan through durable workers and cost diagnostics.
+
+Ai: Changed the extractor port to return its typed batch together with exact
+per-content provider usage. The canonical extraction service now accumulates
+application-level correction usage before writing receipts, preventing guessed,
+zero, or double-counted token/latency facts.
+
+## 2026-07-11 17:05 CST
+State: `feature/idea-foundation` extended through immutable lifecycle and
+relationship coverage contracts; no version/tag/release change.
+
+zma: Continue implementing the complete Stand Out plan after rough human review,
+preserving review traces and stopping only at a human test or inspection gate.
+
+Ai: Added compare-and-swap generation cutover/rollback contracts, exact coverage
+stamps, retention and resumable migration schemas, typed relationship candidates,
+null-valid inference, deterministic accepted edge publication, and bounded alias
+posting compaction. Added lifecycle tests proving no-relation coverage, accepted
+edge generation identity, verified-only cutover, and rollback. Verified 24 portable
+tests, strict pyright, and package builds under the repo-local Python 3.13 runtime.
+
+## 2026-07-11 17:14 CST
+State: `feature/idea-foundation` external-safe artifact/auth failure vocabulary;
+no version/tag/release change.
+
+zma: Continue the complete Stand Out implementation through platform uploads and
+external surfaces.
+
+Ai: Added stable artifact-integrity/readiness, API-key, scope, and rate-limit error
+codes so REST and MCP adapters can redact infrastructure failures while preserving
+exact client behavior.
+
+## 2026-07-11 18:02 CST
+State: `feature/idea-foundation` queue routing contract corrected before worker
+composition; no version/tag/release change.
+
+zma: Continue implementing the complete Stand Out worker and relationship flow.
+
+Ai: Added a distinct `INFER_RELATIONSHIP` workload identity so typed model
+inference cannot collide with `PROJECT_RELATIONSHIP`, which remains the derived
+Neo4j projection job. This preserves one behavior per canonical queue contract.
+
+## 2026-07-11 18:34 CST
+State: `feature/idea-foundation` reversible alias-posting compaction contract;
+no version/tag/release change.
+
+zma: Continue the complete Stand Out implementation until live human gates.
+
+Ai: Added explicit `origin_concept_key` identity for derived canonical alias
+postings, a compound uniqueness contract, and bounded clear behavior. Compaction
+can now accelerate merged reads while unmerge deletes only derived rows and
+preserves every original posting.
+
+## 2026-07-11 18:40 CST
+State: `feature/idea-foundation` exact neighborhood evolution contract; no
+version, tag, release label, or main merge.
+
+zma: Continue the CX plan while the raw-provider repos remain owned by another
+worker, and stop only at live human verification.
+
+Ai: Replaced key-only relationship coverage with bounded typed neighbor
+summaries carrying exact similarity, and stamped the configured consecutive
+absence threshold. This gives platform persistence enough canonical evidence to
+rotate current/previous neighborhoods and retire semantic-only edges with
+hysteresis after revectoring instead of churning on one missing sweep. Verified
+24 tests, strict pyright, and package builds.
+
+## 2026-07-11 18:45 CST
+State: `feature/idea-foundation` generation-idempotent relationship absence
+tracking; no release metadata or main merge.
+
+zma: Continue autonomous self-evolution implementation before human gates.
+
+Ai: Added the last applied absence-neighborhood generation to canonical
+relationship state. Outer Mongo adapters can now increment an exited edge at
+most once per generation even when both endpoint coverage jobs observe the same
+exit, preventing premature retirement from duplicate/symmetric sweeps.
+
+## 2026-07-11 20:51 CST
+State: `fix/standalone-concept-model-retry` aligns model correction with the
+canonical standalone-concept policy; no version/tag/release/main merge.
+
+zma: Complete the live Amazon-to-idea vertical robustly and keep strict concept quality.
+
+Ai: Added the non-raising `standalone_concept_error` contract and made the
+publication validator delegate to it. Platform model adapters can now issue one
+typed `ModelRetry` for generic or provider-like facets without catching errors,
+repairing output, or duplicating the domain policy.
+
+## 2026-07-12 01:49 CST
+State: `feature/idea-foundation` resumable migration identity contract; no
+version, tag, or release metadata.
+
+zma: Finish and reconcile the complete Stand Out plan, including comprehensive
+real-database tests and merge verified work to main.
+
+Ai: Strengthened `MigrationCheckpoint` with a named migration, canonical
+collection owner, integer target schema version, and stable-key progress. This
+lets the platform run bounded Mongo migration jobs without startup-time corpus
+rewrites or ambiguous string versions. Added schema validation coverage and
+documented the contract.
+
+## 2026-07-13 04:30
+State: branch `feature/search-media-projection` (9f51c06) pushed; consumed by keble-data-platform via rev pin; NOT merged to main (train order: merge after the platform branch lands).
+
+zma (via /goal, STAND_OUT_IMPROVE1): public idea search must show the idea WITH its image/video and type-specific metadata.
+
+Ai: domain/search.py gains SearchMediaKind + IdeaSearchMedia (payload-owned derivation: PRODUCT first model-ready image, VIDEO original + first key-frame poster + duration, IMAGE normalized variant) + IdeaSearchSourceSummary (PRODUCT title/price, WEB_PAGE title); both ride IdeaSearchItem as wire-compatible optional fields. IdeaSearchService gains SearchContentReaderProtocol (application/ports.py) and hydrates media per page from ONE batched content read; InMemoryIdeaStore implements the port (contents dict + aget_contents). README notes images as a critical asset. 42 tests, pyright 0.
+
+## 2026-07-13 15:35 CST
+State: `refactor/financial-usage-receipts`; provider-neutral receipt cleanup,
+verified locally; no version, tag, release, or main merge.
+
+zma: Execute the financial-accuracy and 100K-ingestion plan; remove duplicate
+monetary truth from `keble-idea` while retaining strong usage contracts.
+
+Ai: Removed `ProcessingUsage.cost_micros`, introduced receipt schema version 2,
+and kept corrective-run merging limited to measured token/image/duration units.
+Platform `CostFact` and `CostAllocation` are now the only intended monetary
+owners. Added contract/unit guards, updated README, and verified 43 tests,
+strict pyright, wheel, and sdist.
+
+## 2026-07-13 20:31 CST
+State: `refactor/financial-usage-receipts`; Stage-B scheduling contract update,
+no version, tag, publication, or main merge.
+
+zma: Execute the financial-accuracy and 100K-ingestion plan, including bounded,
+deduplicated, prioritized relationship work and comprehensive tests.
+
+Ai: Added bounded provider-neutral relationship priority score/time fields to
+canonical concepts. Platform publication can now project idempotent posting
+quality and recency without changing concept identity, letting the graph queue
+prioritize high-value/recent concepts while Stage A remains independent.
+
+## 2026-07-14 00:50 CST
+State: `refactor/financial-usage-receipts`; queue-timing contract completion;
+no version, tag, or publication.
+
+zma: Complete the financial-accuracy and 100K-ingestion plan, including honest
+queue/provider/elapsed timing and comprehensive tests.
+
+Ai: Extended the existing provider-neutral `EnvelopeExecutionIdentity` with
+optional chronological queued/leased timestamps. Platform adapters can now
+copy durable queue timing into financial attempts without adding provider or
+pricing ownership to Keble Idea. Added schema validation, README/guideline
+documentation, and verified 45 tests plus Ruff and strict Pyright.
+
+## 2026-07-14 18:50 CST
+State: `feature/trending-commerce-discovery`; package release candidate 0.2.0.
+
+zma:
+- Add first-class Amazon and TikTok trending ingestion without fitting creators,
+  sellers, or videos into product schemas.
+- Preserve strong provider-neutral types, durable media identity, safe search
+  projections, comprehensive tests, and immutable downstream dependency pins.
+
+Ai:
+- Added distinct creator/seller profile and short-video content payloads while
+  retaining the existing product contract for product boards.
+- Projected acquired profile images, video thumbnails, and safe display titles
+  into search without persisting provider names or transient playback URLs.
+- Expanded content-kind filtering, added validation/projection tests, documented
+  the conversion boundary, and prepared the 0.2.0 contract release.
+
+## 2026-07-15 10:04 CST
+State: `feature/durable-job-failures`; Idea 0.3.0 canonical failure/recovery
+checkpoint, pending Platform adapter proof and the cross-repo release train.
+
+zma:
+- Execute the complete universal provider-failure plan with durable delays,
+  payment/quota evidence, full mocks/live tests, and all prior work on `main`.
+
+Ai:
+- Replaced loose queue `error_code`/`retry_classification` fields with one frozen
+  `JobFailureSnapshot`, sanitized physical usage/capacity evidence, and an XOR
+  processor result; added resumable `BLOCKED` state.
+- Added immutable versioned recovery profiles, revision-guarded active pointer,
+  required per-job snapshot/attempt/deadline, stable deterministic retry timing,
+  excessive-hint blocking, and finite retry/block/dead-letter decisions.
+- Added the reference profile repository protocol/in-memory CAS implementation,
+  bounded index shapes, public exports, README/agent/code guidance, and focused
+  contract coverage. Portable suite and strict Pyright currently pass; real
+  Mongo proof belongs to the Platform adapter phase and remains open.

keble_idea-0.3.0/CODE_GUIDELINES.md

@@ -0,0 +1,87 @@
+# Keble Idea Code Guidelines
+
+## Ownership
+
+The package owns semantic/domain contracts and caller-owned protocols. The
+platform owns concrete infrastructure and provider conversions. Dependency
+direction is checked through AST architecture tests.
+
+## Typed conversion flow
+
+Use one explicit conversion chain:
+
+```text
+platform source DTO -> ContentMongoObject -> ExtractionService
+-> ContentPublication -> platform persistence adapter
+```
+
+AI output must remain separate from persisted identity and audit fields.
+Prompt fields and output schemas match exactly. Stable keys are internal
+identity; display names are UX only.
+
+## Persistence and projections
+
+Mongo mutations use deterministic keys, unique indexes, optimistic revisions,
+and single-document atomic transitions. Multi-document work is a resumable
+saga. Qdrant and Neo4j are derived projections repaired from Mongo truth.
+
+## Search
+
+Search Qdrant concepts, expand Neo4j to depth at most two, resolve aliases,
+read scalar postings through the compound ESR index, hydrate bounded Mongo
+ideas, and freeze the ranked frontier in a Redis TTL session. Never add corpus
+counts, offset paging, story-array scans, or unbounded graph queries.
+
+## Commerce content boundaries
+
+Keep commerce resource kinds structurally distinct after provider conversion:
+products use `ProductContentPayload`, creators and sellers use
+`CommerceProfileContentPayload` plus `CommerceProfileType`, and ranked videos
+use `ShortVideoContentPayload`. Do not fit profiles or videos into product
+fields, persist transient playback URLs, or put provider-specific response bags
+into the idea-owned schemas. The platform owns raw-provider conversion and
+artifact acquisition before constructing these payloads.
+
+## Queue execution evidence
+
+Keep logical envelope identity, physical attempt number, original admission
+time, and physical lease time together in `EnvelopeExecutionIdentity`.
+Provider packages consume this provider-neutral contract; they must not infer
+queue wait from model duration or introduce pricing/provider fields here.
+
+## Failure and recovery ownership
+
+Persist one immutable `JobFailureSnapshot` with optional sanitized
+`UpstreamFailureEvidence`. Idea owns the job/recovery shape; Data Infra owns its
+wire failure and Platform owns conversion, durable scheduling, gates, incidents,
+and infrastructure. Never import Data Infra/provider exception types into this
+package or preserve parallel loose error strings.
+
+Every envelope snapshots an append-only recovery profile, attempt limit, and
+deadline. Use `recovery_decision_for_failure` for the only retry/block/dead-
+letter decision and `retry_at_for_failure` for stable deterministic timing.
+Excessive provider hints block rather than retrying early. Payment requires
+reviewed billing/account evidence; transport status and capacity alone do not
+qualify.
+
+Side effects if changes:
+
+- queue Mongo updates must validate the entire envelope after every transition;
+- profile activation changes only future admissions unless manual retry opts in;
+- failed physical usage remains exact or explicitly billing unknown.
+
+## Generations and relationship evolution
+
+Treat generation specifications as immutable records. Activate only verified
+generations through a compare-and-swap pointer that retains one rollback-safe
+generation. Stamp derived coverage with exact dependency generations rather
+than a boolean. A typed null relationship is a completed inference result, and
+posting compaction must remain bounded, resumable, and unnecessary for correct
+query-time alias resolution.
+
+## Side-effect notes
+
+Every changed public schema, protocol, service, and processor documents the
+downstream adapters, workers, projections, API/client, or tests affected by a
+contract change. Preserve human `reviewed` and `read` markers on untouched
+blocks.

keble_idea-0.3.0/PKG-INFO

@@ -0,0 +1,187 @@
+Metadata-Version: 2.4
+Name: keble-idea
+Version: 0.3.0
+Summary: Provider-neutral idea extraction, concept graph, generation, and search contracts.
+Author-email: zma <bob0103779@gmail.com>
+Requires-Python: >=3.13
+Requires-Dist: keble-helpers<2,>=1.52.2
+Requires-Dist: pydantic<3.0.0,>=2.10.0
+Description-Content-Type: text/markdown
+
+# Keble Idea
+
+`keble-idea` owns the provider-neutral contracts and application behavior for
+public content snapshots, standout ideas, semantic concepts, relationships,
+immutable generations, durable work ledgers, and concept-first hybrid search.
+
+It deliberately does not import FastAPI, Celery, MongoDB, Redis, Qdrant,
+Neo4j, raw providers, or `keble-data-infra`. The host application implements
+the protocols under `keble_idea.application.ports` and converts raw source
+responses into idea-owned schemas at its outer boundary.
+
+Version 0.3.0 adds the canonical persisted job-failure and recovery-policy
+contract. It reuses only the provider-neutral failure/capacity/usage vocabulary
+from `keble-helpers`; it does not import Data Infra wire types or provider
+exceptions. Platform must convert each outer failure once into an immutable
+`JobFailureSnapshot` and may not restore the removed `error_code` or
+`retry_classification` fields.
+
+## Architecture
+
+```text
+raw provider HTTP responses
+        -> platform-owned adapters
+        -> keble-idea schemas and services
+        -> platform-owned persistence/projection adapters
+```
+
+MongoDB is canonical in the host application. Qdrant stores canonical concept
+vectors only, Neo4j stores the concept graph only, Redis stores TTL search
+frontiers, and RabbitMQ delivers identifiers for Mongo-backed queue envelopes.
+Ideas are retrieved through scalar idea-to-concept postings rather than idea
+vectors or multikey story-array scans.
+
+Cost-bearing extraction and relationship protocols receive an
+`EnvelopeExecutionIdentity` copied from the atomically leased Mongo envelope.
+The envelope key identifies logical work; `attempt_number` identifies the
+physical broker/worker retry. Optional `queued_at` and `leased_at` preserve the
+exact durable queue interval without introducing provider or pricing ownership.
+Financial adapters include the identity in provider attempt keys so redelivery
+cannot overwrite or hide a second possible spend.
+
+## Development
+
+Python 3.13 and `uv` are required.
+
+```bash
+uv sync
+uv run pytest -q -m "not live and not slow and not eval and not local_stack and not db_stack and not container"
+npx --yes pyright .
+uv build
+```
+
+The portable test lane uses only reusable in-memory adapters from
+`keble_idea.testing`. Stateful integration adapters belong to the platform and
+must isolate and clean MongoDB, Redis, Qdrant, Neo4j, RabbitMQ, and artifact
+state after each test.
+
+## Contract rules
+
+- Persist only HIGH and MEDIUM ideas. LOW candidate details are discarded.
+- Write exactly one minimal receipt for every content/generation terminal state.
+- Keep receipts provider-neutral: physical token/image/duration usage remains in
+  `ProcessingUsage`, while monetary pricing and per-content allocation belong
+  exclusively to the platform-owned financial ledger. Receipt schema version 2
+  rejects the removed `cost_micros` field.
+- Use the eight finite semantic facets; provider/source metadata is not a concept.
+- Use `standalone_concept_error` at model boundaries so invalid facet labels
+  receive a typed correction before publication applies the same strict policy.
+- Merge only HIGH-confidence PURPOSE concepts automatically and keep it reversible.
+- Vectorize concepts only unless a separately approved retrieval eval changes the design.
+- Use immutable generations and active pointers; never mutate active indexes in place.
+- Persist named, collection-owned integer-version migration checkpoints so outer
+  workers resume by stable record key and never migrate a corpus at startup.
+- Preserve original ASR and OCR independently from English-normalized fields;
+  unknown-language OCR belongs in `ocr_text_original`, never guessed into
+  `ocr_text_en` from the transcript language.
+- Cutover only verified generations through compare-and-swap pointers and preserve
+  the immediately previous verified generation for exact rollback.
+- Stamp relationship coverage with the exact embedding, neighborhood, and
+  relationship generations plus bounded neighbor similarities; a valid
+  null/no-relation result still completes coverage. Persisted current/previous
+  neighbor sets let outer adapters apply entry/exit hysteresis without all-pairs
+  comparison.
+- Keep `relationship_priority_score` and `relationship_priority_at` as bounded,
+  provider-neutral scheduling hints. The platform derives them idempotently from
+  posting quality/time so Stage-B graph budget reaches high-value/recent concepts
+  first; they never change semantic concept identity or Stage-A search readiness.
+- Compact alias postings asynchronously in bounded resumable pages; merge and
+  unmerge correctness must never depend on compaction finishing first. Derived
+  rows retain `origin_concept_key`, so unmerge removes them without deleting the
+  original posting identities.
+- Search is bounded, concept-first, count-free, and cursor-session based.
+- Cursor signatures bind expiry, frontier revision, and the complete generation/profile identity.
+- Operator lifecycle is canonical and typed (schema version 2): content moves
+  `ACTIVE -> ARCHIVED -> ACTIVE` and enters `PURGING` terminally; ideas move
+  `ACTIVE -> RETIRED -> ACTIVE` and enter `PURGING` terminally. `archived_at` /
+  `retired_at` must match the state, transitions go through
+  `assert_lifecycle_transition`, and public reads exclude non-ACTIVE rows.
+- Operator job commands use the canonical status sets: retry targets
+  `TERMINAL_JOB_STATUSES`, cancel targets `CANCELLABLE_JOB_STATUSES`
+  (never LEASED work), and purge work routes through `IdeaJobType.PURGE`.
+
+## Durable provider recovery
+
+Each queue envelope snapshots the complete active
+`ProviderRecoveryProfileSnapshot`, its matching `attempt_limit`, and an absolute
+deadline. Automatic retry retains that snapshot. A manual retry also preserves
+it unless the operator explicitly requests the active profile. Profile records
+are append-only and activation uses a revision-guarded pointer; changing the
+pointer affects future admissions only.
+
+The initial profile is fixed at eight attempts, a 72-hour deadline, 30-second
+fallback doubled to a one-hour cap, a 24-hour maximum provider hint,
+deterministic 0-10% jitter, five unavailable failures in 60 seconds, a five-
+minute open circuit, one half-open probe, and blocked release batches of 100.
+Those are defaults for the first persisted profile, not module constants that
+silently mutate existing jobs.
+
+`retry_at_for_failure` anchors delay and jitter to the failure's immutable
+observation time. Re-evaluating the same envelope attempt therefore returns the
+same not-before timestamp. A provider hint above the submitted 24-hour policy
+becomes `BLOCKED`; attempt/deadline exhaustion or permanent failure becomes
+`DEAD_LETTERED`; operator action becomes `BLOCKED`. Only a retryable failure
+within both budgets becomes `RETRY_SCHEDULED`.
+
+`UpstreamFailureEvidence` retains bounded status/code/request identity, opaque
+quota scope, exact capacity, physical-attempt count, usage, and a completeness
+bit. Missing or `USAGE_ONLY` evidence on a physical call is billing unknown.
+Payment requires a reviewed billing-code/account-status source; HTTP 402,
+negative capacity, quota, and rate limiting are never sufficient by themselves.
+
+Side effects if changes:
+
+- Platform Mongo queue/profile/incident adapters and central indexes implement
+  these exact field paths and compare-and-swap semantics;
+- Data Platform job and commerce-operation APIs expose the same failure/policy
+  facts through generated clients and translated frontend labels;
+- finance records failed physical usage exactly once while pre-provider gate
+  rejection remains zero spend;
+- `EnvelopeProcessResult` must continue to contain exactly one result reference
+  or failure snapshot.
+
+## Images are a critical asset
+
+Content payloads treat visuals as first-class evidence:
+`ProductContentPayload.image_artifacts` carries model-ready product photos,
+`CommerceProfileContentPayload.image_artifacts` carries acquired creator or
+seller portraits, `ShortVideoContentPayload.thumbnail_artifacts` carries safe
+ranked-video thumbnails without transient playback URLs, and
+`VideoContentPayload` preserves uploaded video bytes plus key frames.
+`IdeaSearchMedia.from_payload` projects the representative image/video (with
+`IdeaSearchSourceSummary` for type-specific display facts) onto every search
+item so consuming platforms can render media-rich cards. Artifact references
+stay checksum-addressed (`ArtifactRef`) — delivery URLs are always resolved by
+the serving platform at read time.
+
+## Commerce discovery content
+
+Trending commerce ingestion stays provider-neutral after the platform boundary:
+
+```python
+profile = CommerceProfileContentPayload(
+    profile_type=CommerceProfileType.CREATOR,
+    display_name="Rocket Lab Notes",
+    image_artifacts=[acquired_profile_image],
+)
+
+video = ShortVideoContentPayload(
+    title="A compact model-rocket launch guide",
+    thumbnail_artifacts=[acquired_thumbnail],
+)
+```
+
+Products keep `ProductContentPayload`; creators and sellers share the typed
+profile payload with a stable `profile_type`; channel-wide ranked videos use a
+separate short-video payload. Provider names, provider response bags, volatile
+playback URLs, and raw enum labels do not enter these durable contracts.