broader-vids 0.1.0 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,396 +1,398 @@
1
- # Architecture
2
-
3
- Broader Vids has one Python engine.
4
- The terminal, Codex skill, and local dashboard use that engine.
5
- The Node launcher installs and starts the engine.
6
- Per-profile locks coordinate launcher commands and setup.
7
- Ordinary commands can share the runtime. Setup requires exclusive access.
8
- Active engine operations and dashboard servers block setup.
9
- The launcher releases locks after normal completion or command failure.
10
- A stale or unreadable lock blocks later work until the operator checks for surviving processes and repairs the lock.
11
- The launcher has no stale-lock repair command and does not implement recommendation logic.
12
-
13
- The GitHub marketplace and explicit npm installer both install the prepared npm plugin.
14
- The npm installer registers a small host catalog without GitHub access or profile access.
15
- Setup installs the bundled engine into external profile storage after plugin installation.
16
-
17
- ## Project files
18
-
19
- ```text
20
- broader-vids/
21
- README.md Purpose, remote installation, and first use
22
- LICENSE Broader Vids license
23
- plugin.json Portable plugin identity
24
- .agents/plugins/
25
- marketplace.json Remote catalog with a pinned npm plugin source
26
- bin/
27
- broader-vids.cjs Node setup and process launcher
28
- plugin-install.cjs Explicit npm plugin registration through Codex
29
- runtime-lock.cjs Shared runtime use and exclusive installation
30
- profile-data.cjs Preview, reset, and complete data removal
31
- skills/
32
- broader-vids/ Recommendations and feedback instructions
33
- broader-vids-setup/ Guided installation and profile selection
34
- broader-vids-dashboard/ Local dashboard access and shutdown
35
- broader-vids-reset/ Fresh profile with tools and sign-in retained
36
- broader-vids-clean/ Complete local data removal
37
- src/broader_vids/
38
- cli/ Command arguments and dispatch
39
- discovery/ Public searches and creator expansion
40
- exploration/ Topic hierarchy and query planning
41
- evidence/ Captions, frames, transcription, and assessment
42
- account_switch.py Staged sign-in, verified replacement, and recovery
43
- recommendations/ Screening, ranking, batches, and reports
44
- retries.py Shared candidate retry counts and cancellation recovery
45
- learning/ Feedback models and calibration
46
- web/ Local server and browser assets
47
- data/queries.json Curated search terms
48
- onboarding.py Atomic platform choice and setup completion
49
- preflight.py Shared readiness and incomplete-feedback checks
50
- preflight_checks.py Bounded tool and authenticated-access probes
51
- preferences.py Profile and video actions
52
- creators.py Creator rules and immutable round snapshots
53
- blacklist.py Authenticated Bilibili blacklist import
54
- identity.py Platform, video, and part identities
55
- policy.py Default weights and duration rules
56
- parsing.py Platform response parsing
57
- http.py Request limits and request records
58
- processes.py Process deadlines and cleanup
59
- time_budget.py Optional round time limits and JSON representation
60
- windows_job.py Windows process ownership
61
- owned_process.py Worker startup gate
62
- storage.py Schema 12 database and exclusive locks
63
- maintenance.py Evidence cleanup
64
- token_usage.py Durable model-attempt receipts and stage reports
65
- python/
66
- core-requirements.txt Locked keyword-conversion dependency
67
- evidence-requirements.txt Locked media and browser dependencies
68
- asr-requirements.txt Locked transcription dependencies
69
- *.whl Generated engine package, excluded from Git
70
- scripts/
71
- test.py Isolated parallel regression runner
72
- verify_package.py Distribution checks
73
- build_plugin.py Clean marketplace payload and plugin ZIP
74
- tests/ Automated tests
75
- docs/
76
- WORKFLOW.md Recommendation steps
77
- ARCHITECTURE.md This document
78
- SCORING.md Scoring and learning models
79
- REFERENCE.md Commands and project maintenance
80
- .github/workflows/ci.yml Automated repository checks
81
- pyproject.toml, uv.lock Python package and dependency definitions
82
- package.json Node launcher and package file list
83
- package-lock.json Locked Node development dependencies
84
- ```
85
-
86
- ## Processing flow
87
-
88
- ```mermaid
89
- flowchart LR
90
- A[Codex skill or terminal] --> S[Guided setup when incomplete]
91
- S --> T[Shared readiness and feedback checks]
92
- T -- Required failure --> U[Repair instructions without recommendation work]
93
- T -- Feedback needed --> V[Dashboard and stop or explicit continuation]
94
- T -- Ready --> B[Topic planning and discovery]
95
- V -- Continue this round --> B
96
- B --> C[Collect 100 videos passing hard metadata floors]
97
- P[Pending recommendations] --> C
98
- C --> D[35 inspection candidates per platform]
99
- D --> E[Captions and selected frames]
100
- E --> F[Quality and full ranking]
101
- F --> G[10 recommendations per platform]
102
- G --> H[Chat recommendations and local dashboard]
103
- H --> I[Explicit feedback]
104
- I --> J[Learning models]
105
- J --> F
106
- H --> P
107
- ```
108
-
109
- The web layer displays saved results and submits explicit user actions.
110
- It does not contain a separate scoring algorithm.
111
- The skill directs the host to call the engine.
112
- `worker_context.py` resolves the invoking thread's model and reasoning effort from local turn metadata.
113
- New runs save that pair and pass it to planning and assessment CLI workers.
114
- `recommendations/collection.py` continues candidate collection until 100 pass `screening.py` duration and raw engagement floors, or a saved limit stops collection.
115
- Screening policy 5 combines metadata screening with learned interest and balances query coverage across thirty-five content-assessment slots.
116
- Final selection favors distinct narrow topics and queries before it compares full scores.
117
- `recommendations/diversity.py` implements these deterministic selection rules.
118
- An insufficient range of assessed topics permits repeats and records the relaxation in each affected selection.
119
- Each new round saves one learning-model reference for screening, preview, and final selection.
120
- Resumed rounds retain their saved plan, policies, and remaining allowances.
121
- Refill records retain query origins and anchors even when the discovery cache reaches its separate limit.
122
-
123
- The recommendation collector uses `collection_policy.py` for retrieval groups and saved targets.
124
- `collection_batches.py` scans bounded batches, fills group deficits, and preserves unfinished batch progress.
125
- `collection.py` supplies targeted search continuations and resumes unfinished collection batches.
126
- `budgets.py` defines request defaults and applies explicit request-cap increases under run and platform locks.
127
- Request-cap changes update settings and add an audit event in one transaction.
128
- They do not rewrite saved recommendation state or reset resource use.
129
-
130
- ## Dashboard files and state
131
-
132
- The Python server serves local HTML, CSS, and JavaScript modules.
133
- HTTP/1.1 connections reuse sockets for asset and API requests.
134
- Idle connections wait at most one second for another request. Active requests use a thirty-second socket timeout.
135
- Shutdown disconnects clients and waits for request handlers to close their database connections before closing the account controller.
136
- The server rejects ambiguous request lengths and closes connections after error responses.
137
- It permits only explicit static asset paths.
138
- The dashboard uses native form controls and dialogs.
139
-
140
- | File under `web/static/` | Responsibility |
141
- | --- | --- |
142
- | `index.html`, `dashboard.css` | Application shell, themes, and responsive layout |
143
- | `dashboard.js` | Navigation, refresh coordination, and user actions |
144
- | `state.js` | Server records, selections, scroll positions, and local drafts |
145
- | `api.js` | Local requests, action tokens, and response errors |
146
- | `token-usage.js` | Saved-round token reports and scoped dialog refresh |
147
- | `bilibili-account.js` | Account-switch controls, scoped status polling, and cancellation |
148
- | `views.js` | Compact video rows and exclusion management |
149
- | `review.js` | Four independent rating groups and the creator checkbox |
150
- | `score.js`, `ui.js` | Technical score presentation and shared controls |
151
-
152
- Recommendations, exclusions, and calibration have separate views.
153
- The desktop list and review panel scroll independently.
154
- Narrow windows use a review dialog with Back navigation.
155
- Score details use a separate desktop dialog or a secondary mobile view.
156
-
157
- The dashboard uses one system font and shared text-size variables.
158
- Page titles, section titles, headings, body text, controls, captions, and small metadata each have a defined size.
159
- Buttons, action links, navigation, and form controls share the control size.
160
- Compact buttons reduce padding without reducing their text size.
161
- The Bilibili panel places its external management link in the header and groups local actions below the account fields.
162
- An idle account switch has no explanatory status text.
163
- Local GET requests have a thirty-second limit, and writes have a ninety-second limit.
164
- Account polling permits one status read at a time and skips unchanged interface updates.
165
- A failed account-switch state remains readable so its controls can recover.
166
- HTTP failures and failed writes still display errors.
167
-
168
- Saved records remain separate from drafts.
169
- Video identity joins feedback across recommendations and calibration.
170
- Each successful action updates its affected records.
171
- Refresh requests carry a read version and a mutation generation.
172
- Only one refresh group runs at a time.
173
- A refresh requested during an active read waits until that read finishes.
174
- Old video records stay in browser memory only while the server returns them, an earlier list uses them, or they have an unsaved rating draft.
175
- The interface rejects responses from before a newer mutation.
176
- It pauses background refresh while the tab is hidden or a write is active.
177
- Refresh preserves active controls and pane positions.
178
- Drafts survive navigation within the tab, but they require Save before the page closes.
179
-
180
- The engine stores token usage as versioned `token_usage` events in the existing database.
181
- Each attempt has one call ID. Later events update that attempt's known status and usage.
182
- The report reads the last event for each call ID.
183
- Parallel workers record the owning recommendation ID and the source evidence ID.
184
- New rounds use exact owner receipts. Historical rounds use saved checkpoint usage where available.
185
- A derived database index supports report queries without a schema-version change.
186
- The ledger stores counters and identifiers, without copying prompts, frames, transcripts, or credentials.
187
- Reports do not depend on recommendation publication or media files.
188
- A missing measurement remains unknown after interruption.
189
- Reset and Clean remove usage with recommendation history.
190
- Preference import and export do not transfer usage history.
191
-
192
- ## Module responsibilities
193
-
194
- | Area | Main files | Responsibility |
195
- | --- | --- | --- |
196
- | Discovery | `queries.py`, `languages.py`, `sources.py`, `state.py`, `service.py` | Build searches, check keyword scripts, parse results, and retain discovery progress |
197
- | Exploration | `domains.py`, `coverage.py`, `rotation.py`, `proposals.py`, `planning.py` | Select underused leaf topics, coordinate reservations, and plan queries |
198
- | Evidence | `service.py`, `media.py`, `downloader.py`, `captions.py`, `browser.py` | Resolve playable videos and collect bounded evidence |
199
- | Evidence workers | `preparation_worker.py`, `asr_worker.py` | Separate media preparation and optional transcription processes |
200
- | Assessment | `assessment.py`, `worker_instructions.txt`, `reports.py`, `retention.py` | Check model output, report evidence limits, and remove completed media |
201
- | Recommendations | `service.py`, `batch.py`, `parallel.py`, `timing.py` | Coordinate platform jobs, worker limits, and recovery |
202
- | Ranking | `engagement.py`, `metadata.py`, `originality.py`, `quality.py`, `ranking.py`, `diversity.py`, `personalization.py` | Screen candidates and select the final list |
203
- | Learning | `features.py`, `topic_labels.py`, `inference.py`, `scoped.py`, `depth.py`, `service.py` | Capture predictors, fit models, and calculate predictions |
204
- | Preference transfer | `learning/transfer.py`, `learning/posterior.py`, `learning/baseline.py`, `web/static/preferences.js` | Transfer learned parameters without video records |
205
- | Feedback | `feedback.py`, `calibration.py`, `evaluation.py`, `evidence.py`, `annotations.py` | Store judgments, reserve test videos, and record evidence corrections |
206
-
207
- File names in this table are relative to their area under `src/broader_vids/`.
208
-
209
- ## Data storage
210
-
211
- The engine resolves one external data directory for each command.
212
- The installed package remains separate from this directory.
213
- See [data directory selection](REFERENCE.md#choose-a-data-directory).
214
-
215
- | Stored data | Purpose |
216
- | --- | --- |
217
- | `.broader-vids-profile.json` | Profile ownership marker for safe lifecycle operations |
218
- | `onboarding.json` | Versioned default platform and setup completion, separate from learned parameters |
219
- | `broader-vids.sqlite3` | Jobs, request records, profile, video state, feedback, models, and calibration |
220
- | `evidence/` | Captions, frames, scorecards, retained hashes, and unfinished media |
221
- | `recommendations/` | Local Markdown digests and JSON reports |
222
- | `feedback/` | Local content descriptor records |
223
- | `runs/` | Discovery reports and request traces |
224
- | `access/bilibili/profile/` | Separate browser profile for Bilibili sign-in |
225
- | `runtimes/`, `wheels/` | External engine environments and cached wheels for the Node launcher |
226
-
227
- Onboarding uses an atomic versioned file protected by a separate configuration lease.
228
- It does not change SQLite schema 12 or the learning revision.
229
- Preference transfer excludes this file.
230
- Preflight counts unique recommendation exposures against current structured ratings.
231
- Cleared fields count as incomplete again.
232
- Its isolated subprocess checks have a deadline and owned process cleanup.
233
- Bilibili checks use the existing browser profile without reading cookies into model inputs.
234
- Account switching stages a fresh profile under `access/bilibili/switches/`.
235
- A bounded worker owns the browser, blacklist, and Bilibili recommendation leases.
236
- The dashboard polls its status without blocking navigation or rating drafts.
237
- After verification, a file journal and database receipt coordinate session activation.
238
- Recovery restores the previous files before a database commit or removes the old session after a commit.
239
- Completed and cancelled workers remove their temporary profiles and staged blacklist database.
240
- Failed cleanup retains the journal and blocks browser use until recovery succeeds.
241
- New and resumed recommendation commands record automated-check time.
242
- That time counts toward an explicit total allowance when one exists.
243
-
244
- This release supports fresh empty databases (schema 0) and existing schema 12 profiles.
245
- The engine rejects other database schemas without changing them.
246
- Schema 12 stores current structured ratings and action records.
247
- Explicit tag rules reside in the user profile.
248
- Creator rules reside in creator_exclusions.
249
- Creator_policy stores their revision and active Bilibili account ID.
250
- Creator_sync stores each account's import status and last successful sync.
251
- Creator rules have a separate revision from the learning profile.
252
- Round state stores versioned creator-rule and learning-model references.
253
- Public recommendation reports expose rule counts rather than imported creator IDs.
254
- Topic anchors remain stable across rating edits and clears.
255
- Model snapshots retain the identities of their training inputs.
256
- Frozen evaluation records retain the tested model identity.
257
-
258
- Preference transfer accepts format 4 with learning model version 8 and feature version 1.
259
- Manual creator rules transfer separately from learning parameters and replace the manual lists transactionally on import.
260
- The export omits account-specific blacklist copies and sign-in data.
261
- It includes a bounded exploration summary with at most 1,024 nodes and aggregate counts and round ages.
262
- It excludes individual video history.
263
- It does not change SQLite schema 12.
264
- A completed `preference_import` job stores the imported parameter baseline and local feedback cutoff.
265
- Fitting combines that baseline with active ratings saved after the cutoff.
266
- Historical ratings remain available to the dashboard.
267
- Import holds both platform locks and the learning lock.
268
- Import prepares the restored model before the write transaction.
269
- A short transaction checks the preview revision and commits the replacement atomically.
270
- Concurrent feedback remains available and prevents a stale import from committing.
271
- Export reuses an exact cached model when available.
272
- Input validation runs once per terminal or HTTP request.
273
- Failed imports remove unused recovery files.
274
- Import also removes obsolete model caches while preserving models referenced by recommendations or frozen evaluations.
275
- An uncertain commit retains its recovery file.
276
- Parameter-only recovery files reside in `backups/`.
277
- The dashboard uses authenticated local POST operations for export, preview, and import.
278
- The transfer dialog preserves rating drafts and blocks stale polling responses during an operation.
279
-
280
- Evidence corrections add derived records.
281
- They do not overwrite original scorecards or user ratings.
282
-
283
- The Preferences page uses `preferences.js` to coordinate `/api/interests` and the preference-transfer endpoints.
284
- One controller serializes interest saves and file operations.
285
- A successful import returns the saved profile and updates the editor without another page.
286
- `tag_rules.py` combines manual tag exclusions with removable built-in defaults.
287
- The profile stores disabled built-in tags separately from manual tags.
288
- Preference imports preserve these local exclusion choices.
289
- Tag saves update rules without fitting a learning model.
290
- The dashboard serializes tag writes per platform and rejects older tag snapshots.
291
- An interest save preserves a draft when only exclusion metadata changed.
292
- Profile revisions prevent one window from replacing a newer interest edit.
293
- Interest writes hold both platform locks and preserve learned parameters and historical video data.
294
-
295
- ## Concurrency and recovery
296
-
297
- One batch coordinates both platforms.
298
- The batch shares worker limits and an optional total time limit.
299
- New rounds have no overall deadline by default.
300
- The time policy keeps internal unlimited values out of saved JSON.
301
- It allows four preparation workers in total, with at most two per platform.
302
- It allows up to four model workers.
303
- Prepared evidence bypasses the preparation queue.
304
- Preparation allows up to five minutes after its worker slot becomes available.
305
- Assessment allows up to three minutes after its model slot becomes available.
306
- Queue waits count toward an explicit parent time allowance when one exists.
307
- They do not consume those separate stage allowances.
308
- Metadata workers use separate database connections, a shared cookie jar, and a shared request gate.
309
- YouTube serializes metadata retrieval so each response can update the visitor session before the next request.
310
- Bilibili retains two concurrent metadata workers.
311
- Metadata request starts use a one-second interval.
312
- Pacing waits count toward an optional saved deadline.
313
- Access failures cancel queued retrieval before another YouTube collector starts.
314
- The gate preserves request pacing and prevents concurrent reservation of the last request slot.
315
- Platform locks prevent conflicting recommendation jobs.
316
- Short database transactions protect user actions and final result writes.
317
- Dashboard lookup checks recommendation rounds from newest to oldest and stops at the first usable list.
318
- Feedback resolves the displayed video evidence before it acquires its database write transaction.
319
- Earlier pending videos retain their saved topic and depth context.
320
-
321
- The engine checks profile, feedback, and queue changes before it saves a final recommendation list.
322
- A conflicting change requires another ranking calculation.
323
- An interrupted job retains its candidate position and remaining allowances.
324
- Continuation does not reset those allowances.
325
- New rounds save an inspection limit of 35 videos per platform.
326
- Resumed rounds retain their saved inspection limit.
327
-
328
- Each worker belongs to an owned process group.
329
- Windows uses a Job Object to stop the worker's child processes.
330
- Cancellation checks prevent new work after cancellation or an explicit deadline.
331
- The engine retains unfinished work when access or assessment fails.
332
- Candidate failures remain local to that candidate.
333
- YouTube and Bilibili share the same limit of three retries for recoverable candidate exceptions.
334
- The engine saves attempt counts before worker dispatch. Continuation cannot grant an extra attempt.
335
- Manual cancellation preserves that allowance.
336
- Exhausted candidates become unavailable while other candidates continue.
337
- Caption access challenges stop acquisition for the affected platform.
338
- Already prepared assessments retain their individual timeouts and any explicit shared deadline.
339
- Other platform access challenges and shared model failures stop the applicable workers.
340
- Reports preserve the first failure separately from later cancellation messages.
341
- Preparation checkpoints record transfer progress for interrupted-byte accounting.
342
- Unknown usage retains an explicit upper bound.
343
- Paused rounds can retain a provisional preview without creating exposure history.
344
- Batch heartbeats and live elapsed calculations keep progress reports current.
345
-
346
- ## Evidence lifetime
347
-
348
- The engine removes a downloaded full video after it checks the completed assessment.
349
- It retains captions, frames, hashes, and scorecards for later use.
350
- Unfinished jobs retain media for recovery.
351
- Cleanup preserves user-supplied source files.
352
-
353
- Optional transcription processes audio in chunks of at most ten minutes.
354
- It releases each decoded buffer before the next chunk.
355
- The transcription process releases its model when it exits.
356
-
357
- ## Installation boundary
358
-
359
- The portable manifest points to five skills for recommendations, setup, the dashboard, Reset, and Clean.
360
- The GitHub marketplace catalog uses the name `broader-vids` and pins its npm source to version `0.1.0`.
361
- The npm archive and plugin ZIP carry the same engine and public assets.
362
- The host installs a cache copy and does not need the working checkout's private files.
363
- Guided setup installs an external runtime through the Node launcher.
364
- A receipt stores the installed wheel digest, installed dependency groups, and their requirement-file digests after setup.
365
- The launcher checks this receipt under the runtime lease before normal commands.
366
- A mismatch or an older receipt requires setup, including after a package rebuild with the same version.
367
- Setup removes the old receipt before installation and writes the new one atomically after all tools succeed.
368
- Failed setup leaves the runtime unverified. `dashboard stop` bypasses the digest check so repair can proceed.
369
- The launcher's saved profile selection also applies to source commands.
370
- Profile data remains separate from installed copies and their upgrades.
371
-
372
- Dashboard lifecycle operations use a per-profile operating-system lock.
373
- They do not hold a database write transaction while stopping the server.
374
- A private token and public instance identity authenticate shutdown and server reuse.
375
- The loopback listener uses exclusive binding on Windows.
376
- An occupied preferred port causes a free-port fallback.
377
-
378
- The plugin runs on the user's computer. Recommendation rounds start on request.
379
- The current project does not provide cloud execution, a hosted dashboard, or a remote MCP service.
380
-
381
- ## Profile lifecycle
382
-
383
- Reset and Clean use the external Node launcher rather than a Python process inside the removable runtime.
384
- Both actions preview their scope and require the exact absolute path before application.
385
- A marker identifies managed profiles. Existing completed profiles also have a metadata-based recognition path.
386
- The launcher rejects protected roots and root directory links.
387
- Unknown files block complete removal and survive Reset.
388
- The exclusive runtime lease blocks concurrent launcher operations.
389
- Active engine locks, browser sign-in, and dashboard processes block destructive changes.
390
- Reset stages its managed files before deletion and restores them if staging fails.
391
- An unfinished reset blocks new database access until recovery completes.
392
- Clean removes the runtime only after engine processes finish.
393
- It releases its control lease before removing the empty control directory and profile directory.
394
- Those final removals are nonrecursive and preserve any concurrently recreated directory.
395
- Reset retains tools, transcription models, and Bilibili sign-in.
396
- Neither operation removes a user export outside the selected data directory or changes a remote blacklist.
1
+ # Architecture
2
+
3
+ Broader Vids has one Python engine.
4
+ The terminal, Codex skill, and local dashboard use that engine.
5
+ The Node launcher installs and starts the engine.
6
+ Per-profile locks coordinate launcher commands and setup.
7
+ Ordinary commands can share the runtime. Setup requires exclusive access.
8
+ Active engine operations and dashboard servers block setup.
9
+ The launcher releases locks after normal completion or command failure.
10
+ A stale or unreadable lock blocks later work until the operator checks for surviving processes and repairs the lock.
11
+ The launcher has no stale-lock repair command and does not implement recommendation logic.
12
+
13
+ The GitHub marketplace and explicit npm installer both install the prepared npm plugin.
14
+ The npm installer registers a small host catalog without GitHub access or profile access.
15
+ Setup installs the bundled engine into external profile storage after plugin installation.
16
+
17
+ ## Project files
18
+
19
+ ```text
20
+ broader-vids/
21
+ README.md Purpose, remote installation, and first use
22
+ LICENSE Broader Vids license
23
+ plugin.json Portable plugin identity
24
+ .agents/plugins/
25
+ marketplace.json Remote catalog tracking the latest public npm release
26
+ bin/
27
+ broader-vids.cjs Node setup and process launcher
28
+ plugin-install.cjs Explicit npm plugin registration through Codex
29
+ runtime-lock.cjs Shared runtime use and exclusive installation
30
+ profile-data.cjs Preview, reset, and complete data removal
31
+ skills/
32
+ broader-vids/ Recommendations and feedback instructions
33
+ broader-vids-setup/ Guided installation and profile selection
34
+ broader-vids-dashboard/ Local dashboard access and shutdown
35
+ broader-vids-reset/ Fresh profile with tools and sign-in retained
36
+ broader-vids-clean/ Complete local data removal
37
+ src/broader_vids/
38
+ cli/ Command arguments and dispatch
39
+ discovery/ Public searches and creator expansion
40
+ exploration/ Topic hierarchy and query planning
41
+ evidence/ Captions, frames, transcription, and assessment
42
+ account_switch.py Staged sign-in, verified replacement, and recovery
43
+ recommendations/ Screening, ranking, batches, and reports
44
+ retries.py Shared candidate retry counts and cancellation recovery
45
+ learning/ Feedback models and calibration
46
+ web/ Local server and browser assets
47
+ data/queries.json Curated search terms
48
+ onboarding.py Atomic platform choice and setup completion
49
+ preflight.py Shared readiness and incomplete-feedback checks
50
+ preflight_checks.py Bounded tool and authenticated-access probes
51
+ preferences.py Profile and video actions
52
+ creators.py Creator rules and immutable round snapshots
53
+ blacklist.py Authenticated Bilibili blacklist import
54
+ identity.py Platform, video, and part identities
55
+ policy.py Default weights and duration rules
56
+ parsing.py Platform response parsing
57
+ http.py Request limits and request records
58
+ processes.py Process deadlines and cleanup
59
+ time_budget.py Optional round time limits and JSON representation
60
+ windows_job.py Windows process ownership
61
+ owned_process.py Worker startup gate
62
+ storage.py Schema 12 database and exclusive locks
63
+ maintenance.py Evidence cleanup
64
+ token_usage.py Durable model-attempt receipts and stage reports
65
+ python/
66
+ core-requirements.txt Locked keyword-conversion dependency
67
+ evidence-requirements.txt Locked media and browser dependencies
68
+ asr-requirements.txt Locked transcription dependencies
69
+ *.whl Generated engine package, excluded from Git
70
+ scripts/
71
+ release_ci.py Patch planning, publication checks, and GitHub assets
72
+ versioning.py Validated release metadata synchronization
73
+ test.py Isolated parallel regression runner
74
+ verify_package.py Distribution checks
75
+ build_plugin.py Clean marketplace payload and plugin ZIP
76
+ tests/ Automated tests
77
+ docs/
78
+ WORKFLOW.md Recommendation steps
79
+ ARCHITECTURE.md This document
80
+ SCORING.md Scoring and learning models
81
+ REFERENCE.md Commands and project maintenance
82
+ .github/workflows/ci.yml Automated repository checks
83
+ pyproject.toml, uv.lock Python package and dependency definitions
84
+ package.json Node launcher and package file list
85
+ package-lock.json Locked Node development dependencies
86
+ ```
87
+
88
+ ## Processing flow
89
+
90
+ ```mermaid
91
+ flowchart LR
92
+ A[Codex skill or terminal] --> S[Guided setup when incomplete]
93
+ S --> T[Shared readiness and feedback checks]
94
+ T -- Required failure --> U[Repair instructions without recommendation work]
95
+ T -- Feedback needed --> V[Dashboard and stop or explicit continuation]
96
+ T -- Ready --> B[Topic planning and discovery]
97
+ V -- Continue this round --> B
98
+ B --> C[Collect 100 videos passing hard metadata floors]
99
+ P[Pending recommendations] --> C
100
+ C --> D[35 inspection candidates per platform]
101
+ D --> E[Captions and selected frames]
102
+ E --> F[Quality and full ranking]
103
+ F --> G[10 recommendations per platform]
104
+ G --> H[Chat recommendations and local dashboard]
105
+ H --> I[Explicit feedback]
106
+ I --> J[Learning models]
107
+ J --> F
108
+ H --> P
109
+ ```
110
+
111
+ The web layer displays saved results and submits explicit user actions.
112
+ It does not contain a separate scoring algorithm.
113
+ The skill directs the host to call the engine.
114
+ `worker_context.py` resolves the invoking thread's model and reasoning effort from local turn metadata.
115
+ New runs save that pair and pass it to planning and assessment CLI workers.
116
+ `recommendations/collection.py` continues candidate collection until 100 pass `screening.py` duration and raw engagement floors, or a saved limit stops collection.
117
+ Screening policy 5 combines metadata screening with learned interest and balances query coverage across thirty-five content-assessment slots.
118
+ Final selection favors distinct narrow topics and queries before it compares full scores.
119
+ `recommendations/diversity.py` implements these deterministic selection rules.
120
+ An insufficient range of assessed topics permits repeats and records the relaxation in each affected selection.
121
+ Each new round saves one learning-model reference for screening, preview, and final selection.
122
+ Resumed rounds retain their saved plan, policies, and remaining allowances.
123
+ Refill records retain query origins and anchors even when the discovery cache reaches its separate limit.
124
+
125
+ The recommendation collector uses `collection_policy.py` for retrieval groups and saved targets.
126
+ `collection_batches.py` scans bounded batches, fills group deficits, and preserves unfinished batch progress.
127
+ `collection.py` supplies targeted search continuations and resumes unfinished collection batches.
128
+ `budgets.py` defines request defaults and applies explicit request-cap increases under run and platform locks.
129
+ Request-cap changes update settings and add an audit event in one transaction.
130
+ They do not rewrite saved recommendation state or reset resource use.
131
+
132
+ ## Dashboard files and state
133
+
134
+ The Python server serves local HTML, CSS, and JavaScript modules.
135
+ HTTP/1.1 connections reuse sockets for asset and API requests.
136
+ Idle connections wait at most one second for another request. Active requests use a thirty-second socket timeout.
137
+ Shutdown disconnects clients and waits for request handlers to close their database connections before closing the account controller.
138
+ The server rejects ambiguous request lengths and closes connections after error responses.
139
+ It permits only explicit static asset paths.
140
+ The dashboard uses native form controls and dialogs.
141
+
142
+ | File under `web/static/` | Responsibility |
143
+ | --- | --- |
144
+ | `index.html`, `dashboard.css` | Application shell, themes, and responsive layout |
145
+ | `dashboard.js` | Navigation, refresh coordination, and user actions |
146
+ | `state.js` | Server records, selections, scroll positions, and local drafts |
147
+ | `api.js` | Local requests, action tokens, and response errors |
148
+ | `token-usage.js` | Saved-round token reports and scoped dialog refresh |
149
+ | `bilibili-account.js` | Account-switch controls, scoped status polling, and cancellation |
150
+ | `views.js` | Compact video rows and exclusion management |
151
+ | `review.js` | Four independent rating groups and the creator checkbox |
152
+ | `score.js`, `ui.js` | Technical score presentation and shared controls |
153
+
154
+ Recommendations, exclusions, and calibration have separate views.
155
+ The desktop list and review panel scroll independently.
156
+ Narrow windows use a review dialog with Back navigation.
157
+ Score details use a separate desktop dialog or a secondary mobile view.
158
+
159
+ The dashboard uses one system font and shared text-size variables.
160
+ Page titles, section titles, headings, body text, controls, captions, and small metadata each have a defined size.
161
+ Buttons, action links, navigation, and form controls share the control size.
162
+ Compact buttons reduce padding without reducing their text size.
163
+ The Bilibili panel places its external management link in the header and groups local actions below the account fields.
164
+ An idle account switch has no explanatory status text.
165
+ Local GET requests have a thirty-second limit, and writes have a ninety-second limit.
166
+ Account polling permits one status read at a time and skips unchanged interface updates.
167
+ A failed account-switch state remains readable so its controls can recover.
168
+ HTTP failures and failed writes still display errors.
169
+
170
+ Saved records remain separate from drafts.
171
+ Video identity joins feedback across recommendations and calibration.
172
+ Each successful action updates its affected records.
173
+ Refresh requests carry a read version and a mutation generation.
174
+ Only one refresh group runs at a time.
175
+ A refresh requested during an active read waits until that read finishes.
176
+ Old video records stay in browser memory only while the server returns them, an earlier list uses them, or they have an unsaved rating draft.
177
+ The interface rejects responses from before a newer mutation.
178
+ It pauses background refresh while the tab is hidden or a write is active.
179
+ Refresh preserves active controls and pane positions.
180
+ Drafts survive navigation within the tab, but they require Save before the page closes.
181
+
182
+ The engine stores token usage as versioned `token_usage` events in the existing database.
183
+ Each attempt has one call ID. Later events update that attempt's known status and usage.
184
+ The report reads the last event for each call ID.
185
+ Parallel workers record the owning recommendation ID and the source evidence ID.
186
+ New rounds use exact owner receipts. Historical rounds use saved checkpoint usage where available.
187
+ A derived database index supports report queries without a schema-version change.
188
+ The ledger stores counters and identifiers, without copying prompts, frames, transcripts, or credentials.
189
+ Reports do not depend on recommendation publication or media files.
190
+ A missing measurement remains unknown after interruption.
191
+ Reset and Clean remove usage with recommendation history.
192
+ Preference import and export do not transfer usage history.
193
+
194
+ ## Module responsibilities
195
+
196
+ | Area | Main files | Responsibility |
197
+ | --- | --- | --- |
198
+ | Discovery | `queries.py`, `languages.py`, `sources.py`, `state.py`, `service.py` | Build searches, check keyword scripts, parse results, and retain discovery progress |
199
+ | Exploration | `domains.py`, `coverage.py`, `rotation.py`, `proposals.py`, `planning.py` | Select underused leaf topics, coordinate reservations, and plan queries |
200
+ | Evidence | `service.py`, `media.py`, `downloader.py`, `captions.py`, `browser.py` | Resolve playable videos and collect bounded evidence |
201
+ | Evidence workers | `preparation_worker.py`, `asr_worker.py` | Separate media preparation and optional transcription processes |
202
+ | Assessment | `assessment.py`, `worker_instructions.txt`, `reports.py`, `retention.py` | Check model output, report evidence limits, and remove completed media |
203
+ | Recommendations | `service.py`, `batch.py`, `parallel.py`, `timing.py` | Coordinate platform jobs, worker limits, and recovery |
204
+ | Ranking | `engagement.py`, `metadata.py`, `originality.py`, `quality.py`, `ranking.py`, `diversity.py`, `personalization.py` | Screen candidates and select the final list |
205
+ | Learning | `features.py`, `topic_labels.py`, `inference.py`, `scoped.py`, `depth.py`, `service.py` | Capture predictors, fit models, and calculate predictions |
206
+ | Preference transfer | `learning/transfer.py`, `learning/posterior.py`, `learning/baseline.py`, `web/static/preferences.js` | Transfer learned parameters without video records |
207
+ | Feedback | `feedback.py`, `calibration.py`, `evaluation.py`, `evidence.py`, `annotations.py` | Store judgments, reserve test videos, and record evidence corrections |
208
+
209
+ File names in this table are relative to their area under `src/broader_vids/`.
210
+
211
+ ## Data storage
212
+
213
+ The engine resolves one external data directory for each command.
214
+ The installed package remains separate from this directory.
215
+ See [data directory selection](REFERENCE.md#choose-a-data-directory).
216
+
217
+ | Stored data | Purpose |
218
+ | --- | --- |
219
+ | `.broader-vids-profile.json` | Profile ownership marker for safe lifecycle operations |
220
+ | `onboarding.json` | Versioned default platform and setup completion, separate from learned parameters |
221
+ | `broader-vids.sqlite3` | Jobs, request records, profile, video state, feedback, models, and calibration |
222
+ | `evidence/` | Captions, frames, scorecards, retained hashes, and unfinished media |
223
+ | `recommendations/` | Local Markdown digests and JSON reports |
224
+ | `feedback/` | Local content descriptor records |
225
+ | `runs/` | Discovery reports and request traces |
226
+ | `access/bilibili/profile/` | Separate browser profile for Bilibili sign-in |
227
+ | `runtimes/`, `wheels/` | External engine environments and cached wheels for the Node launcher |
228
+
229
+ Onboarding uses an atomic versioned file protected by a separate configuration lease.
230
+ It does not change SQLite schema 12 or the learning revision.
231
+ Preference transfer excludes this file.
232
+ Preflight counts unique recommendation exposures against current structured ratings.
233
+ Cleared fields count as incomplete again.
234
+ Its isolated subprocess checks have a deadline and owned process cleanup.
235
+ Bilibili checks use the existing browser profile without reading cookies into model inputs.
236
+ Account switching stages a fresh profile under `access/bilibili/switches/`.
237
+ A bounded worker owns the browser, blacklist, and Bilibili recommendation leases.
238
+ The dashboard polls its status without blocking navigation or rating drafts.
239
+ After verification, a file journal and database receipt coordinate session activation.
240
+ Recovery restores the previous files before a database commit or removes the old session after a commit.
241
+ Completed and cancelled workers remove their temporary profiles and staged blacklist database.
242
+ Failed cleanup retains the journal and blocks browser use until recovery succeeds.
243
+ New and resumed recommendation commands record automated-check time.
244
+ That time counts toward an explicit total allowance when one exists.
245
+
246
+ This release supports fresh empty databases (schema 0) and existing schema 12 profiles.
247
+ The engine rejects other database schemas without changing them.
248
+ Schema 12 stores current structured ratings and action records.
249
+ Explicit tag rules reside in the user profile.
250
+ Creator rules reside in creator_exclusions.
251
+ Creator_policy stores their revision and active Bilibili account ID.
252
+ Creator_sync stores each account's import status and last successful sync.
253
+ Creator rules have a separate revision from the learning profile.
254
+ Round state stores versioned creator-rule and learning-model references.
255
+ Public recommendation reports expose rule counts rather than imported creator IDs.
256
+ Topic anchors remain stable across rating edits and clears.
257
+ Model snapshots retain the identities of their training inputs.
258
+ Frozen evaluation records retain the tested model identity.
259
+
260
+ Preference transfer accepts format 4 with learning model version 8 and feature version 1.
261
+ Manual creator rules transfer separately from learning parameters and replace the manual lists transactionally on import.
262
+ The export omits account-specific blacklist copies and sign-in data.
263
+ It includes a bounded exploration summary with at most 1,024 nodes and aggregate counts and round ages.
264
+ It excludes individual video history.
265
+ It does not change SQLite schema 12.
266
+ A completed `preference_import` job stores the imported parameter baseline and local feedback cutoff.
267
+ Fitting combines that baseline with active ratings saved after the cutoff.
268
+ Historical ratings remain available to the dashboard.
269
+ Import holds both platform locks and the learning lock.
270
+ Import prepares the restored model before the write transaction.
271
+ A short transaction checks the preview revision and commits the replacement atomically.
272
+ Concurrent feedback remains available and prevents a stale import from committing.
273
+ Export reuses an exact cached model when available.
274
+ Input validation runs once per terminal or HTTP request.
275
+ Failed imports remove unused recovery files.
276
+ Import also removes obsolete model caches while preserving models referenced by recommendations or frozen evaluations.
277
+ An uncertain commit retains its recovery file.
278
+ Parameter-only recovery files reside in `backups/`.
279
+ The dashboard uses authenticated local POST operations for export, preview, and import.
280
+ The transfer dialog preserves rating drafts and blocks stale polling responses during an operation.
281
+
282
+ Evidence corrections add derived records.
283
+ They do not overwrite original scorecards or user ratings.
284
+
285
+ The Preferences page uses `preferences.js` to coordinate `/api/interests` and the preference-transfer endpoints.
286
+ One controller serializes interest saves and file operations.
287
+ A successful import returns the saved profile and updates the editor without another page.
288
+ `tag_rules.py` combines manual tag exclusions with removable built-in defaults.
289
+ The profile stores disabled built-in tags separately from manual tags.
290
+ Preference imports preserve these local exclusion choices.
291
+ Tag saves update rules without fitting a learning model.
292
+ The dashboard serializes tag writes per platform and rejects older tag snapshots.
293
+ An interest save preserves a draft when only exclusion metadata changed.
294
+ Profile revisions prevent one window from replacing a newer interest edit.
295
+ Interest writes hold both platform locks and preserve learned parameters and historical video data.
296
+
297
+ ## Concurrency and recovery
298
+
299
+ One batch coordinates both platforms.
300
+ The batch shares worker limits and an optional total time limit.
301
+ New rounds have no overall deadline by default.
302
+ The time policy keeps internal unlimited values out of saved JSON.
303
+ It allows four preparation workers in total, with at most two per platform.
304
+ It allows up to four model workers.
305
+ Prepared evidence bypasses the preparation queue.
306
+ Preparation allows up to five minutes after its worker slot becomes available.
307
+ Assessment allows up to three minutes after its model slot becomes available.
308
+ Queue waits count toward an explicit parent time allowance when one exists.
309
+ They do not consume those separate stage allowances.
310
+ Metadata workers use separate database connections, a shared cookie jar, and a shared request gate.
311
+ YouTube serializes metadata retrieval so each response can update the visitor session before the next request.
312
+ Bilibili retains two concurrent metadata workers.
313
+ Metadata request starts use a one-second interval.
314
+ Pacing waits count toward an optional saved deadline.
315
+ Access failures cancel queued retrieval before another YouTube collector starts.
316
+ The gate preserves request pacing and prevents concurrent reservation of the last request slot.
317
+ Platform locks prevent conflicting recommendation jobs.
318
+ Short database transactions protect user actions and final result writes.
319
+ Dashboard lookup checks recommendation rounds from newest to oldest and stops at the first usable list.
320
+ Feedback resolves the displayed video evidence before it acquires its database write transaction.
321
+ Earlier pending videos retain their saved topic and depth context.
322
+
323
+ The engine checks profile, feedback, and queue changes before it saves a final recommendation list.
324
+ A conflicting change requires another ranking calculation.
325
+ An interrupted job retains its candidate position and remaining allowances.
326
+ Continuation does not reset those allowances.
327
+ New rounds save an inspection limit of 35 videos per platform.
328
+ Resumed rounds retain their saved inspection limit.
329
+
330
+ Each worker belongs to an owned process group.
331
+ Windows uses a Job Object to stop the worker's child processes.
332
+ Cancellation checks prevent new work after cancellation or an explicit deadline.
333
+ The engine retains unfinished work when access or assessment fails.
334
+ Candidate failures remain local to that candidate.
335
+ YouTube and Bilibili share the same limit of three retries for recoverable candidate exceptions.
336
+ The engine saves attempt counts before worker dispatch. Continuation cannot grant an extra attempt.
337
+ Manual cancellation preserves that allowance.
338
+ Exhausted candidates become unavailable while other candidates continue.
339
+ Caption access challenges stop acquisition for the affected platform.
340
+ Already prepared assessments retain their individual timeouts and any explicit shared deadline.
341
+ Other platform access challenges and shared model failures stop the applicable workers.
342
+ Reports preserve the first failure separately from later cancellation messages.
343
+ Preparation checkpoints record transfer progress for interrupted-byte accounting.
344
+ Unknown usage retains an explicit upper bound.
345
+ Paused rounds can retain a provisional preview without creating exposure history.
346
+ Batch heartbeats and live elapsed calculations keep progress reports current.
347
+
348
+ ## Evidence lifetime
349
+
350
+ The engine removes a downloaded full video after it checks the completed assessment.
351
+ It retains captions, frames, hashes, and scorecards for later use.
352
+ Unfinished jobs retain media for recovery.
353
+ Cleanup preserves user-supplied source files.
354
+
355
+ Optional transcription processes audio in chunks of at most ten minutes.
356
+ It releases each decoded buffer before the next chunk.
357
+ The transcription process releases its model when it exits.
358
+
359
+ ## Installation boundary
360
+
361
+ The portable manifest points to five skills for recommendations, setup, the dashboard, Reset, and Clean.
362
+ The GitHub marketplace catalog uses the name `broader-vids` and follows the npm `latest` distribution tag.
363
+ The npm archive and plugin ZIP carry the same engine and public assets.
364
+ The host installs a cache copy and does not need the working checkout's private files.
365
+ Guided setup installs an external runtime through the Node launcher.
366
+ A receipt stores the installed wheel digest, installed dependency groups, and their requirement-file digests after setup.
367
+ The launcher checks this receipt under the runtime lease before normal commands.
368
+ A mismatch or an older receipt requires setup, including after a package rebuild with the same version.
369
+ Setup removes the old receipt before installation and writes the new one atomically after all tools succeed.
370
+ Failed setup leaves the runtime unverified. `dashboard stop` bypasses the digest check so repair can proceed.
371
+ The launcher's saved profile selection also applies to source commands.
372
+ Profile data remains separate from installed copies and their upgrades.
373
+
374
+ Dashboard lifecycle operations use a per-profile operating-system lock.
375
+ They do not hold a database write transaction while stopping the server.
376
+ A private token and public instance identity authenticate shutdown and server reuse.
377
+ The loopback listener uses exclusive binding on Windows.
378
+ An occupied preferred port causes a free-port fallback.
379
+
380
+ The plugin runs on the user's computer. Recommendation rounds start on request.
381
+ The current project does not provide cloud execution, a hosted dashboard, or a remote MCP service.
382
+
383
+ ## Profile lifecycle
384
+
385
+ Reset and Clean use the external Node launcher rather than a Python process inside the removable runtime.
386
+ Both actions preview their scope and require the exact absolute path before application.
387
+ A marker identifies managed profiles. Existing completed profiles also have a metadata-based recognition path.
388
+ The launcher rejects protected roots and root directory links.
389
+ Unknown files block complete removal and survive Reset.
390
+ The exclusive runtime lease blocks concurrent launcher operations.
391
+ Active engine locks, browser sign-in, and dashboard processes block destructive changes.
392
+ Reset stages its managed files before deletion and restores them if staging fails.
393
+ An unfinished reset blocks new database access until recovery completes.
394
+ Clean removes the runtime only after engine processes finish.
395
+ It releases its control lease before removing the empty control directory and profile directory.
396
+ Those final removals are nonrecursive and preserve any concurrently recreated directory.
397
+ Reset retains tools, transcription models, and Bilibili sign-in.
398
+ Neither operation removes a user export outside the selected data directory or changes a remote blacklist.