@heripo/research-radar 6.3.1 → 6.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +17 -12
- package/dist/index.d.ts +48 -1
- package/dist/index.js +417 -34
- package/dist/index.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -255,16 +255,17 @@ formats, omit it for the default, or pass an empty string to suppress it.
|
|
|
255
255
|
|
|
256
256
|
### Optional generation settings
|
|
257
257
|
|
|
258
|
-
| Option
|
|
259
|
-
|
|
|
260
|
-
| `logger`
|
|
261
|
-
| `publishDate`
|
|
262
|
-
| `customFetch`
|
|
263
|
-
| `templateOptions`
|
|
264
|
-
| `promptProvider`
|
|
265
|
-
| `excavationReportSource`
|
|
266
|
-
| `journalIssueSource`
|
|
267
|
-
| `
|
|
258
|
+
| Option | Behavior |
|
|
259
|
+
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
260
|
+
| `logger` | Core `AppLogger` implementation, such as `console` |
|
|
261
|
+
| `publishDate` | A real calendar date in `YYYY-MM-DD` format; invalid values throw. Defaults to the current date in `Asia/Seoul` (KST), regardless of server timezone |
|
|
262
|
+
| `customFetch` | A `typeof fetch` implementation for crawling and parser API requests, such as a proxy adapter; does not configure LLM requests |
|
|
263
|
+
| `templateOptions` | Default or KRAS newsletter branding and Markdown sections (see below) |
|
|
264
|
+
| `promptProvider` | Core `PromptProvider` overriding the package's own LLM prompts entirely; omit to use the package's prompts, which fall back to core's defaults |
|
|
265
|
+
| `excavationReportSource` | Supplies 국가유산청 발굴조사 보고서 entries from the application; that board is then never crawled. Omit to crawl it as before |
|
|
266
|
+
| `journalIssueSource` | Supplies newly published journal issues (such as KCI) from the application, one article per issue. Omit and that target collects nothing |
|
|
267
|
+
| `heritageBidVerdictStore` | Keeps 나라장터 triage verdicts between runs so the overlapping half of the window is not judged again. Omit and each run judges the whole window |
|
|
268
|
+
| `previewNewsletter` | Fetch a saved `Newsletter` and send it through a supplied core `EmailService` |
|
|
268
269
|
|
|
269
270
|
`previewNewsletter` requires `fetchNewsletterForPreview: () => Promise<Newsletter>`, `emailService` (with `send(message)`), and `emailMessage` (core `EmailMessage` without `subject`, `html`, or `text`). The core fills those three fields and skips preview delivery when no newsletter was created. Ensure the callback fetches the issue saved by this run.
|
|
270
271
|
|
|
@@ -331,6 +332,8 @@ await generateNewsletter({
|
|
|
331
332
|
});
|
|
332
333
|
```
|
|
333
334
|
|
|
335
|
+
Every model resolved here — including one supplied through `contentGeneration` — is wrapped so the provider's prompt cache applies on both sides. Each stage's long instruction is sent as a system message ahead of the article, which OpenAI caches automatically once it clears 1,024 tokens; Anthropic caches nothing without an explicit breakpoint, so the wrapper marks that system message with `cache_control`. `providerOptions` is namespaced, so OpenAI and Google requests are unchanged. Anthropic does not cache a block below a minimum length, and that minimum differs per model by several times between the small and large ones, so no length check is made here: a block under it is processed without caching rather than rejected. Each call reports its cache reads and writes through the `llm.prompt.cache` log event, which is what answers whether a given stage and model actually cache.
|
|
336
|
+
|
|
334
337
|
The optional `contentGeneration: { provider, apiKey, model? }` compatibility path still supports Anthropic and Google for newsletter generation. `models.generateNewsletter` takes precedence when both are supplied. If `models.generateNewsletter` is omitted or `undefined`, the existing `contentGeneration` provider/model selection applies; without either option, generation uses `llmConfig.models.generateNewsletter`. `openAIApiKey` remains required even when supplying model objects for every stage.
|
|
335
338
|
|
|
336
339
|
[src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.1.1 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
|
|
@@ -348,7 +351,9 @@ The optional `contentGeneration: { provider, apiKey, model? }` compatibility pat
|
|
|
348
351
|
|
|
349
352
|
Only the excavation status board stays commented out, as low-value fragmented data. Boards that a site's robots.txt restricts are configured normally and refused at runtime by the robots.txt check, so the configuration does not have to track each site's policy by hand. With current policies 14 of the 73 targets are refused.
|
|
350
353
|
|
|
351
|
-
Three targets are read from data.go.kr open APIs instead of scraped: 나라일터 (`PblJobService`) and 알리오 (`recruitment`) under Employment, and 나라장터 (`BidPublicInfoService`) under Business. Each is a single round of requests, and `publicDataApiKey` supplies the service key — omit it and they answer with an empty list without making a request. The boards carry every public-sector vacancy and procurement notice in the country, so both are narrowed before analysis. The job boards use `src/crawling/heritage-job-filter.ts`, whose exclusions are role nouns and hold up as patterns; roughly 2% of postings survive, about 2.6 a day. 나라장터 instead uses `src/crawling/heritage-triage.ts`, which asks a cheap model about 100 titles per request: heritage vocabulary cannot be settled by substring in Korean, and the deterministic filter remains behind it as the fallback when a batch fails.
|
|
354
|
+
Three targets are read from data.go.kr open APIs instead of scraped: 나라일터 (`PblJobService`) and 알리오 (`recruitment`) under Employment, and 나라장터 (`BidPublicInfoService`) under Business. Each is a single round of requests, and `publicDataApiKey` supplies the service key — omit it and they answer with an empty list without making a request. The boards carry every public-sector vacancy and procurement notice in the country, so both are narrowed before analysis. The job boards use `src/crawling/heritage-job-filter.ts`, whose exclusions are role nouns and hold up as patterns; roughly 2% of postings survive, about 2.6 a day. 나라장터 instead uses `src/crawling/heritage-triage.ts`, which asks a cheap model about 100 titles per request: heritage vocabulary cannot be settled by substring in Korean, and the deterministic filter remains behind it as the fallback when a batch fails. A response is accepted whether it arrives as the schema object, a bare array or fenced text, so a provider's output conventions do not silently hand the batch back to the regex; only parsed JSON counts, never integers scraped from prose, and the index list is taken whole or not at all — trimming a stray `"2"` out of `[0, "2"]` leaves a list that reads exactly like a real verdict rejecting that row.
|
|
355
|
+
|
|
356
|
+
Verdicts are cached in two places. Within a run they are held in memory, which is what core's retry needs — core aborts the crawl fetch after 10 seconds and tries again, and without it every attempt would pay for the window twice. Across runs they are held by the optional `heritageBidVerdictStore`: the window is 48 hours and the run is daily, so half of each window was judged yesterday, and because triage only passes its survivors on, that half is almost entirely rejections that never reach `saveCrawledArticles` and cannot be recognised from the article table. Keys are hashes covering the notice as the model sees it plus a fingerprint of the prompt, schema and model id, so editing the selection rules or switching providers retires the old rows rather than reusing them. Verdicts that came from the deterministic fallback are never persisted, and `crawl.g2b.triage.cache` reports where each call's verdicts came from. The store cannot fail the crawl: an error is reported and the run judges normally, and a lookup that has not answered within about two seconds is left behind — core aborts the crawl fetch at ten seconds without racing the fetch promise, so a `load()` that never settles would otherwise hang the run. A late answer is still used, but one stall takes the store out of that run.
|
|
352
357
|
|
|
353
358
|
Newly published journal issues are not crawled. The application supplies them through `journalIssueSource` — heripo-web reads them from its KCI paper catalog — and the `KCI_학술지_신간` target serves them from that source, so its URL is never requested. Each issue (권호) becomes one article: its `uniqId` is the issue's `externalId` and its URL the issue's `url`. `usedArticles` carry only the application's article id and URL, so the application records which issues a saved newsletter introduced through the `uniqId` it stored in `saveCrawledArticles`, or through the issue URL. The article body, which scoring sees, lists every paper with its authors, keywords and an abstract truncated to 300 characters, after the keywords several papers share. Without a source the target answers with an empty list. This replaces the former crawl of the 국립문화유산연구원 헤리티지:역사와 과학 and 보존과학연구 boards.
|
|
354
359
|
|
|
@@ -425,7 +430,7 @@ npm run lint # lint source files
|
|
|
425
430
|
npm run lint:fix # lint with autofix
|
|
426
431
|
npm run lint:ci # quiet CI lint
|
|
427
432
|
npm run typecheck # TypeScript type-check (source and tests)
|
|
428
|
-
npm test # offline model
|
|
433
|
+
npm test # offline regression tests (model wiring, triage, deadlines, journals)
|
|
429
434
|
|
|
430
435
|
# formatting
|
|
431
436
|
npm run format # format src/ with Prettier
|
package/dist/index.d.ts
CHANGED
|
@@ -318,6 +318,43 @@ interface NewsletterRepository {
|
|
|
318
318
|
id: string | number;
|
|
319
319
|
}>;
|
|
320
320
|
}
|
|
321
|
+
/**
|
|
322
|
+
* Storage for 나라장터 triage verdicts, so a judgement outlives its run.
|
|
323
|
+
*
|
|
324
|
+
* The 48-hour window and a daily run overlap by half, and the overlap is almost
|
|
325
|
+
* entirely rejections — notices the model threw out, which never reach
|
|
326
|
+
* `saveCrawledArticles` and so cannot be recognised from the article table.
|
|
327
|
+
* Without somewhere to keep them, every run pays to re-judge yesterday's half.
|
|
328
|
+
*
|
|
329
|
+
* Keys are 64-character lowercase hex. They cover the notice as the model sees
|
|
330
|
+
* it **and** a fingerprint of the prompt, schema and model id, so editing the
|
|
331
|
+
* selection rules or switching providers starts a fresh generation of keys and
|
|
332
|
+
* never reuses a verdict the current configuration did not make. Old rows go
|
|
333
|
+
* stale rather than wrong: prune anything older than a few days.
|
|
334
|
+
*
|
|
335
|
+
* A run asks for about 2,100 keys at once, which is past what some clients will
|
|
336
|
+
* put in a URL — chunk `load`, or send it as a request body. Chunk it in
|
|
337
|
+
* parallel: `load` has roughly two seconds before the crawl stops waiting for
|
|
338
|
+
* it and judges the window itself, and a lookup that misses that deadline once
|
|
339
|
+
* is not consulted again for the rest of the run. A late answer is still used
|
|
340
|
+
* if it arrives, so a slow lookup costs time rather than correctness.
|
|
341
|
+
*
|
|
342
|
+
* Neither call is allowed to fail the crawl: an error is reported and the run
|
|
343
|
+
* judges normally. That holds whether the failure arrives as a rejection or is
|
|
344
|
+
* thrown outright before a promise is ever returned, so an implementation does
|
|
345
|
+
* not have to defend its own synchronous paths.
|
|
346
|
+
*/
|
|
347
|
+
interface HeritageBidVerdictStore {
|
|
348
|
+
/**
|
|
349
|
+
* Verdicts for the keys that are known. Unknown keys are simply absent.
|
|
350
|
+
*/
|
|
351
|
+
load(keys: readonly string[]): Promise<ReadonlyMap<string, boolean>>;
|
|
352
|
+
/**
|
|
353
|
+
* Persists verdicts the model made. Verdicts that came from the
|
|
354
|
+
* deterministic fallback are never passed here.
|
|
355
|
+
*/
|
|
356
|
+
save(verdicts: ReadonlyMap<string, boolean>): Promise<void>;
|
|
357
|
+
}
|
|
321
358
|
|
|
322
359
|
/**
|
|
323
360
|
* Uses the required OpenAI key for every default model. Each task accepts an
|
|
@@ -452,6 +489,16 @@ interface NewsletterGeneratorDependencies {
|
|
|
452
489
|
* nothing while every other target is unaffected.
|
|
453
490
|
*/
|
|
454
491
|
publicDataApiKey?: string;
|
|
492
|
+
/**
|
|
493
|
+
* Storage for 나라장터 triage verdicts (optional).
|
|
494
|
+
*
|
|
495
|
+
* The window is 48 hours and the run is daily, so half of every window was
|
|
496
|
+
* already judged by the previous run — almost all of it rejections, which
|
|
497
|
+
* never reach `saveCrawledArticles` and so cannot be recognised from the
|
|
498
|
+
* article table. Supply a store and that half is read back instead of being
|
|
499
|
+
* sent to the model again; omit it and each run judges the whole window.
|
|
500
|
+
*/
|
|
501
|
+
heritageBidVerdictStore?: HeritageBidVerdictStore;
|
|
455
502
|
/**
|
|
456
503
|
* LLM prompt overrides (optional).
|
|
457
504
|
*
|
|
@@ -989,4 +1036,4 @@ declare function fetchResearchRadarDeadlineSource({ source, customFetch, publicD
|
|
|
989
1036
|
declare function refreshResearchRadarDeadlines(config: ResearchRadarDeadlineRefreshConfig): Promise<ResearchRadarDeadlineRefreshResult>;
|
|
990
1037
|
|
|
991
1038
|
export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DEADLINE_REMINDER_LEAD_DAYS, DEADLINE_REMINDER_LIMIT, DateService, JOURNAL_ISSUE_LIST_URL, TaskService, contentOptions, createCrawlingTargetGroups, createDeadlineReminderSelector, extractResearchRadarDeadlines, fetchResearchRadarDeadlineSource, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, maximumImportanceScoreByDomain, newsletterConfig, refreshResearchRadarDeadlines, renderVerifiedDeadlineReminders, researchRadarPromptProvider, robotsExemptOrigins };
|
|
992
|
-
export type { ArticleRepository, ContentGenerationConfig, ContentOptions, DeadlineReminderSelector, DeadlineReminderSelectorOptions, DeadlineVerificationFailure, EligibleDeadlineReminder, ExcavationReport, ExcavationReportSource, JournalIssue, JournalIssuePaper, JournalIssueSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterModelConfig, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, ResearchRadarDeadlineExtractionConfig, ResearchRadarDeadlineRefreshConfig, ResearchRadarDeadlineRefreshResult, ResearchRadarDeadlineSource, ResearchRadarDeadlineSourceOptions, ResearchRadarDeadlineSourceResult, SourceGroup, SourceItem, TagRepository, TaskRepository, VerifiedDeadlineReminder, WelcomeTemplateOptions };
|
|
1039
|
+
export type { ArticleRepository, ContentGenerationConfig, ContentOptions, DeadlineReminderSelector, DeadlineReminderSelectorOptions, DeadlineVerificationFailure, EligibleDeadlineReminder, ExcavationReport, ExcavationReportSource, HeritageBidVerdictStore, JournalIssue, JournalIssuePaper, JournalIssueSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterModelConfig, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, ResearchRadarDeadlineExtractionConfig, ResearchRadarDeadlineRefreshConfig, ResearchRadarDeadlineRefreshResult, ResearchRadarDeadlineSource, ResearchRadarDeadlineSourceOptions, ResearchRadarDeadlineSourceResult, SourceGroup, SourceItem, TagRepository, TaskRepository, VerifiedDeadlineReminder, WelcomeTemplateOptions };
|
package/dist/index.js
CHANGED
|
@@ -2,14 +2,14 @@ import { createAnthropic } from '@ai-sdk/anthropic';
|
|
|
2
2
|
import { createGoogleGenerativeAI } from '@ai-sdk/google';
|
|
3
3
|
import { createOpenAI } from '@ai-sdk/openai';
|
|
4
4
|
import { DateType, isArticleDeadlineGrounded, GenerateNewsletter, extractArticleDeadlines } from '@llm-newsletter-kit/core';
|
|
5
|
-
import { jsonSchema, generateText, Output } from 'ai';
|
|
5
|
+
import { jsonSchema, generateText, Output, NoObjectGeneratedError, wrapLanguageModel } from 'ai';
|
|
6
|
+
import { createHash } from 'node:crypto';
|
|
6
7
|
import * as cheerio from 'cheerio';
|
|
7
8
|
import TurndownService from 'turndown';
|
|
8
9
|
import { JSDOM } from 'jsdom';
|
|
9
10
|
import juice from 'juice';
|
|
10
11
|
import safeMarkdown2Html from 'safe-markdown2html';
|
|
11
12
|
import DOMPurify from 'dompurify';
|
|
12
|
-
import { createHash } from 'node:crypto';
|
|
13
13
|
|
|
14
14
|
/**
|
|
15
15
|
* Deterministic pre-filter for public job-posting APIs.
|
|
@@ -134,6 +134,24 @@ function isHeritageBidCandidate({ institution, title, classification, }) {
|
|
|
134
134
|
|
|
135
135
|
const DEFAULT_BATCH_SIZE = 100;
|
|
136
136
|
const DEFAULT_CONCURRENCY = 24;
|
|
137
|
+
/**
|
|
138
|
+
* How long a store lookup may hold the crawl up.
|
|
139
|
+
*
|
|
140
|
+
* The store is optional and the crawl is not, so a lookup that stalls must not
|
|
141
|
+
* take the run with it. Core aborts the crawl fetch at 10 seconds and only
|
|
142
|
+
* calls `abort()` — it does not race the fetch promise — so nothing outside
|
|
143
|
+
* this function will ever end a `load()` that simply never settles.
|
|
144
|
+
*
|
|
145
|
+
* Two seconds is the budget rather than a round number. The API collection in
|
|
146
|
+
* the same fetch takes about 5.8 seconds and triage's own wave 5 to 8, so a
|
|
147
|
+
* lookup costing more than about two turns the saving into a loss; and a limit
|
|
148
|
+
* much above four would let core's abort land first every time, which leaves
|
|
149
|
+
* the lookup looking fine and makes all five attempts pay the stall again. A
|
|
150
|
+
* lookup of some 2,100 keys should answer well inside a second.
|
|
151
|
+
*/
|
|
152
|
+
const DEFAULT_STORE_LOAD_TIMEOUT_MS = 2_000;
|
|
153
|
+
/** A lookup that outlasted its budget, as distinct from one that failed. */
|
|
154
|
+
const TIMED_OUT = Symbol('store.load timed out');
|
|
137
155
|
/**
|
|
138
156
|
* Asks for the indices to keep rather than a verdict per row.
|
|
139
157
|
*
|
|
@@ -142,7 +160,7 @@ const DEFAULT_CONCURRENCY = 24;
|
|
|
142
160
|
* mistake without failing. Indices carry their own alignment, and anything out
|
|
143
161
|
* of range or repeated is discarded rather than trusted.
|
|
144
162
|
*/
|
|
145
|
-
const
|
|
163
|
+
const KEEP_JSON_SCHEMA = {
|
|
146
164
|
type: 'object',
|
|
147
165
|
properties: {
|
|
148
166
|
keep: {
|
|
@@ -153,7 +171,8 @@ const KEEP_SCHEMA = jsonSchema({
|
|
|
153
171
|
},
|
|
154
172
|
required: ['keep'],
|
|
155
173
|
additionalProperties: false,
|
|
156
|
-
}
|
|
174
|
+
};
|
|
175
|
+
const KEEP_SCHEMA = jsonSchema(KEEP_JSON_SCHEMA);
|
|
157
176
|
const SYSTEM_PROMPT = `너는 대한민국 나라장터 입찰공고를 국가유산 뉴스레터용으로 선별한다.
|
|
158
177
|
|
|
159
178
|
이것은 **선별** 단계이지 평가 단계가 아니다. 중요도와 기사 가치는 뒤의 채점
|
|
@@ -208,35 +227,102 @@ function buildUserPrompt$2(candidates) {
|
|
|
208
227
|
});
|
|
209
228
|
return `다음 ${candidates.length}건 중 국가유산 업무와 관련될 가능성이 있는 공고의 번호만 골라라.\n\n${lines.join('\n')}`;
|
|
210
229
|
}
|
|
230
|
+
/**
|
|
231
|
+
* Reads the index list out of whatever shape the response arrived in.
|
|
232
|
+
*
|
|
233
|
+
* The output spec parses JSON but does not enforce the schema, so a model that
|
|
234
|
+
* answers with the bare `[0, 2]` instead of `{"keep": [0, 2]}` reaches us as a
|
|
235
|
+
* perfectly valid result whose `.keep` is `undefined`. Both shapes mean the
|
|
236
|
+
* same thing and both are accepted.
|
|
237
|
+
*
|
|
238
|
+
* Every element must be an integer, or the whole list is refused. Dropping the
|
|
239
|
+
* bad ones instead would turn `[0, "2"]` into `[0]`, which keeps row 0, rejects
|
|
240
|
+
* row 2 and is indistinguishable downstream from a model that considered row 2
|
|
241
|
+
* and said no — `usable` is true, so the out-of-range guard never fires, the
|
|
242
|
+
* batch counts as the model's own work and the store persists it. `["0", "2"]`
|
|
243
|
+
* is the same failure with every row silently rejected.
|
|
244
|
+
*/
|
|
245
|
+
function readKeep(value) {
|
|
246
|
+
const keep = Array.isArray(value)
|
|
247
|
+
? value
|
|
248
|
+
: (value ?? undefined)?.keep;
|
|
249
|
+
if (!Array.isArray(keep)) {
|
|
250
|
+
return undefined;
|
|
251
|
+
}
|
|
252
|
+
// An empty list stays legitimate: a batch of road works has nothing to keep.
|
|
253
|
+
return keep.every((index) => Number.isInteger(index))
|
|
254
|
+
? keep
|
|
255
|
+
: undefined;
|
|
256
|
+
}
|
|
257
|
+
/**
|
|
258
|
+
* Recovers the index list from a response the output spec could not parse.
|
|
259
|
+
*
|
|
260
|
+
* Structured output is not uniform across providers: a model may fence its
|
|
261
|
+
* JSON, or — where the provider has no native schema support and the JSON tool
|
|
262
|
+
* was declined — answer in plain text. Falling straight back to
|
|
263
|
+
* {@link isHeritageBidCandidate} in those cases would quietly reinstate the
|
|
264
|
+
* regex this stage exists to replace, so the response is given one honest
|
|
265
|
+
* second reading first.
|
|
266
|
+
*
|
|
267
|
+
* Only parsed JSON counts. Scraping integers out of prose would read
|
|
268
|
+
* "100건 중 3번과 15번" as keeping rows 100, 3 and 15, which looks like a
|
|
269
|
+
* verdict and is not one.
|
|
270
|
+
*/
|
|
271
|
+
function salvageKeep(text) {
|
|
272
|
+
if (!text) {
|
|
273
|
+
return undefined;
|
|
274
|
+
}
|
|
275
|
+
const unfenced = text
|
|
276
|
+
.replace(/^\s*```(?:json)?\s*/i, '')
|
|
277
|
+
.replace(/\s*```\s*$/, '');
|
|
278
|
+
const start = unfenced.search(/[[{]/);
|
|
279
|
+
if (start === -1) {
|
|
280
|
+
return undefined;
|
|
281
|
+
}
|
|
282
|
+
const end = unfenced.lastIndexOf(unfenced[start] === '[' ? ']' : '}');
|
|
283
|
+
if (end <= start) {
|
|
284
|
+
return undefined;
|
|
285
|
+
}
|
|
286
|
+
try {
|
|
287
|
+
return readKeep(JSON.parse(unfenced.slice(start, end + 1)));
|
|
288
|
+
}
|
|
289
|
+
catch {
|
|
290
|
+
return undefined;
|
|
291
|
+
}
|
|
292
|
+
}
|
|
211
293
|
/** Applies the deterministic filter, which is what triage falls back to. */
|
|
212
294
|
function deterministicVerdicts(candidates) {
|
|
213
|
-
return
|
|
295
|
+
return {
|
|
296
|
+
verdicts: candidates.map((candidate) => isHeritageBidCandidate(candidate)),
|
|
297
|
+
fromModel: false,
|
|
298
|
+
};
|
|
214
299
|
}
|
|
215
300
|
async function triageBatch(candidates, { model, onFallback }, signal) {
|
|
301
|
+
let keep;
|
|
216
302
|
try {
|
|
217
|
-
const
|
|
303
|
+
const result = await generateText({
|
|
218
304
|
model,
|
|
219
305
|
output: Output.object({ schema: KEEP_SCHEMA }),
|
|
220
306
|
system: SYSTEM_PROMPT,
|
|
221
307
|
prompt: buildUserPrompt$2(candidates),
|
|
222
308
|
abortSignal: signal,
|
|
309
|
+
// The mode core settled on after Anthropic truncated structured output
|
|
310
|
+
// containing quotation marks. This schema holds only integers, but the
|
|
311
|
+
// JSON tool is the path Claude is known to answer reliably, and other
|
|
312
|
+
// providers ignore the key.
|
|
313
|
+
providerOptions: { anthropic: { structuredOutputMode: 'jsonTool' } },
|
|
223
314
|
});
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
if (Number.isInteger(index) && index >= 0 && index < candidates.length) {
|
|
228
|
-
verdicts[index] = true;
|
|
229
|
-
usable = true;
|
|
230
|
-
}
|
|
315
|
+
let output;
|
|
316
|
+
try {
|
|
317
|
+
output = result.output;
|
|
231
318
|
}
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
return deterministicVerdicts(candidates);
|
|
319
|
+
catch {
|
|
320
|
+
// `output` is a getter, and it throws when the step produced nothing the
|
|
321
|
+
// spec could read at all — a provider that answered with a tool call, or
|
|
322
|
+
// with no text. The raw text is still on the result either way.
|
|
323
|
+
output = undefined;
|
|
238
324
|
}
|
|
239
|
-
|
|
325
|
+
keep = readKeep(output) ?? salvageKeep(result.text);
|
|
240
326
|
}
|
|
241
327
|
catch (error) {
|
|
242
328
|
// A cancelled crawl must fail the fetch, not quietly answer from the regex:
|
|
@@ -246,9 +332,106 @@ async function triageBatch(candidates, { model, onFallback }, signal) {
|
|
|
246
332
|
if (signal?.aborted) {
|
|
247
333
|
throw error;
|
|
248
334
|
}
|
|
249
|
-
|
|
335
|
+
// Other providers reject inside `generateText` instead, carrying the raw
|
|
336
|
+
// text on the error rather than on a result.
|
|
337
|
+
keep = NoObjectGeneratedError.isInstance(error)
|
|
338
|
+
? salvageKeep(error.text)
|
|
339
|
+
: undefined;
|
|
340
|
+
if (keep === undefined) {
|
|
341
|
+
onFallback?.(error instanceof Error ? error.message : String(error), candidates.length);
|
|
342
|
+
return deterministicVerdicts(candidates);
|
|
343
|
+
}
|
|
344
|
+
}
|
|
345
|
+
if (keep === undefined) {
|
|
346
|
+
onFallback?.('the response held no readable index list', candidates.length);
|
|
250
347
|
return deterministicVerdicts(candidates);
|
|
251
348
|
}
|
|
349
|
+
const verdicts = new Array(candidates.length).fill(false);
|
|
350
|
+
let usable = false;
|
|
351
|
+
for (const index of keep) {
|
|
352
|
+
if (Number.isInteger(index) && index >= 0 && index < candidates.length) {
|
|
353
|
+
verdicts[index] = true;
|
|
354
|
+
usable = true;
|
|
355
|
+
}
|
|
356
|
+
}
|
|
357
|
+
// An empty selection is a legitimate answer for a batch of road works, but
|
|
358
|
+
// a selection that is entirely out of range is a malformed response wearing
|
|
359
|
+
// the right shape, and it must not read as "reject everything".
|
|
360
|
+
if (!usable && keep.length > 0) {
|
|
361
|
+
onFallback?.('every returned index was out of range', candidates.length);
|
|
362
|
+
return deterministicVerdicts(candidates);
|
|
363
|
+
}
|
|
364
|
+
return { verdicts, fromModel: true };
|
|
365
|
+
}
|
|
366
|
+
/**
|
|
367
|
+
* Calls a store method so that a synchronous throw arrives as a rejection.
|
|
368
|
+
*
|
|
369
|
+
* The interface returns promises, but the body behind it is the application's
|
|
370
|
+
* own code and nothing stops it throwing before it ever reaches an `await` — a
|
|
371
|
+
* client that was never initialised, a config read that fails. Thrown that way
|
|
372
|
+
* the error sidesteps both guards here, because the lookup's `try` is entered
|
|
373
|
+
* after `load()` has already been called and the write's `.catch` is attached
|
|
374
|
+
* to a promise that was never returned. An optional cache would then fail the
|
|
375
|
+
* crawl it is specifically not allowed to fail.
|
|
376
|
+
*/
|
|
377
|
+
function callStore(work) {
|
|
378
|
+
return Promise.resolve().then(work);
|
|
379
|
+
}
|
|
380
|
+
/**
|
|
381
|
+
* Waits for a store lookup, but not past what the crawl can afford.
|
|
382
|
+
*
|
|
383
|
+
* The lookup itself takes no `AbortSignal` — the interface is the
|
|
384
|
+
* application's to implement and keeping it to two plain methods is the point
|
|
385
|
+
* — so the budget is enforced here instead. An abort is rethrown rather than
|
|
386
|
+
* reported: core has stopped waiting, and the reasoning is the same as for a
|
|
387
|
+
* cancelled batch.
|
|
388
|
+
*/
|
|
389
|
+
async function loadWithinBudget(load, timeoutMs, signal) {
|
|
390
|
+
if (signal?.aborted) {
|
|
391
|
+
throw signal.reason;
|
|
392
|
+
}
|
|
393
|
+
let timer;
|
|
394
|
+
let onAbort;
|
|
395
|
+
try {
|
|
396
|
+
return await new Promise((resolve, reject) => {
|
|
397
|
+
timer = setTimeout(() => resolve(TIMED_OUT), timeoutMs);
|
|
398
|
+
if (signal) {
|
|
399
|
+
onAbort = () => reject(signal.reason);
|
|
400
|
+
signal.addEventListener('abort', onAbort, { once: true });
|
|
401
|
+
}
|
|
402
|
+
load.then(resolve, reject);
|
|
403
|
+
});
|
|
404
|
+
}
|
|
405
|
+
finally {
|
|
406
|
+
clearTimeout(timer);
|
|
407
|
+
if (onAbort) {
|
|
408
|
+
signal?.removeEventListener('abort', onAbort);
|
|
409
|
+
}
|
|
410
|
+
}
|
|
411
|
+
}
|
|
412
|
+
/**
|
|
413
|
+
* Identifies what the verdicts in a store were made by.
|
|
414
|
+
*
|
|
415
|
+
* A verdict is only reusable while the thing that produced it is unchanged, so
|
|
416
|
+
* the fingerprint covers the selection rules (the system prompt), the shape the
|
|
417
|
+
* answer is asked for (the user prompt line and the schema) and the model. Edit
|
|
418
|
+
* any of them and every key changes, which retires the old rows instead of
|
|
419
|
+
* silently serving yesterday's judgement under today's rules. The alternative —
|
|
420
|
+
* a version constant bumped by hand — is one edit away from being wrong, and
|
|
421
|
+
* the failure is invisible.
|
|
422
|
+
*/
|
|
423
|
+
function fingerprintOf(model) {
|
|
424
|
+
const modelIdentity = typeof model === 'string' ? model : `${model.provider}:${model.modelId}`;
|
|
425
|
+
return createHash('sha256')
|
|
426
|
+
.update([
|
|
427
|
+
SYSTEM_PROMPT,
|
|
428
|
+
// Rendering one empty row pins the line format and `identityOf` along
|
|
429
|
+
// with the instruction, without pinning the batch size.
|
|
430
|
+
buildUserPrompt$2([{ institution: '', title: '' }]),
|
|
431
|
+
JSON.stringify(KEEP_JSON_SCHEMA),
|
|
432
|
+
modelIdentity,
|
|
433
|
+
].join('\u0000'))
|
|
434
|
+
.digest('hex');
|
|
252
435
|
}
|
|
253
436
|
/**
|
|
254
437
|
* Builds the LLM triage that replaces the regex as 나라장터's gate.
|
|
@@ -262,6 +445,14 @@ async function triageBatch(candidates, { model, onFallback }, signal) {
|
|
|
262
445
|
* request costs about 15 calls a day, so the judgement can be made by something
|
|
263
446
|
* that reads them.
|
|
264
447
|
*
|
|
448
|
+
* Verdicts are cached in two places. The in-memory map spans one run and exists
|
|
449
|
+
* for core's retry: core aborts the first attempt at 10 seconds and tries again
|
|
450
|
+
* with a longer budget, and without it every attempt would pay for the whole
|
|
451
|
+
* window. An optional {@link HeritageBidVerdictStore} spans runs, which is what
|
|
452
|
+
* the 48-hour window and a daily schedule need — half of each window was judged
|
|
453
|
+
* yesterday, and almost all of that half is rejections, which never reach the
|
|
454
|
+
* article table and so cannot be recognised any other way.
|
|
455
|
+
*
|
|
265
456
|
* It never fails open: an error, or a response whose indices are unusable, falls
|
|
266
457
|
* back to {@link isHeritageBidCandidate} for that batch rather than admitting
|
|
267
458
|
* every notice into per-article scoring. Omitting triage entirely leaves the
|
|
@@ -269,17 +460,77 @@ async function triageBatch(candidates, { model, onFallback }, signal) {
|
|
|
269
460
|
* LLM key.
|
|
270
461
|
*/
|
|
271
462
|
const createHeritageBidTriage = (options) => {
|
|
272
|
-
const { batchSize = DEFAULT_BATCH_SIZE, concurrency = DEFAULT_CONCURRENCY } = options;
|
|
273
|
-
|
|
274
|
-
//
|
|
275
|
-
//
|
|
463
|
+
const { batchSize = DEFAULT_BATCH_SIZE, concurrency = DEFAULT_CONCURRENCY, store, storeLoadTimeoutMs = DEFAULT_STORE_LOAD_TIMEOUT_MS, onStoreError, onCacheStats, } = options;
|
|
464
|
+
const fingerprint = fingerprintOf(options.model);
|
|
465
|
+
// Keying on exactly what the model is shown is what keeps the cache honest:
|
|
466
|
+
// key on less and two notices it judged separately become one, and the second
|
|
467
|
+
// verdict silently overwrites the first. The hash keeps every key the same
|
|
468
|
+
// length, because the store is asked about some 2,100 of them at once.
|
|
469
|
+
const keyOf = (candidate) => createHash('sha256')
|
|
470
|
+
.update([fingerprint, ...identityOf(candidate)].join('\u0000'))
|
|
471
|
+
.digest('hex');
|
|
276
472
|
const verdicts = new Map();
|
|
277
|
-
|
|
473
|
+
/** Model verdicts this process has not handed to the store yet. */
|
|
474
|
+
const unsaved = new Set();
|
|
475
|
+
/**
|
|
476
|
+
* Set once a lookup has outlasted its budget.
|
|
477
|
+
*
|
|
478
|
+
* Core retries the crawl fetch up to five times, so a store that stalls
|
|
479
|
+
* would otherwise cost the full budget on every one of them. One stall is
|
|
480
|
+
* enough to conclude the store is not helping this run.
|
|
481
|
+
*/
|
|
482
|
+
let storeLookupDisabled = false;
|
|
278
483
|
return async (candidates, signal) => {
|
|
279
484
|
if (candidates.length === 0) {
|
|
280
485
|
return [];
|
|
281
486
|
}
|
|
282
|
-
const
|
|
487
|
+
const keys = new Map(candidates.map((c) => [c, keyOf(c)]));
|
|
488
|
+
const fromMemory = candidates.filter((c) => verdicts.has(keys.get(c))).length;
|
|
489
|
+
if (store && !storeLookupDisabled) {
|
|
490
|
+
const missing = [
|
|
491
|
+
...new Set(candidates
|
|
492
|
+
.map((c) => keys.get(c))
|
|
493
|
+
.filter((key) => !verdicts.has(key))),
|
|
494
|
+
];
|
|
495
|
+
if (missing.length > 0) {
|
|
496
|
+
// The lookup keeps its own promise even when the wait below gives up
|
|
497
|
+
// on it, so a store that is merely slow still fills the cache for
|
|
498
|
+
// core's next attempt. A verdict the model has made since is the
|
|
499
|
+
// fresher of the two and is left alone.
|
|
500
|
+
const loading = callStore(() => store.load(missing)).then((rows) => {
|
|
501
|
+
for (const [key, verdict] of rows) {
|
|
502
|
+
if (!verdicts.has(key)) {
|
|
503
|
+
verdicts.set(key, verdict);
|
|
504
|
+
}
|
|
505
|
+
}
|
|
506
|
+
return rows;
|
|
507
|
+
});
|
|
508
|
+
// A rejection that lands after the wait gave up is reported there, not
|
|
509
|
+
// here; this only keeps it from going unhandled.
|
|
510
|
+
loading.catch(() => undefined);
|
|
511
|
+
try {
|
|
512
|
+
// Awaited, unlike the write: the whole point is to judge less, and a
|
|
513
|
+
// verdict that arrives after the batches are built saves nothing.
|
|
514
|
+
const loaded = await loadWithinBudget(loading, storeLoadTimeoutMs, signal);
|
|
515
|
+
if (loaded === TIMED_OUT) {
|
|
516
|
+
storeLookupDisabled = true;
|
|
517
|
+
onStoreError?.(`load did not answer within ${storeLoadTimeoutMs}ms`, 'load');
|
|
518
|
+
}
|
|
519
|
+
}
|
|
520
|
+
catch (error) {
|
|
521
|
+
// A cancelled crawl must propagate; core has already given up.
|
|
522
|
+
if (signal?.aborted) {
|
|
523
|
+
throw error;
|
|
524
|
+
}
|
|
525
|
+
// A store that is down must not stop the crawl; the run simply pays
|
|
526
|
+
// for what it could not read. A rejection is cheap, so unlike a
|
|
527
|
+
// stall it does not take the store out of the run.
|
|
528
|
+
onStoreError?.(error instanceof Error ? error.message : String(error), 'load');
|
|
529
|
+
}
|
|
530
|
+
}
|
|
531
|
+
}
|
|
532
|
+
const fromStore = candidates.filter((c) => verdicts.has(keys.get(c))).length - fromMemory;
|
|
533
|
+
const pending = candidates.filter((c) => !verdicts.has(keys.get(c)));
|
|
283
534
|
const batches = [];
|
|
284
535
|
for (let index = 0; index < pending.length; index += batchSize) {
|
|
285
536
|
batches.push(pending.slice(index, index + batchSize));
|
|
@@ -290,14 +541,38 @@ const createHeritageBidTriage = (options) => {
|
|
|
290
541
|
const batch = batches[next++];
|
|
291
542
|
const decided = await triageBatch(batch, options, signal);
|
|
292
543
|
batch.forEach((candidate, index) => {
|
|
293
|
-
|
|
544
|
+
const key = keys.get(candidate);
|
|
545
|
+
verdicts.set(key, decided.verdicts[index]);
|
|
546
|
+
if (decided.fromModel) {
|
|
547
|
+
unsaved.add(key);
|
|
548
|
+
}
|
|
294
549
|
});
|
|
295
550
|
}
|
|
296
551
|
});
|
|
297
552
|
// A rejection here propagates out of the fetch, which is what an aborted
|
|
298
553
|
// crawl needs; completed batches stay cached for the next attempt.
|
|
299
554
|
await Promise.all(workers);
|
|
300
|
-
|
|
555
|
+
onCacheStats?.({
|
|
556
|
+
candidates: candidates.length,
|
|
557
|
+
fromMemory,
|
|
558
|
+
fromStore,
|
|
559
|
+
judged: pending.length,
|
|
560
|
+
});
|
|
561
|
+
if (store && unsaved.size > 0 && !signal?.aborted) {
|
|
562
|
+
const pendingWrite = new Map([...unsaved].map((key) => [key, verdicts.get(key)]));
|
|
563
|
+
unsaved.clear();
|
|
564
|
+
// Deliberately not awaited. This runs inside the crawl fetch that core
|
|
565
|
+
// times out at 10 seconds, and the saving is meant to make that budget
|
|
566
|
+
// easier to hit, not harder; the run continues for minutes afterwards, so
|
|
567
|
+
// the write has time to land.
|
|
568
|
+
void callStore(() => store.save(pendingWrite)).catch((error) => {
|
|
569
|
+
for (const key of pendingWrite.keys()) {
|
|
570
|
+
unsaved.add(key);
|
|
571
|
+
}
|
|
572
|
+
onStoreError?.(error instanceof Error ? error.message : String(error), 'save');
|
|
573
|
+
});
|
|
574
|
+
}
|
|
575
|
+
return candidates.map((c) => verdicts.get(keys.get(c)) ?? false);
|
|
301
576
|
};
|
|
302
577
|
};
|
|
303
578
|
|
|
@@ -875,6 +1150,87 @@ const createJournalPaperSelector = (options) => {
|
|
|
875
1150
|
};
|
|
876
1151
|
};
|
|
877
1152
|
|
|
1153
|
+
const EPHEMERAL = { type: 'ephemeral' };
|
|
1154
|
+
/**
|
|
1155
|
+
* Marks the instruction block so Anthropic caches it, and reports what hit.
|
|
1156
|
+
*
|
|
1157
|
+
* Both halves of this exist because the two providers cache on opposite
|
|
1158
|
+
* defaults. The prompts are already shaped for it — core sends each stage's
|
|
1159
|
+
* long instruction as `instructions`, which the AI SDK turns into a system
|
|
1160
|
+
* message ahead of the article — so OpenAI's automatic prefix cache has been
|
|
1161
|
+
* hitting that block all along at no cost to us. Anthropic caches **nothing**
|
|
1162
|
+
* without an explicit `cache_control` breakpoint, so the same prompt that is
|
|
1163
|
+
* free to repeat on GPT is billed in full on Claude, every article.
|
|
1164
|
+
*
|
|
1165
|
+
* The breakpoint goes on the last system message rather than anywhere later:
|
|
1166
|
+
* Anthropic caches the prefix up to and including the marked block, so marking
|
|
1167
|
+
* the user turn would cache the article too and never hit twice.
|
|
1168
|
+
*
|
|
1169
|
+
* Wrapping is unconditional on purpose. `providerOptions` is namespaced, so
|
|
1170
|
+
* OpenAI and Google ignore the Anthropic key, and an application that injects
|
|
1171
|
+
* its own `LanguageModel` into any slot gets the same treatment without having
|
|
1172
|
+
* to know which provider this package expected.
|
|
1173
|
+
*
|
|
1174
|
+
* Anthropic will not cache a block below a minimum length, and that minimum is
|
|
1175
|
+
* **per model** — published figures differ by several times between the small
|
|
1176
|
+
* and large models, so do not hard-code one here or reason from a remembered
|
|
1177
|
+
* number. A block under it is processed without caching rather than rejected,
|
|
1178
|
+
* which is why no length check is made: a breakpoint that cannot fire costs
|
|
1179
|
+
* nothing, and a length check against the wrong number would disable one that
|
|
1180
|
+
* could. The rendered scoring prompt is around 5,200 characters and the
|
|
1181
|
+
* 나라장터 triage prompt is 832; whether either clears the configured model's
|
|
1182
|
+
* minimum is answered by `cacheWrite` in the reported usage, not by this
|
|
1183
|
+
* comment.
|
|
1184
|
+
*/
|
|
1185
|
+
const withPromptCaching = (model, { onUsage } = {}) => {
|
|
1186
|
+
if (typeof model === 'string') {
|
|
1187
|
+
// A bare model id carries no provider configuration to wrap; the caller
|
|
1188
|
+
// resolves it against a provider first.
|
|
1189
|
+
return model;
|
|
1190
|
+
}
|
|
1191
|
+
return wrapLanguageModel({
|
|
1192
|
+
model,
|
|
1193
|
+
middleware: {
|
|
1194
|
+
transformParams: ({ params }) => {
|
|
1195
|
+
let lastSystemIndex = -1;
|
|
1196
|
+
params.prompt.forEach((message, index) => {
|
|
1197
|
+
if (message.role === 'system') {
|
|
1198
|
+
lastSystemIndex = index;
|
|
1199
|
+
}
|
|
1200
|
+
});
|
|
1201
|
+
if (lastSystemIndex === -1) {
|
|
1202
|
+
return Promise.resolve(params);
|
|
1203
|
+
}
|
|
1204
|
+
const prompt = params.prompt.map((message, index) => index === lastSystemIndex
|
|
1205
|
+
? {
|
|
1206
|
+
...message,
|
|
1207
|
+
providerOptions: {
|
|
1208
|
+
...message.providerOptions,
|
|
1209
|
+
anthropic: {
|
|
1210
|
+
...message.providerOptions?.anthropic,
|
|
1211
|
+
cacheControl: EPHEMERAL,
|
|
1212
|
+
},
|
|
1213
|
+
},
|
|
1214
|
+
}
|
|
1215
|
+
: message);
|
|
1216
|
+
return Promise.resolve({ ...params, prompt });
|
|
1217
|
+
},
|
|
1218
|
+
wrapGenerate: async ({ doGenerate, model: wrapped }) => {
|
|
1219
|
+
const result = await doGenerate();
|
|
1220
|
+
const { cacheRead, cacheWrite, noCache } = result.usage.inputTokens;
|
|
1221
|
+
onUsage?.({
|
|
1222
|
+
provider: wrapped.provider,
|
|
1223
|
+
modelId: wrapped.modelId,
|
|
1224
|
+
cacheRead: cacheRead ?? 0,
|
|
1225
|
+
cacheWrite: cacheWrite ?? 0,
|
|
1226
|
+
noCache: noCache ?? 0,
|
|
1227
|
+
});
|
|
1228
|
+
return result;
|
|
1229
|
+
},
|
|
1230
|
+
},
|
|
1231
|
+
});
|
|
1232
|
+
};
|
|
1233
|
+
|
|
878
1234
|
/**
|
|
879
1235
|
* Reads what data.go.kr said when it refused a request.
|
|
880
1236
|
*
|
|
@@ -6472,10 +6828,21 @@ function createNewsletterGenerator(dependencies) {
|
|
|
6472
6828
|
const openai = createOpenAI({
|
|
6473
6829
|
apiKey: dependencies.openAIApiKey,
|
|
6474
6830
|
});
|
|
6831
|
+
// Every model this run uses goes through here, so the Anthropic cache
|
|
6832
|
+
// breakpoint reaches core's calls — which this package cannot otherwise
|
|
6833
|
+
// reach — as well as its own.
|
|
6834
|
+
const cached = (model, task) => withPromptCaching(model, {
|
|
6835
|
+
onUsage: (usage) => {
|
|
6836
|
+
dependencies.logger?.debug({
|
|
6837
|
+
event: 'llm.prompt.cache',
|
|
6838
|
+
data: { task, ...usage },
|
|
6839
|
+
});
|
|
6840
|
+
},
|
|
6841
|
+
});
|
|
6475
6842
|
// Resolve each task separately so even explicit undefined keeps its default.
|
|
6476
6843
|
const resolveModel = (task) => {
|
|
6477
|
-
const
|
|
6478
|
-
return typeof
|
|
6844
|
+
const configured = dependencies.models?.[task] ?? llmConfig.models[task];
|
|
6845
|
+
return cached(typeof configured === 'string' ? openai(configured) : configured, task);
|
|
6479
6846
|
};
|
|
6480
6847
|
const dateService = new DateService(dependencies.publishDate);
|
|
6481
6848
|
const taskService = new TaskService(dependencies.taskRepository);
|
|
@@ -6489,23 +6856,39 @@ function createNewsletterGenerator(dependencies) {
|
|
|
6489
6856
|
// deterministic filter stays behind it as the fallback.
|
|
6490
6857
|
createHeritageBidTriage({
|
|
6491
6858
|
model: resolveModel('heritageBidTriage'),
|
|
6859
|
+
store: dependencies.heritageBidVerdictStore,
|
|
6492
6860
|
onFallback: (reason, batchSize) => {
|
|
6493
6861
|
dependencies.logger?.info({
|
|
6494
6862
|
event: 'crawl.g2b.triage.fallback',
|
|
6495
6863
|
data: { reason, batchSize },
|
|
6496
6864
|
});
|
|
6497
6865
|
},
|
|
6866
|
+
onStoreError: (reason, operation) => {
|
|
6867
|
+
dependencies.logger?.error({
|
|
6868
|
+
event: 'crawl.g2b.triage.store.failed',
|
|
6869
|
+
data: { reason, operation },
|
|
6870
|
+
});
|
|
6871
|
+
},
|
|
6872
|
+
onCacheStats: (stats) => {
|
|
6873
|
+
dependencies.logger?.info({
|
|
6874
|
+
event: 'crawl.g2b.triage.cache',
|
|
6875
|
+
data: stats,
|
|
6876
|
+
});
|
|
6877
|
+
},
|
|
6498
6878
|
}), loadJournalIssues);
|
|
6499
6879
|
const analysisProvider = new AnalysisProvider({
|
|
6500
6880
|
classifyTags: resolveModel('classifyTags'),
|
|
6501
6881
|
analyzeImages: resolveModel('analyzeImages'),
|
|
6502
6882
|
determineImportance: resolveModel('determineImportance'),
|
|
6883
|
+
// Cached like every other stage: core sends this one a long instruction
|
|
6884
|
+
// too, and a model named here would otherwise be the one place a Claude
|
|
6885
|
+
// model pays full price for it on every article.
|
|
6503
6886
|
extractDeadlines: dependencies.extractDeadlineOptions
|
|
6504
6887
|
? dependencies.extractDeadlineOptions.model === undefined
|
|
6505
6888
|
? resolveModel('determineImportance')
|
|
6506
|
-
: typeof dependencies.extractDeadlineOptions.model === 'string'
|
|
6889
|
+
: cached(typeof dependencies.extractDeadlineOptions.model === 'string'
|
|
6507
6890
|
? openai(dependencies.extractDeadlineOptions.model)
|
|
6508
|
-
: dependencies.extractDeadlineOptions.model
|
|
6891
|
+
: dependencies.extractDeadlineOptions.model, 'extractDeadlines')
|
|
6509
6892
|
: undefined,
|
|
6510
6893
|
includeEvents: dependencies.extractDeadlineOptions?.includeEvents,
|
|
6511
6894
|
}, dependencies.articleRepository, dependencies.tagRepository);
|
|
@@ -6530,7 +6913,7 @@ function createNewsletterGenerator(dependencies) {
|
|
|
6530
6913
|
const contentModel = dependencies.models?.generateNewsletter !== undefined
|
|
6531
6914
|
? resolveModel('generateNewsletter')
|
|
6532
6915
|
: dependencies.contentGeneration
|
|
6533
|
-
? createContentGenerationModel(dependencies.contentGeneration)
|
|
6916
|
+
? cached(createContentGenerationModel(dependencies.contentGeneration), 'generateNewsletter')
|
|
6534
6917
|
: resolveModel('generateNewsletter');
|
|
6535
6918
|
const contentGenerateProvider = new ContentGenerateProvider(contentModel, dependencies.articleRepository, dependencies.newsletterRepository, templateOptions, resolvedBrandName,
|
|
6536
6919
|
// Reminder selection judges relevance the way importance scoring does, so
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sources":[],"sourcesContent":[],"names":[],"mappings":"
|
|
1
|
+
{"version":3,"file":"index.js","sources":[],"sourcesContent":[],"names":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;"}
|
package/package.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"name": "@heripo/research-radar",
|
|
3
3
|
"private": false,
|
|
4
4
|
"type": "module",
|
|
5
|
-
"version": "6.3.
|
|
5
|
+
"version": "6.3.2",
|
|
6
6
|
"description": "AI-driven intelligence for Korean cultural heritage. This package serves as both a ready-to-use newsletter service and a practical implementation example for the LLM-Newsletter-Kit.",
|
|
7
7
|
"main": "dist/index.js",
|
|
8
8
|
"types": "dist/index.d.ts",
|