@heripo/research-radar 6.3.3 → 6.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -67,10 +67,10 @@ For academic publications:
67
67
  ## Installation
68
68
 
69
69
  ```bash
70
- npm install @heripo/research-radar '@llm-newsletter-kit/core@~3.1.1'
70
+ npm install @heripo/research-radar '@llm-newsletter-kit/core@~3.2.0'
71
71
  ```
72
72
 
73
- **Requirements**: Node.js 24.22.1 or newer within 24.x, or >= 26.12.1, and an ESM application. Node.js 25 is not supported. The package exports `dist/index.js` and TypeScript declarations (`dist/index.d.ts`), with a JavaScript sourcemap. The core engine is a peer dependency; the current supported range is `~3.1.1`. Core 3.1.1 is required for recent-title guarding and source-grounded deadline extraction. It also includes the crawl-timeout retry handling and generation self-verification retry cap required by Radar.
73
+ **Requirements**: Node.js 24.22.1 or newer within 24.x, or >= 26.12.1, and an ESM application. Node.js 25 is not supported. The package exports `dist/index.js` and TypeScript declarations (`dist/index.d.ts`), with a JavaScript sourcemap. The core engine is a peer dependency; the current supported range is `~3.2.0`. Core 3.2.0 is required for byte-based image analysis with `analyzeImagesOptions.customFetch` and integer importance scores. Recent-title guarding, source-grounded deadline extraction, the crawl-timeout retry handling and the generation self-verification retry cap that Radar relies on came with 3.1.1.
74
74
 
75
75
  An OpenAI API key is always required and supplies all default models. The optional compatibility path for Anthropic or Google newsletter generation additionally requires that provider's key. Keys are passed explicitly to the library; load environment variables in your application.
76
76
 
@@ -151,8 +151,8 @@ Candidate selection belongs to your repository. Persist `usedArticles` associati
151
151
 
152
152
  ### Recent titles and source deadlines
153
153
 
154
- These APIs use the published core 3.1.1 package. The peer dependency requires
155
- `~3.1.1`, and `npm ci` installs the locked package without a local core link.
154
+ These APIs use the published core 3.2.0 package. The peer dependency requires
155
+ `~3.2.0`, and `npm ci` installs the locked package without a local core link.
156
156
 
157
157
  - Implement optional `NewsletterRepository.findRecentPublishedTitles({ fromInclusive, toExclusive })`
158
158
  to enable the title guard. Radar supplies the seven calendar days before the
@@ -255,17 +255,18 @@ formats, omit it for the default, or pass an empty string to suppress it.
255
255
 
256
256
  ### Optional generation settings
257
257
 
258
- | Option | Behavior |
259
- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
260
- | `logger` | Core `AppLogger` implementation, such as `console` |
261
- | `publishDate` | A real calendar date in `YYYY-MM-DD` format; invalid values throw. Defaults to the current date in `Asia/Seoul` (KST), regardless of server timezone |
262
- | `customFetch` | A `typeof fetch` implementation for crawling and parser API requests, such as a proxy adapter; does not configure LLM requests |
263
- | `templateOptions` | Default or KRAS newsletter branding and Markdown sections (see below) |
264
- | `promptProvider` | Core `PromptProvider` overriding the package's own LLM prompts entirely; omit to use the package's prompts, which fall back to core's defaults |
265
- | `excavationReportSource` | Supplies 국가유산청 발굴조사 보고서 entries from the application; that board is then never crawled. Omit to crawl it as before |
266
- | `journalIssueSource` | Supplies newly published journal issues (such as KCI) from the application, one article per issue. Omit and that target collects nothing |
267
- | `heritageBidVerdictStore` | Keeps 나라장터 triage verdicts between runs so the overlapping half of the window is not judged again. Omit and each run judges the whole window |
268
- | `previewNewsletter` | Fetch a saved `Newsletter` and send it through a supplied core `EmailService` |
258
+ | Option | Behavior |
259
+ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
260
+ | `logger` | Core `AppLogger` implementation, such as `console` |
261
+ | `publishDate` | A real calendar date in `YYYY-MM-DD` format; invalid values throw. Defaults to the current date in `Asia/Seoul` (KST), regardless of server timezone |
262
+ | `customFetch` | A `typeof fetch` implementation for crawling and parser API requests, such as a proxy adapter; does not configure LLM requests |
263
+ | `templateOptions` | Default or KRAS newsletter branding and Markdown sections (see below) |
264
+ | `promptProvider` | Core `PromptProvider` overriding the package's own LLM prompts entirely; omit to use the package's prompts, which fall back to core's defaults |
265
+ | `excavationReportSource` | Supplies 국가유산청 발굴조사 보고서 entries from the application; that board is then never crawled. Omit to crawl it as before |
266
+ | `journalIssueSource` | Supplies newly published journal issues (such as KCI) from the application, one article per issue. Omit and that target collects nothing |
267
+ | `submittedPaperSource` | Supplies author-submitted papers (theses, journal papers outside KCI) entered after review. Rendered in the journal section at save time, never scored |
268
+ | `heritageBidVerdictStore` | Keeps 나라장터 triage verdicts between runs so the overlapping half of the window is not judged again. Omit and each run judges the whole window |
269
+ | `previewNewsletter` | Fetch a saved `Newsletter` and send it through a supplied core `EmailService` |
269
270
 
270
271
  `previewNewsletter` requires `fetchNewsletterForPreview: () => Promise<Newsletter>`, `emailService` (with `send(message)`), and `emailMessage` (core `EmailMessage` without `subject`, `html`, or `text`). The core fills those three fields and skips preview delivery when no newsletter was created. Ensure the callback fetches the issue saved by this run.
271
272
 
@@ -336,7 +337,9 @@ Every model resolved here — including one supplied through `contentGeneration`
336
337
 
337
338
  The optional `contentGeneration: { provider, apiKey, model? }` compatibility path still supports Anthropic and Google for newsletter generation. `models.generateNewsletter` takes precedence when both are supplied. If `models.generateNewsletter` is omitted or `undefined`, the existing `contentGeneration` provider/model selection applies; without either option, generation uses `llmConfig.models.generateNewsletter`. `openAIApiKey` remains required even when supplying model objects for every stage.
338
339
 
339
- [src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.1.1 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
340
+ Core 3.2 downloads each article's images and sends their bytes instead of URLs, refusing non-public addresses. When the application supplies `customFetch`, images are downloaded through it behind a robots.txt gate, so a proxy or TLS workaround a board needs applies to its images too; that gate's robots.txt lookup never follows a redirect to another host. Core checks every image URL and redirect hop before calling that fetch, but checking the address at connect time, against DNS rebinding, is then up to the application's fetch. Without `customFetch`, core's own download is used, which checks addresses as it connects but skips robots.txt for images. The analysis prompt asks for two labelled paragraphs: `시각 맥락` describes what the images show, and `이미지 속 정보` carries legible text read from them — event names, dates, venues, organisers and deadlines that a poster-only notice leaves out of its body. The generation prompt may cite `이미지 속 정보` to fill in such facts, prefers the body when the two disagree, and treats everything else in the image analysis as context only. Deadline extraction still ignores image analysis.
341
+
342
+ [src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.2.0 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
340
343
 
341
344
  ## Crawling targets and parsers
342
345
 
@@ -357,7 +360,7 @@ Verdicts are cached in two places. Within a run they are held in memory, which i
357
360
 
358
361
  Newly published journal issues are not crawled. The application supplies them through `journalIssueSource` — heripo-web reads them from its KCI paper catalog — and the `KCI_학술지_신간` target serves them from that source, so its URL is never requested. Each issue (권호) becomes one article: its `uniqId` is the issue's `externalId` and its URL the issue's `url`. `usedArticles` carry only the application's article id and URL, so the application records which issues a saved newsletter introduced through the `uniqId` it stored in `saveCrawledArticles`, or through the issue URL. The article body, which scoring sees, lists every paper with its authors, keywords and an abstract truncated to 300 characters, after the keywords several papers share. Without a source the target answers with an empty list. This replaces the former crawl of the 국립문화유산연구원 헤리티지:역사와 과학 and 보존과학연구 boards.
359
362
 
360
- The generation model does not write these issues into the body. `ContentGenerateProvider` renders a `📖 새로 나온 논문` section itself, placed after `🎓 학술대회·학술행사` and before `📋 발굴조사보고서 (신규 제출)`. When that section is missing it goes before the next fixed category present (`⛏️ 발굴현장 공개`, `💼 채용·공고`, `📑 사업·입찰 공고`, then `📌 마무리`). Each issue links to its KCI table of contents and shows, as a table with their authors, the papers its readers would want to read, most interesting first; the rest are collapsed into a count and their topics. The criterion is the readers' interest, not academic discipline — a 역사학 paper can be left out and a non-archaeology one shown — and the readers are 한국고고학회 회원 in KRAS mode. The importance model only chooses and orders paper indices and names the topics of the papers it leaves out — titles, links and authors come from the injected data. If the selection fails or times out, every paper is listed; if the run cannot load the issues, the articles go to the model as ordinary candidates. The issue articles stay among the model's candidates, marked with the content type `자동 섹션: 📖 새로 나온 논문`: the prompt keeps them out of the body sections but weighs them like any other news for the title and briefing, so a new issue can lead the newsletter on the same terms. They count toward the publication criteria and come back in `usedArticles` like the rest. As a deterministic guard, any line the model still writes about an injected issue or one of its papers outside the briefing is removed before saving (`newsletter.journals.body.removed` is logged), along with table headers and headings left empty.
363
+ The generation model does not write these issues into the body. `ContentGenerateProvider` renders a `📖 새로 나온 논문` section itself, placed after `🎓 학술대회·학술행사` and before `📋 발굴조사보고서 (신규 제출)`. When that section is missing it goes before the next fixed category present (`⛏️ 발굴현장 공개`, `💼 채용·공고`, `📑 사업·입찰 공고`, then `📌 마무리`). Each issue links to its KCI table of contents and shows, as a table with their authors, the papers its readers would want to read, most interesting first; the rest are collapsed into a count and their topics. The criterion is the readers' interest, not academic discipline — a 역사학 paper can be left out and a non-archaeology one shown — and the readers are 한국고고학회 회원 in KRAS mode. The importance model only chooses and orders paper indices and names the topics of the papers it leaves out — titles, links and authors come from the injected data. If the selection fails or times out, every paper is listed; if the run cannot load the issues, the articles go to the model as ordinary candidates. The issue articles stay among the model's candidates, marked with the content type `자동 섹션: 📖 새로 나온 논문`: the prompt keeps them out of the body sections but weighs them like any other news for the title and briefing, so a new issue can lead the newsletter on the same terms. They count toward the publication criteria and come back in `usedArticles` like the rest. Submitted papers are a separate path. `submittedPaperSource` supplies theses and journal papers outside KCI that authors applied for and an administrator entered after review. They are not articles: they are not crawled, scored, counted toward the publication criteria or shown to any model, so they never appear in the title or briefing. When an issue is saved they are rendered after the KCI issues under `저자 신청 논문` — a table of title (linked only for http(s) URLs), authors and journal or university, like a KCI issue's, all escaped — and the ones rendered come back in `saveNewsletter`'s `submittedPapers` for the application to mark introduced. A run that saves no newsletter never reads the source, and a failing or stalled source (60-second limit) leaves them out of that issue with `newsletter.submitted_papers.load.failed` logged. In KRAS mode the section ends with how authors of papers outside KCI — theses or journal papers of comparable standing, within six months of publication, once per paper — can apply through the 한국고고학회 newsletter's Google Form (`PAPER_SUBMISSION_FORM_URL`). The closing 뉴스레터 출처 section states, in both modes, that the KCI issues come from the KCI Open API rather than the listed sources; in KRAS mode it adds that submitted papers are reviewed before being introduced and repeats the form link. As a deterministic guard, any line the model still writes about an injected issue or one of its papers outside the briefing is removed before saving (`newsletter.journals.body.removed` is logged), along with table headers and headings left empty.
361
364
 
362
365
  Crawling fetches pass through that check (`src/crawling/robots.ts`) before reaching the network: a disallowed request is refused rather than sent, and a missing or unreachable robots.txt allows it. `robotsExemptOrigins` in `src/config/index.ts` lists origins exempted from the check, each with its reason.
363
366
 
package/dist/index.d.ts CHANGED
@@ -197,6 +197,45 @@ interface JournalIssue {
197
197
  * generation run.
198
198
  */
199
199
  type JournalIssueSource = () => Promise<JournalIssue[]>;
200
+ /**
201
+ * A paper whose author applied to have it introduced, entered by an
202
+ * administrator after review — a thesis or a journal paper outside KCI.
203
+ *
204
+ * Submitted papers are not articles: they are not crawled, scored or shown to
205
+ * a model. When a newsletter is saved, every one the source supplied is
206
+ * rendered in the journal section after the KCI issues and handed back in
207
+ * `saveNewsletter`'s `submittedPapers`, so the application can mark it
208
+ * introduced. A run that saves no newsletter consumes none of them.
209
+ */
210
+ interface SubmittedPaper {
211
+ /**
212
+ * Stable identifier from the application. It comes back unchanged in
213
+ * `saveNewsletter`'s `submittedPapers`, which is how the application records
214
+ * that the paper was introduced.
215
+ */
216
+ id: string;
217
+ /** Paper title. */
218
+ title: string;
219
+ /** Author names in publication order. */
220
+ authors: string[];
221
+ /**
222
+ * Where the paper appeared: the journal for a journal paper, the university
223
+ * for a thesis.
224
+ * @example "고려대학교"
225
+ */
226
+ venue: string;
227
+ /**
228
+ * Page where readers can see the paper or its record. Only http(s) URLs are
229
+ * linked; anything else renders the title without a link.
230
+ */
231
+ url: string;
232
+ }
233
+ /**
234
+ * Supplies the submitted papers to introduce in this run. Called at most once,
235
+ * when the newsletter is saved. A failing or stalled source leaves them out of
236
+ * that newsletter rather than failing it.
237
+ */
238
+ type SubmittedPaperSource = () => Promise<SubmittedPaper[]>;
200
239
  /**
201
240
  * Repository interface for tag management
202
241
  */
@@ -309,11 +348,14 @@ interface NewsletterRepository {
309
348
  * @param input - Input parameters
310
349
  * @param input.newsletter - Newsletter data
311
350
  * @param input.usedArticles - List of used articles
351
+ * @param input.submittedPapers - Submitted papers rendered in this
352
+ * newsletter; empty when none were supplied or the source failed
312
353
  * @returns Saved newsletter ID
313
354
  */
314
355
  saveNewsletter(input: {
315
356
  newsletter: Newsletter;
316
357
  usedArticles: ArticleForGenerateContent[];
358
+ submittedPapers: SubmittedPaper[];
317
359
  }): Promise<{
318
360
  id: string | number;
319
361
  }>;
@@ -481,6 +523,18 @@ interface NewsletterGeneratorDependencies {
481
523
  * saved newsletter introduced.
482
524
  */
483
525
  journalIssueSource?: JournalIssueSource;
526
+ /**
527
+ * Supplies submitted papers to introduce in this run (optional): theses and
528
+ * journal papers outside KCI that authors applied for and an administrator
529
+ * entered after review.
530
+ *
531
+ * They are rendered in the journal section after the KCI issues when the
532
+ * newsletter is saved, without being crawled, scored or shown to a model,
533
+ * and do not count toward the publication criteria. The ones rendered come
534
+ * back in `saveNewsletter`'s `submittedPapers`; a run that saves no
535
+ * newsletter consumes none.
536
+ */
537
+ submittedPaperSource?: SubmittedPaperSource;
484
538
  /**
485
539
  * data.go.kr service key for the 나라일터 and 알리오 job boards (optional).
486
540
  *
@@ -624,6 +678,13 @@ interface AnalysisModels {
624
678
  extractDeadlines?: LanguageModel;
625
679
  /** Event extraction is enabled only together with extractDeadlines. */
626
680
  includeEvents?: boolean;
681
+ /**
682
+ * Fetch used to download article images for image analysis. Core sends the
683
+ * image bytes rather than their URLs, so this should apply the same proxy
684
+ * and robots.txt policy as crawling. Omitted, core's own download is used,
685
+ * which checks each address as it connects.
686
+ */
687
+ imageFetch?: typeof fetch;
627
688
  }
628
689
  /**
629
690
  * Analysis provider implementation
@@ -636,9 +697,7 @@ declare class AnalysisProvider implements AnalysisProvider$1 {
636
697
  classifyTagOptions: {
637
698
  model: LanguageModel;
638
699
  };
639
- analyzeImagesOptions: {
640
- model: LanguageModel;
641
- };
700
+ analyzeImagesOptions: AnalysisProvider$1['analyzeImagesOptions'];
642
701
  extractDeadlineOptions?: AnalysisProvider$1['extractDeadlineOptions'];
643
702
  determineScoreOptions: {
644
703
  model: LanguageModel;
@@ -706,8 +765,10 @@ type DeadlineReminderSelectorOptions = {
706
765
  */
707
766
  declare const createDeadlineReminderSelector: (options: DeadlineReminderSelectorOptions) => DeadlineReminderSelector;
708
767
 
768
+ /** Memoized access to one run's injected items. */
769
+ type SourceLoader<T> = () => Promise<T[]>;
709
770
  /** Memoized access to one run's injected journal issues. */
710
- type JournalIssueLoader = () => Promise<JournalIssue[]>;
771
+ type JournalIssueLoader = SourceLoader<JournalIssue>;
711
772
 
712
773
  /** Which papers of an issue the table shows, and which it summarizes. */
713
774
  type JournalPaperSelection = {
@@ -739,6 +800,13 @@ type JournalIssueRendering = {
739
800
  onUnavailable?: (reason: string) => void;
740
801
  /** Reports body lines about the issues removed despite the prompt. */
741
802
  onBodyMentionsRemoved?: (removed: number) => void;
803
+ /**
804
+ * The run's submitted papers, rendered after the issues when the newsletter
805
+ * is saved. They are not candidates and never reach the model.
806
+ */
807
+ loadSubmittedPapers?: SourceLoader<SubmittedPaper>;
808
+ /** Reports that the submitted papers could not be loaded and were left out. */
809
+ onSubmittedPapersUnavailable?: (reason: string) => void;
742
810
  };
743
811
  /**
744
812
  * Content generation provider implementation
@@ -805,7 +873,13 @@ declare class ContentGenerateProvider implements ContentGenerateProvider$1 {
805
873
  * an issue never appears both as a model-written row and in the table.
806
874
  */
807
875
  private removeJournalIssueMentions;
876
+ /**
877
+ * Renders the held issues and the submitted papers. `submittedPapers` are the
878
+ * ones actually rendered, so the repository marks exactly those introduced.
879
+ */
808
880
  private renderJournalIssues;
881
+ /** A failed source leaves the papers out of this newsletter, never fails it. */
882
+ private loadSubmittedPapers;
809
883
  private renderDeadlineReminders;
810
884
  }
811
885
 
@@ -1036,4 +1110,4 @@ declare function fetchResearchRadarDeadlineSource({ source, customFetch, publicD
1036
1110
  declare function refreshResearchRadarDeadlines(config: ResearchRadarDeadlineRefreshConfig): Promise<ResearchRadarDeadlineRefreshResult>;
1037
1111
 
1038
1112
  export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DEADLINE_REMINDER_LEAD_DAYS, DEADLINE_REMINDER_LIMIT, DateService, JOURNAL_ISSUE_LIST_URL, TaskService, contentOptions, createCrawlingTargetGroups, createDeadlineReminderSelector, extractResearchRadarDeadlines, fetchResearchRadarDeadlineSource, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, maximumImportanceScoreByDomain, newsletterConfig, refreshResearchRadarDeadlines, renderVerifiedDeadlineReminders, researchRadarPromptProvider, robotsExemptOrigins };
1039
- export type { ArticleRepository, ContentGenerationConfig, ContentOptions, DeadlineReminderSelector, DeadlineReminderSelectorOptions, DeadlineVerificationFailure, EligibleDeadlineReminder, ExcavationReport, ExcavationReportSource, HeritageBidVerdictStore, JournalIssue, JournalIssuePaper, JournalIssueSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterModelConfig, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, ResearchRadarDeadlineExtractionConfig, ResearchRadarDeadlineRefreshConfig, ResearchRadarDeadlineRefreshResult, ResearchRadarDeadlineSource, ResearchRadarDeadlineSourceOptions, ResearchRadarDeadlineSourceResult, SourceGroup, SourceItem, TagRepository, TaskRepository, VerifiedDeadlineReminder, WelcomeTemplateOptions };
1113
+ export type { ArticleRepository, ContentGenerationConfig, ContentOptions, DeadlineReminderSelector, DeadlineReminderSelectorOptions, DeadlineVerificationFailure, EligibleDeadlineReminder, ExcavationReport, ExcavationReportSource, HeritageBidVerdictStore, JournalIssue, JournalIssuePaper, JournalIssueSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterModelConfig, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, ResearchRadarDeadlineExtractionConfig, ResearchRadarDeadlineRefreshConfig, ResearchRadarDeadlineRefreshResult, ResearchRadarDeadlineSource, ResearchRadarDeadlineSourceOptions, ResearchRadarDeadlineSourceResult, SourceGroup, SourceItem, SubmittedPaper, SubmittedPaperSource, TagRepository, TaskRepository, VerifiedDeadlineReminder, WelcomeTemplateOptions };