@heripo/research-radar 5.3.4 → 6.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -67,10 +67,10 @@ For academic publications:
67
67
  ## Installation
68
68
 
69
69
  ```bash
70
- npm install @heripo/research-radar '@llm-newsletter-kit/core@~3.0.6'
70
+ npm install @heripo/research-radar '@llm-newsletter-kit/core@~3.1.0'
71
71
  ```
72
72
 
73
- **Requirements**: Node.js 24.15.0 or newer within 24.x, or >= 26.0.0, and an ESM application. Node.js 25 is not supported. The package exports `dist/index.js` and TypeScript declarations (`dist/index.d.ts`), with a JavaScript sourcemap. The core engine is a peer dependency; the current supported range is `~3.0.6`. The floor moved to 3.0.6 because the 나라장터 target runs its triage inside a crawl fetch that core times out, and before 3.0.6 core classified a timeout as non-retryable, so that target failed on the first attempt every run. 3.0.5 remains required for the self-verification retry cap the generation prompt relies on.
73
+ **Requirements**: Node.js 24.15.0 or newer within 24.x, or >= 26.0.0, and an ESM application. Node.js 25 is not supported. The package exports `dist/index.js` and TypeScript declarations (`dist/index.d.ts`), with a JavaScript sourcemap. The core engine is a peer dependency; the current supported range is `~3.1.0`. Core 3.1.0 is required for recent-title guarding and source-grounded deadline extraction. It also includes the crawl-timeout retry handling and generation self-verification retry cap required by Radar.
74
74
 
75
75
  An OpenAI API key is always required and supplies all default models. The optional compatibility path for Anthropic or Google newsletter generation additionally requires that provider's key. Keys are passed explicitly to the library; load environment variables in your application.
76
76
 
@@ -149,6 +149,80 @@ All interfaces are exported from the package root and defined in [src/types/depe
149
149
 
150
150
  Candidate selection belongs to your repository. Persist `usedArticles` associations as needed to exclude previously published content. Issue numbering is initialized before the pipeline starts; coordinate concurrent runs in your application or database. `TaskService` tracks an active task within one service instance, not across processes or separate `generateNewsletter()` calls.
151
151
 
152
+ ### Recent titles and source deadlines
153
+
154
+ These APIs use the published core 3.1.0 package. The peer dependency requires
155
+ `~3.1.0`, and `npm ci` installs the locked package without a local core link.
156
+
157
+ - Implement optional `NewsletterRepository.findRecentPublishedTitles({ fromInclusive, toExclusive })`
158
+ to enable the title guard. Radar supplies the seven calendar days before the
159
+ publication date (exclusive). Return titles only from actually published issues
160
+ of the same newsletter/audience. Drafts and failed sends are excluded. The core
161
+ compares normalized titles and allows at most two title-only repairs.
162
+ - Pass `extractDeadlineOptions: {}` to extract deadlines during article analysis
163
+ with the importance model, or `{ model }` for a separate OpenAI ID/AI SDK model.
164
+ Persist `article.deadlines` in `updateAnalysis`: `[]` means extracted without
165
+ supported deadlines; `undefined` means not extracted/failed. Clear the cache
166
+ when the source changes. Omit the option to keep existing analysis calls.
167
+ - `extractResearchRadarDeadlines({ model, article: { title, detailContent } })`
168
+ provides the same original-source extraction for host-managed backfill and
169
+ verification. It does not fetch URLs or activate reminders.
170
+
171
+ The host activates reminders only after an article is actually included in a
172
+ published issue. Core's `usedArticles` is the candidate array, including articles
173
+ the editor may omit; it is not evidence of actual inclusion. The host selects
174
+ actually included articles from the preceding 60 days for the one-time backfill,
175
+ stores deadlines separately, and selects only imminent reminders for each issue.
176
+ Historical issue bodies are not passed into generation.
177
+
178
+ Before reminding, call `refreshResearchRadarDeadlines({ source: { title, url,
179
+ targetUrl }, model, customFetch?, publicDataApiKey? })`. It fetches the registered
180
+ detail through the same adapters and robots rules as crawling, with a 15-second
181
+ total network budget. Job APIs fetch one posting; 나라장터 queries only that
182
+ notice number's revisions and returns the latest revision's URL. No daily list,
183
+ triage, scoring or persistence runs. `fetchResearchRadarDeadlineSource()` is
184
+ available separately for source-only reads (`status: 'fetched'`).
185
+
186
+ Dates and times must match their verbatim evidence. Even an unchanged source
187
+ hash cannot reuse a cache that fails the current grounding rules; it is extracted
188
+ again. Unsupported values hold the reminder.
189
+
190
+ - `status: 'verified'` includes the fresh article, deadlines, `sourceHash`,
191
+ `checkedAt` and model usage. Explicit extensions use the new date; explicit
192
+ cancellations have `status: 'cancelled'`. Passing `previous: { sourceHash,
193
+ deadlines }` from a successful refresh skips extraction only if a fresh fetch
194
+ has the same hash. Invalidate this cache when extraction policy changes.
195
+ - `status: 'unverified'` includes a reason, without deadlines. Missing dates,
196
+ access failures, unsupported boards and extraction failures hold reminders;
197
+ they never cancel a stored action. Even a verified source does not confirm all
198
+ past actions: the host must match each action and hold unmatched ones. A
199
+ separate announcement at another URL is outside this targeted check.
200
+
201
+ Pass host-selected verified actions to
202
+ `renderVerifiedDeadlineReminders(reminders, publicationDate)` and place its result
203
+ in `templateOptions.deadlineRemindersMarkdown`. It emits a date-ordered section
204
+ of at most five active deadlines within seven days, preserving supplied times
205
+ without inventing 23:59. The provider appends it to saved Markdown and HTML after
206
+ generation, so reminders add no LLM context. The host owns D-7/D-1 scheduling,
207
+ repeat suppression and publication history. Empty sections are omitted.
208
+
209
+ Prefer `templateOptions.verifiedDeadlineReminders` with freshly verified historical
210
+ actions. The provider merges these with source-grounded deadlines from articles
211
+ actually linked in the generated body, sorts them, and displays at most five
212
+ active deadlines within seven days in one linked section. For a refreshed source,
213
+ only freshly verified actions are eligible: unmatched cached actions are held,
214
+ never inferred to be active or cancelled from free-form `title` or `kind` labels.
215
+ G2B revisions share a source identity by notice number. Pass each distinct action
216
+ as a freshly verified result to retain multiple actions from that source. Other
217
+ sources without supplied verification continue using grounded article deadlines.
218
+ The older
219
+ `deadlineRemindersMarkdown` option remains available for pre-rendered sections.
220
+ The generation prompt no longer produces a separate closing schedule list.
221
+ The colleague-sharing notice follows the highlighted deadline section in both
222
+ saved Markdown and HTML; no eligible deadlines means no empty section.
223
+ Set `templateOptions.shareNoticeMarkdown` to customize this notice in both
224
+ formats, omit it for the default, or pass an empty string to suppress it.
225
+
152
226
  ### Optional generation settings
153
227
 
154
228
  | Option | Behavior |
@@ -228,7 +302,7 @@ await generateNewsletter({
228
302
 
229
303
  The optional `contentGeneration: { provider, apiKey, model? }` compatibility path still supports Anthropic and Google for newsletter generation. `models.generateNewsletter` takes precedence when both are supplied. If `models.generateNewsletter` is omitted or `undefined`, the existing `contentGeneration` provider/model selection applies; without either option, generation uses `llmConfig.models.generateNewsletter`. `openAIApiKey` remains required even when supplying model objects for every stage.
230
304
 
231
- [src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.0.6 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
305
+ [src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.1.0 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
232
306
 
233
307
  ## Crawling targets and parsers
234
308
 
@@ -382,16 +456,18 @@ npm run health-check:proxy # Load the same .env and pass --proxy
382
456
  npm run health-check -- --skip-khs-excavation # Skip KHS excavation report/site-open targets
383
457
  ```
384
458
 
385
- **What it checks per target**:
459
+ **What it checks per crawling target**:
386
460
 
387
461
  - `parseList()`: Non-empty array; checks the first item for non-empty title/date and a detailUrl starting with `http`
388
462
  - `parseDetail()`: Fetches the first detail item and checks trimmed `detailContent` length >= 20
389
463
 
464
+ For the public-data APIs (ALIO, 나라일터, 나라장터), it sends the configured `PUBLIC_DATA_API_KEY` directly to each list endpoint with `numOfRows=1` and checks only for HTTP 200. 나라장터 probes both service and construction endpoints. Response bodies, result codes, list contents, and details are not validated. These three targets are skipped when the key is missing.
465
+
390
466
  Use repeatable `--skip-target=<id-or-name>` or `--skip-target <id-or-name>` to exclude targets by exact ID or name. `npm run health-check -- --help` lists options without crawling. This is a live-site smoke check of the first item per target, not full article validation; it does not invoke LLM analysis. The script exits with code 1 if any checked target fails.
391
467
 
392
468
  **Output**: Console table summary + compact text summary for CI integrations.
393
469
 
394
- **CI**: [.github/workflows/parser-health-check.yml](./.github/workflows/parser-health-check.yml) runs daily at 08:00 UTC (17:00 KST), or manually, on an `org-linux` runner with a 30-minute job timeout. It skips the two KHS excavation report/site-open targets. The health-check composes the same fetch as production — robots.txt gate, then the KRAS detail adapter — so disallowed boards are reported as skipped rather than failed: 16 skipped and 55 checked with the current configuration. The detail check tries up to three list items, so one unreadable post at the top of a board does not fail the target. Slack notifications require the `SLACK_BOT_TOKEN` secret and `SLACK_ALERT_DEV_CHANNEL` repository variable. Forks need a matching runner and notification configuration to use this workflow unchanged. The CLI also writes GitHub Actions outputs and a job summary when their environment variables are present.
470
+ **CI**: [.github/workflows/parser-health-check.yml](./.github/workflows/parser-health-check.yml) runs daily at 08:00 UTC (17:00 KST), or manually, on an `org-linux` runner with a 30-minute job timeout. It skips the two KHS excavation report/site-open targets. For crawling targets, the health-check composes the same fetch as production — robots.txt gate, then the KRAS detail adapter — so disallowed boards are reported as skipped rather than failed: 16 skipped and 55 checked with the current configuration. The detail check tries up to three list items, so one unreadable post at the top of a board does not fail the target. Slack notifications require the `SLACK_BOT_TOKEN` secret and `SLACK_ALERT_DEV_CHANNEL` repository variable. Forks need a matching runner and notification configuration to use this workflow unchanged. The CLI also writes GitHub Actions outputs and a job summary when their environment variables are present.
395
471
 
396
472
  ## 🤝 Contributing
397
473
 
package/dist/index.d.ts CHANGED
@@ -1,7 +1,16 @@
1
- import { UrlString, ParsedTarget, CrawlingTargetGroup, CrawlingTarget, UnscoredArticle, ArticleForUpdateByAnalysis, ArticleForGenerateContent, Newsletter, IsoDateString, AppLogger, EmailService, EmailMessage, PromptProvider, DateService as DateService$1, TaskService as TaskService$1, AnalysisProvider as AnalysisProvider$1, ContentGenerateProvider as ContentGenerateProvider$1, HtmlTemplate, CrawlingProvider as CrawlingProvider$1, GenerateNewsletterConfig } from '@llm-newsletter-kit/core';
2
- import { LanguageModel } from 'ai';
1
+ import { ArticleDeadline, UrlString, IsoDateString, ParsedTarget, CrawlingTargetGroup, CrawlingTarget, UnscoredArticle, ArticleForUpdateByAnalysis, ArticleForGenerateContent, Newsletter, AppLogger, EmailService, EmailMessage, PromptProvider, DateService as DateService$1, TaskService as TaskService$1, AnalysisProvider as AnalysisProvider$1, ContentGenerateProvider as ContentGenerateProvider$1, HtmlTemplate, CrawlingProvider as CrawlingProvider$1, GenerateNewsletterConfig, ExtractArticleDeadlinesConfig, ExtractArticleDeadlinesResult } from '@llm-newsletter-kit/core';
2
+ import { LanguageModel, LanguageModelUsage } from 'ai';
3
3
  import { OpenAIProvider } from '@ai-sdk/openai';
4
4
 
5
+ /** Host-selected, published action matched against a freshly verified source. */
6
+ interface VerifiedDeadlineReminder {
7
+ deadline: ArticleDeadline;
8
+ url: UrlString;
9
+ verificationStatus: 'verified';
10
+ }
11
+ /** Deterministic display, no LLM context. Host owns D-7/D-1 selection and history. */
12
+ declare function renderVerifiedDeadlineReminders(reminders: VerifiedDeadlineReminder[], publicationDate: IsoDateString, now?: Date): string;
13
+
5
14
  /**
6
15
  * Repository interface for task management
7
16
  */
@@ -115,6 +124,18 @@ interface TagRepository {
115
124
  * Base template options shared by all newsletter variants.
116
125
  */
117
126
  interface BaseNewsletterTemplateOptions {
127
+ /**
128
+ * Deterministic reminder section from renderVerifiedDeadlineReminders().
129
+ * Appended to saved Markdown and HTML after generation; never fed to the LLM.
130
+ */
131
+ deadlineRemindersMarkdown?: string;
132
+ /** Freshly verified historical actions, merged with deadlines from included articles. */
133
+ verifiedDeadlineReminders?: VerifiedDeadlineReminder[];
134
+ /**
135
+ * Custom colleague-sharing Markdown, placed after the unified deadline section
136
+ * in both saved Markdown and HTML. Omit for the default; use '' to suppress it.
137
+ */
138
+ shareNoticeMarkdown?: string;
118
139
  /**
119
140
  * Markdown content for KRAS news section.
120
141
  * Converted to HTML and injected into the newsletter template.
@@ -182,6 +203,15 @@ interface WelcomeTemplateOptions {
182
203
  * Repository interface for newsletter management
183
204
  */
184
205
  interface NewsletterRepository {
206
+ /**
207
+ * Titles of actually published issues in [fromInclusive, toExclusive).
208
+ * Scope the query to this newsletter/audience; exclude drafts and failed sends.
209
+ * Providing this method enables the preceding seven calendar days title guard.
210
+ */
211
+ findRecentPublishedTitles?(range: {
212
+ fromInclusive: IsoDateString;
213
+ toExclusive: IsoDateString;
214
+ }): Promise<string[]>;
185
215
  /**
186
216
  * Get the next issue order number
187
217
  * @returns Next issue order
@@ -266,6 +296,15 @@ interface NewsletterGeneratorDependencies {
266
296
  openAIApiKey: string;
267
297
  /** OpenAI model IDs or AI SDK model instances for every LLM task (optional). */
268
298
  models?: NewsletterModelConfig;
299
+ /**
300
+ * Opt in to original-article deadline extraction. An omitted model reuses the
301
+ * importance model; a string is resolved through the supplied OpenAI key.
302
+ * Repositories must persist deadlines, including an empty array. Omit to keep
303
+ * the existing analysis calls. This does not activate reminders on candidates.
304
+ */
305
+ extractDeadlineOptions?: {
306
+ model?: string | LanguageModel;
307
+ };
269
308
  /**
270
309
  * Non-OpenAI content generation configuration (optional).
271
310
  *
@@ -426,6 +465,7 @@ interface AnalysisModels {
426
465
  classifyTags: LanguageModel;
427
466
  analyzeImages: LanguageModel;
428
467
  determineImportance: LanguageModel;
468
+ extractDeadlines?: LanguageModel;
429
469
  }
430
470
  /**
431
471
  * Analysis provider implementation
@@ -441,6 +481,7 @@ declare class AnalysisProvider implements AnalysisProvider$1 {
441
481
  analyzeImagesOptions: {
442
482
  model: LanguageModel;
443
483
  };
484
+ extractDeadlineOptions?: AnalysisProvider$1['extractDeadlineOptions'];
444
485
  determineScoreOptions: {
445
486
  model: LanguageModel;
446
487
  minimumImportanceScoreRules: Array<{
@@ -476,13 +517,16 @@ declare class AnalysisProvider implements AnalysisProvider$1 {
476
517
  declare class ContentGenerateProvider implements ContentGenerateProvider$1 {
477
518
  private readonly articleRepository;
478
519
  private readonly newsletterRepository;
520
+ private readonly templateOptions?;
479
521
  private _issueOrder;
522
+ private readonly deadlineRemindersMarkdown;
480
523
  readonly model: LanguageModel;
524
+ readonly fetchRecentTitles?: ContentGenerateProvider$1['fetchRecentTitles'];
481
525
  /** HTML template with markers for title and content injection */
482
526
  htmlTemplate: HtmlTemplate;
483
527
  /** Newsletter brand name (defaults to config, can be overridden via constructor) */
484
528
  newsletterBrandName: string;
485
- constructor(model: LanguageModel, articleRepository: ArticleRepository, newsletterRepository: NewsletterRepository, templateOptions?: NewsletterTemplateOptions, brandName?: string);
529
+ constructor(model: LanguageModel, articleRepository: ArticleRepository, newsletterRepository: NewsletterRepository, templateOptions?: NewsletterTemplateOptions | undefined, brandName?: string);
486
530
  /** LLM temperature setting for content generation */
487
531
  temperature: number;
488
532
  /** Subscribe page URL */
@@ -685,5 +729,61 @@ declare const llmConfig: {
685
729
  */
686
730
  declare const researchRadarPromptProvider: PromptProvider;
687
731
 
688
- export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DateService, TaskService, contentOptions, createCrawlingTargetGroups, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, maximumImportanceScoreByDomain, newsletterConfig, researchRadarPromptProvider, robotsExemptOrigins };
689
- export type { ArticleRepository, ContentGenerationConfig, ContentOptions, ExcavationReport, ExcavationReportSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterModelConfig, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, SourceGroup, SourceItem, TagRepository, TaskRepository, WelcomeTemplateOptions };
732
+ type ResearchRadarDeadlineExtractionConfig = Omit<ExtractArticleDeadlinesConfig, 'outputLanguage'> & {
733
+ outputLanguage?: string;
734
+ };
735
+ /** Shared by host-managed backfill and targeted source verification. No DB writes. */
736
+ declare function extractResearchRadarDeadlines(config: ResearchRadarDeadlineExtractionConfig): Promise<ExtractArticleDeadlinesResult>;
737
+
738
+ interface ResearchRadarDeadlineSource {
739
+ title: string;
740
+ url: UrlString;
741
+ /** Original board URL retained when the article was collected. */
742
+ targetUrl: UrlString;
743
+ }
744
+ type DeadlineVerificationFailure = 'unsupported-source' | 'invalid-source-url' | 'source-unavailable' | 'empty-source' | 'no-supported-deadlines' | 'extraction-failed';
745
+ type FetchedSource = {
746
+ status: 'fetched';
747
+ article: {
748
+ title: string;
749
+ detailContent: string;
750
+ url: UrlString;
751
+ };
752
+ sourceHash: string;
753
+ checkedAt: string;
754
+ };
755
+ type UnverifiedSource = {
756
+ status: 'unverified';
757
+ reason: DeadlineVerificationFailure;
758
+ checkedAt: string;
759
+ };
760
+ type ResearchRadarDeadlineSourceResult = FetchedSource | UnverifiedSource;
761
+ type ResearchRadarDeadlineRefreshResult = (Omit<FetchedSource, 'status'> & {
762
+ status: 'verified';
763
+ deadlines: ArticleDeadline[];
764
+ usage: LanguageModelUsage | undefined;
765
+ }) | UnverifiedSource;
766
+ interface ResearchRadarDeadlineSourceOptions {
767
+ source: ResearchRadarDeadlineSource;
768
+ customFetch?: typeof fetch;
769
+ publicDataApiKey?: string;
770
+ /** Total budget for the original page and parser API requests, not LLM calls. */
771
+ timeoutMs?: number;
772
+ }
773
+ type ResearchRadarDeadlineRefreshConfig = ResearchRadarDeadlineSourceOptions & Omit<ResearchRadarDeadlineExtractionConfig, 'article'> & {
774
+ /** Cache only a successful extraction from this source. A fresh fetch is mandatory. */
775
+ previous?: {
776
+ sourceHash: string;
777
+ deadlines: ArticleDeadline[];
778
+ };
779
+ };
780
+ /**
781
+ * Fetches only the registered detail source and any API requests its parser
782
+ * requires. Does not enumerate lists, look up existing URLs, score, or save.
783
+ */
784
+ declare function fetchResearchRadarDeadlineSource({ source, customFetch, publicDataApiKey, timeoutMs, }: ResearchRadarDeadlineSourceOptions): Promise<ResearchRadarDeadlineSourceResult>;
785
+ /** Only explicit, grounded results are verified. Missing dates are never cancellation. */
786
+ declare function refreshResearchRadarDeadlines(config: ResearchRadarDeadlineRefreshConfig): Promise<ResearchRadarDeadlineRefreshResult>;
787
+
788
+ export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DateService, TaskService, contentOptions, createCrawlingTargetGroups, extractResearchRadarDeadlines, fetchResearchRadarDeadlineSource, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, maximumImportanceScoreByDomain, newsletterConfig, refreshResearchRadarDeadlines, renderVerifiedDeadlineReminders, researchRadarPromptProvider, robotsExemptOrigins };
789
+ export type { ArticleRepository, ContentGenerationConfig, ContentOptions, DeadlineVerificationFailure, ExcavationReport, ExcavationReportSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterModelConfig, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, ResearchRadarDeadlineExtractionConfig, ResearchRadarDeadlineRefreshConfig, ResearchRadarDeadlineRefreshResult, ResearchRadarDeadlineSource, ResearchRadarDeadlineSourceOptions, ResearchRadarDeadlineSourceResult, SourceGroup, SourceItem, TagRepository, TaskRepository, VerifiedDeadlineReminder, WelcomeTemplateOptions };