@heripo/research-radar 5.0.6 → 5.2.0

package/README.md

@@ -23,7 +23,7 @@ Previously reported service metrics were $0.2–1 per issue and 15% CTR. These a
 
 - Type-safe TypeScript with strict interfaces
 - Provider pattern for swapping components (Crawling/Analysis/Content/Email)
-- 59 active crawling targets across heritage agencies, museums, academic societies
+- 74 active crawling targets across heritage agencies, museums, academic societies, filtered at runtime by robots.txt
 - Multi LLM providers: OpenAI GPT-5 (analysis) + selectable content generation (OpenAI / Anthropic / Google)
 - Built-in retries, chain options, preview emails
 
@@ -70,7 +70,7 @@ For academic publications:
 npm install @heripo/research-radar '@llm-newsletter-kit/core@~3.0.0'
 ```
 
-**Requirements**: Node.js >= 24 and an ESM application. The package exports `dist/index.js` and TypeScript declarations (`dist/index.d.ts`), with a JavaScript sourcemap. The core engine is a peer dependency; the current supported range is `~3.0.0`.
+**Requirements**: Node.js >= 24 and an ESM application. The package exports `dist/index.js` and TypeScript declarations (`dist/index.d.ts`), with a JavaScript sourcemap. The core engine is a peer dependency; the current supported range is `~3.0.5`. 3.0.5 is the floor because the newsletter generation prompt this package ships relies on the self-verification retry cap added there.
 
 Article analysis requires an OpenAI API key. Content generation requires a key for the selected provider (OpenAI / Anthropic / Google); OpenAI can use the same key for both. Keys are passed explicitly to the library; load environment variables in your application.
 
@@ -136,13 +136,15 @@ Candidate selection belongs to your repository. Persist `usedArticles` associati
 
 ### Optional generation settings
 
-| Option              | Behavior                                                                                                                                             |
-| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `logger`            | Core `AppLogger` implementation, such as `console`                                                                                                   |
-| `publishDate`       | A real calendar date in `YYYY-MM-DD` format; invalid values throw. Defaults to the current date in `Asia/Seoul` (KST), regardless of server timezone |
-| `customFetch`       | A `typeof fetch` implementation for crawling and parser API requests, such as a proxy adapter; does not configure LLM requests                       |
-| `templateOptions`   | Default or KRAS newsletter branding and Markdown sections (see below)                                                                                |
-| `previewNewsletter` | Fetch a saved `Newsletter` and send it through a supplied core `EmailService`                                                                        |
+| Option                   | Behavior                                                                                                                                             |
+| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `logger`                 | Core `AppLogger` implementation, such as `console`                                                                                                   |
+| `publishDate`            | A real calendar date in `YYYY-MM-DD` format; invalid values throw. Defaults to the current date in `Asia/Seoul` (KST), regardless of server timezone |
+| `customFetch`            | A `typeof fetch` implementation for crawling and parser API requests, such as a proxy adapter; does not configure LLM requests                       |
+| `templateOptions`        | Default or KRAS newsletter branding and Markdown sections (see below)                                                                                |
+| `promptProvider`         | Core `PromptProvider` overriding the package's own LLM prompts entirely; omit to use the package's prompts, which fall back to core's defaults       |
+| `excavationReportSource` | Supplies 국가유산청 발굴조사 보고서 entries from the application; that board is then never crawled. Omit to crawl it as before                       |
+| `previewNewsletter`      | Fetch a saved `Newsletter` and send it through a supplied core `EmailService`                                                                        |
 
 `previewNewsletter` requires `fetchNewsletterForPreview: () => Promise<Newsletter>`, `emailService` (with `send(message)`), and `emailMessage` (core `EmailMessage` without `subject`, `html`, or `text`). The core fills those three fields and skips preview delivery when no newsletter was created. Ensure the callback fetches the issue saved by this run.
 
@@ -165,16 +167,16 @@ Defaults below describe the checked-in code, not provider recommendations. This
 
 | Stage              | Provider  | Default model            |
 | ------------------ | --------- | ------------------------ |
-| Tag classification | OpenAI    | `gpt-5-mini`             |
-| Image analysis     | OpenAI    | `gpt-5.1`                |
-| Importance scoring | OpenAI    | `gpt-5.1`                |
-| Content generation | OpenAI    | `gpt-5.4`                |
+| Tag classification | OpenAI    | `gpt-5.6-luna`           |
+| Image analysis     | OpenAI    | `gpt-5.6-terra`          |
+| Importance scoring | OpenAI    | `gpt-5.6-terra`          |
+| Content generation | OpenAI    | `gpt-5.6-sol`            |
 | Content generation | Anthropic | `claude-sonnet-4-6`      |
 | Content generation | Google    | `gemini-3.1-pro-preview` |
 
 Select content generation with `contentGeneration: { provider, apiKey, model? }`. `model` overrides that provider's default. Analysis models are configured in [analysis.provider.ts](./src/providers/analysis.provider.ts).
 
-[src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.0.4 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
+[src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.0.5 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
 
 ## Crawling targets and parsers
 
@@ -182,16 +184,20 @@ Select content generation with `contentGeneration: { provider, apiKey, model? }`
 
 | Group      | Active | Commented out |
 | ---------- | -----: | ------------: |
-| News       |     48 |            10 |
-| Business   |      4 |             0 |
-| Employment |      7 |             3 |
-| **Total**  | **59** |        **13** |
+| News       |     57 |             1 |
+| Business   |      5 |             0 |
+| Employment |     12 |             0 |
+| **Total**  | **74** |         **1** |
 
-The 13 commented targets are excluded from runtime configuration: one excavation status board is marked as low-value fragmented data, and 12 museum boards are marked as restricted by robots.txt. Their parser code remains in the repository.
+Only the excavation status board stays commented out, as low-value fragmented data. Boards that a site's robots.txt restricts are configured normally and refused at runtime by the robots.txt check, so the configuration does not have to track each site's policy by hand. With current policies 14 of the 74 targets are refused.
+
+Three targets are read from data.go.kr open APIs instead of scraped: 나라일터 (`PblJobService`) and 알리오 (`recruitment`) under Employment, and 나라장터 (`BidPublicInfoService`) under Business. Each is a single round of requests, and `publicDataApiKey` supplies the service key — omit it and they answer with an empty list without making a request. The boards carry every public-sector vacancy and procurement notice in the country, so both are narrowed before analysis. The job boards use `src/crawling/heritage-job-filter.ts`, whose exclusions are role nouns and hold up as patterns; roughly 2% of postings survive, about 2.6 a day. 나라장터 instead uses `src/crawling/heritage-triage.ts`, which asks a cheap model about 100 titles per request: heritage vocabulary cannot be settled by substring in Korean, and the deterministic filter remains behind it as the fallback when a batch fails.
+
+Crawling fetches pass through that check (`src/crawling/robots.ts`) before reaching the network: a disallowed request is refused rather than sent, and a missing or unreachable robots.txt allows it. `robotsExemptOrigins` in `src/config/index.ts` lists origins exempted from the check, each with its reason.
 
 Sources include the Korea Heritage Service, National Research Institute of Cultural Heritage, National Research Institute of Maritime Heritage, Korea Heritage Agency, Korea Association of Archaeological Heritage, archaeological societies, and national museums.
 
-[src/parsers/](./src/parsers/) contains 20 organization-specific parser modules plus shared date and URL utilities. List parsers return `ParsedTargetListItem[]` (title, date, detail URL, date type, and optional source ID); detail parsers return Markdown `detailContent` and attachment/image flags. Parsers can be synchronous or asynchronous. KRAS, Yeongnam Archaeological Society, and maritime heritage sources use additional API requests for client-rendered content.
+[src/parsers/](./src/parsers/) contains 23 organization-specific parser modules plus shared date and URL utilities. List parsers return `ParsedTargetListItem[]` (title, date, detail URL, date type, and optional source ID); detail parsers return Markdown `detailContent` and attachment/image flags. Parsers can be synchronous or asynchronous. KRAS, Yeongnam Archaeological Society, and maritime heritage sources use additional API requests for client-rendered content.
 
 `CrawlingProvider` uses a maximum concurrency of 5 and wraps the supplied fetch to route KRAS public detail URLs to its detail API, while retaining public URLs in article metadata. When constructing your own pipeline, use the provider's fetch together with its target groups.
 
@@ -203,7 +209,7 @@ import { getSourceList } from '@heripo/research-radar';
 const groups = getSourceList(); // [{ id, name, sources: [{ id, name, url }] }]
 ```
 
-`createCrawlingTargetGroups(customFetch?)`, `getSourceList()`, `contentOptions`, `newsletterConfig`, and `llmConfig` are public exports. The package also exports the three provider classes, `DateService`, `TaskService`, and their public configuration/dependency types through [src/index.ts](./src/index.ts).
+`createCrawlingTargetGroups(customFetch?)`, `getSourceList()`, `contentOptions`, `newsletterConfig`, `llmConfig`, and `researchRadarPromptProvider` are public exports. `ExcavationReport` and `ExcavationReportSource` are exported as types. The package also exports the three provider classes, `DateService`, `TaskService`, and their public configuration/dependency types through [src/index.ts](./src/index.ts).
 
 ## Email templates
 
@@ -332,7 +338,7 @@ Use repeatable `--skip-target=<id-or-name>` or `--skip-target <id-or-name>` to e
 
 **Output**: Console table summary + compact text summary for CI integrations.
 
-**CI**: [.github/workflows/parser-health-check.yml](./.github/workflows/parser-health-check.yml) runs daily at 08:00 UTC (17:00 KST), or manually, on an `org-linux` runner with a 30-minute job timeout. It skips the two KHS excavation report/site-open targets (57 targets checked with the current configuration). Slack notifications require the `SLACK_BOT_TOKEN` secret and `SLACK_ALERT_DEV_CHANNEL` repository variable. Forks need a matching runner and notification configuration to use this workflow unchanged. The CLI also writes GitHub Actions outputs and a job summary when their environment variables are present.
+**CI**: [.github/workflows/parser-health-check.yml](./.github/workflows/parser-health-check.yml) runs daily at 08:00 UTC (17:00 KST), or manually, on an `org-linux` runner with a 30-minute job timeout. It skips the two KHS excavation report/site-open targets. The health-check composes the same fetch as production — robots.txt gate, then the KRAS detail adapter — so disallowed boards are reported as skipped rather than failed: 16 skipped and 55 checked with the current configuration. The detail check tries up to three list items, so one unreadable post at the top of a board does not fail the target. Slack notifications require the `SLACK_BOT_TOKEN` secret and `SLACK_ALERT_DEV_CHANNEL` repository variable. Forks need a matching runner and notification configuration to use this workflow unchanged. The CLI also writes GitHub Actions outputs and a job summary when their environment variables are present.
 
 ## 🤝 Contributing
 
@@ -390,7 +396,7 @@ const contentGeneration: ContentGenerationConfig = {
 };
 ```
 
-Default models: openai=`gpt-5.4`, anthropic=`claude-sonnet-4-6`, google=`gemini-3.1-pro-preview`
+Default models: openai=`gpt-5.6-sol`, anthropic=`claude-sonnet-4-6`, google=`gemini-3.1-pro-preview`
 
 To change analysis providers, update both `src/providers/analysis.provider.ts` (provider type and models) and `src/newsletter-generator.ts` (provider construction), using a compatible AI SDK provider. Also adapt domain-specific minimum-score rules in the analysis provider, output language and expert fields in config, package metadata, and the GitHub Actions runner/Slack settings for your fork.
 

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-import { UrlString, ParsedTarget, CrawlingTargetGroup, CrawlingTarget, UnscoredArticle, ArticleForUpdateByAnalysis, ArticleForGenerateContent, Newsletter, AppLogger, EmailService, EmailMessage, DateService as DateService$1, IsoDateString, TaskService as TaskService$1, AnalysisProvider as AnalysisProvider$1, ContentGenerateProvider as ContentGenerateProvider$1, HtmlTemplate, CrawlingProvider as CrawlingProvider$1, GenerateNewsletterConfig } from '@llm-newsletter-kit/core';
+import { UrlString, ParsedTarget, CrawlingTargetGroup, CrawlingTarget, UnscoredArticle, ArticleForUpdateByAnalysis, ArticleForGenerateContent, Newsletter, IsoDateString, AppLogger, EmailService, EmailMessage, PromptProvider, DateService as DateService$1, TaskService as TaskService$1, AnalysisProvider as AnalysisProvider$1, ContentGenerateProvider as ContentGenerateProvider$1, HtmlTemplate, CrawlingProvider as CrawlingProvider$1, GenerateNewsletterConfig } from '@llm-newsletter-kit/core';
 import { OpenAIProvider } from '@ai-sdk/openai';
 import { LanguageModel } from 'ai';
 
@@ -54,6 +54,53 @@ interface ArticleRepository {
      */
     findCandidatesForNewsletter(): Promise<ArticleForGenerateContent[]>;
 }
+/**
+ * A single excavation report supplied by the application instead of crawled.
+ *
+ * Field names mirror what the 국가유산청 report board exposes, so an application
+ * that already stores these rows can hand them over without reshaping them.
+ */
+interface ExcavationReport {
+    /**
+     * The board's own identifier (`ecexmRcno`). Used both as the article's unique
+     * id and to build its public detail URL, so it must match the value the board
+     * uses — otherwise previously crawled reports are re-saved as duplicates.
+     * @example "202609157717"
+     */
+    externalId: string;
+    /**
+     * Report title (보고서명).
+     * @example "태안 태안읍성 -남문지 및 연지-"
+     */
+    title: string;
+    /**
+     * Submission date (제출일) in ISO format (YYYY-MM-DD).
+     * @example "2026-09-15"
+     */
+    submittedDate: IsoDateString;
+    /**
+     * Report detail fields as label/value pairs, rendered into the article body in
+     * insertion order. Supply whatever the source holds — 허가번호, 유적명,
+     * 발간기관, 조사시도시군구, 조사기간, 유적성격/시대구분 and so on.
+     * Empty values are skipped.
+     */
+    fields: Record<string, string | null | undefined>;
+    /**
+     * Whether the report has a downloadable file. Defaults to true, matching the
+     * crawled parser.
+     */
+    hasAttachedFile?: boolean;
+}
+/**
+ * Supplies excavation reports from the application instead of crawling them.
+ *
+ * When provided, the 국가유산청 발굴조사 보고서 board is served from this
+ * function and never requested over the network; the rest of the crawl is
+ * unaffected. When omitted, the board is crawled as before.
+ *
+ * Called at most once per generation run.
+ */
+type ExcavationReportSource = () => Promise<ExcavationReport[]>;
 /**
  * Repository interface for tag management
  */
@@ -178,7 +225,7 @@ interface PreviewNewsletterOptions {
  * Each provider uses a sensible default model that can be overridden.
  *
  * Default models:
- * - openai: `gpt-5.4`
+ * - openai: `gpt-5.6-sol`
  * - anthropic: `claude-sonnet-4-6`
  * - google: `gemini-3.1-pro-preview`
  */
@@ -226,6 +273,30 @@ interface NewsletterGeneratorDependencies {
     templateOptions?: NewsletterTemplateOptions;
     /** Custom fetch function for crawling (e.g., proxy-based fetch). Optional. */
     customFetch?: typeof fetch;
+    /**
+     * Supplies 국가유산청 발굴조사 보고서 entries from the application (optional).
+     *
+     * When provided, that board is read from this function and never crawled,
+     * which lets an application that already stores the reports reuse them. Omit
+     * it to keep crawling the board as before. No other target is affected.
+     */
+    excavationReportSource?: ExcavationReportSource;
+    /**
+     * data.go.kr service key for the 나라일터 and 알리오 job boards (optional).
+     *
+     * Pass the encoded key exactly as the portal supplies it. Both boards are read
+     * through open APIs rather than scraped; omit the key and they collect
+     * nothing while every other target is unaffected.
+     */
+    publicDataApiKey?: string;
+    /**
+     * LLM prompt overrides (optional).
+     *
+     * When provided, this replaces Research Radar's own prompt provider entirely
+     * rather than merging with it. Omit it to use the package's tuned prompts,
+     * which in turn fall back to core's defaults for any stage they do not define.
+     */
+    promptProvider?: PromptProvider;
 }
 /**
  * Newsletter generation execution function
@@ -419,6 +490,22 @@ declare class ContentGenerateProvider implements ContentGenerateProvider$1 {
     }>;
 }
 
+type HeritageBidCandidate = {
+    /** Issuing and requesting institutions, joined. */
+    institution: string;
+    /** Notice title. */
+    title: string;
+    /** `pubPrcrmntMidClsfcNm`, when the notice carries one. */
+    classification?: string | null;
+};
+
+/**
+ * Decides which 나라장터 notices are worth scoring, one batch at a time.
+ *
+ * Returns one verdict per candidate, in the order they were given.
+ */
+type HeritageBidTriage = (candidates: HeritageBidCandidate[], signal?: AbortSignal) => Promise<boolean[]>;
+
 /**
  * Crawling provider implementation
  * - Defines crawling targets
@@ -433,7 +520,7 @@ declare class CrawlingProvider implements CrawlingProvider$1 {
     customFetch?: typeof fetch;
     /** Crawling target groups configuration */
     crawlingTargetGroups: CrawlingTargetGroup[];
-    constructor(articleRepository: ArticleRepository, customFetch?: typeof fetch);
+    constructor(articleRepository: ArticleRepository, customFetch?: typeof fetch, excavationReportSource?: ExcavationReportSource, logger?: AppLogger, publicDataApiKey?: string, bidTriage?: HeritageBidTriage);
     /**
      * Fetch existing articles by URLs to avoid duplicate crawling
      * @param articleUrls - URLs to check
@@ -489,6 +576,38 @@ declare const contentOptions: ContentOptions;
  * Newsletter brand configuration
  */
 declare const newsletterConfig: NewsletterConfig;
+/**
+ * Maximum importance score per heritage domain (`tag1`).
+ *
+ * The newsletter is archaeology-first: archaeology and cultural heritage keep
+ * the full 1-10 range, while natural and intangible heritage are capped so they
+ * cannot crowd out archaeological coverage. `기타` — material that is not
+ * heritage at all — is capped lowest, because the scoring prompt's academic-value
+ * floor is domain-blind and would otherwise lift things like a general journal's
+ * call for papers into the top half of the scale.
+ *
+ * The cap is applied deterministically after scoring, in
+ * `AnalysisProvider.update()`, rather than asked for in the prompt, so the
+ * ceiling always holds. It never reaches 1: a score of 1 means "exclude from the
+ * newsletter" in the consuming application's candidate query, so capping to 1
+ * would delete these articles instead of demoting them.
+ *
+ * Domains absent from this map are not capped.
+ */
+declare const maximumImportanceScoreByDomain: Record<string, number>;
+/**
+ * Origins exempted from the robots.txt check.
+ *
+ * Each entry deliberately overrides what the site publishes, so it needs a
+ * reason and should be revisited when that site's robots.txt changes.
+ *
+ * - `http://www.yngogo.or.kr` (영남고고학회): its board is rendered from
+ *   `/module/ntt/unity/selectNttListAjax.ink`, and robots.txt carries a blanket
+ *   `Disallow: /module`. The rule reads as protecting an internal path rather
+ *   than the public board it happens to serve, and there is no other route to
+ *   the listing, so the society's boards are collected under this exemption.
+ */
+declare const robotsExemptOrigins: readonly string[];
 /**
  * LLM configuration
  */
@@ -500,5 +619,35 @@ declare const llmConfig: {
     };
 };
 
-export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DateService, TaskService, contentOptions, createCrawlingTargetGroups, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, newsletterConfig };
-export type { ArticleRepository, ContentGenerationConfig, ContentOptions, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, SourceGroup, SourceItem, TagRepository, TaskRepository, WelcomeTemplateOptions };
+/**
+ * Research Radar's LLM prompt overrides.
+ *
+ * Each builder here **replaces** core's built-in prompt for that stage rather
+ * than extending it — a `PromptBuilder` returns the whole prompt string. That is
+ * deliberate: core's defaults instruct the model to use emoticons, render
+ * importance as star ratings, and add share-of-coverage statistics, all of which
+ * this newsletter removes. Appending contradicting rules to those defaults
+ * degrades the output, so a replacement is written from scratch instead.
+ *
+ * **Contract a replacement must still satisfy.** Core's output schema is fixed,
+ * and `generateNewsletter` regenerates the whole newsletter whenever the model
+ * reports a failure. A replacement prompt has to steer the model toward:
+ *
+ * - `title`: 20–70 characters
+ * - `isWrittenInOutputLanguage`, `copyrightVerified`, `factAccuracy`: true
+ * - `titleContext` (KRAS mode): the phrase must appear in the title
+ *
+ * Core caps that loop at 5 attempts, so a prompt that ignores the contract costs
+ * up to 5 full generations on the most expensive model in the pipeline.
+ *
+ * Tune these against real articles with core's playground
+ * (`npm run playground:generate-newsletter` in ../llm-newsletter-kit-core) and
+ * diff them against the defaults for free with `playground:verify-prompts`.
+ * The playground loads this module from `dist`, so run `npm run build` here
+ * after every edit — its source cannot be imported directly across repos
+ * because of the `~/*` path alias.
+ */
+declare const researchRadarPromptProvider: PromptProvider;
+
+export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DateService, TaskService, contentOptions, createCrawlingTargetGroups, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, maximumImportanceScoreByDomain, newsletterConfig, researchRadarPromptProvider, robotsExemptOrigins };
+export type { ArticleRepository, ContentGenerationConfig, ContentOptions, ExcavationReport, ExcavationReportSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, SourceGroup, SourceItem, TagRepository, TaskRepository, WelcomeTemplateOptions };